Chapter 12: Reference Architectures and Design Reviews
1. Chapter Overview
The preceding chapters treat product fit, prompts, context, harnesses, actions, evaluation, trust, release, and operations as separate engineering disciplines. Architecture is where their decisions meet—and where contradictions among them become system failures.
Chapter 12 owns that integration. It shows how to recognize and compose six recurring patterns—retrieval-augmented generation, copilots, AI search, workflow agents, human-in-the-loop systems, and event-driven AI—then review the composition as one production system.
The primary engineering decision is:
Which pattern or composition matches the system's responsibility, authority, and operating constraints, and is the integrated design coherent enough to approve?
A reference architecture is not a deployment recipe. It gives reviewers a known shape of responsibilities, boundaries, invariants, and expected failure behavior against which to test a proposal. Implementations can differ; the architecture argument cannot. That argument identifies who initiates work, what may be decided or changed, which source is authoritative, where state lives, how failure propagates, and what evidence supports each consequential claim.
The review consumes artifacts from Chapters 3 through 11 and asks whether their interfaces form one defensible system. It does not re-teach retrieval, orchestration, tool security, evaluation, deployment, or incident response, nor does it expand into a case study or implementation tutorial.
2. Primary Engineering Problem
Architecture labels often arrive before requirements. Once a team announces a RAG system, an agent, or a copilot, the product and its controls start bending around that label. The design process has been reversed.
The six patterns in this chapter are not mutually exclusive categories:
- RAG describes a knowledge relationship: generation is conditioned on evidence selected for this request.
- A copilot describes a user relationship: the system assists inside a human-owned workflow.
- AI search describes a product responsibility: help a user discover and inspect relevant information.
- A workflow agent describes a control shape: the system adaptively selects and executes bounded steps.
- Human-in-the-loop describes where human judgment or authority enters a flow.
- Event-driven AI describes initiation and timing: work begins from an event rather than an active user request.
A production system might therefore be an event-driven, retrieval-grounded copilot with a pre-action approval gate. The composition may be justified, but each modifier creates another seam where identity, authority, state, evidence, or failure behavior can change.
The design problem has four parts:
- State the responsibility without using a pattern name.
- Select the least powerful dominant pattern that fulfills that responsibility.
- Add only the structural modifiers justified by material requirements.
- Prove that the resulting contracts compose within the system's consequence and operating envelope.
Conceptual fit is only the beginning. A human approval pattern fails when peak arrival exceeds review capacity. An event-driven design fails when retries can duplicate irreversible effects. RAG cannot stand in for authoritative truth when the product cannot distinguish retrieved text from the source of record. A workflow agent is indefensible when its goal, action authority, or stopping condition has no enforceable bound.
Design review turns those concerns into an engineering decision, not a plausibility check on a diagram. The resulting record states whether the composition is coherent, what evidence and residual risk support that conclusion, and which future changes invalidate it.
3. Core Mental Model
Responsibility first, modifiers second
Start with a verb that states the product responsibility: discover, synthesize, recommend, draft, classify, decide, prepare, or act. Eight dimensions then expose the architecture hidden behind that verb.
| Dimension | Questions the architecture must answer |
|---|---|
| Primary responsibility | What outcome is the system accountable for, and what is explicitly a non-goal? |
| Initiator and interaction | Is work initiated by a user, schedule, event, or callback? Is it synchronous, asynchronous, or long-running? |
| Knowledge relationship | Does the system use no external grounding, retrieval-selected evidence, structured authoritative data, or a controlled mixture? |
| Control shape | Is execution single-pass, deterministic, adaptively planned within a workflow, or directed by a human? |
| Authority | May the system inform, propose, prepare, approve, commit, or act autonomously? |
| Human placement | Is the human an operator, collaborator, pre-action approver, exception handler, sampled reviewer, or accountable decision owner? |
| State and effect | Is the result stateless, durably queued, workflow stateful, a persistent mutation, or an external side effect? |
| Consequence and reversibility | Who can be affected, how quickly can harm spread, and can work be cancelled, compensated, corrected, or restored? |
The dominant pattern best explains both the responsibility and its most important control boundary; everything else is a modifier. An AI-search product may use RAG for optional synthesis, yet search remains dominant because discovery, corpus coverage, and ranking are the promise. A copilot may use a workflow agent to prepare a change, but it remains a copilot only while a person meaningfully owns the commit.
If every catalog pattern requires strained exceptions, forcing a fit hides the real design. Define a locally governed pattern or simplify the product responsibility instead.
The common architecture contract
Pattern descriptions and design proposals share a minimum claim set:
- responsibility, non-goals, initiating condition, actors, affected parties, and accountable owner;
- canonical success, abstention, escalation, partial-failure, recovery, and high-consequence flows;
- source-of-record, source-of-authority, control, data, trust, human-authority, state, and external-effect boundaries;
- the invariant that must remain true for the pattern to be acceptable;
- durable identities for the request, workflow, decision, release, evidence, action, and outcome where applicable;
- state, concurrency, retry, cancellation, compensation, and reconciliation behavior;
- demand, latency, freshness, dependency, human-capacity, cost, isolation, and recovery assumptions;
- safe degraded modes and explicit stop conditions;
- evidence references, known blind spots, unresolved gaps, and owners; and
- migration conditions and changes that require another design review.
No single large document is required. What matters is that consequential claims are findable, versioned, and mutually consistent.
Selection is a falsifiable argument
For every proposed composition, compare at least one credible simpler alternative. Add authority, adaptability, asynchrony, or another pattern only when all three statements are true:
- The added capability creates material value that a simpler design cannot economically provide.
- Its consequence and failure propagation remain within declared limits.
- The team can enforce and observe the additional contracts in production.
An architecture argument has already failed if it needs unlimited reviewer capacity, perfect retrieval, failure-free dependencies, instantaneous feedback, or reliable compliance with an unenforced instruction.
Primary responsibility selects a base architecture while retrieval, workflow, event, and human-approval modifiers create a reviewable composition.
System view
Responsibility Selects the Base; Modifiers Create Review Seams
Text equivalent
- Begin with the primary responsibility stated as a product outcome rather than an architecture label.
- If the responsibility is evidence return through discovery, ranking, and source inspection, use AI search as the base pattern.
- If the responsibility is judgment support and a capable person retains meaningful decision ownership, use a copilot as the base pattern.
- If the responsibility requires adaptive work with bounded external effect, consider a workflow agent only after a simpler deterministic path fails the value test.
- An event trigger is a modifier that adds admission, delivery, replay, ordering, policy-time, and aggregate-impact contracts.
- Retrieval is a modifier that adds evidence selection, provenance, permission, freshness, and unsupported-claim contracts.
- Human approval is a modifier only when a qualified reviewer has adequate evidence, authority, independence, and capacity to interrupt the target failure.
- A workflow modifier adds adaptive step selection, durable state, budgets, checkpoints, stop conditions, and reconciliation.
- The selected composition contains one responsibility-led base and only the modifiers whose value and contracts can be demonstrated.
- Each base-pattern choice and approved modifier enters the selected composition with an explicit owner and fitness envelope.
- Review the seams for identity continuity, state ownership, trust changes, authority changes, failure propagation, fallback behavior, and decision-grade evidence.
- If any seam broadens responsibility or authority beyond the accepted product decision, reject the composition or reopen the upstream decision.
- The approved architecture is the least powerful composition whose integrated contracts remain enforceable and observable in production.
4. System Boundary
The review boundary follows one representative item from initiation to a user-visible result or external effect, then through correction or recovery. Ownership by another team does not move a component outside that boundary when its contract can change the outcome.
Minimum review views
Six views make the integrated claim reviewable. They may share a diagram when the result stays legible; no particular notation is required.
- Responsibility and context view: users, affected parties, business process, inputs, outputs, systems of record, accountable owners, and explicit exclusions.
- Behavior and control-flow view: success, abstention, escalation, partial failure, approval, validation, fallback, stop, and recovery paths.
- Data and trust view: sensitive and untrusted flows, source authority, tenant and permission changes, retention boundaries, and policy-enforcement locations.
- State and effect view: durable state, queues, attempts, idempotency scope, human decisions, writes, external actions, compensation, and reconciliation.
- Runtime and dependency view: execution modes, effective release identity, capacity boundaries, failure domains, material dependencies, and degraded modes.
- Evidence and operations view: claims, evidence references, telemetry joins, decision and action receipts, health signals, owners, and recovery proof.
Every material node and edge needs an owner and a contract. A component inventory without behavior paths is incomplete. So is a sequence diagram that omits trust, state, or evidence boundaries.
Architecture contract map
Chapter 12 integrates the handbook's existing artifacts; it does not replace them.
| Source artifact | Claim consumed by architecture review | Interface question |
|---|---|---|
| Chapter 3: AI Fit and Risk Assessment Worksheet | Responsibility, consequence, blast radius, accountable human | Does the pattern carry more authority or reach than the accepted product decision permits? |
| Chapter 4: Production Prompt Specification | Instruction, refusal, and output contracts | Where is this version enforced, and do fallbacks preserve its contract? |
| Chapter 5: Context Assembly Plan | Source authority, permissions, freshness, and context identity | Can the resulting claim be traced to evidence authorized for this request? |
| Chapter 6: Harness Control-Loop Diagram | State, routing, validation, retry, fallback, approval, and stop control | Which invariant is enforced at each transition, including recovery paths? |
| Chapter 7: Tool Permission Matrix | Action eligibility, approval, idempotency, and effect receipts | Can any route, retry, or delegation exceed the declared action envelope? |
| Chapter 8: Evaluation Plan | Claims, scenarios, thresholds, uncertainty, and applicability | Does the evidence cover this exact composition, population, and effective release? |
| Chapter 9: AI Threat Model | Trust boundaries, policy constraints, abuse cases, and residual risk | Do data and authority flows match the approved assumptions and enforcement points? |
| Chapter 10: AI Release Manifest | Composite identity, assignment, compatibility, and recovery target | Can operators identify and control the behavior actually serving or processing work? |
| Chapter 11: AI Operations Runbook | Observable identity, objectives, containment, and recovery verification | Can the team detect a violated invariant, bound impact, and prove recovery? |
For each pattern-critical claim, the map identifies the supplying artifact and version, owner, enforcement or verification point, evidence, unresolved gap, and change trigger. A contradiction is an architecture finding. When the tool matrix permits what the threat model assumes impossible, the team does not have two locally valid documents; it has one unsafe system.
Decision boundaries
Architecture review determines whether the integrated structure and its interfaces are coherent. It does not absorb downstream authority: Chapter 8 still owns whether evaluation evidence satisfies a release claim; Chapter 9 owns governance and security acceptance; Chapter 10 owns promotion and rollback mechanics; and Chapter 11 owns incident command and recovery verification. Design approval cannot substitute for any of them.
5. Design Principles
Principle 1: Select from responsibility, not fashion
The required outcome, authority, and consequence come before the pattern name. A label that does not explain the system's actual responsibility has no design value.
Principle 2: Prefer the least powerful sufficient composition
Authority, adaptive planning, asynchrony, and pattern count multiply complexity. Each addition must earn its place through material value and enforceable contracts.
Principle 3: Protect an architecture-level invariant
Give each pattern one concise condition whose violation makes the architecture unacceptable. This invariant connects boundaries, controls, and evidence more effectively than a component list does.
Principle 4: Separate evidence, judgment, and effect
Evidence retrieval, proposal generation, decision authorization, and effect commitment carry different authority. They may share an implementation, but the architecture cannot hide the transition among them.
Principle 5: Treat humans as capacity-constrained system participants
A human becomes a control only when role, evidence, independence, authority, workload, fallback, and accountability are explicit. An “approve” button alone is not a control model.
Principle 6: Review seams before components
Locally correct components can still disagree about identity, authority, state, retries, source truth, or degraded behavior. Those seams produce some of the most dangerous failures, so ownership must cover the composite outcome rather than stop at team boundaries.
Principle 7: Make fitness assumptions bounded and owned
Demand, latency, source freshness, reviewer capacity, dependency availability, cost, consistency, isolation, and recovery belong in bounded ranges with owners and evidence. “Scalable” and “highly available” say nothing reviewable on their own.
Principle 8: Define safe failure before normal operation
Before describing the happy path, decide what happens when each material dependency or evidence channel fails: refuse, abstain, queue, become read-only or draft-only, fall back to manual work, use bounded cached data, or stop.
Principle 9: Preserve identity across time and effects
Queues, retries, human handoffs, fallbacks, and recovery must preserve user, tenant, source, workflow, decision, policy, release, action, and outcome identities. Lose that continuity and neither authority nor evidence can be reconstructed.
Principle 10: Make approval expire
Approval lasts only while its stated assumptions hold. New authority, source class, event trigger, human placement, tenancy, consequence, execution timing, fallback, or material composition reopens the decision.
6. Architecture Patterns
The six cards use a common structure so teams can compare contracts rather than implementation stacks.
Pattern 1: Retrieval-Augmented Generation
Responsibility and fit. RAG conditions a generated result on evidence selected for the current request. It fits when useful output requires synthesis across unstructured or semi-structured sources and the product can communicate the limits of that evidence. It is not a substitute for structured authoritative access when a precise current field, balance, entitlement, or status is the required answer.
Canonical flow. An authorized request establishes user, tenant, purpose, and policy identity. The Context Assembly Plan governs evidence selection, while the harness carries evidence identity through generation and validates the output contract. It then returns a result with an appropriate evidence relationship or abstains. Feedback and outcomes retain the request, evidence, prompt, release, and result identities.
Invariant. The result must not imply support, authority, or scope beyond evidence selected and authorized for this request.
Boundary and fitness. The architecture view exposes source-of-record ownership, ingestion and query-time versions, permission enforcement, retrieval result, evidence-to-claim link, structured-data handoff, and conflicting or missing-evidence path. Fitness depends on corpus scope, indexing lag, permission-change latency, retrieval and generation latency, context limits, source volatility, dependency availability, and cost per supported outcome.
Safe failure. Missing, conflicting, stale, unauthorized, or untraceable evidence leads to scoped language, source display, authoritative structured lookup, user clarification, or abstention according to consequence. A fallback must not silently remove provenance while preserving the same confident interface. A source outage may permit bounded cached evidence only when age and authority remain visible and acceptable.
Review evidence and evolution. The review needs source coverage, permission behavior, unsupported-answer paths, evidence-to-claim traces, and applicable Chapter 8 evidence. New source classes, cross-tenant retrieval, changed synthesis authority, a new fallback, or use of the result to trigger action requires re-review.
Pattern 2: Copilot
Responsibility and fit. A copilot assists a person inside a human-owned workflow by informing, recommending, drafting, or preparing a change. It fits when the user has enough context, time, authority, and meaningful options to assess the result. It does not fit when acceptance is effectively mandatory, review evidence is inaccessible, or the system commits consequential effects while the interface still implies human ownership.
Canonical flow. A user initiates work in a known task context. The copilot gathers permitted context, produces advice or a preview, shows material evidence and uncertainty, and preserves edit, reject, defer, and escalation paths. Any commit of a prepared change remains a separate, attributable action with its own authorization and receipt.
Invariant. The user remains the meaningful owner of any outcome represented as human-decided; system behavior must not make acceptance the only practical path.
Boundary and fitness. The design distinguishes advice, draft, prepared mutation, and committed effect. It states user qualifications, decision rights, evidence visible at the decision point, reversibility, workflow latency, interruption cost, acceptance defaults, and the population that cannot review meaningfully. Fitness depends on whether assistance reduces effort without displacing required judgment or creating review volume the workflow cannot absorb.
Safe failure. When grounding, model, or tool dependencies degrade, the copilot becomes read-only, draft-only, explicitly limited, or unavailable. It does not preserve a commit path after losing the evidence that made review meaningful. Undo, correction, and escalation remain available for completed work within the declared consequence window.
Review evidence and evolution. Representative preview-to-commit traces reveal more than a raw acceptance rate. Sample rejected and edited outputs, evidence visibility, time on review, automation-bias risks, and downstream outcomes. Default acceptance, bulk application, automatic commit, reduced evidence display, or migration from user initiation to event initiation materially changes the architecture.
Pattern 3: AI Search
Responsibility and fit. AI search owns discovery over a declared corpus or set of systems. Its primary promise is that relevant material can be found, ranked, inspected, and corrected. Optional synthesis may use RAG, but synthesis does not erase the search responsibility. Search is preferable to answer generation when coverage, source diversity, user exploration, or direct source inspection matters more than a concise synthesized response.
Canonical flow. Query interpretation starts with user, tenant, and filter context. Eligible corpora then feed retrieval and ranking, and the interface exposes source identity, scope, and freshness. Optional synthesis remains visibly distinct from ranked results; users can still open sources, reformulate, filter, and report omissions or errors.
Invariant. The experience must not present a well-supported subset as complete discovery when material eligible sources or result classes are absent.
Boundary and fitness. The architecture names corpus owners, inclusion and exclusion rules, indexing identity and lag, permission filters, ranking objective, diversity and coverage expectations, query interpretation boundary, synthesis boundary, and correction path. Fitness depends on corpus change rate, query mix, latency, tenancy, ranking scale, freshness, and the consequence of systematic omission.
Safe failure. When an index, corpus, filter, or ranking dependency is incomplete, the system marks affected scope, narrows the result set explicitly, falls back to source browsing, or withholds synthesis. “No results” must be distinguishable from missing or stale coverage.
Review evidence and evolution. Useful evidence covers each corpus and material slice, ranking and filter behavior, source-opening paths, indexing lag, empty-result semantics, and optional synthesis. Adding an authoritative corpus, changing the ranking objective, hiding sources, or using search output as an automated decision input triggers re-review.
Pattern 4: Workflow Agent
Responsibility and fit. A workflow agent adaptively chooses steps within a bounded goal and may use tools to prepare or perform work. It fits when task variation cannot be enumerated economically as a deterministic workflow and adaptive selection delivers material, measurable value inside a controllable action envelope. A deterministic workflow remains the default when steps, branches, and recovery can be specified at reasonable cost.
Canonical flow. An authorized actor supplies a goal with validated transformations, success conditions, non-goals, budget, and expiry. A deterministic outer workflow owns durable state; within it, the agent may propose or select bounded steps. State-specific validation and approval govern plans and actions. Identity follows every attempt, delegation, tool call, external effect, checkpoint, and outcome through completion, cancellation, compensation, or reconciliation.
Invariant. No adaptive decision may expand the approved goal, data, cost, time, delegation, or action authority without a new authorized transition.
Boundary and fitness. The design exposes plan representation, mutation rules, progress measures, uncertainty, budgets, checkpoints, stop and escalation conditions, workflow and attempt identity, tool authority by state, concurrency, cycle prevention, retry ownership, compensation, and orphan handling. Fitness includes workflow duration, fan-out, dependency volatility, state consistency, action reversibility, supervision capacity, unit economics, and recovery complexity.
Safe failure. Dependency or evidence loss pauses at a durable checkpoint, narrows the plan, switches to read-only preparation, requests human direction, or stops. Timeouts and cancellation revoke future work but do not pretend completed external effects disappeared. Recovery reconciles queued, in-flight, completed, duplicated, and orphaned actions before resumption.
Review evidence and evolution. Walk a changed plan, rejected action, duplicate attempt, cancellation, dependency outage, and irreversible-effect proposal. Each path should expose goal-boundary enforcement, action receipts, state recovery, and applicable evaluation evidence. New tools, broader goal transformation, delegation, longer duration, higher budgets, or autonomous commit reopens review.
Pattern 5: Human-in-the-Loop System
Responsibility and fit. Human-in-the-loop places human judgment or authority at a named point in a flow. The person may be a collaborator, pre-action approver, exception handler, sampled reviewer, or accountable decision owner. These placements are not interchangeable. The pattern fits only when the human can actually break the targeted failure path.
Canonical flow. Eligibility and routing determine what enters the human queue and which population does not. A reviewer receives the exact proposal, evidence, alternatives, uncertainty, policy context, and authority needed for the decision. The record binds reviewer identity and role to versioned evidence, choice, edits, required rationale, expiry, and resulting action. Disagreement, escalation, appeal, and correction remain explicit paths.
Invariant. A consequential outcome that relies on human control cannot proceed without a qualified, informed, attributable decision made within the control's declared capacity and independence constraints.
Boundary and fitness. The architecture states reviewer qualifications, accountability, conflicts, coverage hours, arrival rate, service-time distribution, queue limit, abandonment, escalation, sampling logic, and overload behavior. It distinguishes pre-action approval from exception and sampled post-action review. Raw acceptance is not assumed to be a quality label, and a sampled reviewer is not represented as the owner of unseen outcomes.
Safe failure. When reviewers are unavailable, disagree, fall behind, or show evidence of rubber-stamping, the system queues within a bounded window, narrows eligibility, routes to an independent reviewer, becomes draft-only, follows a separately authorized manual path, or stops. Bypass paths are explicit, time-bounded, and governed by consequence.
Review evidence and evolution. Peak-capacity assumptions need queue simulations or operating evidence, not optimism. The review also examines decision quality, overrides, escalations, bypasses, unseen populations, and whether human judgment actually interrupts the target failure. Moving a gate after an effect, replacing full review with sampling, changing reviewer qualifications, or increasing automated eligibility requires re-review.
Pattern 6: Event-Driven AI
Responsibility and fit. Event-driven AI initiates classification, drafting, recommendation, or action from a system event, schedule, or callback. It fits when work does not require an active user request and when admission, state, delivery, and replay semantics can bound aggregate impact. It is unsafe when one behavioral error can propagate through an unbounded stream or when duplicate external effects cannot be prevented or reconciled.
Canonical flow. An authenticated producer emits a versioned event with event time, observation time, source, tenant, and stable identity. Admission evaluates eligibility under the effective policy and release. Partitioning, ordering, lateness, and deduplication rules govern processing, which records distinct identities for the attempt, decision, output, delivery, acknowledgement, and external effect. Terminal status is explicit: completed, rejected, quarantined, dead-lettered, cancelled, compensated, or awaiting reconciliation.
Invariant. Reordering, redelivery, replay, or recovery must not create an effect that was ineligible under the governing state or duplicate an effect beyond the declared idempotency boundary.
Boundary and fitness. The design states producer ownership, schema compatibility, authenticity, ordering scope, watermark and lateness behavior, policy-time semantics, partitioning, admission, buffering, backpressure, throttling, load shedding, poison-event isolation, retry, replay, cancellation, compensation, and reconciliation. Fitness depends on steady and burst volume, backlog tolerance, state freshness, effect reversibility, downstream capacity, tenant isolation, and recovery time for queued work.
Safe failure. Incompatible or suspicious events are quarantined. Saturation invokes bounded buffering, per-tenant throttling, narrowed eligibility, or load shedding rather than uncontrolled propagation. A dependency outage may keep work queued only while policy, freshness, capacity, and expiry remain valid. Recovery defines which queued events use their original release and policy, which are re-evaluated, and which must never replay.
Review evidence and evolution. Exercise duplicate, late, reordered, poison, burst, replay, schema-change, stale-state, and partial-delivery paths. The evidence must distinguish no eligible work from stalled or missing processing. New event producers, changed eligibility time, higher batch authority, cross-tenant partitions, or action-producing replay requires re-review.
Composition rules
Most production systems combine patterns. The combination is acceptable only if it preserves the strongest relevant invariant.
For each modifier, record the capability gained, the simpler alternative rejected, and the new contracts incurred. At every seam, specify:
- identity entering and leaving the handoff;
- state owner and concurrency rule;
- authority before and after the handoff;
- data and trust transformation;
- failure, retry, timeout, and fallback behavior;
- evidence produced and evidence that may be lost; and
- the owner accountable for the integrated outcome.
Composition makes risk multiplicative. An event-driven agent using stale retrieval can repeat a wrong action across an entire backlog; that failure is more than the sum of event, agent, and RAG risks. Fallbacks deserve particular scrutiny. A RAG copilot that loses retrieval and continues generating has silently changed pattern. A human-gated agent whose retry bypasses the gate has silently changed authority.
The composition also needs its own fitness envelope: demand and burst profile, response and completion objectives, source freshness, state consistency, dependency budgets, reviewer capacity, cost per eligible and completed outcome, isolation requirements, degraded modes, recovery expectations, uncertainty, and assumption owners. Unbounded queues, reviewer demand, fan-out, retries, or recovery work make the design unfit.
7. Failure Modes
Review architecture failures as broken invariants with observable consequences.
| Initiating condition | Violated invariant and propagation | Evidence and containment | Preventive contract |
|---|---|---|---|
| A fashionable label is selected before responsibility | Authority or product promise exceeds the actual need; controls are designed for the label | Compare the responsibility statement and simpler option; remove unjustified modifiers | Responsibility-led selection and explicit non-goals |
| Evidence is stale, incomplete, conflicting, or unauthorized | A RAG result implies support it does not possess and may spread into decisions or actions | Preserve evidence identity, mark affected results, disable synthesis or abstain | Source authority, permission, freshness, and evidence-to-claim contract |
| Search coverage omits a source class | Ranked results appear complete while a population is systematically invisible | Expose corpus health and missing scope; narrow claims or fall back to source browsing | Corpus inclusion, coverage, freshness, and empty-result semantics |
| A copilot preview becomes a one-click or automatic commit | Interface language preserves human ownership while practical authority moves to the system | Disable commit, restore preview and meaningful alternatives, inspect affected outcomes | Separation of advice, prepared change, authorization, and effect |
| Adaptive work exceeds its goal or budget | An agent expands scope through plan mutation, delegation, loops, or retries | Pause durable workflows, revoke authority, reconcile effects and orphaned work | Goal, state, budget, action-envelope, and stop contracts |
| Review demand exceeds human capacity | Queues grow, reviewers rubber-stamp, bypasses expand, or unseen work proceeds | Narrow eligibility, become draft-only, add qualified capacity, or stop | Capacity envelope, queue limits, independence, and overload behavior |
| Events arrive late, duplicated, reordered, or replayed | Governing state changes or external effects repeat across a batch | Quarantine or throttle partitions, stop effects, reconcile receipts and queued work | Event identity, policy-time, ordering, idempotency, and replay contract |
| Identity breaks across a queue, fallback, human gate, or action | Authorization and evidence cannot be tied to the effective release or outcome | Halt consequential paths, reconstruct only where receipts permit, declare unknown scope | End-to-end user, tenant, workflow, decision, release, and effect identity |
| A fallback removes evidence or approval | The system continues under a weaker architecture while presenting the original promise | Disable the fallback, bound affected traffic, communicate degraded capability | Explicit pattern-changing fallback and consequence limit |
| Locally correct components hold inconsistent contracts | Permission, release, context, evaluation, or recovery assumptions conflict at a seam | Treat contradiction as a blocking integrated finding and contain the broader authority | Versioned architecture contract map with interface owners |
| A shared dependency fails across composite paths | Correlated failure defeats apparently independent controls | Isolate affected tenants or patterns, reduce authority, and use independent evidence where defined | Failure-domain, isolation, and degraded-mode contract |
| Product behavior evolves without review | A suggestion becomes an action, a human gate becomes sampling, or sync work moves to a queue | Revert or contain the unapproved capability and reassess impacted outcomes | Material-change triggers and expiring design approval |
Reachable components do not prove recovery. Recovery also accounts for queued and in-flight work, durable state, human decisions, user-visible results, external effects, correction, and evidence that the architecture invariant has been restored.
8. Tradeoffs
A useful tradeoff names the default, the conditions for deviation, and the owner authorized to accept the consequence.
| Decision | Preferred default | Justified deviation | Required proof / authority |
|---|---|---|---|
| Structured authoritative access, search results, or retrieval-grounded generation | Use structured access for precise current facts; expose ranked results for discovery | Synthesis across selected evidence creates material value and unsupported claims can be bounded | Source authority and failure behavior; product and architecture owners |
| Ranked discovery or synthesized answer | Preserve discovery and source inspection when coverage matters | Concision materially improves the task and evidence-to-claim behavior is fit for consequence | Coverage plus synthesis evidence; product owner |
| Advice, draft, prepared change, or autonomous commit | Stop at the lowest authority that completes the user need | Additional authority removes material friction and remains reversible or strongly controlled | Outcome value, action containment, and risk acceptance; accountable business and action owners |
| Deterministic workflow or bounded adaptive planning | Use deterministic branches where task variation can be enumerated economically | Variation is material, adaptive choice measurably helps, and goal/action envelopes are enforceable | Comparative evidence, budget and recovery paths; workflow owner |
| Pre-action approval, exception review, or sampled post-action review | Put review before consequential, hard-to-reverse effects | Later or sampled review has bounded consequence and detection/remediation occur within tolerance | Capacity, sampling coverage, correction window, and accepted residual risk; accountable decision owner |
| Synchronous or asynchronous/event-driven execution | Prefer synchronous behavior when the task and dependency budget permit | Duration, burst absorption, decoupling, or offline initiation materially requires durable work | Queue, identity, expiry, replay, and recovery proof; runtime and product owners |
| Combined decision and action or separated stages | Separate proposals from consequential effects | Atomic automation is necessary and the combined boundary has equivalent authorization and evidence | End-to-end action trace and containment; action owner |
| Shared service or isolated cells | Share when failure and data boundaries are compatible | Consequence, tenancy, jurisdiction, dependency, or recovery needs demand isolation | Failure-domain and cost evidence; architecture and trust owners |
| Rich composition or explicit product limitation | Keep the fewest patterns and modifiers | Each addition delivers material value and its seam contracts can be owned, tested, and operated | Option comparison and complexity budget; design decision owner |
Approval authority must match the consequence. An architecture lead may accept implementation complexity, but only the accountable product, security, compliance, or operational owner may accept consequences assigned to that role by earlier chapters.
9. Production Checklist
Design-review lifecycle
- Charter the decision. State the requested decision, owner, scope, timing, system responsibility, consequence, and material exclusions.
- Declare the pattern argument. Name the dominant pattern, modifiers, fitness envelope, and at least one credible simpler alternative.
- Qualify the pre-read. Provide current architecture views, the contract map, representative paths, open assumptions, and exact artifact versions. Return an incomplete package before the review meeting.
- Assign reviewer roles. Include owners able to challenge product responsibility, architecture, data and trust, state and action, evidence, runtime, and operations. Identify who advises, who may block, and who decides.
- Walk behavior. Trace success, abstention, escalation, partial failure, high-consequence effect, containment, and recovery. Use real identities and contract boundaries, not only component names.
- Challenge assumptions. Test capacity, dependency, state, authority, source truth, degraded mode, correlated failure, cost, and change assumptions. Seek disconfirming evidence.
- Record findings. Give each finding a consequence, severity, owner, due date, verification method, and blocking status.
- Issue a disposition. Use approve, approve with conditions, return for redesign, reject, or defer for missing evidence. Record residual risks, authorized acceptors, dissent, conditions, and expiry.
- Verify closure. An owner supplies closure evidence; the designated reviewer verifies it. A meeting promise is not closure.
- Reopen on material change. Responsibility, authority, human placement, source class, event trigger, timing, tenancy, consequence, fallback, or composition changes invalidate affected parts of the decision.
Pre-read completeness gate
Do not convene a review until the package can show—not merely assert—the following:
- a responsibility statement, non-goals, affected parties, consequence, and accountable owner;
- the dominant pattern, modifiers, rejected simpler option, and disqualifying conditions;
- all six minimum views with owners and contracts for material nodes and edges;
- the pattern invariant and its enforcement and observation points;
- a versioned contract map for Chapters 3–11 with contradictions and missing interfaces visible;
- representative success, refusal, escalation, partial-failure, effect, correction, and recovery traces;
- source, workflow, decision, policy, release, action, and outcome identity where applicable;
- bounded demand, latency, freshness, dependency, reviewer, cost, isolation, and recovery assumptions;
- safe behavior for unavailable evidence, dependencies, reviewers, queues, and control channels;
- evidence coverage and its absent populations, paths, dependencies, and uncertainty;
- containment and reconciliation for queued, in-flight, persisted, visible, and externally effective work; and
- material-change triggers, proposed decision authority, residual risks, and expiry.
10. Design Review Questions
- Show the responsibility-to-pattern argument. Which simpler architecture was rejected, and what evidence justified the added power?
- Which pattern is dominant, which are modifiers, and where does each modifier change identity, state, trust, authority, failure, or evidence?
- What single invariant makes this architecture acceptable? Where is it enforced, observed, and restored after failure?
- Trace one item through success, abstention, human escalation, partial failure, external effect, correction, and recovery. Where can identity be lost?
- Which system is authoritative at each decision point? What happens when sources conflict, are stale, or are unavailable?
- What does the system do before and after a human decision? Can the reviewer genuinely reject, defer, edit, escalate, undo, or appeal?
- Demonstrate that human and machine capacity assumptions hold at peak and under backlog. What is shed, queued, narrowed, or stopped first?
- Show how retry, replay, fallback, cancellation, and recovery preserve eligibility and avoid duplicate or unauthorized effects.
- Which artifact from Chapters 3–11 supplies each critical claim? Where do versions or assumptions disagree?
- Which populations, failure paths, dependencies, and effective releases are absent from current evidence?
- What correlated failure can cross pattern seams or shared dependencies, and what bounds its impact?
- Which mutable assumption would make this design invalid tomorrow, and how will that change trigger re-review?
- What findings block approval, who may accept each residual consequence, and what exact evidence closes a condition?
11. Main Artifact: Reference Architecture Catalog
The Reference Architecture Catalog is one governed artifact expressed through eight coordinated views. Those views may live in separate files, but they share one version and decision history. Chapter 14 can package blank forms; Chapter 12 defines the substance they must preserve.
View 1: Catalog governance
Catalog governance records identity, owner, version, applicable product and consequence classes, approved pattern definitions, known limits, review cadence, and change history. A local deviation needs its own owner, rationale, expiry, and superseding path. The downstream handoff to the Case Study Analysis Format carries the catalog version, design-review decision ID, accepted conditions, unresolved findings, and material-change triggers. Chapter 14's Complete AI Systems Engineering Toolkit may package these forms, but the identities and decision semantics remain intact. Without a governed definition, a pattern name is only vocabulary.
View 2: Selection matrix
| Pattern | Role in composition | Best fit | Avoid / disqualify when | Core invariant |
|---|---|---|---|---|
| RAG | Knowledge and synthesis modifier; sometimes dominant for grounded answers | Generated output must synthesize selected unstructured evidence | Precise structured truth is required, permissions cannot be enforced, or unsupported output cannot be bounded | Result implies no more support than authorized selected evidence provides |
| Copilot | User relationship and authority pattern | A capable user meaningfully owns review and outcome | Acceptance is ceremonial or effects occur before informed user action | Human ownership represented by the interface is real in the flow |
| AI search | Discovery responsibility pattern | Corpus discovery, ranking, coverage, and source inspection are central | Missing coverage cannot be exposed or synthesis is mistaken for complete discovery | Presented scope does not conceal material eligible omissions |
| Workflow agent | Adaptive control and action pattern | Material task variation needs bounded adaptive step selection | Goals, budgets, state, tools, stops, or effects cannot be bounded | Adaptive choice never expands the authorized envelope |
| Human-in-the-loop | Human judgment or control modifier | A qualified human can interrupt the targeted failure path | Capacity, authority, evidence, independence, or safe overload behavior is absent | Required consequential outcomes receive meaningful human control |
| Event-driven AI | Initiation, timing, and scale modifier | Work must begin asynchronously from events and aggregate effects can be contained | Delivery or replay can duplicate unbounded effects or governing state is unknowable | Redelivery and recovery preserve eligibility and effect uniqueness |
The selection record also captures primary responsibility, initiator, knowledge relationship, control shape, authority, human placement, state and effect, consequence, reversibility, and the reason no simpler option suffices.
View 3: Common architecture contract
For the proposed design, record:
- canonical flows and the six architecture views;
- material nodes, edges, owners, interfaces, and invariants;
- identity continuity across request, source, workflow, decision, release, action, and outcome;
- fitness ranges for demand, latency, freshness, consistency, dependencies, reviewers, cost, isolation, and recovery;
- safe degraded modes, containment units, reconciliation, and stop conditions;
- evidence references, blind spots, open assumptions, and owners; and
- migration, coexistence, expiry, and re-review triggers.
The source contract map belongs in this view. A chapter number or document link without an exact artifact identity is insufficient.
| Source artifact | Identity / version | Accepted decision / constraints / validity | Interface owner | Enforcement / verification point / evidence | Unresolved gap / change trigger |
|---|---|---|---|---|---|
| AI Fit and Risk Assessment Worksheet | |||||
| Production Prompt Specification | |||||
| Context Assembly Plan | |||||
| Harness Control-Loop Diagram | |||||
| Tool Permission Matrix | |||||
| Evaluation Plan | |||||
| AI Threat Model | |||||
| AI Release Manifest | |||||
| AI Operations Runbook |
The integrated review is a monotonic handoff: it may expose contradictions, narrow a composition, block approval, or require stronger evidence. It cannot silently broaden a source artifact's scope, eligibility, population, authority, data use, guarantee, accepted risk, or validity envelope. The owning artifact must approve any broadening before the architecture decision relies on it.
View 4: Pattern cards
Each selected pattern uses the common fields from Section 6: responsibility and fit; canonical flow; invariant; boundaries and fitness; safe failure; and review evidence and evolution. Link to the authoritative artifacts from Chapters 3–11 rather than copying their methods. Cards for patterns absent from the proposed composition remain catalog references, not mandatory paperwork.
View 5: Composition map
| Seam | Upstream pattern / contract | Downstream pattern / contract | Identity / state handoff | Authority change | Failure / fallback | Integrated owner |
|---|---|---|---|---|---|---|
| Example: search results to optional synthesis | AI search owns eligible corpus and ranked result identity | RAG owns evidence-conditioned synthesis | Query, tenant, corpus, index, result, and evidence identities preserved | Discovery becomes generated interpretation, not action | Missing corpus disables completeness claim; lost evidence disables synthesis | Named end-to-end product owner |
A completed map covers correlated failure, partial completion, recovery, and the complexity budget for every added modifier.
View 6: Option comparison record
| Criterion | Simpler option | Proposed option | Assumption range / evidence | Consequence if wrong | Decision owner |
|---|---|---|---|---|---|
| Responsibility coverage | |||||
| Authority and reversibility | |||||
| Source truth and coverage | |||||
| State and recovery | |||||
| Human capacity | |||||
| Latency, scale, and cost | |||||
| Evidence and operational control |
View 7: Design-review record
The review record captures the charter, decision owner, reviewer rights, artifact versions, walkthroughs, findings, conditions, residual risks and their authorized acceptors, dissent, disposition, decision date, expiry, and material-change triggers. Every condition names its consequence, owner, due date, verification evidence, and closure authority.
Valid dispositions are:
- Approve: the architecture argument is coherent and no blocking condition remains.
- Approve with conditions: bounded work may proceed while named, time-limited conditions are closed before the specified consequence or stage increases.
- Return for redesign: the responsibility, pattern, composition, or invariant is not credible in its current form.
- Reject: the design cannot satisfy the accepted responsibility and consequence within feasible controls.
- Defer for missing evidence: the design may be plausible, but the decision cannot be made from the supplied evidence.
View 8: Closure and evolution record
Closure and evolution records track each finding, owner, proposed change, evidence, verifier, closure date, approved deviation, expiry, and superseding review. Changes to responsibility, authority, human placement, source class, event initiation, execution timing, tenancy, consequence, pattern composition, or fallback explicitly reopen the decision.
Compact selection walkthrough
Consider a bounded need: helping an engineer work with a large internal knowledge corpus.
For discovery, AI search is the dominant pattern. Its contract prioritizes corpus coverage, ranking, source visibility, and a reliable path to the source. Optional synthesis is a RAG modifier, not a new source of truth. When an index is stale, the safe mode exposes the missing scope or returns the user to source browsing.
For a grounded draft that the engineer meaningfully reviews, the dominant pattern is a copilot with a RAG modifier. This composition adds evidence-to-claim identity, edit and reject paths, preview-to-commit separation, and proof that the user can assess the result. It fails the fit test when workload or expertise makes review ceremonial.
For investigation, system changes, and completion reporting across varying tasks, a workflow agent becomes a candidate. It adds durable state, adaptive planning, tool authority, budgets, checkpoints, action receipts, and reconciliation. Task variation must create measurable value that neither a deterministic workflow nor a copilot can provide. Even then, accepted consequences may still require pre-action approval rather than autonomous commit.
Suppose the team selects AI search with optional RAG synthesis. The decision records search as dominant, RAG as a modifier, direct source inspection as the degraded mode, and “no completeness claim when an eligible corpus is unavailable” as the integrated invariant. Approval remains conditional until corpus-health identity is joined to every synthesized response before exposure expands. The decision is durable because it says what was chosen, why the more powerful agent option was rejected, what remains open, who owns it, and which future changes reopen review.
12. Key Takeaways
- Reference architectures are governed contracts for responsibility, boundaries, invariants, and failure—not branded component diagrams.
- The six patterns describe different architecture dimensions and often compose; name the dominant responsibility pattern and each modifier explicitly.
- Authority, adaptability, asynchrony, and extra patterns are justified only when material value and enforceable evidence outweigh their complexity.
- Seams among the artifacts from Chapters 3–11 need integrated review because locally correct contracts can still form an unsafe system.
- Human review, retrieval, retries, fallbacks, queues, and recovery must be evaluated as production behavior with bounded capacity and consequence.
- A design review is complete only after it records a disposition, conditions, residual risk, expiry, closure evidence, and material-change triggers.
- The Reference Architecture Catalog is a living decision record for selecting, comparing, composing, and reviewing patterns, not a recipe collection.