Chapter 10: Deployment and Runtime Architecture
1. Chapter Overview
An AI system can change behavior without a conventional application deployment. Edit a prompt bundle, resolve a model alias differently, refresh a context index, select a new route, or use a runtime flag to expand exposure, and quality, latency, cost, authority, or risk may move while the application commit stays put.
The primary engineering decision is how to assemble the complete behavioral configuration into a reproducible release candidate, promote the same candidate through controlled environments, assign it to runtime work, and recover a known-good state when the result is unacceptable.
This chapter owns deployment topology, composite versioning, prompt/model/context-index releases, runtime routing, feature flags, configuration distribution, progressive exposure, rollback, and CI/CD properties for AI systems. It consumes the versionable prompt, context, harness, and tool contracts from Chapters 4 through 7; valid evaluation evidence from Chapter 8; security, privacy, governance, and dependency constraints from Chapter 9; and operational signals from Chapter 11. It does not recreate those decisions.
The main artifact is an AI Release Manifest: one living control record that describes what was intended, what was actually deployed, who or what received it, why each transition occurred, and how recovery was verified.
2. Primary Engineering Problem
Traditional release practice often assumes that a build artifact is the deployable unit and an environment is a location in which it runs. In an AI system, behavior emerges from a dependency graph that may include:
- application and harness code;
- prompt templates and included policy fragments;
- allowed model identities and parameter sets;
- context sources, transformation pipelines, and index snapshots;
- tool schemas and permission-policy identities;
- routing, fallback, and admission rules;
- runtime flags and environment or tenant bindings;
- serving topology, capacity limits, and external dependencies.
Those parts form a dependency graph, not a menu of independent versions. A prompt may assume a tool schema; a harness, a particular output contract; an index, a compatible metadata filter. A routing policy may select only model configurations covered by the prompt and evaluation evidence. Changing one node can invalidate compatibility or evidence elsewhere in the graph.
The engineering problem is therefore:
Build, promote, expose, and recover an immutable composite candidate whose dependencies and effective runtime state are known.
A release record cannot stop at what the team intended. Production also has to show what became effective. Configuration propagation, mutable external services, stale workers, long-running workflows, partial index cutovers, and operators under pressure all sit between those two states. Release architecture makes those failure surfaces explicit.
Material change is a dependency question
File type and diff size say little about release depth. A one-line alias change can alter every request, while a large refactor may preserve behavior. Assess every proposed change against the declared dependency graph:
- Identify affected behaviors, requirements, risk claims, workflows, cohorts, tenants, and regions.
- Identify dependent prompts, indexes, schemas, policies, evaluators, tools, runtime components, and persisted state.
- Determine required rebuilds, migrations, compatibility checks, environment checks, and recovery work.
- Classify existing evaluation and assurance evidence as applicable, supplement-required, or invalid; Chapters 8 and 9 define those judgments.
- Select rollout depth, observation window, approval authority, and rollback or roll-forward readiness.
- Record any skipped gate with a reason, accountable owner, compensating control, and expiry.
Apply the same assessment to changes initiated outside the repository: model retirement, silent service revision, corpus mutation, permission or credential-scope changes, dependency-policy updates, and infrastructure-limit changes. A system that cannot detect an external dependency change cannot claim to reproduce the candidate.
3. Core Mental Model
Deployment is a controlled transition from declared candidate to effective runtime assignment.
The composite candidate
A candidate has one immutable release identifier plus a graph of immutable, or strongest-available, component identities. Each component record carries provenance, compatibility constraints, and dependency edges. Candidate identity remains independent of the environment; endpoints, credentials, capacity, and residency bindings are applied separately and recorded.
Prefer content-addressed identities or immutable build references. Mutable names such as latest, production, default-model, or current-index are locators, not identities. If an external service exposes no stable digest, record the resolved identity, resolution time, observed capabilities, revision semantics, detection mechanism, and residual reproducibility limit. Chapter 9 decides whether the dependency is acceptable; Chapter 10 decides whether its runtime identity is visible enough for promotion and recovery.
The candidate should include identities for:
- source revision, build output, build process, provenance, and integrity evidence;
- prompt bundle, included templates, policy fragments, and output-contract versions;
- allowed model configurations, parameters, runtime revisions, and alias-resolution rules;
- context corpus cutoff, ingestion, parsing, chunking, embedding, filtering, ranking, and index snapshot;
- harness, validation, routing-policy, tool-schema, and permission-policy versions;
- runtime configuration schema, declared values, flags, and approved overrides;
- infrastructure or serving-profile assumptions that materially affect capacity, latency, fallback, or behavior;
- parent candidate, known-good recovery target, dependency edges, and compatibility constraints.
Evaluation results, governance approvals, and operational signals are not executable components of the candidate. They are evidence and constraints bound to its exact identity.
Promotion, exposure, and assignment are different states
Deploying bits, making a variant eligible, assigning traffic, and making it the default are separate operations. Represent them as an auditable state machine rather than a collection of overloaded flags:
- Assembled — the candidate and dependency graph are immutable.
- Gates pending — required evidence, compatibility checks, and approvals are unresolved.
- Environment approved — the candidate may enter a named environment with declared bindings.
- Deployed — the candidate is present but receives no authoritative production decisions.
- Exercised — it receives synthetic, shadow, or otherwise non-authoritative traffic under explicit side-effect rules.
- Initially exposed — a stable, named cohort receives authoritative behavior.
- Held — exposure is fixed for a declared observation and outcome-latency window.
- Expanded — exposure moves through predefined stages.
- Steady state — the candidate has its intended default assignment.
- Constrained, paused, rolled back, superseded, or retired — exposure or eligibility is reduced and closure obligations begin.
For every transition, record the human or automation identity, preconditions, evidence snapshot, target environment, assignment, time, policy version, result, and reversal path. CI/CD assembles once, enforces gates, promotes the same candidate, applies controlled bindings, and issues a deployment receipt. A pipeline that quietly rebuilds a materially different candidate in each environment has broken the evidence chain.
The five planes of release architecture
Five cooperating planes make the release state understandable:
- Artifact plane: immutable code, prompts, model-policy records, indexes, harnesses, schemas, and policy assets.
- Release plane: composite identity, dependency graph, material-change assessment, evidence links, provenance, and approvals.
- Control plane: environment promotion, configuration distribution, cohort assignment, routing policy, exposure transitions, and recovery commands.
- Data plane: request and workflow execution pinned to an effective release identity.
- Evidence plane: deployment receipts, effective-state observations, release-health inputs, and transition decisions.
The central invariant is simple: every authoritative request or durable workflow can identify the effective composite release and route decision that governed it. Without that invariant, reproduction and composite recovery are guesses.
A versioned composite moves through evidence-gated promotion, routing eligibility, cohort assignment, staged exposure, and compatible rollback.
System view
Composite Release Advances Through Controlled Exposure and Recovery
Text equivalent
- The build identity names the immutable output and provenance that every environment must use.
- The composite manifest binds that build to prompt, model-policy, context-index, harness, tool-policy, routing, configuration, and infrastructure identities under one release ID.
- The release owner submits the exact manifest to an evidence gate; applicable evaluation, compatibility, security, governance, and expiry conditions must pass.
- A successful gate authorizes promotion of the same candidate into a named environment with controlled bindings, but promotion alone assigns no production traffic.
- Routing eligibility restricts runtime choice to approved composite destinations and records the policy that made each destination available.
- Cohort assignment admits only the declared population and pins conversations, retries, and durable workflows with stable affinity.
- Staged exposure gives the assigned cohort authoritative behavior only within the current minimum, maximum, and time window.
- Release health evaluates candidate- and stage-bound signals, denominators, delayed outcomes, zero-tolerance events, and missing-evidence rules.
- The release-health authority advances, holds, constrains, or reverses exposure according to the predeclared contract.
- Recovery selects a compatible, authorized rollback target that can interpret shared state, preserve in-flight work, and carry restored load.
- The assignment controller directs new work to the recovery target while incompatible state or irreversible effects follow their declared containment, compensation, or roll-forward path.
- The deployment receipt records promotion, effective exposure, transition decisions, data-plane convergence, and verified recovery so operators can distinguish actual state from control-plane intent.
4. System Boundary
Chapter 10 owns the mechanics that turn approved, versionable inputs into effective runtime behavior.
Prompt, model, and tool assets enter as contracts
Chapter 4 defines prompt structure and prompt-specific versioning, Chapter 6 defines harness behavior, and Chapter 7 defines tool authority, idempotency, and compensation. Deployment records the resulting immutable identities, compatibility edges, promotion state, exposure, and recovery behavior without redesigning those internals.
Context design and index release are separate decisions
Chapter 5 decides what information should be assembled, how it is ranked, and which permission and freshness properties apply. Here, the release process builds and promotes that declared index implementation without exposing a partial or incompatible state.
An index release records source scope and cutoff, transformation-pipeline identities, schema compatibility, permission projection, completeness and lag, duplicate and tombstone handling, and the effective query identity. Prefer isolated build and atomic routing or alias cutover over in-place mutation when reproducibility or rollback matters. Dual-write, shadow-query, or blue-green strategies can reduce uncertainty, but shadow paths must not create authoritative state or external effects unless explicitly designed to do so.
Permission revocations and privacy deletions require special treatment during and after cutover. Retaining an old snapshot may support reproduction and rollback but conflict with deletion or retention obligations supplied by Chapter 9. The release plan must state which obligation wins, how deleted content is removed across retained snapshots, and when a snapshot ceases to be a valid recovery target.
Evaluation and governance constrain promotion
Chapter 8 owns evaluation methodology, uncertainty, thresholds, and evidence validity; Chapter 9 owns threat, privacy, dependency, governance, and residual-risk decisions. Deployment binds those outputs to the exact candidate and blocks a transition when a required input is absent, stale, or invalidated by material change.
A security approval is not a free-form checkbox. It is a versioned constraint set with scope, conditions, invalidation triggers, owner, and expiry. The release controller enforces those conditions during promotion and routing without inventing new policy.
Release health is not an observability platform
Chapter 11 owns telemetry collection, alerting, incident command, and ongoing service objectives. This chapter selects the approved signals that govern a particular exposure transition. The resulting release-health contract binds a candidate and stage to a population, denominator, slice, baseline, threshold, window, outcome delay, missing-data rule, decision authority, and action.
Loss of required evidence is not success. Depending on consequence, missing or indeterminate evidence should hold, constrain, or reverse exposure.
5. Design Principles
Principle 1: Identify the complete behavior, not only the build
One immutable release identifier must resolve to every behavior-affecting asset and dependency. A mutable alias may help with discovery, but approval waits until that alias resolves to a recorded identity.
Principle 2: Build once and promote the same candidate
Build in an isolated, repeatable process, capture provenance, enforce policy gates, and then promote the same candidate through controlled environment bindings. Rebuilding per environment severs the chain between evidence and production behavior.
Principle 3: Let dependencies determine release depth
Material-change assessment determines what must be rebuilt, re-evaluated, re-approved, migrated, observed, and made recoverable. Do not make every change pay every cost, and do not let a small diff bypass a large dependency fan-out.
Principle 4: Separate eligibility, assignment, and exposure
A deployed candidate is not automatically eligible for every workflow. An eligible destination is not automatically assigned. An assignment rule is not approval to expand exposure. Keep these states explicit and queryable.
Principle 5: Preserve cohort and workflow integrity
Deterministic assignment and declared affinity keep a conversation, durable job, retry chain, or approval flow from silently changing release identity mid-execution. Define how the allocation treats anonymous and multi-device users, and apply tenant, region, risk, plan, and residency constraints before it runs.
Principle 6: Make runtime configuration controlled and observable
Give every behavior-affecting setting a schema, owner, classification, allowed range, dependency set, source of truth, precedence, safe default, validation rule, distribution semantics, expiry where appropriate, and recovery behavior. Each request or workflow then carries the effective-configuration identity.
Principle 7: Route only among approved composites
A routing policy must identify allowed destinations, decision inputs and precedence, eligibility constraints, affinity, fallback order, overload behavior, policy identity, and rollback target. Best available model is not a runtime contract.
Principle 8: Design mixed-version behavior before rollout
Progressive delivery creates mixed versions by design. Specify read/write compatibility, migration order, in-flight work, shared state, cache and derived-data treatment, external effects, and points of no return before exposure begins.
Principle 9: Treat recovery as a verified transition
Rollback is not complete when a controller accepts a command. Verify effective state across the data plane, restore sufficient capacity, handle candidate-created state, and record the result. Where rollback cannot undo state or effects, pre-plan containment, compensation, or roll-forward.
6. Architecture Patterns
Pattern 1: Composite manifest and dependency graph
Any system with more than one independently changeable behavioral asset needs this pattern. The manifest protects reproducibility; the graph protects compatibility and change-impact reasoning. Candidate assembly fails if a required edge is unresolved or a mutable dependency lacks explicit resolution semantics.
At scale, asset owners maintain their component contracts while one release owner remains accountable for the assembled candidate. Every asset need not share an owner, but the effective composite needs one authoritative view.
Pattern 2: Promotion controller with deployment receipts
The controller enforces the state machine, environment bindings, gate policy, and transition authority. Each successful or failed transition produces a receipt containing intended candidate, actual effective identities, topology target, assignment state, configuration convergence, actor, time, and deviations.
Control-plane unavailability should fail safely. Existing approved assignments may continue while their effective state remains known; new promotions and unverified mutations stop. An emergency path needs pre-authorized scope, explicit identity, expiry, evidence capture, and retrospective review.
Pattern 3: Stable cohort and workflow assignment
Progressive exposure requires more than a percentage flag. Each stage declares:
- eligible and excluded populations;
- unit and deterministic rule of assignment;
- affinity duration for requests, conversations, jobs, and retries;
- tenant, geography, risk, capability, and residency constraints;
- minimum and maximum exposure;
- start time, observation window, and delayed-outcome allowance;
- advancement, hold, rollback, and insufficient-evidence conditions;
- authority for automatic and manual transitions.
The assignment receipt should make sample pollution and accidental default exposure detectable. Chapter 8 determines whether the resulting evidence supports a behavioral claim; this chapter ensures the intended population actually received the candidate consistently.
Pattern 4: Versioned runtime-routing contract
A routing contract lists approved destinations by composite identity, never only by a mutable model label. It declares decision inputs, precedence, prohibited inputs, eligibility, deterministic or adaptive behavior, affinity, fallbacks, admission and overload rules, and maximum permitted degradation.
When adaptation is allowed, the released policy includes its choice set, objective, constraints, update cadence, and evaluation boundary. Shadow routes state whether they may write memory, enqueue work, invoke tools, or affect external state; the safe default permits no authoritative side effect. Every routing decision emits a route receipt with enough evidence to reproduce why a destination was eligible and selected.
Chapter 6 owns general request-level orchestration. This pattern owns assignment among deployed behavioral variants and prevents the harness from selecting an unapproved combination.
Pattern 5: Typed configuration control plane
Classify configuration as release-bound, environment-bound, tenant-bound, operational, secret, or emergency override. Define precedence among manifest values, environment bindings, flags, and overrides. Validate before acceptance and again before activation when runtime context can change validity.
The control plane states whether activation is atomic. When partial application is unavoidable, it defines the convergence deadline, acknowledgement behavior, stale-client policy, and combinations that remain compatible. Drift detection compares intended and effective state. Flags and emergency overrides carry an owner, purpose, approval, creation time, expiry, cleanup obligation, and last-known-good value.
Secrets belong behind versioned references, not inside the manifest. A value rotation need not create a new behavioral candidate, but a changed principal, scope, endpoint, or privilege assumption triggers material-change review.
Pattern 6: Isolated index build and atomic cutover
Build a new index under a unique identity rather than mutating the active index when behavior, access, or rollback requirements justify it. Reconcile source count, deletions, permission projection, schema compatibility, freshness watermark, duplicates, and transformation failures before eligibility. Record shadow-query or comparison evidence when used.
Cutover binds queries atomically or with declared propagation semantics, and every query records the effective index identity. Retain the prior snapshot only while it remains compatible, authorized, and operationally supportable. Retirement removes stale routes, storage, derived data, and obsolete recovery references.
Pattern 7: Environment-equivalence contract
Exact production replicas are often infeasible and sometimes unsafe. Compare material properties instead:
- candidate identity and environment bindings;
- model capability and revision semantics;
- prompt, policy, index pipeline, ranking, permissions, and freshness behavior;
- harness paths, tool schemas, approval behavior, routing, flags, and timeouts;
- topology, capacity, identity, tenancy, network, residency, and dependency limits;
- state, migration, and telemetry needed to judge the release.
For every substitution—synthetic data, reduced corpus, mocked side effect, smaller capacity, alternate endpoint—record why it is acceptable, what behavior it cannot represent, the compensating production control, and residual uncertainty. Promote the same candidate and make substitutions through controlled bindings.
Pattern 8: Compatibility-aware recovery
For each changed component, declare backward and forward compatibility, mixed-version reads and writes, migration phases, in-flight behavior, candidate-created state, cache and derived-data cleanup, external effects, recovery target capacity, and verification signals. Mark the point of no return.
Expand-contract migrations let old and new releases coexist when the state model permits them. Pin long-running work to a compatible release or define an explicit handoff protocol. Once a candidate has emitted irreversible actions, rollback can stop new exposure but cannot erase the effects; use the compensation contracts from Chapter 7 or contain and roll forward.
A compact release walkthrough
Suppose a team revises a prompt to use a new metadata field, rebuilds the context index to supply that field, and changes routing so a bounded workflow can use a newly approved model configuration.
The pipeline assembles one candidate containing immutable prompt, index-pipeline, index-snapshot, harness, model-policy, and routing identities. The dependency graph marks the prompt as incompatible with the old metadata schema. Material-change analysis requires an index rebuild, targeted compatibility checks, applicable Chapter 8 evidence, Chapter 9 constraint confirmation, a stable internal cohort, and a recovery plan for conversations already pinned to the parent release.
The candidate enters production with no authoritative assignment. Shadow queries verify index completeness but cannot write memory or invoke tools. Once the gates pass, deterministic tenant-and-workflow assignment exposes the initial cohort. The controller holds that stage until the declared observation windows for quality, latency, cost, and delayed outcomes are complete. Each request records candidate, route-policy, model, prompt, effective-configuration, and index identities.
If the candidate writes new-format workflow state before a release-health condition fails, sending all traffic back to the parent may be unsafe. New workflows return to the parent, compatible in-flight work stays pinned, and incompatible state is isolated for transformation or roll-forward. The deployment receipt verifies data-plane convergence and records why recovery required more than a simple rollback. Those are the mechanics hidden by the phrase “roll back the prompt.”
7. Failure Modes
Failure mode 1: Mutable identity masquerades as a version
Initiating condition: a manifest records aliases such as latest or current-index. Broken property: reproducibility. Evidence: resolution and effective-state receipts. Response: freeze exposure and resolve the actual composite. Prevention: immutable identities with explicit external revision semantics.
Failure mode 2: Partial deployment creates mixed composites
Initiating condition: workers converge slowly or reject configuration. Broken property: candidate atomicity and compatibility. Evidence: per-cell effective identity and convergence status supplied through Chapter 11. Response: hold assignment, isolate incompatible cells, or restore last known good. Prevention: compatibility envelopes, acknowledgements, deadlines, and safe stale-client behavior.
Failure mode 3: A workflow changes identity mid-execution
Initiating condition: routing is recalculated for a retry, continuation, or background step. Broken property: workflow consistency. Evidence: route and workflow receipts. Response: repin to a compatible release or stop safely. Prevention: durable affinity and explicit handoff semantics.
Failure mode 4: Configuration becomes an unreviewed launch path
Initiating condition: a flag or override expands exposure outside the promotion controller. Broken property: transition authority. Evidence: intended-versus-effective configuration and audit record. Response: restore precedence, constrain exposure, and expire the override. Prevention: typed configuration, policy gates, and no hidden defaults.
Failure mode 5: Routing selects an unapproved combination
Initiating condition: capacity fallback, adaptive policy, or alias resolution expands the choice set. Broken property: eligibility. Evidence: route receipt and destination identity. Response: fail closed or use the declared degraded route. Prevention: bounded destination sets and versioned fallback order.
Failure mode 6: Index cutover exposes incomplete or unauthorized content
Initiating condition: in-place mutation, partial permission projection, stale deletions, or propagation lag. Broken property: index completeness and access compatibility. Evidence: reconciliation receipt and effective query identity. Response: return queries to an authorized retained snapshot or disable the affected path. Prevention: isolated build, validation, atomic cutover, and deletion-aware retention.
Failure mode 7: Shadow execution creates real effects
Initiating condition: a duplicated path writes memory, enqueues work, or calls a tool. Broken property: non-authoritative exercise. Evidence: side-effect ledger from the action path. Response: stop shadowing and compensate where possible. Prevention: read-only capability boundaries and explicit shadow contracts.
Failure mode 8: Rollback restores code but not behavior
Initiating condition: prompt, model route, index, configuration, or state remains on the candidate. Broken property: composite recovery. Evidence: effective-state verification across all planes. Response: execute the component recovery sequence or contain and roll forward. Prevention: one recovery target graph and tested composite recovery.
Failure mode 9: The rollback target is incompatible or under-capacity
Initiating condition: state has migrated, dependencies have retired, or old capacity has been removed. Broken property: recoverability. Evidence: recovery exercise, capacity reservation, and compatibility check. Response: constrain traffic, isolate state, or roll forward. Prevention: recovery-target lifecycle and explicit point-of-no-return decisions.
Failure mode 10: Environment substitution hides a production-only behavior
Initiating condition: staging uses different data shape, permissions, tools, model revision, or capacity. Broken property: evidence applicability. Evidence: equivalence record and production stage result. Response: hold expansion and apply the compensating control. Prevention: property-level equivalence rather than a parity checkbox.
Failure mode 11: Required evidence disappears during exposure
Initiating condition: telemetry loss, outcome delay, or sample contamination makes a stage indeterminate. Broken property: advancement decision. Evidence: release-health completeness status. Response: hold, constrain, or reverse as predeclared. Prevention: missing-data semantics and a maximum decision time.
Failure mode 12: Emergency state becomes permanent architecture
Initiating condition: an override, fallback, or temporary route has no expiry or owner. Broken property: controlled configuration. Evidence: exception-age and drift records. Response: retire or formalize the state through normal promotion. Prevention: scoped authority, automatic expiry, and closure ownership.
8. Tradeoffs
Release decisions should be driven by consequence, reversibility, dependency fan-out, state compatibility, evidence delay, cohort isolation, external volatility, operator burden, recovery objectives, and retention constraints.
Bundled composite versus independently deployable assets
Keep one composite identity even when assets deploy independently. Coordinate promotion when compatibility is narrow or recovery must be atomic. Independent promotion is justified only when contracts are broad, dependency edges are explicit, evidence remains applicable, and each component has isolated rollback. Match approval authority to the combined reachable consequence, not the number of repositories.
Immutable pinning versus controlled dynamic resolution
Resolve behavior-critical dependencies immutably by default. Dynamic resolution is reasonable when freshness or availability is a product requirement and the choice set, constraints, detection, evidence boundary, and recovery behavior are explicit. Whatever resolves at runtime must be recorded; convenience alone does not justify mutability.
Blue-green, canary, rolling, shadow, or immediate cutover
Blue-green favors atomic reversal but doubles capacity and complicates state. Canary reduces blast radius but depends on stable cohorts and timely evidence. Rolling delivery is efficient when versions are mutually compatible. Shadowing provides comparative evidence but risks cost, privacy exposure, and side effects. Immediate cutover should be reserved for low-consequence, highly reversible changes or urgent containment with appropriate authority.
Centralized control versus domain-owned promotion
Prefer shared invariants with federated asset ownership. Central control improves uniformity but can become a bottleneck and obscure domain judgment. Domain-owned promotion works when teams can produce standard identities, receipts, gates, and recovery evidence. Any deviation from shared invariants needs an explicit owner and review because it weakens system-wide reconstruction.
Automatic advancement versus human approval
Automate transitions when signals are timely, decision rules are predeclared, reversibility is high, and false advancement is bounded. Require human approval when outcomes are delayed, evidence needs interpretation, state changes are hard to reverse, or consequence is asymmetric. Automation should execute a decision contract, not invent one after observing results.
Rollback versus roll-forward
Rollback is the default before incompatible state or irreversible effects cross the point of no return. Roll forward when the parent cannot read new state, dependencies have changed irreversibly, or compensation is safer than reversal. The release authority must accept that recovery uncertainty before exposure reaches a stage where the alternative disappears.
Fast override versus manifest-only mutation
Operational overrides can shorten containment time, but they create configuration drift. Permit them only with scoped authority, validation, a safe default, an immutable record, expiry, and a path to reconcile effective state. High-consequence behavior should not rely on an unrecorded console mutation.
Environment fidelity versus cost and data exposure
Seek equivalence in behaviorally material properties rather than identical scale or sensitive data. Higher fidelity is justified when capacity, authorization, state, regional dependencies, or corpus shape materially affect behavior. Where only production can supply the evidence, compensate with narrow exposure, conservative advancement, and explicit residual uncertainty.
9. Production Checklist
Candidate and impact
- [ ] One immutable release identifier resolves to every behavior-affecting asset and dependency.
- [ ] Mutable and external dependencies record resolution, observation, revision, and reproducibility limits.
- [ ] Provenance, integrity, parent release, compatibility constraints, and known-good target are recorded.
- [ ] Material-change analysis identifies affected behavior, rebuilds, migrations, evidence, approvals, rollout depth, and recovery work.
Topology and environment
- [ ] Runtime topology records entry points, sync and async paths, placement, state, isolation, scaling unit, backpressure, failure domains, control-plane dependencies, and degraded modes.
- [ ] Environment bindings are separate from candidate identity.
- [ ] Material equivalence, substitutions, invalid representations, compensating controls, and residual gaps are recorded.
- [ ] Capacity exists for initial exposure, steady state, and the declared recovery target.
Promotion, routing, and configuration
- [ ] CI/CD builds once, enforces gates, promotes the same candidate, and issues transition receipts.
- [ ] Promotion, eligibility, assignment, exposure, hold, and recovery states have distinct authority and reversal paths.
- [ ] Cohorts have explicit eligibility, stable assignment, workflow affinity, stage windows, and insufficient-evidence behavior.
- [ ] Routing lists approved destinations, decision precedence, constraints, affinity, fallback, overload behavior, and receipt fields.
- [ ] Runtime configuration defines schema, ownership, class, precedence, validation, propagation, effective identity, expiry, drift, safe default, and rollback.
State, index, and recovery
- [ ] Mixed-version reads and writes, migrations, in-flight work, candidate-created state, caches, derived data, and external effects are addressed.
- [ ] Index source cutoff, pipeline identity, schema and permission compatibility, completeness, cutover, query identity, retention, and retirement are recorded.
- [ ] Rollback, containment, compensation, and roll-forward conditions are explicit.
- [ ] Composite recovery has been exercised at a blast radius appropriate to the consequence, and effective restoration can be verified.
Evidence and closure
- [ ] Chapter 8 evidence and Chapter 9 constraints are bound to the exact candidate and checked for invalidation or expiry.
- [ ] Each exposure stage has candidate, cohort, denominator, slice, window, threshold, outcome delay, missing-data rule, authority, and action.
- [ ] Chapter 11 can supply the required effective-state and release-health signals without making logs a substitute for the release record.
- [ ] Deployment receipts, deviations, emergency changes, final decision, retirement duties, and follow-up owners are retained.
10. Design Review Questions
- Which immutable identifier reconstructs the complete candidate, and which dependencies remain only partially reproducible?
- What dependency edges make this change material, and which evidence or approvals did those edges invalidate?
- Was the same candidate promoted, or was any behaviorally material asset rebuilt or resolved differently by environment?
- Which topology units receive the candidate, where is state owned, and what failure domain bounds partial deployment?
- Which properties are equivalent to production, which are substituted, and what production behavior remains unrepresented?
- What are the permitted promotion and exposure transitions, their authorities, preconditions, evidence snapshots, and reversal paths?
- What is the unit of cohort assignment, how long is affinity preserved, and what happens to durable work during a transition?
- Which composite destinations may routing select, in what precedence order, and what receipt can reproduce a decision?
- What is the effective configuration source of truth, how is convergence proven, and what happens to stale clients or invalid values?
- How is a new context index reconciled, authorized, cut over, identified per query, retained, and retired?
- Which reads, writes, migrations, caches, memories, jobs, and external effects prevent a simple rollback?
- Does the recovery target remain compatible, available, authorized, and sufficiently provisioned?
- Which exact health signals govern each stage, and what happens when results are delayed, sparse, contaminated, or missing?
- Which Chapter 9 constraints limit routing, exposure, regions, data, dependencies, or emergency authority?
- What did the deployment receipt prove became effective, and how would an engineer verify restoration independently of control-plane intent?
11. Main Artifact: AI Release Manifest
The AI Release Manifest is one living artifact with coordinated views. It references the Production Prompt Specification, Context Assembly Plan, Harness Control-Loop Diagram, Tool Permission Matrix, Evaluation Plan, AI Threat Model, and AI Operations Runbook rather than duplicating them. Fields marked as not applicable require a reason.
View 1: Release charter
| Field | Record |
|---|---|
| AI Release Manifest identity and schema version | |
| Immutable release ID and creation time | |
| Decision, purpose, and intended outcome | |
| Release owner and transition authorities | |
| Parent and known-good recovery release | |
| Target environments, regions, tenants, and workflows | |
| Intended steady-state population | |
| Source AI Fit and Risk Assessment Worksheet identity, version, accepted decision IDs, and consequence constraints | |
| Source Production Prompt Specification, Context Assembly Plan, Harness Control-Loop Diagram, and Tool Permission Matrix identities and versions | |
| Source Evaluation Plan identity, version, evaluation run, and decision-record ID | |
| Source AI Threat Model identity, version, governance decision ID, constraints, and expiry | |
| AI Operations Runbook handoff: release-health contract, effective-state identities, receipts, and recovery proof | |
| External deadlines or dependency events |
Release assembly and exposure are monotonic handoffs. Chapter 10 may select a narrower eligible cohort, route, environment, or degraded mode, and it may block promotion when evidence is missing. It may not silently broaden upstream eligibility, data use, action authority, accepted risk, behavioral guarantees, or evidence validity. A required broadening returns to the owning artifact and produces a new accepted decision before release.
View 2: Composite identity and dependency graph
| Component | Immutable / resolved identity | Provenance / integrity | Dependencies / compatibility | Environment binding | Reproducibility limit |
|---|---|---|---|---|---|
| Application/build | |||||
| Prompt bundle and included fragments | |||||
| Allowed model configurations | |||||
| Context corpus and index snapshot | |||||
| Ingestion, transformation, and ranking pipeline | |||||
| Harness and validation contracts | |||||
| Tool schemas and permission policy | |||||
| Routing policy | |||||
| Runtime configuration schema and declared values | |||||
| Serving profile and material infrastructure assumptions |
Record alias-resolution time, observed capabilities, revision semantics, and change detection for every external component without a stable immutable identity. Attach the machine-readable dependency graph where maintained.
View 3: Material-change and gate matrix
| Changed asset / event | Affected behavior / claim / workflow / cohort | Rebuild / migration / compatibility work | Evidence status / Chapter 8 action | Chapter 9 decision / constraint | Rollout / recovery consequence | Owner / status |
|---|---|---|---|---|---|---|
For every omitted gate, record justification, compensating control, approving authority, and expiry.
View 4: Runtime topology and capacity envelope
| Concern | Declared runtime state |
|---|---|
| Entry points and sync/async execution paths | |
| Regions, jurisdictions, cells, workers, queues, and schedulers | |
| Model endpoints, context stores, tool gateways, and control-plane dependencies | |
| Tenant and isolation boundaries consumed from Chapter 9 | |
| Stateful components, ownership, replication, and consistency | |
| Scaling unit, concurrency, admission, backpressure, queue age, and timeout budget | |
| Dependency failure domains and degraded modes | |
| Assignment point and propagation through durable work | |
| Promotion/rollback unit and reserved recovery capacity |
View 5: Environment-equivalence record
| Material property | Target production behavior | Preproduction representation | Substitution / gap | Unrepresented invalid behavior | Compensating control | Residual uncertainty / owner |
|---|---|---|---|---|---|---|
| Model revision semantics | ||||||
| Corpus, index, ranking, and permissions | ||||||
| Harness, tools, approvals, and routing | ||||||
| Identity, tenancy, network, and residency | ||||||
| State, migrations, capacity, and timeouts | ||||||
| Release-health telemetry |
View 6: Routing and configuration contract
| Contract area | Record |
|---|---|
| Approved destinations by composite release ID | |
| Decision inputs, precedence, defaults, and prohibited inputs | |
| Tenant, region, risk, data, workflow, and capability eligibility | |
| Deterministic/adaptive behavior and bounded choice set | |
| Conversation, workflow, retry, and asynchronous affinity | |
| Fallback order, fail-open/closed rule, overload behavior, and maximum degradation | |
| Shadow side-effect constraints | |
| Route policy identity, owner, approval, receipt, and rollback target | |
| Configuration classes, schema, source of truth, and precedence | |
| Validation, atomicity/partial semantics, propagation, acknowledgement, and stale-client behavior | |
| Effective-configuration identity, safe defaults, expiry, drift, and recovery | |
| Secret bindings and material scope/identity assumptions |
View 7: Promotion and exposure plan
| Stage / transition | Environment / topology target | Eligible / excluded population | Assignment unit / affinity | Min / max exposure | Observation / outcome-delay window | Advance / hold / reverse conditions | Actor / authority | Reversal path |
|---|---|---|---|---|---|---|---|---|
| Assembled → gates pending | ||||||||
| Approved → deployed | ||||||||
| Deployed → exercised | ||||||||
| Exercised → initially exposed | ||||||||
| Hold/expansion stages | ||||||||
| Steady state/retirement |
Link each transition to pipeline action or command reference, evidence snapshot, and resulting receipt. Emergency transitions record pre-authorized scope and expiry.
View 8: Compatibility, migration, and recovery plan
| Changed component/state | Backward/forward compatibility | Mixed-version read/write rule | Migration and point of no return | In-flight work | Candidate-created state/cache/index effects | External effects/compensation | Rollback, containment, or roll-forward action | Verification |
|---|---|---|---|---|---|---|---|---|
Record recovery-time and data-loss expectations, target capacity and dependency availability, retained index snapshots, cleanup duties, and the latest recovery-exercise result.
View 9: Release-health contract
| Claim/failure condition | Candidate and stage | Population/denominator/slice | Signal source and owning chapter/system | Baseline, threshold, direction, uncertainty | Window and outcome delay | Minimum evidence/zero-tolerance event | Missing or indeterminate behavior | Automatic/manual action and authority | Terminal result |
|---|---|---|---|---|---|---|---|---|---|
Quality measurement and evidence validity remain in Chapter 8; security and governance constraints remain in Chapter 9; signal collection and incident operations remain in Chapter 11. This view binds their approved outputs to release transitions.
View 10: Deployment receipt and closure record
| Field | Actual result |
|---|---|
| Candidate and environment binding requested | |
| Effective component, route-policy, configuration, and index identities by topology unit | |
| Transition actor, policy, start/end time, and result | |
| Cohort assignment and exposure actually achieved | |
| Configuration convergence and drift result | |
| Release-health stage decisions and evidence references | |
| Deviations, emergency changes, owners, and expiry | |
| Rollback/containment/roll-forward execution and verified effective state | |
| Final steady-state or retirement decision | |
| Snapshot, flag, route, capacity, and derived-data cleanup | |
| Follow-up owners and due dates |
The manifest closes only when effective state matches the final decision, temporary release machinery has an owner or has been removed, and recovery or retirement obligations are explicit.
12. Key Takeaways
- The release unit is an immutable composite behavior identity, not merely an application build.
- Dependency-aware material-change analysis determines rebuilds, evidence, approvals, rollout depth, and recovery work.
- Deployment topology, environment bindings, routing, runtime configuration, and context-index cutover are part of release correctness.
- Deploying, making eligible, assigning, exposing, expanding, and retiring are distinct auditable transitions.
- Progressive exposure is meaningful only with stable cohorts, workflow affinity, decision windows, and missing-evidence behavior.
- Rollback must account for mixed versions, persisted state, in-flight work, external effects, index and cache cleanup, and recovery capacity.
- The AI Release Manifest joins intent, effective runtime state, transition evidence, and verified recovery without duplicating evaluation, governance, or incident practice.