Skip to contentThe Observability LayerSearch

Enterprise handbook · Section 20 of 30

18. Twelve detailed implementation specifications

These specifications are proposed practice. Each contains its objective, required behavior, implementation, maturity progression, ownership, evidence, handbook mapping, optional historical anchors, and source basis. E01-E12 identify the mechanisms discussed throughout the main chapters. They are not additional statutory requirements.

Evidence table: Specification, Behavior to demonstrate, Handbook practices, Starter tests
SpecificationBehavior to demonstrateHandbook practicesStarter tests
E01 Join the deployed configuration to its approval evidenceEach production trajectory identifies an approved configuration and current mandate; material configuration changes reopen relevant release evidence.H-GOV-01, H-DES-01, H-MON-04, H-ASR-01T01, T02, T03
E02 Enforce authority at the consequential-effect boundaryEach consequential commit passes current resource authorization, limits, and any required action-bound approval outside the agent's control.H-DES-02, H-RUN-01, H-RUN-02, H-RUN-03T04, T05, T06
E03 Verify completion from external business stateCritical completion claims require an independent postcondition check appropriate to the claimed business outcome.H-EVL-01, H-RUN-04, H-MON-01T07, T08, T09
E04 Preserve relevant safety state across the workflowApproved safety state persists across retries, compaction, restarts, model changes, and child-agent delegation.H-DES-04, H-RUN-03, H-MAS-03T10, T11, T12
E05 Evaluate delayed influence and repair its descendantsRetrieval and memory tests include later activation, derivative creation, and repair that remains effective after reuse.H-DES-03, H-DES-04, H-EVL-03T13, T14, T15
E06 Validate affected interactions before approving changeMaterial changes receive targeted joint tests covering shared authority, state, data, and consequential effects.H-DES-01, H-EVL-05, H-MON-04, H-TPR-04T16, T17, T18
E07 Limit monitor assurance to demonstrated operating conditionsEach monitor claim states observed signals, tested threats, operating threshold, error rates, timing, and failure response.H-EVL-04, H-MON-02, H-RUN-05T19, T20, T21
E08 Prove protocol and adapter enforcement coverageAdopted versions and adapters have a conformance record and a tested inventory of all effect routes.H-DES-02, H-RUN-01, H-TPR-01T22, T23, T24
E09 Test multi-agent outcomes across threat and topologyDelegated workflows receive end-to-end testing and maintain constrained child authority and aggregate effect evidence.H-MAS-01, H-MAS-02, H-MAS-03, H-MAS-04T25, T26, T27
E10 Measure customer outcomes across the complete decision pathCovered workflows measure aligned eligible cohorts, intermediate exclusions, decision errors, and redress outcomes under an approved fairness protocol.H-FCO-01, H-FCO-02, H-FCO-05T28, T29, T30
E11 Demonstrate human correction competence and approval integrityReviewers have necessary evidence and authority, demonstrate competence in seeded tests, and approve a bounded action or batch.H-GOV-05, H-RUN-02, H-FCO-04T31, T32, T33
E12 Link assurance claims to protected operational evidenceEach material claim links to an approved configuration, a tested mechanism, independently observable outcomes, and a named owner.H-MON-01, H-MON-03, H-ASR-01, H-ASR-04, H-ASR-05T34, T35, T36

E01. Join the deployed configuration to its approval evidence

Objective: Ensure the system being operated is the system that was approved.

Control: Each production trajectory identifies an approved configuration and current mandate; material configuration changes reopen relevant release evidence.

Implementation: Generate a manifest at build/release time. Store it independently of agent memory. Resolve provider aliases where possible and expose unresolved version visibility as a limitation.

Maturity: Baseline records the manifest; Enhanced automates production comparison; Frontier identifies affected evaluation coverage on change.

Ownership: First line business and platform own deployment identity; second line validates materiality and release gates; third line tests the linkage.

Evidence: Manifest, configuration digest, production attestation, change record, and re-evaluation decision.

Handbook practices: H-GOV-01, H-DES-01, H-MON-04, H-ASR-01

Original anchors: GOV-03, GOV-11, EVL-16, TPR-03.

Source basis: Historical TOL context [S02, S03, S07]; interaction research [S13]. Exact implementation is proposed.

E02. Enforce authority at the consequential-effect boundary

Objective: Prevent approved credentials or a misleading review from enabling an unauthorized effect.

Control: Each consequential commit passes current resource authorization, limits, and any required action-bound approval outside the agent's control.

Implementation: Normalize the action, bind review to its digest, revalidate after review, enforce idempotency, and deny raw bypass routes. An asynchronous transaction requires its own cancellation and reconciliation logic.

Maturity: Baseline mediates privileged tools; Enhanced covers subprocesses and background work; Frontier verifies enforcement routes continuously.

Ownership: First line platform and resource owners enforce; second line approves consequence policy; third line re-performs bypass tests.

Evidence: Authorized/denied calls, payload-drift tests, approval records, resource receipts, and bypass inventory.

Handbook practices: H-DES-02, H-RUN-01, H-RUN-02, H-RUN-03

Original anchors: DES-08, DES-09, RUN-15, RUN-17.

Source basis: MCP security requirements [S28], AISI incident [S38], and runtime-contract framing [S24]. Design details are proposed.

E03. Verify completion from external business state

Objective: Prevent false completion and downstream reliance on an unverified result.

Control: Critical completion claims require an independent postcondition check appropriate to the claimed business outcome.

Implementation: Declare the expected state, query or observe it independently, reconcile partial effects, and retain failure or uncertainty rather than overwriting it with a success narrative.

Maturity: Baseline checks critical outcomes; Enhanced records reusable evidence contracts; Frontier propagates verified status and invalidation to dependent workflows.

Ownership: First line process owner defines success; platform supplies the verifier; second line challenges validity; third line samples completed cases.

Evidence: Postcondition specification, receipts, independent state checks, failed-completion cases, and reconciliation.

Handbook practices: H-EVL-01, H-RUN-04, H-MON-01

Original anchors: EVL-07, EVL-08, RUN-18, MON-01.

Source basis: Runtime-contract evidence framing [S24] and financial evidence evaluation [S22]. The postcondition contract is proposed.

E04. Preserve relevant safety state across the workflow

Objective: Prevent limits and unresolved risks from disappearing at task boundaries.

Control: Approved safety state persists across retries, compaction, restarts, model changes, and child-agent delegation.

Implementation: Use an external workflow ledger for budget consumption, revocation, pending approvals, policy denials, and quarantined artifacts. Set retention by purpose and record class.

Maturity: Baseline retains per-run state; Enhanced maintains cross-session lineage; Frontier tests patient, distributed attempts and recovery after restoration.

Ownership: First line platform and process owner operate the ledger; second line approves retention and escalation; third line tests state continuity.

Evidence: Restart and delegation tests, global budget records, revocation propagation, and restoration checks.

Handbook practices: H-DES-04, H-RUN-03, H-MAS-03

Original anchors: RUN-09, RUN-14, MAS-07.

Source basis: Loop-state research [S18] and multi-agent control scope [S06]. Enterprise persistence and retention rules are proposed.

E05. Evaluate delayed influence and repair its descendants

Objective: Keep contextual contamination from becoming enduring authority.

Control: Retrieval and memory tests include later activation, derivative creation, and repair that remains effective after reuse.

Implementation: Track lineage from sources to summaries, skills, caches, and child artifacts. Revoke influence through the lineage and test benign as well as malicious conditional instructions.

Maturity: Baseline separates instructions from data; Enhanced gates memory promotion; Frontier verifies descendant repair under continued adaptation.

Ownership: First line data and platform owners maintain provenance; second line privacy/security define sensitive uses; third line samples repair completeness.

Evidence: Trigger-turn tests, admission decisions, lineage, quarantine records, and post-repair recurrence tests.

Handbook practices: H-DES-03, H-DES-04, H-EVL-03

Original anchors: DES-03, DES-14, EVL-10, RUN-16.

Source basis: Delayed-injection study [S15] and evolving-harness interactions [S13]. The repair workflow is proposed.

E06. Validate affected interactions before approving change

Objective: Detect joint failures hidden by isolated component validation.

Control: Material changes receive targeted joint tests covering shared authority, state, data, and consequential effects.

Implementation: Compare baseline, isolated changes, and combinations. Keep optimization data separate from release holdout data, record selected coverage, and retain newly identified failures.

Maturity: Baseline tests direct dependencies; Enhanced tests prioritized pairs; Frontier adds higher-order and persistent-state interactions with documented coverage limits.

Ownership: First line engineering supplies candidates; second line validation owns challenge and thresholds; third line reviews change decisions.

Evidence: Candidate manifests, interaction matrix, controlled comparison outcomes, holdout lineage, and release minutes.

Handbook practices: H-DES-01, H-EVL-05, H-MON-04, H-TPR-04

Original anchors: EVL-16, MAS-09, MON-12, TPR-03.

Source basis: Controlled harness-composition experiments [S13]. Risk-prioritized coverage rules are proposed.

E07. Limit monitor assurance to demonstrated operating conditions

Objective: Prevent an impressive detection statistic from overstating operational prevention.

Control: Each monitor claim states observed signals, tested threats, operating threshold, error rates, timing, and failure response.

Implementation: Evaluate unseen categories, schema changes, adaptive attacks, context loss, and capacity limits. Test the monitor with its actual trigger and feedback logic. Keep hard authorization enforcement independent.

Maturity: Baseline measures threshold errors; Enhanced adds holdouts and delay tests; Frontier maintains repeated adaptive red-team evidence.

Ownership: First line operates monitoring; second line validates the claim; third line verifies the evidence and response mechanism.

Evidence: Confusion counts, labeled corpus, latency distribution, coverage exclusions, outage tests, and residual failures.

Handbook practices: H-EVL-04, H-MON-02, H-RUN-05

Original anchors: GOV-13, RUN-10, MON-07, MON-08.

Source basis: Monitor generalization [S17], harness optimization [S12], and AISI monitor red teaming [S40]. Operating policy is proposed.

E08. Prove protocol and adapter enforcement coverage

Objective: Keep protocol integration from creating hidden control bypasses.

Control: Adopted versions and adapters have a conformance record and a tested inventory of all effect routes.

Implementation: Validate request identity, issuer/audience, state handles, cached tools, approval exchanges, and downgrade paths. Test supported ACS hooks and unsupported routes separately.

Maturity: Baseline pins versions; Enhanced automates conformance and bypass tests; Frontier correlates live route coverage with deployed capabilities.

Ownership: First line integration/resource teams enforce; second line security approves standards; third line samples claimed routes.

Evidence: Protocol versions, adapter matrix, token-negative tests, hook coverage, and bypass outcomes.

Handbook practices: H-DES-02, H-RUN-01, H-TPR-01

Original anchors: DES-13, RUN-15, TPR-06.

Source basis: MCP release/security [S27, S28] and ACS documentation [S29, S30]. Adapter acceptance is proposed.

E09. Test multi-agent outcomes across threat and topology

Objective: Bound consequences that emerge only across several agents.

Control: Delegated workflows receive end-to-end testing and maintain constrained child authority and aggregate effect evidence.

Implementation: Vary topology, shared memory, scheduling, worker compromise, coordinated violations, and ordinary errors. Include learned links only where used. Compare with a simpler baseline.

Maturity: Baseline records topology and lineage; Enhanced varies threat/topology pairs; Frontier adds distributed, cross-session adaptation scenarios.

Ownership: First line orchestrator/process owner governs the system; second line validates aggregate risk; third line challenges system-level sign-off.

Evidence: Topology manifest, delegation grants, artifact lineage, joint outcomes, and global budget tests.

Handbook practices: H-MAS-01, H-MAS-02, H-MAS-03, H-MAS-04

Original anchors: MAS-01, MAS-03, MAS-05, MAS-10.

Source basis: ORBIT [S14], latent-link study [S16], and loop-state research [S18]. Acceptance rules are proposed.

E10. Measure customer outcomes across the complete decision path

Objective: Detect uneven treatment that occurs before final decision scoring.

Control: Covered workflows measure aligned eligible cohorts, intermediate exclusions, decision errors, and redress outcomes under an approved fairness protocol.

Implementation: Register stages and eligible populations; compare evidence burden, routing, review delay, and results. Investigate differences and test alternative designs before attributing cause.

Maturity: Baseline scopes covered decisions; Enhanced captures stage-level outcomes; Frontier joins production evidence to remediation and alternative-design tests.

Ownership: First line business owns outcomes; second line compliance/fairness validates the protocol; third line tests lineage and remediation.

Evidence: Cohort definitions, stage counts, uncertainty, reason records, alternatives, and lookback decisions.

Handbook practices: H-FCO-01, H-FCO-02, H-FCO-05

Original anchors: FCO-01, FCO-03, FCO-07, FCO-09.

Source basis: Historical TOL fairness [S09], ARIA simulations [S23], and applicable decision-rights sources [S34, S35, S36, S37]. Analysis design is proposed.

E11. Demonstrate human correction competence and approval integrity

Objective: Make human oversight capable of changing consequential outcomes.

Control: Reviewers have necessary evidence and authority, demonstrate competence in seeded tests, and approve a bounded action or batch.

Implementation: Test factual, authorization, and customer-rights errors under ordinary and peak load. Bind approval to scope and expiry. Preserve reasons for override and non-override.

Maturity: Baseline trains reviewers and binds approvals; Enhanced measures error and queue outcomes; Frontier tests competence decay and difficult edge cases.

Ownership: First line business staffs review; second line risk/compliance defines acceptance; third line samples operating effectiveness.

Evidence: Reviewer rubric, planted-case outcomes, queue results, override records, and payload-drift rejection.

Handbook practices: H-GOV-05, H-RUN-02, H-FCO-04

Original anchors: RUN-02, RUN-17, FCO-08, FCO-10.

Source basis: Historical TOL runtime/fairness [S04, S09] and applicable review-rights sources [S34, S35, S36]. Competence tests are proposed.

Objective: Make important control claims reconstructable and challengeable.

Control: Each material claim links to an approved configuration, a tested mechanism, independently observable outcomes, and a named owner.

Implementation: Correlate policy decisions, approvals, resource receipts, and state checks. Protect integrity and access, reconcile missing events, and rehearse containment and recovery.

Maturity: Baseline supplies a complete case record; Enhanced automates reconciliation; Frontier maintains continuous evidence expiry and challenge.

Ownership: First line platform/process owns evidence; second line validates claims and access; third line re-performs selected tests.

Evidence: Claim ledger, trace custody, event reconciliation, recovery drill, and unresolved limitations.

Handbook practices: H-MON-01, H-MON-03, H-ASR-01, H-ASR-04, H-ASR-05

Original anchors: MON-01, MON-03, MON-10, ASR-10.

Source basis: Existing evidence controls [S05], runtime-contract framing [S24], and telemetry documentation [S42]. Assurance design is proposed.