Skip to contentThe Observability LayerSearch

Enterprise handbook · Section 1 of 31

Executive Summary

An enterprise AI agent can do more than generate an answer. It can select tools, retrieve information, retain state, delegate work, and change a business system. Responsible AI therefore has to govern the complete operating system and the consequences of its actions.

The central recommendation is to approve a bounded deployment configuration: a defined business mandate, accountable owner, permitted actions, controlled tools and memory, tested oversight, and independently observable outcomes. Approval should identify the configuration that was evaluated and specify the changes that invalidate its evidence.

  • Establish authority before execution. A model's proposed action must pass an independent authorization check. Consequential approvals must bind the actual action, affected resources, permitted variation, and expiry.
  • Verify outcomes after execution. Check the external business state before declaring completion. Reconcile partial, uncertain, and duplicated effects. An agent's explanation is supporting evidence, not independent outcome verification.
  • Measure prevention and useful work together. Keep critical prohibited effects, business utility, review burden, customer outcomes, and recovery as separate acceptance conditions. A system that refuses every task has not demonstrated useful operation.
  • Maintain assurance throughout operation. Changes to prompts, memory, models, tools, permissions, monitors, and delegation can change behavior. Incidents and material changes reopen the relevant control claims.

This handbook supplies the knowledge and artifacts needed to put that approach into practice: definitions and risk assessment, a control architecture, operating and assurance guidance, worked workflows, a 90-day implementation sequence, 45 baseline control practices, 12 detailed implementation specifications, and 36 proposed acceptance tests. The control practices and tests are original proposed enterprise designs. Their presence is not evidence of effectiveness in an enterprise.

Recent research reinforces the need to test complete configurations. In one study, component updates that each passed on their own failed when enabled together in three separately constructed benchmark panels: 22 of 157 eligible cases in AgentDojo, 2 of 11 in Agent-SafetyBench and 19 of 1,243 in Agent Security Bench. The panels were built differently, so their counts are reported separately and not combined. A proactive-monitoring benchmark reported 40.74% as its best optimal-window result: the best evaluated model gave the correct risk label and interrupted within the annotated timing window in roughly four of ten synthetic trajectories. An interruption that came too early counts as a miss on this metric, so 40.74% is not a harm rate. These findings concern the specified experiments. They do not estimate an institution's production failure rate. [S11, S13]