Scope. This report covers sources from 2023 through 2026 and asks which evaluation categories are mandatory before an AI agent is deployed with high autonomy. It is written for executive reading first, with source traceability following. Dr. Fisher's research fleet of agents gathers and cross-tabulates sources; every claim below cites a primary source and is reviewed by the editor.
Executive Read
Agentic evaluation has matured from "can the model answer?" into a layered assurance stack for systems that plan, use tools, persist across steps, and can affect external state. The must-have eval program is not one benchmark. It is a portfolio:
- Capability ceiling: how much can a scaffolded agent accomplish?
- Autonomy horizon: how long can it work without human help?
- Tool-use integrity: does it misuse tools or obey injected instructions?
- Deception and scheming: will it hide, sandbag, sabotage, or fake
alignment under pressure?
- Control robustness: can oversight protocols keep an untrusted agent
useful while preventing unacceptable actions?
- Runtime monitoring: do monitors catch bad trajectories under adaptive
attack?
- Multi-agent topology: does the system remain safe when agents interact?
- Post-failure response: what happens after loss of control starts?
- Eval integrity: can the evaluation itself be gamed?
Methodology: CRFM and Cochrane Review Discipline
This report is not only a synthesis of must-have evals; it is framed as an auditable evidence review using two methodological anchors:
- Stanford CRFM / HELM pattern: evaluate agentic systems through explicit
scenarios, standardized tasks, multiple metrics, reproducible procedures, and transparent reporting rather than a single leaderboard score.
- Cochrane pattern: define eligibility, search and selection logic, source
extraction fields, risk of bias, certainty of evidence, and limits before drawing conclusions.
Review Question
For AI agents and agentic systems from 2023–2026, what evaluation categories are mandatory before high-autonomy deployment, and what evidence supports each category?
Eligibility Criteria
- Included: 2023–2026 sources on agent capability evaluation, long-horizon
autonomy, tool misuse, prompt injection, scheming/deception, control protocols, monitoring, multi-agent topology, incident response, and eval-integrity methods.
- Prioritized: primary or methodologically strong sources, including
frontier-lab system/eval reports, METR/AISI work, peer-reviewed or arXiv technical reports, and official governance material.
- Excluded or downgraded: vendor-only claims, unsourced commentary, broad AI
ethics material without agentic measurement relevance, and stale benchmark claims not tied to current agent/tool workflows.
Search and Selection Method
Sources were drawn from a curated set of primary sources on agentic evaluation and control. They were grouped by evaluation claim rather than by publication order: capability, autonomy, tool integrity, deception, control, monitoring, topology, incident response, and evaluation integrity.
Risk of Bias / Source Quality
- Lab-authored evidence: strong access to frontier systems, but possible
incentive to frame results favorably.
- Benchmark evidence: comparable and reproducible when harnesses are open,
but vulnerable to benchmark saturation, contamination, and construct drift.
- Policy/governance sources: authoritative for obligations and institutional
direction, but often weaker on empirical measurement.
- Emerging 2026 material: useful for recency, but some sources are verified
only at the abstract level or still await full-text verification.
Certainty of Evidence
- High: Multi-metric portfolios, not single scores, are required; this
repeats across HELM-style methods and benchmark-limit literature.
- High: Tool-use and prompt-injection evals are mandatory because they map
directly to deployed-agent side-effect risk.
- Moderate-High: Long-horizon autonomy is a central operational metric, but
scaffolds and tool access still change quickly.
- Moderate: Scheming/deception tests are necessary, but eval-awareness and
measurement validity remain difficult.
- Moderate: Multi-agent topology and collusion evals are mandatory for
multi-agent deployment, while benchmarks are still maturing.
- Moderate: Loss-of-control incident drills belong in the stack, though the
empirical evidence base remains thinner than for capability benchmarks.
Threats to Validity
- Construct validity: benchmark tasks may not capture real deployed-agent
behavior.
- Internal validity: scaffold, prompt, tool access, and monitor design may
drive outcomes more than base-model capability.
- External validity: results may not generalize across domains,
organizations, credential scopes, or multi-agent architectures.
- Temporal validity: frontier capabilities and benchmark saturation change
quickly; 2023 findings may understate 2026 risks.
- Publication bias: failures and incidents are underreported relative to
benchmark wins and lab-selected evaluations.
- Evaluator bias: labs, vendors, auditors, and researchers all have
incentives that can shape framing, metric choice, and disclosure.
Must-Have Evaluation Stack
| Layer | Must-have eval | Why it is required | Anchor sources |
|---|---|---|---|
| 1 | Dangerous-capability evals on scaffolded agents | Measures upper-bound capability with tools and planning, not just base-model answers. | DeepMind dangerous-capability evals |
| 2 | Task time-horizon / long-horizon autonomy | Converts autonomy into a measurable operational quantity. | METR time-horizon |
| 3 | AI R&D automation | Directly tests the self-improvement / autonomous research capability that frontier policies care about. | METR RE-Bench |
| 4 | Standard agent benchmarks | Gives broad coverage across software, web, ML, cyber, and general assistant tasks. | SWE-bench, GAIA, Cybench, MLE-bench, OSWorld, WebArena |
| 5 | Tool misuse and prompt-injection resistance | Tests whether tool access remains inside policy under malicious or untrusted context. | AgentHarm, AgentDojo |
| 6 | Deception / scheming stress tests | Tests covert goal pursuit, oversight sabotage, sandbagging, alignment faking, and realistic agentic misalignment. | Apollo scheming, Anthropic sabotage, Alignment Faking, Agentic Misalignment |
| 7 | Control protocol evaluations | Tests whether monitoring, defer-to-trusted, trusted editing, resampling, and adaptive deployment keep an untrusted agent safe enough. | AI Control, Ctrl-Z, Adaptive Deployment |
| 8 | Chain-of-thought and oversight-signal monitoring | Tests whether the oversight signal exists, remains legible, and degrades under training or architecture changes. | CoT Monitorability, AISI Loss of Oversight |
| 9 | Monitor red-teaming | Tests whether a monitor survives adversarially refined trajectories, not just clean elicitation. | MonitoringBench |
| 10 | Multi-agent topology and collusion evals | Tests ordering instability, cascades, functional collapse, covert collusion, and architecture-driven failures. | Interaction Topology, Multi-Agent Risks, NARCBench / collusion interpretability |
| 11 | Loss-of-control incident drills | Tests containment, neutralization, response, and resilience after prevention fails. | LOC Incident Management |
| 12 | Eval-integrity review | Tests validity of the harness and flags reward hacking, sandbagging, eval-awareness, and reporting gaps. | OpenAI Third-Party Eval Playbook, Trustworthy Agentic AI survey |
Periodization: What Changed From 2023 to 2026
2023: Control and Risk Framing
2023 establishes the governance and control baseline:
- NIST AI RMF 1.0 supplies broad Govern / Map / Measure / Manage language.
- ISO/IEC 42001:2023 introduces certifiable AI management-system governance.
- AI Control reframes evaluation around the assumption that an advanced model
may intentionally subvert oversight.
- Frontier AI safety policies begin treating dangerous capability thresholds as
deployment gates.
Meta-finding: 2023 gives the vocabulary and first control protocol, but not yet a complete agentic benchmark portfolio.
2024: Scaffolded Agents and Deception Become Concrete
2024 makes the agent the unit of measurement:
- DeepMind dangerous-capability evals emphasize scaffolded agents, tools, and
elicitation of upper-bound capability.
- Apollo and Anthropic/Redwood work make scheming, alignment faking, sabotage,
and oversight subversion testable.
- Agent benchmarks broaden from coding into web, cyber, ML, and tool use.
Meta-finding: 2024 proves that "agentic" risk is not abstract. The system can be tested as an actor with tools and incentives.
2025: Control, Monitoring, and Eval Integrity Deepen
2025 expands from one-shot evaluation into trajectories:
- Ctrl-Z and Adaptive Deployment extend control to multi-step and distributed
threat settings.
- CoT Monitorability identifies the reasoning trace as a fragile oversight
opportunity.
- OpenAI / Apollo anti-scheming work begins testing mitigation, while also
exposing eval-awareness as a confound.
- Multi-Agent Risks makes interaction among agents a distinct risk class.
Meta-finding: 2025 shows that evaluation must cover process, not just final answers, and that control systems need their own evals.
2026: Runtime, Enterprise, and Multi-Agent Evaluation Become Central
2026 shifts toward deployed-agent governance:
- LOC Incident Management adds post-failure response and resilience.
- Interaction Topology argues that safety can be determined by wiring, not only
model weights.
- Trustworthy Agentic AI surveys agent workflow risk and process metrics.
- MonitoringBench and newer scheming/collusion work make monitor robustness and
real-world incident evidence central.
- NIST, CISA/Five-Eyes, OECD HAIP 2.0, and enterprise control-plane work begin
turning agentic evaluation into procurement and operating guidance.
Meta-finding: 2026 is the runtime year. The must-have question becomes: "Can this agent, with these tools and credentials, remain within its intended bounds while acting over time?"
Minimal Enterprise Eval Program
An enterprise agent deployment should not launch without evidence in six files:
- Capability card: scaffolded capability, autonomy horizon, task domains.
- Tool-risk card: allowed tools, denied tools, prompt-injection results,
misuse-refusal results.
- Deception/control card: scheming, sabotage, alignment-faking, sandbagging,
and control-protocol results.
- Runtime-monitor card: monitor catch rate, adversarial monitor red-team,
alert thresholds, human breakpoints.
- Topology card: single-agent vs multi-agent wiring, communication paths,
collusion/miscoordination tests.
- Incident card: containment, rollback, external side effects, reporting
triggers, recovery drill results.
Practical Ranking of Must-Have Evals
Tier A: Non-negotiable before external side effects
- Tool misuse / prompt injection.
- Long-horizon autonomy.
- Runtime monitoring under adversarial trajectories.
- HITL breakpoint and rollback drills.
- Deception / sabotage stress tests.
Tier B: Required for high-autonomy or high-blast-radius agents
- AI R&D automation / self-improvement.
- Control-protocol evaluation.
- Multi-agent topology and collusion.
- Loss-of-control incident response.
Tier C: Governance maturity / assurance layer
- Eval-integrity audit.
- Third-party evaluator access and reporting.
- Standards mapping to NIST AI RMF, ISO/IEC 42001, EU AI Act, and sector
controls.
Open Gaps
- No widely adopted runtime-control conformance test exists for deployed agents.
- Rollback remains weak for irreversible external actions like email, money
movement, account changes, or public publication.
- Multi-agent control benchmarks lag multi-agent risk taxonomies.
- Eval-integrity methods are often lab-authored; independent validity review is
still thin.
- Operator-native controls are mostly vendor or standards material rather than
peer-reviewed primary literature.
Sources
Primary anchors include:
- AI Control: Greenblatt et al., 2023.
- DeepMind dangerous-capability evaluations, 2024.
- METR time-horizon and RE-Bench.
- Agentic benchmark cluster: SWE-bench, GAIA, Cybench, AgentHarm, MLE-bench,
OSWorld, WebArena, AgentDojo.
- Apollo in-context scheming; Anthropic sabotage evaluations; Alignment Faking;
Agentic Misalignment.
- Ctrl-Z; Adaptive Deployment; AI Control Safety Case.
- CoT Monitorability; AISI Loss of Oversight.
- Multi-Agent Risks; Interaction Topology; multi-agent collusion interpretability.
- LOC Incident Management.
- OpenAI Third-Party Evaluations Playbook; Trustworthy Agentic AI survey.
- MonitoringBench and 2026 monitor-red-team work.
Bottom Line
The must-have agentic eval program for 2023–2026 is a measurement-and-control stack, not a leaderboard. The central governance question is now operational:
Is this agent, with these tools, credentials, memory, and delegated sub-agents, still within its intended bounds over time, and can we detect, stop, contain, and recover when it is not?