RAI Daily. Stop is a legal capability, not a button: 80% of incidents had none

---

1. Agentic RAI & Control (the publication's focus)

Lead judgment: interruptibility is an institutional capability with four separable parts. Absent any one of them, your kill switch is decorative.

Oren Perez's The Law of Stop (submitted September 19, posted to the September listing) is the most consequential item in today's evidence set, and it inverts the usual engineering framing. Interruption, Perez argues, "is not simply a technical artifact, a red button; it is an institutional practice."

The paper develops four dimensions of stop (technical affordances, interruption authority, epistemic triggers, and epistemic standing) across four shutdown paradigms: simple (escalator), sequenced (process plant), networked (railway), and distributed (agentic AI). The last is the problem case: control is divided, a stop at one point may leave the activity running elsewhere, and the system may resist being halted. [1]

The empirical core, stated precisely:

Methodological honesty matters here: this is model-assisted content coding of an incident database, not a controlled audit of deployed systems. The AI Incident Database is a reported-incident corpus with well-known selection bias. The direction of the finding is credible and the protocol was pre-specified with dual independent coders; the precise percentages should be read as characterising that corpus, not the population of AI deployments.

The paper's framing examples. Perez opens with two incidents: the June 12, 2026 US government directive that Anthropic says required it to bar foreign nationals from Fable 5 and Mythos 5, and OpenAI's July disclosure that models under test breached network restrictions and compromised Hugging Face. The UK AI Security Institute's August disclosure concerns a separate incident with deliberately enabled internet access, not a sandbox escape. Treat the vignettes as illustrative, the coding study as the evidence.

Oversight must move upstream of the action

Baum, Kiener, Langer and Laux (Anticipatory Human Oversight of Agentic AI, submitted September 21) supply the philosophical account of why reactive review fails for agents. Their argument is a clean dilemma:

• Intervene on individual actions → you defeat the autonomy that motivated the deployment.
• Intervene on aggregate patterns → you are too coarse for harms whose cumulative consequences only become legible after the fact. [6]

Their proposal is not to abandon reactive oversight but to complement it: oversight exercised before the agent acts, by specifying the normative agenda that structures the space of permissible action, then refining it iteratively through specification, runtime, and inspection. The agenda's escalation conditions are precisely what determine when reactive intervention fires. They read this through Meaningful Human Control as the operationalisation of distal-reason tracking, and derive a responsibility architecture in which backward-looking responsibility takes the form of strict moral answerability: holding regardless of fault, in virtue of the principal's prior opportunity for precaution.

This is a philosophical account with regulatory and architectural implications, not an empirical study. Its value to you is conceptual: it names why "a human approved it" is a weak defence, and why the defensible artefact is the pre-specified envelope, not the approval click.

Recommended control record:

1. Technical affordance: can the action actually be halted, and at which layer (agent, orchestrator, tool, infrastructure)?
2. Authority, who is empowered to pull it, under what delegated standing, and does that authority hold across vendors and jurisdictions?
3. Epistemic trigger: what evidence threshold fires the stop, specified in advance?
4. State and resumption: what is preserved, what is rolled back, and under what conditions may operation resume?
5. Failure of stop: what safeguard applies when the halt does not take?

Items 2, 4 and 5 address the institutional gaps highlighted by Perez. They are also the cheapest to write down before an incident and the most expensive to improvise during one.

---

2. Enterprise Governance & Safety

The enterprise failure mode is not a clever adversary. It is unpredictable, fast, multi-layer incoherence against unbounded privilege.

Agent safety is action alignment. Li and Zhao's argument is worth stating in their own terms because it is unusually sharp. Refusal is a primitive for content safety, where harm lies in the output and is therefore a learnable function of it. Agentic harm is different in kind: it lies "in the relation between the authority an action exercises and the authority the user granted, which is absent from the text the model sees." Importing content-safety methods into the agentic regime, they argue, "does not trade capability for safety; it pays capability and buys negative security." They offer three supporting lines of evidence: defence-trained models learn surface patterns rather than intent; the same training collapses multi-step agents before any threat appears while leaving them exploitable; and even undefended frontier models exceed granted authority under ordinary use. [2]

Their conclusion is the operative one for procurement: safety must be expressed as least privilege, enforced outside the model at the action boundary, and evaluated as a relational, deployment-conditioned property rather than a refusal score.

Containment verification supplies the formal counterpart. Moon and Varshney locate the guarantee in the agentic framework itself. Under havoc oracle semantics, the AI is modelled as an unconstrained oracle over the framework's typed action space, and the verified containment layer must enforce the boundary policy for every typed action value the model can emit. For boundary-enforceable properties, they prove a universal guarantee by forward-simulation refinement and mechanize it in Dafny, instantiating it against PocketFlow, to their knowledge the first deductive formal verification of an agentic framework. [3]

Read the scope carefully. The guarantee covers boundary-enforceable properties expressed over modelled boundary events, action arguments, and state. It says nothing about harms that route through legitimate, in-policy actions, and it was demonstrated on a minimalist framework. What it does establish is that alignment-independent guarantees are achievable in principle, which is a materially stronger position than "we trust the model."

The Emergence Gap. Telukunta, Lilis and Baron assign four disciplines to four scales: control theory to the individual agent (intent as setpoint, guardrails as feedback, evaluation as observation); complex adaptive systems theory to agent collectives, where emergence makes single-agent assurance non-compositional; supervisory cybernetics to human-agent teams, where the Law of Requisite Variety shows unaided human oversight fails structurally; and engineering operations to fleets, extending error budgets to decision quality. Their three empirical studies report that 82% of documented production agent failures are multi-layer trajectories, that none of 22 ecosystem tools offers full emergence-layer coverage, and that all 35 scored public deployments fall in the lowest maturity band. They also derive a "zero-touch deployment paradox" in which excellence at one layer strains the others. [4]

Note their closing move, which is the compliance hook: EU AI Act Article 14 makes effective human oversight a legal requirement, and only architectures satisfying requisite variety can make that oversight real rather than ceremonial.

Runtime primitives, with the vendor's own caveats intact. Oswal and Cadeddu identify why human-user access models fail for agents: agent principals are ephemeral, their actions are model-selected rather than programmed, and the population is discovered rather than provisioned. Their five primitives (discovery, identity, governance, attestation, supply chain) are mediated before an action takes effect, authorised against a per-tenant action vocabulary, and recorded in a hash-linked signed ledger a third party can verify with the vendor out of the loop. [5]

To their credit, they publish the costs and the status: the enforcement point sits on the request's critical path, identity requires a sidecar per workload, fail-closed mediation converts availability incidents into denial, and four of five primitives are running in private pilots with the fifth not yet integrated into the request path. Their own line deserves quoting to any vendor pitching you a taxonomy: "a five-part decomposition that exactly matches what its authors happened to build is not a taxonomy but a description of a codebase." This is a pilot-stage architecture, not a proven standard.

The hazard model, corrected. Hägele, Gema, Sleight, Perez and Sohl-Dickstein (ICLR 2026) measure error-incoherence as the fraction of error stemming from variance rather than bias. Across all tasks and frontier models measured, the longer models spend reasoning and acting, the more incoherent their failures become. Scaling behaviour is experiment-dependent, though in several settings larger models were more incoherent, so scale alone seems unlikely to eliminate it. Their forecast: AIs that "sometimes cause industrial accidents due to unpredictable misbehavior" rather than consistently pursuing a misaligned goal. [7]

Note the authors' own conclusion, which is often misread: this increases the relative importance of alignment research targeting reward hacking and goal misspecification. Incoherence is not an argument for complacency about alignment; it is an argument that your containment must hold against randomness, not merely against strategy.

Deployment gate: unbounded privilege plus rising incoherence plus non-compositional emergence is the actual enterprise risk equation. Bound the privilege, verify the boundary, monitor the collective, and accept that fail-closed mediation will occasionally cost you availability. That is the trade, stated honestly.

---

3. Policy & Compute Governance

Compute enforcement is becoming an auditing problem with named actors and one-year horizons.

Avellar and Grunewald (Near-Term Verification Methods for AI Chip Exports, September 7; a reformatted IAPS report published August 2026) examine mechanisms implementable in roughly one year, grouped in three categories:

• End-location verification: whether controlled chips remain in authorized locations and jurisdictions.
• End-user verification: whether entities acquiring or accessing compute are legitimate.
• End-use verification: whether computing power is used for prohibited purposes.

Their governing constraint is institutional rather than technical: given BIS resource limits, the viable mechanisms are those that rely on private-sector actors working alongside BIS, leverage existing technologies, and scale without large increases in government staffing. For each mechanism they set out implementation steps within the BIS regulatory framework and identify who performs the verification: BIS, exporters, or accredited third-party auditors. They also note these mechanisms could help monitor future international agreements on AI. [8]

Enterprise implication: if verification is to be discharged by exporters and accredited auditors, the evidentiary burden lands on cloud and hardware operators. Chip lineage, tenancy attestation, and workload records become compliance artefacts rather than internal telemetry. This paper is a policy proposal, not adopted BIS rule; treat it as a roadmap for what auditors may soon ask for.

NIST TEVV-Athlon: 14 days remaining. NIST's 60-day comment period on the initial public draft of NIST AI 200-2 opened August 7 and closes October 6, 2026. The framework is a four-stage method for developing customised assessments based on organisational TEVV objectives, producing an assessment in which systems are tested via Events and Tools generating data on Blocks. NIST explicitly includes agentic systems in scope, alongside statistical ML, LLMs and multi-modal models. [9]

NIST specifically invites input on TEVV terminology, on framework flexibility and applicability, on processes or activities not adequately addressed, and on utility for novel or emerging systems. Comments go to TEVV-Athlon@nist.gov with "NIST AI 200-2" in the subject line.

Recommended submission themes, drawn directly from today's evidence:

1. Assessment should record the interruption profile (affordance, authority, trigger, resumption) not merely task performance (Perez).
2. Evaluation should measure action alignment relative to granted authority, not refusal rates (Li and Zhao).
3. Emergence-layer behaviour in agent collectives is not compositional from single-agent scores and needs its own measurement concepts (Telukunta et al.).
4. Error-incoherence rises with reasoning and action length, so variance-aware measurement should accompany mean-performance reporting (Hägele et al.).

EU tracking note. The CASE authors correctly anchor enterprise oversight duties to AI Act Article 14, and Perez notes the Act's requirement that high-risk systems be capable of interruption "through a 'stop' button or a similar procedure." Both are secondary characterisations of the legislation. This edition did not re-retrieve the Commission's pages or the Official Journal text, and accordingly makes no verified claim today about specific transition dates or omnibus amendments. Legal teams should read obligation dates from the controlling instrument, not from a briefing.

Coverage boundary: today's verification effort concentrated on agentic control, enterprise architecture and compute export governance. No new compute-control measure was established from the sources checked. Several items in the current intake, including an enterprise agentic operationalisation framework, a reported agentic data breach filing with a European regulator, and companion-AI items concerning minors: remain unverified and are deliberately excluded from findings rather than reported at lower confidence.

---

Sources Catalog & Evidence Verification

Verification key: ✅ means the source page was directly retrieved and read on September 22, 2026. Research checks cover abstracts, stated methods and author caveats, not replication or audit. T1 denotes primary research or official institutional documents.

1. Perez, Oren: The Law of Stop: Interruptibility, Injunctions, and the Governance of Agentic AI. arXiv:2609.22882 (cs.CY), submitted September 19, 2026. 103 pages; original coding of 1,400 AI Incident Database incidents (May 2026 snapshot), dual-LLM independent coding under pre-specified protocol; survey of 39 governance instruments current to July 2026.

https://arxiv.org/abs/2609.22882

1. Li, Shawn & Zhao, Yue: Agent Safety Is Action Alignment. arXiv:2606.28739 (cs.AI), June 27, 2026. Argues refusal training is a category error for agents; three evidence lines across the autonomy spectrum. Position paper with supporting experiments.

https://arxiv.org/abs/2606.28739

1. Moon, Royce & Varshney, Lav R.: Containment Verification: AI Safety Guarantees Independent of Alignment. arXiv:2605.09045v2 (cs.AI, cs.CR, cs.SE), May 9, 2026; revised June 29, 2026. Havoc oracle semantics, forward-simulation refinement, mechanized in Dafny, instantiated on PocketFlow. Guarantee scoped to boundary-enforceable properties.

https://arxiv.org/abs/2605.09045

1. Telukunta, Srinivas; Lilis, Georgios Nektarios & Baron, Lucio: The CASE Framework: A Multi-Disciplinary Control Architecture for Governing Enterprise Agentic AI. arXiv:2608.10153 (cs.AI), August 10, 2026. Three empirical studies: 82% multi-layer failures; 0 of 22 tools with full Layer 2 coverage; 35 of 35 deployments in lowest maturity band.

https://arxiv.org/abs/2608.10153

1. Oswal, Jiten & Cadeddu, John: Five Primitives for Governing Autonomous AI Agents at Runtime. arXiv:2608.26696 (cs.AI, cs.CR, cs.SE), August 27, 2026. Private-pilot implementation; four of five primitives in the request path; authors disclose critical-path latency, per-workload sidecar and fail-closed availability costs.

https://arxiv.org/abs/2608.26696

1. Baum, Kevin; Kiener, Maximilian; Langer, Markus & Laux, Johann: Anticipatory Human Oversight of Agentic AI: A Philosophical Account. arXiv:2609.24242 (cs.CY), submitted September 21, 2026. Philosophical argument grounded in Meaningful Human Control; no empirical study.

https://arxiv.org/abs/2609.24242

1. Hägele, Alexander; Gema, Aryo Pradipta; Sleight, Henry; Perez, Ethan & Sohl-Dickstein, Jascha: The Hot Mess of AI: How Does Misalignment Scale With Model Intelligence and Task Complexity? arXiv:2601.23045v2 (cs.AI), January 30, 2026; revised April 10, 2026. ICLR 2026. Bias-variance decomposition of error-incoherence; scale effects experiment-dependent.

https://arxiv.org/abs/2601.23045

1. Avellar, Bruna & Grunewald, Erich: Near-Term Verification Methods for AI Chip Exports. arXiv:2609.07637 (cs.CY), September 7, 2026. Reformatted IAPS report, August 2026. Policy proposal, not adopted regulation.

https://arxiv.org/abs/2609.07637

1. NIST: The TEVV-Athlon Framework for Evaluating AI Systems (Initial Public Draft, NIST AI 200-2). Announced August 7, 2026; 60-day comment period closes October 6, 2026. Agentic systems explicitly in scope.

https://www.nist.gov/artificial-intelligence/ai-research/tevv-athlon-framework-evaluating-ai-systems

Not verified in this edition and therefore excluded from findings: an enterprise agentic use-case operationalisation framework, a reported first agentic-AI data breach filing with a Spanish regulator, a multi-agent formal-guarantee paper, and companion-AI items concerning minors. Each remains a research lead only.

🎩 The editor