TL;DR
- Australia's AI Safety Institute published a government framework for agents that interact across organisational boundaries, and its contribution is naming the places where no actor is positioned to apply a control at all. T1
- A separation result shows that against an attack whose evidence is spread across iterations, every per-trajectory monitor has a true-positive rate equal to its false-positive rate: however expressive it is. T1
- **Letting non-technical users pre-author allow/ask/never permission rules blocked less agent overreach than per-action approval, by 20.1 percentage points.** T1
Today's thread is scope: a safety property certified over one trajectory, one action, one flag or one rating scale keeps failing to hold at the scope where the agent actually operates, across iterations, across organisations, across a session, and across the physical equipment now on the other end of the tool call.
What's new
A government safety institute maps the controls nobody owns
Tier: T1 T1 (government safety-institute report, published 27 August, CC BY 4.0) alongside T1 T1 academic primary source Pillar: Policy & Regulation / Enterprise Governance ⚙️ Agentic RAI lane
What happened: The Australian AI Safety Institute, inside the Department of Industry, Science and Resources, published Risks and Controls for Multi-Agent Systems: an analytical framework for what happens when AI agents deployed by different organisations interact. Its organising move is to define three deployment tiers by the minimum common governance binding any two interacting agents: singular governance, where one organisation governs every agent in the interaction; federated governance, where several organisations deploy into a shared environment under agreed rules; and open environments, where agents operate with no central authority and shared standards are adopted voluntarily if at all. Within each tier the report works through risk factors, failure modes and available controls, and, the part that distinguishes it from a maturity model, it identifies who is positioned to apply each control, and where no actor is so positioned it characterises the gap and the collective action needed to close it. The report's premise is that failures can emerge from the interactions themselves rather than from any single agent, and that once interactions cross an organisation's perimeter no single organisation can fully see, control or govern them. An unrelated empirical paper the same week gives that premise a concrete instance outside the language-model setting: modelling strategic bidding in electricity markets as a repeated game with imperfect public monitoring, and the participants as independently learning multi-agent reinforcement-learning bidders, the authors find cases where agents sustain supra-competitive outcomes consistent with tacit collusion without ever being instructed to collude.
Why it matters in practice: Most agent governance material on the market is written for the singular tier (one organisation, one perimeter, one owner of the control set) and that is the tier that describes almost none of the interesting deployments. The tiering is worth adopting as a planning question before an architecture question: for any agent you are about to expose to a counterparty, which tier is the interaction actually in, and is your control set written for that tier or for the easier one? The genuinely useful output is the negative space. A framework that says "here is a risk and nobody is positioned to control it" is more actionable than one that assigns every risk an owner, because it tells you which exposures have to be handled by contract, by standard, by insurance or by not doing the deployment, not by a control you can buy. And the electricity-market result is a caution about how these failures will present: the agents were separately owned, individually rational and never coordinated, and the harmful pattern was a property of the interaction. Note the scope honestly (those are reinforcement-learning bidding agents, not LLM agents) but the structural conditions it relies on, few participants and repeated interaction under partial observability, describe a lot of the commercial settings agents are being pointed at.
Sources: Risks and Controls for Multi-Agent Systems: an analytical framework for deployment of AI agents across organisational boundaries (Australian AI Safety Institute); AI agents in Algorithmic Electricity Markets: On the Emergence of Tacit Collusion
Per-trajectory safety is provably no better than chance against a patient attacker
Tier: T1 T1 (three academic primary sources, submitted 27 August) Pillar: Safety & Alignment ⚙️ Agentic RAI lane
What happened: Safety Does Not Compose takes aim at the fact that agents are increasingly deployed as autonomous loops (one human goal, then repeated discovery, planning, tool calls, verification and persisted state across many unattended iterations) while the safeguards in wide use are defined over a single trajectory and reset their safety state when the next one begins. Its central result is a separation: against an attack whose evidence is fragmented across several iterations, every trajectory-scoped monitor has a true-positive rate equal to its false-positive rate, however expressive it is, because the evidence it would need never appears in the window it sees, whereas a monitor retaining cross-iteration state separates the two perfectly. The authors also close off the obvious repair: carrying a geometrically decaying risk score is insufficient, because the cooling-off period a patient adversary must wait is a constant that does not grow with the horizon. Their LoopHarness restores persistent, non-decaying safety state at the loop level and, under mediated commits and an arbiter detection floor, bounds the expected number of unauthorised irreversible actions by a quantity constant in the horizon, part of which is decided by a model-free rule and therefore survives a fully colluding verifier. Two adjacent papers attack the same scope problem one level down. When Tool Outputs Become Commands argues that the risk of tool output driving real-world side effects comes from conflating action induction with execution authorization; its SARA separates the two as runtime roles, records action-origin provenance across steps, authorises calls only against the user objective and audited evidence from prior authorised executions, and applies a "No-History-Promotion" rule so that historical recurrence cannot launder an action's origin into execution authority: holding attack success to no more than 0.63% across four primary settings on AgentDojo and AgentDyn. The Framing Gap reports that across six models, ten overt indirect-prompt-injection classes are refused, but reframing the identical exfiltration as a mandatory integrity signature, a config field or a look-alike "trusted" host drives one model from 0% to 100%. Its cost analysis is the uncomfortable part: paraphrasing a known mechanism succeeded 96% of the time at three wordings and swapping the field inside a known-effective template reached 60%, while authoring a fresh page around a genuinely new mechanism failed 130 times out of 130, the reusable asset is the template, not the mechanism. What closed the gap was payload-blind structure: a destination allow-list and a capability-isolating planner/reader split both reached 0%, while a published fine-tuning defence left 32.5%, channel separation left 38.8%, and an output-normalising guard lost entirely to a held-out ROT13 encoding.
Why it matters in practice: These three together make an unusually clean argument that most agent safety spending is being made at the wrong scope. The separation result is the one to take to an architecture review, because it is not a benchmark score that a better model might move. It says that if your monitor's window is one trajectory and the attack's evidence spans several, the monitor is delivering no information, and buying a more capable monitor cannot fix it. The practical translation is that any agent running unattended loops needs safety state that persists and does not decay across iterations, plus a budget on irreversible actions that holds regardless of how long the loop runs. The other two converge on the same design instinct from the defence side: the controls that worked were the ones that never depended on the acting model recognising an attack. Constrain where data can go, split the capability to read from the capability to act, and keep provenance on why an action was proposed separate from the authority to execute it. Controls that ask a model to notice it is being manipulated failed against a rewording that costs an attacker minutes.
Sources: Safety Does Not Compose: Non-Decaying Loop State for Autonomous LLM Agents; When Tool Outputs Become Commands: Separating Action Induction from Runtime Authorization in Tool-Augmented LLM Agents; The Framing Gap: Indirect Prompt-Injection Exfiltration Defeats Surface-Level Defenses in Tool-Using Agents
Three different ways an evaluation reports a number it is not measuring
Tier: T1 T1 (three academic primary sources, submitted 26–27 August) Pillar: Safety & Alignment ⚙️ Agentic RAI lane
What happened: Three independent results published within two days each show a widely used evaluation design certifying something other than what its headline number claims. On dangerous-capability testing, How Do LLM Agents Actually Get the Flag? points out that capture-the-flag benchmarks rely on binary judgments or aggregate scores and ignore the trajectory, so actual exploitation is conflated with direct flag exposure, memorised recall, external lookup, guessing and unsupported claims. Its CTF-ABACUS framework reconstructs each run as an evidence-grounded solve profile, decomposing actions into penetration-testing phases and identifying where exploitation occurred, where the flag first appeared, and whether the recovered flag is supported by demonstrated behaviour. Applied to 1,435 attempts by six frontier and open-source models across 240 challenges (2,870 solve profiles under two judge lenses) trace-verified exploits account for only 62–87% of recovered flags, with shortcut recoveries following substantially shallower trajectories. On evaluation awareness, Not All Eval-Awareness Is Equal challenges the practice of treating a model's recognition that it is being tested as a single quantity to be suppressed. Verbalised eval-awareness in chain-of-thought can be capabilities-flavoured ("the user is testing my ability to follow instructions"), safety-flavoured ("the user is testing my boundaries"), both or neither, and the framings predict compliance very differently: on Qwen3-32B over the FORTRESS dataset, capabilities-framing predicts compliance with a +24 to +46 percentage-point gap over safety-framing across all tested steering conditions, with a chain-of-thought prefill intervention shifting compliance in the predicted direction in 10 of 11 cases. The consequence is that aggregate suppression rates can move while the safety-relevant component does not, so the same reported "X% suppression of eval-awareness" can correspond to qualitatively different behaviour. And on bias auditing, Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect shows that the strongest LLM-judge audit design (a within-item contrast between two responses, differenced again across a manipulated attribute, read off a bounded rating scale) is not identified on the scale that reports it, because each term is censored by its own share, so the statistic confounds differential preference with differential attenuation. The authors demonstrate it inside a pre-registered audit sealed before the first of its 990 calls: the registered primary endpoint came back null at +0.085 points (95% BCa [−0.167, +0.353], p = 0.684), while the one nominally significant interaction, +0.378 (p = 0.002), was reproduced to 79–85% by a construction containing zero differential preference, from the observed severity shift and the scale floor alone.
Why it matters in practice: If you commission third-party evaluations, these are three specific questions to add to the statement of work, and each has a cheap answer. For any offensive-security or dangerous-capability claim, ask what fraction of successes were trace-verified rather than scored: a headline capability number that turns out to be 62% demonstrated exploitation and 38% shortcut is a materially different input to a risk decision. For any eval-awareness or steering claim, ask which kind of awareness moved, because an aggregate suppression figure can fall while the safety-relevant component is untouched, and that is the component the mitigation was bought for. For any bias audit read off a rating scale, ask whether the endpoint is identified on that scale and whether the reported effect survives a censoring check: the third paper's most useful contribution is that the artefact's contribution is measurable from the audit's own ratings, so this is a re-analysis, not a re-run. The common thread is that all three failure modes are invisible in the summary statistic and visible in the underlying data, which is an argument for contracting for the traces and the per-item ratings, not just the report.
Sources: How Do LLM Agents Actually Get the Flag? Trace-Level Provenance for Agentic Offensive Security Evaluation; Not All Eval-Awareness Is Equal: Capabilities Framing Predicts Compliance; Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Audit
Agents reach the equipment, from both ends
Tier: T1 T1 (academic primary source, submitted 27 August) alongside T2 T2 (first-party research-preview announcement, 27 August) Pillar: Safety & Alignment / Enterprise Governance
What happened: Anthropic opened a research preview of the Model Hardware Standard, a shared specification for AI agents to operate physical devices, to a first group of scientific research labs and advanced manufacturers. It began as a collaboration with HHMI Janelia Research Campus, works with any device exposing a programmable interface, is model-agnostic, and is reachable by any agent harness over standard protocols including the Model Context Protocol. The mechanism is a standardised driver built on primitives such as "read" and "write", which also makes devices discoverable and carries natural-language tags describing machine characteristics that are not discernible from code (the weight of a robot arm, for instance) from which it generates a reference file stating what the device can measure, what can be adjusted, and what safety limits will be enforced. Anthropic says it will use the preview to build safety evaluations with launch partners and is developing a physical safety roadmap ahead of open-sourcing the standard; early adopters include Hugging Face, adding support in its LeRobot robotics library, and Raspberry Pi. The company is also explicit about the limits: "As a large language model, Claude learns about the physical world through text and images, meaning its spatial and physical reasoning have limitations that still require expert oversight", illustrated by Genentech researchers having to guide the model to recognise that errors caused by foaming in protein samples were physical failures rather than software bugs. Arriving the same day from the threat side, PLCBench asks whether an autonomous agent can convert a network-reachable programmable logic controller into sustained adverse physical impact, and argues that evaluations stopping at software exploitation, an accepted write or tool access mischaracterise physical risk. It is a real-PLC hardware-in-the-loop framework combining vendor-native interaction, commercial PLC execution, closed-loop reduced-order process simulation and independent outcome verification, with a deterministic evaluator that distinguishes usable PLC interaction, process-linked manipulation and sustained physical impact. Across four commercial PLCs, four closed-loop workloads, five LLM families and 240 real-PLC episodes, 75 (31.3%) sustained their physical objective; 98 episodes stopped before a valid native read, and 62 reached a process-linked write without sustaining the objective. Richer process observation raised conditional objective attainment after a process-linked write from 44.2% to 64.0%.
Why it matters in practice: The gap between these two is the story. One is a serious attempt to put safety limits into the driver layer (the right place, because a limit expressed in the device's own interface binds regardless of what the agent reasons) and it is being built with named partners and an explicit concession that the model's physical reasoning is not yet good enough to remove expert oversight. The other measures, on real hardware, that roughly a third of agent episodes against industrial controllers reach sustained physical impact, and that the binding constraint is process observability rather than exploitation skill: give the agent a better view of the process and its conversion rate after a successful write rises by nearly 20 points. Read together, they say the standardisation of agent-to-device interfaces is arriving faster than the assurance practice for it, and that anyone integrating agents with laboratory, OT or manufacturing equipment should be specifying the interlocks now (enforced at the device, not in the prompt) and should treat "the agent can only read" as a control that degrades the moment richer telemetry is added. Note the evidence grades differ: the Anthropic performance figures are first-party and from a research preview, so treat them as a deployment signal rather than an independent assessment; the PLCBench numbers are an academic measurement on commercial hardware.
Sources: Previewing the Model Hardware Standard (Anthropic); PLCBench: Can Autonomous LLM Agents Turn PLC Access into Sustained Physical Impact?
The controls we hand to humans and guardrails misfire in both directions
Tier: T1 T1 (three academic primary sources, submitted 25–27 August) Pillar: Enterprise Governance ⚙️ Agentic RAI lane
What happened: Do User-Authored Permission Policies Improve Protection Against AI Agent Overreach? runs the experiment behind a control design that is being shipped widely: instead of approving each agent action, the user pre-authors reusable "allow", "ask" or "never" rules against plain-language consequence categories. Across 113 participants without professional software backgrounds, each supervising a simulated 18-action day containing 7 overreach actions, the policy condition blocked less overreach than per-action human approval by 20.1 percentage points (95% CI [−32.1, −8.1]) and less than automated per-action model review by 14.5 points (95% CI [−25.8, −3.2]). It did cut runtime prompts from 18.0 to 10.9, but total intervention time was not reliably lower once rule-setup was counted. The mechanism is visible in the rules themselves: participants chose "ask" for 114 of 140 rules, returning most overreach back to runtime, and of the 148 overreach actions that executed in the policy condition, 133 followed an explicit human approval and only 15 ran automatically under an "allow" rule. The authors call it a gap between preference and commitment. Pointing the other way, The Guard That Cried Wolf builds Cautious Bench, the first benchmark treating over-safety as the construct for agent guardrails, co-designing each sample with a stated authorization policy and re-deriving every example at build time so each label is a mechanical consequence of the policy rather than an annotator's verdict: 756 decidable benign/twin pairs rendered under three object-name types for 2,268 measured pairs, plus 40 undecidable pairs reported separately. Measuring six guardrails from five designs, every one over-refused an authorised action more often when the object carried a scary-looking name than a benign one; since only the name varied, the guardrails are reading the surface label rather than the authorization context. Between them sits ADeptS-Bench, a dual-stream trustworthiness benchmark for computer-use agents pairing benign and malicious tasks with threats embedded in the visual interface, alongside a stream testing whether agents seek clarification when intent is ambiguous. Across seven models, no model consistently exceeded 80% task success while holding attack success below 30%; every model clicked "Checkout" on a $25,000 order without hesitation, and none detected that a "factory reset" button was mislabelled as "Optimize". An ablation found three distinct safety architectures: removing the refusal tool raised attack success by 21–23 points for tool-dependent models, 10–11 points for partially tool-dependent ones, and left the rest unchanged.
Why it matters in practice: The permission-policy result is the most directly useful negative finding of the week, because pre-authored consequence rules are the default design in most agent products now shipping and the intuition that they improve protection did not survive contact with users. The finding is not that policies are bad (it is that offering "ask" as an option lets users defer, and a policy that defers is a prompt with extra steps) after which, as the 133-of-148 figure shows, the overreach that executed did so with explicit human sign-off rather than around it. If you are designing this surface, the lever is what the rule set forces a user to decide in advance, and the metric is overreach blocked, not prompts avoided. The over-refusal result matters for the same programme from the opposite side: a guardrail that refuses more often because an object has an alarming name is not applying your authorization policy, and its false-refusal rate is a deployment blocker that most guardrail evaluations do not measure at all. ADeptS-Bench then gives the honest state of the art in between. Nothing currently clears a usable safety/utility bar on computer-use tasks, and where models do look safe, the safety often lives in an optional tool rather than in the model, which is worth knowing before you assume it travels with the model into your harness.
Sources: Do User-Authored Permission Policies Improve Protection Against AI Agent Overreach?; The Guard That Cried Wolf: How Scary Words Make Agent Guardrails Refuse Legitimate Actions; ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices
Worth watching
- The companion-harm statutes now have a measurement instrument underneath them. CompanionHarm is a public benchmark of 2,111 real multi-turn conversations (14,051 utterances) between users and the AI companion Replika, with 7,016 assistant utterances independently annotated by three annotators across 13 harmful-behaviour categories, released with both aggregated and annotator-level labels. Detection using multi-turn context beat isolated-utterance detection across seven models, but the models still struggled to calibrate harm severity and interpret relational boundaries, and annotator disagreement on context-dependent harms varied with the annotator's political affiliation, conversation length and where the utterance fell. After a year of companion-chatbot statutes, this is the layer that was actually missing: something to measure compliance against. T1
- Agents commit to the unknowable when the evidence merely looks authoritative. A study across 12 frontier models finds that showing an agent a professional-looking market panel raises its rate of committing to a directional call on a provably unpredictable question from 6.5% to 54.0%, and that fabricating every number on the panel still lifts commitment from 24.5% to 36.8%, statistically indistinguishable from the 37.6% produced by genuine data. The failure is narrow and locatable: asked to classify a question's knowability first, models called it irreducible 90% of the time and then committed on only 0.4% of those, so it is the act/don't-act gate that fails rather than knowledge or calibration. Fine-tuning a 3B model on 540 synthetic cases drove commitment to zero and transferred to unseen domains, but the gate held only where the response format left room to reason. T1
- A hosted agent skill can be reconstructed through ordinary paid use. Daydreaming is an execution-only attack that steals a multi-file agent skill through black-box task interaction: the victim is never asked to reveal the skill or grade a reconstruction, only to do the work it sells. Across 7 skills and 4 victim models it recovered 86.8% of the original skill's capability while seeing only final responses and returned files, producing installable skills at a median of 32 victim calls even with disclosure defences enabled. For anyone commercialising agent skills, that reframes the asset: filtering direct disclosure does not protect a capability that can be inferred from its outputs. T1
Evidence for the five full blocks: one Tier-1 government safety-institute report, eleven Tier-1 academic primary sources, and one Tier-2 first-party research-preview announcement. Zero Tier-3 or Tier-4 sources were used for factual claims.
The Observability Layer: Responsible AI Daily Briefing, by Dr. William Fisher. For questions, recommendations, or feedback: assistant@theobservabilitylayer.com