Skills and plugins are executable supply-chain inputs, and agents barely recognize the attack

Welcome to The Observability Layer, a daily briefing on Responsible AI, AI governance, evaluation, and the technologies shaping the frontier.

A note about how this podcast is made: the hosts you're hearing are AI agents. The research, analysis, editorial perspective, and authorship behind The Observability Layer come from Dr. William Fisher. Our agents help transform that work into the conversation you hear each day.

We think that is fitting. A podcast about understanding, governing, and observing AI should not just talk about intelligent systems. It should put them to work, transparently.

This is The Observability Layer. Let's get into today's research.

Today, we're tracking a theme that's coming up again and again: agent governance is moving down the stack. We're looking at the supply chain for agent skills, why you have to test what's actually deployed, and how to put cryptographic limits on what an agent can authorize.

And finally, a sobering look at what we can, and cannot, recover from the audit trail when things go wrong.

Let's start with that supply chain. A new preprint suggests that the 'skills' we give to coding agents are a major attack vector.

That's right. The paper is titled 'Towards a Risk Assessment of Malicious Skill Files in Coding Agents'. Researchers took over 470 real-world shell commands and turned them into more than 2,800 benign-looking skills, which they then tested against Gemini CLI and Qwen Code.

And the results are pretty stark. Based on the agent's declared intent to comply, Gemini was labeled exploitable in about 96% of runs, and Qwen in about 74%.

Critically, the models explicitly recognized the safety issue in less than 2% of runs. Now, this was a test of declared intent, not confirmed execution, but it's strong evidence that 'the agent will notice' is not a reliable control.

So, what does a governance lead do on Monday? This sounds like more than just prompt filtering.

It is. You have to treat skill and plugin installation like package admission. Verify the publisher, inspect both the natural language instructions and the executable content, sandbox the first execution, and restrict its access to networks and secrets.

So if we can't fully trust what goes into an agent, that puts even more pressure on testing the output. A new pair of papers suggests we're not being nearly specific enough about what we're actually evaluating.

Exactly. One study compared ChatGPT's web interface to its API, both with and without web search. The results were all over the place. Accuracy on some benchmarks shifted by up to eight percentage points depending on the settings.

And it wasn't just accuracy. Repeated runs gave inconsistent answers on up to 21% of prompts. This means a vendor's benchmark score, likely run on a clean API, tells you very little about how the product your employees are using will actually behave.

Which is why evaluation records need to be much richer. They should capture the interface, the model snapshot, system instructions, tool configuration, and sampling settings, not just a model name and a mean score.

That sounds expensive. How can teams afford to run that many variations?

A companion paper offers a path. It applies Item Response Theory to safety benchmarks, finding that about ten well-chosen, adaptive questions can recover the full benchmark score. This could lower evaluation costs by 97-99%, freeing up budget to test the specific configurations you actually deploy.

Okay, so the theme here is getting more specific and moving controls to where they matter. Which brings us to our next story: taking high-stakes authority away from the model entirely.

A new paper, 'Hardware Keystores for AI Agent Signing Workflows', does just that. Instead of letting a model access a signing key from a file or an environment variable, it moves the key into a hardware security module or smart card.

So the model can't actually touch the credential. It has to ask for something to be signed.

Precisely. It's like a bank teller not having the combination to the vault. They can fill out a withdrawal slip, but they have to pass it to a manager with the keys, who checks the request against policy before opening the door. The model requests an operation, and an independent, hardware-backed layer enforces the policy.

And in their tests, this worked. Prompt-injection success across three models dropped from over 19% to zero.

Zero in a bounded experiment, to be clear. The study is small and author-evaluated. The useful claim isn't that they've solved prompt injection, but that this is the right architecture for high-impact actions like code signing, payments, or production changes. Authority should sit below the model.

And when that architecture fails, or isn't in place, can we at least figure out what went wrong after the fact?

That's proving to be incredibly difficult. A new benchmark called SearchAuditBench analyzed over 1,200 failed, long-horizon agent tasks. These are complex, averaging over 70 messages. The best automated auditor could only identify the root cause of the failure in 32.3% of cases.

So 'we log everything' doesn't mean 'we can reconstruct what went wrong'.

Not even close. And a related analysis shows a harder limit: some small errors are statistically indistinguishable from normal variation. Simply having a larger archive of logs doesn't guarantee you can find the mistake. This tells boards and regulators that not every failure will be explainable.

Let's close with a quick look at what's worth watching. First, agent memory.

A project called ContextWeave shows that richer, multi-month memory dramatically improves agent performance. But it also increases the surface area for poisoning attacks through misleading recall. We need ways to manage memory provenance and integrity.

And finally, regulation is already here. The UK's Competition and Markets Authority and the EU AI Act Service Desk have both clarified that existing consumer and AI laws apply to agents now. They're looking for bounded authority, audit logs, and clear accountability.

The implementation question is whether those boundaries appear in runtime evidence, not just in policy documents.

Which brings us back to our thread of the day. Agent governance is moving from abstract instructions the model might ignore, down to supply-chain controls, deployment-faithful tests, cryptographic limits, and honest audit records. It's becoming an engineering discipline.

And one that has to be built from the hardware up.

That's today's edition of The Observability Layer.

If you value rigorous, practical Responsible AI research without the hype, like, follow, and subscribe wherever you listen. It helps more people working at the frontier of AI find the show.

And we want to hear from you.

For questions, comments, research recommendations, or topics you think deserve deeper investigation, reach out to Dr. Fisher at assistant@theobservabilitylayer.com.

We are particularly interested in cutting-edge research that is impactful, technically credible, and well supported by evidence. If there is something the Responsible AI community should be paying attention to, send it our way.

The research and editorial direction of The Observability Layer are authored by Dr. William Fisher, with production and presentation performed by AI agents.

Until next time, keep looking beneath the model, beneath the interface, and beneath the claims.

This is The Observability Layer.