These reading links connect findings to related control areas. They do not establish that a safeguard works in your setting. Read the study conditions and limitations with the finding.
Researchers at MIT propose Closed-Loop Agent Evaluation: Layered Measurement and Self-Calibration , addressing a fundamental weakness in current AI auditing: static, one-off benchmark runs do not establish whether evaluation findings are trustworthy or improve the next agent version. The framework models deployed evaluation as performance estimation under biased raters and censored outcomes, establishing a four-tier analysis hierarchy (span, trace, conversation, and engagement) along a measurement spine of Execution, Substance, Trajectory, and Results. Its inner loop audits evidence, rules, and metrics, while an outer loop guides agent improvement. The implementation remains offline; online triggers and the release gate are not connected to production.
Related control areas
Ask in practiceDoes the test measure the failure that matters in your setting?
In assistive brain-to-robot control, research on Beyond Accuracy: Information-Theoretic Decoder Evaluation Under Latent Intent shows why single-trial decoder accuracy can miss downstream planning regret. The paper derives an information-theoretic regret predictor ($U_{id}$) from decoder confusion matrices, with findings focused on sequential assistive control.
Related control areas
Ask in practiceDoes the test measure the failure that matters in your setting?
Using LLM agents to evaluate other agents introduces severe structural risks, as highlighted in Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge . The preprint reports that a single false pre-screening note attributed to an authority overturns up to 40% of initially correct verdicts across four judges and five datasets, showing confirmation bias in the tested evaluators.
Related control areas
Ask in practiceWhose outcomes were measured, and can affected people seek correction?
Addressing post-hoc forensic auditing, Attributing Agent Failures to Their History Needs an Executed Counterfactual: Toward a Reporting Standard argues that attributing an agent's behavior to its history requires intervention-verified evidence. The proposed reporting standard calls for restoring an executable checkpoint, altering only the visible history, and continuing execution with a measured replay certificate.
Related control areas
Ask in practiceWhat reviewable evidence supports the claim or deployment decision?
To prevent automated supervisory nodes from corrupting correct agent outputs, Who May Overrule the Agent? Evidence-Gated Authority in LLM Review Institutions proposes explicit evidence-gated jurisdiction rules, setting verifiable boundaries on when a reviewer may overrule an acting agent.
Related control areas
Ask in practiceWhat reviewable evidence supports the claim or deployment decision?
Source tiers describe authority and rigor. Read each finding with its study design, setting, and qualifications. A reported result is not a guarantee of performance elsewhere.
3 graded highlights. Counts describe the labels attached to highlights, not unique sources or confidence in a result.
T11 Primary authoritative
T21 Authoritative secondary
T31 Industry analysis
These labels come from the summary, not from the 2 sections below: this edition carries no per-section tier line. Grades for individual sources are shown with the sources themselves.
Peer review, study design, replication, and uncertainty need to be read separately. Interpret the tiers
Verification
Tier grades the source. Verification grades our checking. Before publication, 4 load-bearing claims were re-checked against the primary sources: 1 confirmed as written, 3 did not hold up.
At a glance
MIT researchers propose closed-loop agent evaluation for biased raters and censored outcomes, with an offline implementation so far.
T1
A preprint reports authority framing overturning up to 40% of initially correct Agent-as-a-Judge verdicts; separate research proposes a standard for history-based failure attribution.
MIT Closed-Loop Evaluation Framework: Researchers at MIT propose Closed-Loop Agent Evaluation: Layered Measurement and Self-Calibration, addressing a fundamental weakness in current AI auditing: static, one-off benchmark runs do not establish whether evaluation findings are trustworthy or improve the next agent version. The framework models deployed evaluation as performance estimation under biased raters and censored outcomes, establishing a four-tier analysis hierarchy (span, trace, conversation, and engagement) along a measurement spine of Execution, Substance, Trajectory, and Results. Its inner loop audits evidence, rules, and metrics, while an outer loop guides agent improvement. The implementation remains offline; online triggers and the release gate are not connected to production.
Latent Intent Information-Theoretic Evaluation: In assistive brain-to-robot control, research on Beyond Accuracy: Information-Theoretic Decoder Evaluation Under Latent Intent shows why single-trial decoder accuracy can miss downstream planning regret. The paper derives an information-theoretic regret predictor ($U_{id}$) from decoder confusion matrices, with findings focused on sequential assistive control.
Confirmation Bias in Agent-as-a-Judge: Using LLM agents to evaluate other agents introduces severe structural risks, as highlighted in Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge. The preprint reports that a single false pre-screening note attributed to an authority overturns up to 40% of initially correct verdicts across four judges and five datasets, showing confirmation bias in the tested evaluators.
Proposed Executed Counterfactual Attribution Standard: Addressing post-hoc forensic auditing, Attributing Agent Failures to Their History Needs an Executed Counterfactual: Toward a Reporting Standard argues that attributing an agent's behavior to its history requires intervention-verified evidence. The proposed reporting standard calls for restoring an executable checkpoint, altering only the visible history, and continuing execution with a measured replay certificate.
Evidence-Gated Review Jurisdiction: To prevent automated supervisory nodes from corrupting correct agent outputs, Who May Overrule the Agent? Evidence-Gated Authority in LLM Review Institutions proposes explicit evidence-gated jurisdiction rules, setting verifiable boundaries on when a reviewer may overrule an acting agent.