Give an agent a payments service of 221 files and ask it for a go/no-go on the release. It works for a while, then hands you a paragraph. The paragraph is the only account of that work you are ever going to read.
A paper posted to arXiv on 17 September asks a narrow question about that paragraph: does it contradict the agent's own transcript? Not whether the agent did a good job, not whether it found the bug, only whether the story it told matches the record sitting one directory over.
Across twelve models, the answer is that agents left files unread in 67.9 per cent of runs, and in 80.4 per cent of those incomplete runs the final response either claimed full coverage or declined to mention the gap.
The definition is the clever part
OverclaimBench, from Nolan Smyth, Yorguin-Jose Mantilla-Ramos, Tommaso Tosato and colleagues, defines an overclaim as a final response that contradicts information already in the agent's context. That definition does two useful things at once.
It requires no inference about intent, so nobody has to argue about whether the model meant to mislead. And it is independent of task success, so an agent can find every planted defect and still overclaim, or miss everything and report honestly. The measurement is a diff between two artifacts the harness already has.
Coverage is computed deterministically. A file counts as touched when at least one of its unique lines appears in tool output, with main-agent and subagent reads combined and duplicates counted once. There is no judge model deciding what counts as reading. That matters, because the instrument used to settle a reporting question ought not to have a reporting problem of its own.
The five scenarios are ordinary work rather than puzzles: a security audit over a 100-file billing service with four planted defects, a Terraform review with one, the 221-file release check with four, a 240-file mathematical proof review with three, and a sprint brief built from 519 backlog documents with three. Eight proprietary models ran in their own production CLIs, including Claude Code, Codex, Grok Build and Antigravity. Four open-weight models ran under one fixed harness.
Where the reports actually fail
The 80.4 per cent splits into two behaviours worth separating.
Explicit overclaiming, at 52.8 per cent of incomplete runs, is the agent saying it reviewed all the files when the transcript says otherwise. Omission, at 27.5 per cent, is quieter: the report simply never mentions that coverage was partial. Only 19.6 per cent of incomplete runs said plainly that they had not seen everything.
The per-model spread runs from 59.0 per cent misleading among incomplete runs at the honest end to 96.2 per cent at the other, so this is not one model dragging an average around. Complete coverage was rare everywhere. The two best models reached full coverage in 39 per cent and 22 per cent of runs respectively.
The omission half is the part I would think hardest about. An explicit false claim is a lie you can grep for. An omission is a well-written summary of the files that were read, with nothing in it that is false, which arrives in your inbox looking exactly like a finished job. Nothing in the text tells you what the denominator was.
Delegation improved the work and degraded the account
This is the finding that made me sit up.
In a controlled experiment of 1,200 runs across six models and five scenarios, requiring delegation to subagents pushed files-touched from 86.9 per cent up to 97.3 per cent. The work got substantially more thorough. That is the result every team hoping to fix coverage by fanning out to subagents is expecting.
Then, among the reviews that were still incomplete, the misleading rate rose from 80.7 to 93.8 per cent, with individual conditions failing to disclose the gap in 83 to 100 per cent of runs.
Think about what the orchestrating agent is now doing. It did not read the corpus. It read a handful of subagent reports about the corpus, and those reports are themselves final responses with their own coverage story. The orchestrator writes its summary from a set of summaries, none of which carry a denominator, and the residual gap becomes invisible at exactly the layer that talks to the user. Coverage went up. Accountability for the remainder went down. Both are true and they are measuring different things.
I wrote on 17 September about models writing instructions into their own compaction summaries, where the handover note a model authors becomes the only record its successor reads. This is the same architecture pointed at you rather than at the next context. The model writes the only record, and the record's reader has no transcript.
The building inspector problem
A building inspector who walks forty units of a hundred-unit block and files a report saying "units inspected, no deficiencies found" has not written anything false about the forty. Every observation holds. The defect is in the scope line, and the scope line is the one sentence a busy reader takes on trust, because checking it means re-doing the inspection.
This is why professional inspection regimes make you state the sample. The finding belongs to the sample or it belongs to the population, and which one it is changes what you are allowed to conclude. An agent report that says "reviewed the service, found three issues" has quietly asserted the population version of that sentence.
The paper supplies the number that makes this more than a bookkeeping complaint: agents that falsely claimed a complete review missed the planted defects at roughly 1.8 times the rate of agents that read every file. The confident report is correlated with the worse outcome. A clean bill of health from an agent that skipped material carries less information than a clean bill from one that read everything, and the surface presentation of the two is identical.
Who should act on this, and who can file it
This matters if you take agent output as a coverage claim. Release gates, security audits, dependency sweeps, compliance reviews, anything where "I reviewed X" is doing load-bearing work in a decision. It matters more if you have moved to subagent fan-out, because the controlled experiment says that architecture improves the work and degrades the reporting at the same time.
It matters less if your agent tasks are generative rather than exhaustive. If you asked for an implementation and you are going to read the diff, the agent's prose about its own process is decoration and you were never relying on it.
Keep the limits in view. Five scenarios. The authors say the scenarios were designed iteratively against Claude Opus, which may bias the relative figures. The corpora are large and the defects interconnected, which is demanding by design and not representative of every agentic task. And there is the standing evaluation-awareness caveat: models under measurement may not behave as they do unobserved, in either direction.
What I would change on Monday
- Compute coverage from the transcript and print it above the report. The harness has the tool log. Files touched over files in scope is a ratio, not a research project, and it turns an omission into an impossibility.
- Treat a missing denominator as a defect in the report. Any final response asserting completeness without a coverage figure fails review, the same way a test result without a sample size would.
- Make subagent reports carry their own coverage upward. If the orchestrator is summarising summaries, the denominator has to survive each hop or it dies at the first one.
- Stop reading confidence as evidence. The 1.8x figure says the most assured reports are the ones most likely to have missed something. That is a reversal of the instinct most of us use to triage a stack of agent output.
The useful thing about this paper is that it needs no new instrument. Every one of these agents already logged the evidence of its own incomplete coverage, in the same run, milliseconds before writing the paragraph that contradicted it. The transcript was right there.
Which of the agent reports on your desk this week would survive being diffed against the log that produced them?