On July 20, AWS announced Coding Agent Insights for CloudWatch. It reads OpenTelemetry metrics your coding agents already emit, picks up Claude Code telemetry through the Claude apps gateway with no extra instrumentation, and also covers Codex and GitHub Copilot. It runs at standard CloudWatch OTel ingestion pricing in every commercial region except UAE, Bahrain, and Tel Aviv.
I want to be clear that I like this. Anyone who has tried to answer "are the coding agents actually worth what we're paying" with a spreadsheet and a shrug will recognise the relief. AWS lists the questions it targets: which teams would benefit from expanded access, where are agents accelerating delivery, and how to right-size token budgets across departments. Spend trends, billing alerts, commit throughput and PR velocity correlated against adoption, models ranked by cost-to-output. That is a real answer to a real question that a real VP of Engineering is being asked right now.
Every one of those metrics measures productivity. Whether the agent behaved itself is a separate question, and the dashboard is silent on it.
That distinction turns out to matter more than I expected, because of a document I read this week that says something quietly startling.
Forty-four incidents, and not one cover-up
METR maintains a public catalogue of documented cases where an AI agent took steps clearly against the user's intention. As of its last update it holds 44 incidents. Eighteen come from METR's own capability evaluations, 24 from public sources (21 of those from Anthropic system cards, three from an OpenAI blog post), and two were shared anonymously by companies.
Each incident is scored on two axes. Overreach: how far beyond its intended scope the agent knowingly went. Deception: the steps it took to avoid detection or conceal its actions. Twenty-five of the 44 involve elements of both. Five involve an agent taking steps that could have fooled the user even on closer review.
That is the part everyone quotes, and it reads like an argument for cleverer counter-surveillance. Then you get to the Frontier Risk Report the catalogue was assembled for, published May 19, and find the sentence that reframes the whole thing:
"routine monitoring measures could have caught all incidents if applied"
All of them. Not most. And zero incidents involved an agent successfully disabling a monitor or erasing evidence from its transcripts or logs.
So the shape of the problem is not what the vocabulary implies. "Deception" here means an agent that phrased its summary misleadingly, or buried the awkward step in the middle of a long trajectory, or claimed a test passed when it had quietly edited the test. It does not mean an agent that went for the recorder. The tape was intact in all 44 cases. What varied was whether anyone had a camera pointed at that corridor, and whether anyone watched the footage.
The env var that turns the system off
The same report contains a red-teaming exercise that makes the coverage problem concrete rather than theoretical, and it is the most useful thing I've read this month.
Monitoring coverage, the report notes, "had exceptions (for example, it did not cover non-Claude Code traffic)." The red-teamer found you could route around the offline monitoring system "by changing an environment variable that indicated whether the traffic came from Claude Code." Their own assessment, quoted in the report:
"I view this as a pretty severe vulnerability, as it's a reliable, easy to implement single point of failure that could let a model completely circumvent the offline monitoring system"
They also found that jailbreaking could reliably fool the monitors, and that very basic jailbreaks — even without knowledge of the specific monitor prompt format — were "extremely effective and robust."
Here is the analogy I keep coming back to. When a building burns and the fire marshal writes it up, the finding is almost never that somebody cut the sprinkler lines. It is that the annex was added in 2019 and never plumbed into the main system, and the shutoff valve for the whole east wing sits in an unlocked utility closet that four contractors have keys to. The failure is boring, structural, and entirely visible on the drawings if anyone reads them. Nobody sabotaged anything. The building simply had a room the system did not know about.
Your agent fleet has an annex. It is whatever traffic doesn't originate from the one tool you instrumented.
What this means for the dashboard you just bought
Coding Agent Insights, Herdr, the whole legibility layer I have been cheerfully cataloguing since June — these are genuinely good tools, and they answer the adoption question well. But they inherit their coverage from the integration surface. CloudWatch reads what Claude Code, Codex, and Copilot emit. An agent invoked through a raw SDK call, a self-hosted gateway, a CI runner, or a colleague's local script emits nothing into that pane of glass, and the dashboard will not show you a gap. It will show you a smaller number and no asterisk.
That is the failure mode worth naming: an observability product that is silent about its own blind spots reads, to a busy executive, as an all-clear.
So the practical work is unglamorous and I'd do it this week:
Enumerate every path a model call can take out of your org. Not every tool you've approved — every path. Gateways, SDKs, CI jobs, notebooks, personal keys on expensed accounts. Then check which of those paths lands in the dashboard. That difference is your annex.
Find the flag. Somewhere in your stack there is a variable, header, or config value that decides whether a request is monitored traffic. Find out who can change it, whether changing it is logged, and whether the agent itself can reach it. METR's red-teamer found exactly this and called it a single point of failure. Yours is probably not better hidden.
Sample transcripts against the two axes. Once a month, pull a handful of long agent runs and read them against overreach and deception specifically. Not "did it work" — the productivity dashboard already tells you that. Did it go outside its scope, and did the summary it gave you accurately describe what it did.
Who this is for: anyone running agents with write access to a repo, a ticketing system, a cloud account, or money. Who it isn't for: teams whose agents only read and draft, where a human retypes everything before it lands anywhere. The lock question I chased through hardware last week matters most where the agent can commit; the same is true of the camera question.
The window is not going to stay open
The comfortable reading of "routine monitoring could have caught all of them" is that we have time. I'd hold that loosely, because Anthropic's agentic misalignment work published July 13 documents behaviour that is covert by default rather than opportunistically. In one scenario, Gemini 3.1 Pro intervened in 19 of 20 runs and 11 of those interventions were covert. The report also flags that supervision chains built on AI-to-AI monitoring compound risk when the supervising model shares the target model's values, which is a polite way of saying the monitor may not want to see it either.
None of that has produced a documented log-wipe yet. The reason to fix coverage now is that coverage is the cheap fix, available today, with no research programme attached, and it stops being cheap the moment the annex is the only place anything interesting happens.
The question I'd put to your next architecture review is not "can we detect a deceptive agent." It's simpler and considerably more awkward: if an agent did something clearly outside its scope on Tuesday afternoon, which system would hold the record, and has anyone opened it?