Here is a number worth sitting with. Three security scanners were pointed at the same 61,990 agent skills. Between them they raised a flag on 24,148 of those listings. All three agreed on 446.
That is 1.8 per cent of the flagged population. The rest is one scanner shouting while the other two shrug.
The number comes from After the Party: Governing What a Viral Agent-Skill Ecosystem Left Behind, posted to arXiv on 15 September by Yunpeng Xiong and Ting Zhang. It is a census of what happens to a package registry after the hype cycle that filled it has moved on, and it is the most useful thing I have read this week on the question of whether a registry gate can be automated at all.
What the boom left in the drawer
The subject is the public skill registry that grew up around the OpenClaw agent during the first half of 2026. Agent skills are natural-language instruction files that steer a host agent toward shell commands, network calls, credential reads, file operations and process control. They are distributed the way packages are distributed, and they are read the way configuration is read, which is to say hardly at all.
The growth curve is the familiar one. The authors reconstruct a snapshot of 33,399 listings on 20 March, then measure 65,175 on 22 June and 68,096 on 14 July. The stock nearly doubled in ninety-one days, a majority of what was visible in June had been created in the preceding two months, and by the end of the study window monthly creation and core-repository activity were both falling off their spring peak.
So far, so ordinary. Every registry has a spring. What the paper does that I have not seen done carefully before is turn around and ask what is still sitting there afterwards, and whether anything in the metadata tells you which parts are safe.
Three findings, and they compound:
Attention concentrated and then left. The top ten per cent of skills took 46.93 per cent of all downloads. Meanwhile 77.86 per cent of listings have zero stars and zero comments. Nobody is looking at three-quarters of the shelf.
The unread part is not the harmless part. The authors hand-authored twelve rules, one per dimension of privileged capability: filesystem reads and writes, shell and code execution, network access, credential access, browser and process control, persistence, destructive actions, external side effects and privilege escalation. Each rule matches both the frontmatter keys where a skill can declare a capability and a short ordered list of regular expressions for how the capability shows up in the artifact text. On that basis, 85.06 per cent of readable skills carry privilege evidence, and 42,160 of the zero-feedback listings — 84.34 per cent of the evaluable ones — carry at least one detected dimension.
Nothing in the metadata predicts survival. This is the quietly devastating one. They tested whether simple features like size or download count predict whether a listing is still there later, and once creation cohort and skill age are controlled, none of the seven directions from the full cohort survives the restriction. The download association reverses sign between observation windows. What popularity mostly tracks in this corpus is when a thing was uploaded.
Union is not consensus
Which brings us back to the scanners.
The three tools were an LLM-based scanner, a static-analysis scanner and VirusTotal. Of the 61,990 listings all three could see, 24,148 were flagged by at least one and 446 by all three. The LLM scanner alone accounted for 15,874.
A disagreement rate like that could mean one of two things, and the difference matters enormously. Either the scanners are looking at different genuine problems and the union is the true answer, or they are each catching a slice and missing most of the rest. To tell which, the authors did the expensive thing: they drew 180 cases with a fixed random seed — 80 flagged, 100 all-clean — and had two annotators, one with seven years and one with five years of software security experience, label them independently and then resolve disagreements by discussion. The reference standard that came out was 69 flag and 111 do-not-flag.
Against that standard, weighted:
| Scanner | Sensitivity | Specificity | |---|---|---| | LLM-based | 61.06% | 81.28% | | Static analysis | 21.67% | 95.38% | | VirusTotal | 25.11% | 84.54% |
The best of the three finds about three-fifths of what experienced humans would flag. The other two find roughly a fifth and a quarter, and buy that with high specificity, which is the honest trade a conservative rule makes: say little, be right when you speak.
This is the shape of a smoke detector that only responds to one kind of smoke. Put three different models on the ceiling, each tuned to a different particle, and the fact that they seldom go off together tells you almost nothing about whether the house is on fire. It tells you the detectors are different. You still need somebody to walk through the rooms.
The PASS that is not a PASS
I wrote on 14 September about the six-hour hold WordPress.org now puts on every plugin release, where AI models and a commercial scanner score the diff and block the risky ones automatically. My complaint then was about what the gate records when a release passes, which is nothing at all. This paper supplies the measurement that makes the complaint concrete.
If a single automated scanner sits at the gate, the sensitivity figure is the miss rate of the whole scheme. At 61 per cent you are letting through something close to two in five of the things a careful reviewer would have stopped. At 22 per cent you are letting through four in five. And because none of those gates emits a durable record of what version of what scanner looked at what artifact on what date, a downstream consumer has no way to distinguish "checked by the good one in September" from "checked by the conservative one in March."
The obvious dodge is to run all three and flag on the union. Then look at the arithmetic: 24,148 flags across 61,990 listings, on a corpus where 85 per cent of readable skills legitimately carry privilege evidence because that is what a skill is for. A gate that flags nearly two-fifths of everything in an ecosystem whose entire purpose is privileged action is a gate that gets clicked through. It moves the failure from "the scanner missed it" to "the human stopped reading the scanner," which is worse, because the second one leaves no trace in the logs.
Who should care, and who can leave this one
This matters a great deal if you are running or building an agent-skill registry, a plugin marketplace, an MCP directory, or any internal catalogue where one team publishes instruction files that another team's agent will execute with real credentials. The paper's arithmetic is your arithmetic.
It matters if you install skills from a public registry. The practical takeaway is to retire download count as a safety signal. The download association reversed sign between observation windows, which means a high number was tracking recency rather than scrutiny.
You can leave this one if your agents run only skills authored inside your own organisation and reviewed on the way in, or if every privileged action is gated by a human approval that is not a rubber stamp. The paper is about the governance of a registry, and if you do not consume from one, the finding is interesting rather than urgent.
One limit on the study. The privilege-evidence rules are pattern matches over declared frontmatter and artifact text, so they measure what a skill announces or visibly contains, not what it does when run. And the human reference standard rests on 180 adjudicated cases, which is enough to separate 61 per cent from 22 per cent but not enough to put a fine point on either. The authors are clear about both. Read the sensitivity figures as orders of magnitude, not as calibrated instruments.
What a governable registry would have to publish
The authors' conclusion is that governing a fast-growing skill registry cannot rest on simple metadata or a single scanner score, and needs robust, transparent measurement and independent validation. That is correct and it is also abstract, so here is what I think it cashes out to in things a registry could actually emit tomorrow:
- The identity of the check, attached to the artifact. Which scanner, which version, which model, what date, what verdict. A PASS with no provenance is indistinguishable from no check.
- A published sensitivity estimate for whatever runs at the gate, measured against a human-adjudicated sample the way this paper did it, and refreshed when the scanner changes. If the operator will not say what fraction it catches, the number a consumer should assume is unknown.
- A last-reviewed date that expires. A check performed in March against a skill still listed in September is an assertion about March. Silence since then is not evidence.
None of that is a research problem. It is a schema and a cron job.
The question I keep turning over is whether the disagreement itself could be made load-bearing rather than embarrassing. Three scanners that overlap on 446 listings out of 24,148 have told you something real: those 446 are where everything agrees, and the ordering of the rest is information a triage queue could use. Nobody publishes it, because a registry has no incentive to advertise that its gate is one opinion among three. Would you install a skill that came with a card saying two of three scanners cleared it and the third did not, and here is which?