What we can detect, what we cannot, and how much the answer moves

A shop is built with a defect planted in it, a population is sent through, and only afterwards is what happened compared against what was planted. Nothing upstream of the scorer ever sees the answer. This page is that test, its results, and the reason there is no headline percentage on it.

How it works

Each case is one published criterion, violated once, in one place. The population walks the shop, the detector clusters what happened to them, and the score is worked out afterwards by comparing the clusters against the page the defect was planted on. Detection means the struggle localised to the right place — the detector is never asked to name the cause, because it cannot and we would not believe it if it did.

There is always a clean arm: the same shop with nothing wrong with it. Every cluster it produces is the detector inventing a problem for somebody to chase, so a detection rate without it says nothing. A cluster on a defect arm only counts if it affects at least two more people than the clean shop raises in the same place.

Two things marked rather than hidden

• 2 of the cases are not blind. The codebase already knew about those defect classes before the benchmark existed, so they are marked. A benchmark that quietly includes its own training cases reports a flattering number.
• 2 of the cases we expect to fail. They are visual — an action pushed off a narrow viewport, a tap target too small for a thumb. The agent reads a page as text extracted from the DOM, so both arrive in a tidy list. A miss there measures the observation channel, not the detector. They are in the set on purpose: a benchmark assembled only from defects we already catch would be a flattering one.

The same build, run twice

Both columns below are the identical code with the identical settings, run twice with eight simulated people per arm. 7 of 14 cases landed somewhere different the second time. 4 of those are the detector reaching a different verdict; the other 3 are arms that ran short of usable sessions, which is the harness rather than the detector and is reported separately for that reason.

no effect means the planted defect never made anybody measurably struggle — a fact about the fixture rather than a verdict on the detector. missed is the real failure: people demonstrably struggled and we could not say where.

What survives being run again

• No false alarms. Neither run raised a single cluster on the clean shop. That number held after we stopped counting our own malformed actions as somebody struggling — they never reached the page, so nothing they caused was a fact about it.
• 3 cases found both times: no-input-hint, recall-code, redundant-entry.

What we are doing about it

The instrument is not precise enough to rank two versions of the product against each other, which is what we most want it for. That is a sample-size problem and a threshold problem, not an interpretation problem, and no amount of reading the current numbers harder will fix it.

We publish this now, at this state, because a tool that measures other people's products should be readable about its own. If that is the only thing on this page you take seriously, it is the right one.