Naming a testing technique does not make an AI agent use it

Dan Luu published an eval on danluu.com this week testing whether telling a coding agent to use a named testing or verification technique actually improves the code it writes. It does not. He ran 26 prompt conditions — TDD, fuzzing, property-based testing, Lean 4, TLA+, Verus, Alloy, SMT solvers, mutation testing, and a control condition with no extra instructions — plus four testing skills, at an average of 80 runs per condition and effort level, all implementing Zstd in Rust. The control condition, Default, scored well above average. So did "Make no mistakes", a prompt Luu included as a joke, which he notes outperformed most conditions because a no-op beats making agents do ineffective things.

The technique name transfers. The judgment does not.

The failure mode is consistent across tools. Agents write the tests they were always going to write, then wrap them in the vocabulary of the technique they were told to use. Told to use Verus, they proved vacuous properties of the form A implies A. Told to use TLA+ on an IMAP implementation, 75 of 80 agents modelled the protocol after writing the code rather than before, and they modelled the parts where the Default condition already had a test pass rate above 99.6%.

TDD changed behaviour the most and helped the least. Agents in the TDD condition had one or more failing tests before substantial implementation in 67 of 160 cases, against 0 of 160 for Default. They wrote roughly twice as many tests. They were still more likely to fail the hard cases, because they wrote tests that could not catch them — four identical bitstreams in a test of a feature that uses four different ones, palindromic inputs in a test of a reversal.

That is the whole finding. The agent can name the method. It cannot locate where the bug will be. Luu notes that when asked directly to identify risky areas, agents did that well — and then still failed to build checks that would catch the risk they had just named.

Shared expertise is not the moat people assume

All three published skills he tested underperformed. One belongs to a skills collection with 250,000 GitHub stars and 38,000 forks. Two official vendor skills he tested separately — one from a large AI lab, one from a smaller company — also made results worse. The skill that scored highest was five bullet points Luu wrote himself in about two minutes, encoding how he personally hunts bugs, and he is explicit that it did not work the way he intended.

The consensus reading of agentic coding is that the scarce input is the prompt, and prompts are copyable, so expertise is about to be commoditised into a repository anyone can clone. The eval says the opposite. What separates a good result from a bad one is not the artefact. It is the person who reads what the agent did, sees that the test is palindromic, and types three more sentences. Luu's own account of what works is exactly that: set up a reasonable structure, then check back and re-prompt.

Your Next Move

Stop grading output on whether it passes. Agents on high effort reliably get their own tests to pass. That is the trap. Pick one thing your team shipped with agent help this month and ask a different question: what specific mistake would this test suite fail to notice? If you cannot answer, you do not have verification, you have a green tick.

Audit the skills and rules files in your stack. Every published testing skill in the eval underperformed, vendor-official ones included. Anything sitting in your repo that reads like a tutorial rather than a correction of default behaviour is costing tokens and buying nothing. Delete it or rewrite it as a nudge away from the specific failure you have watched happen.

Build the thing that does not compress. Knowing where bugs hide in your domain is the layer agents did not reach in any of the 26 conditions. Write down the three failure modes your product actually has in production. That list is your leverage, and it is not on GitHub.