The best coding agent finished 38.8% of real enterprise tasks

Specific Labs published Real-SWE this week, a benchmark that drops frontier coding agents into private production codebases licensed from real companies rather than public repositories. Eight model-and-harness configurations, ten tasks, 640 scored rollouts, eight independent runs per task. The best configuration — Fable 5.1 running in Claude Code — resolved 38.8% of tasks. Second was GPT-6 Astra in Codex CLI at 33.8%, then Gemini 3.8 Flash at 31.2%. The lowest, GPT-5.6 Sol in Codex CLI, managed 16.2%. No model solved every task, and on one task, an analytics stream reducer, all eight models went 0 for 8.

The benchmark gap is the whole story

Public benchmarks measure work that resembles the training data. Real-SWE measures work that does not exist on the internet. Specific's framing is blunt: 99% of tokens inside real enterprises are hidden from frontier models. The tasks are billing, tax calculation, customer migration — changes with financial consequences, spanning services, databases, and third-party tools including Kubernetes, PostgreSQL, Linear and Slack.

The difficulty is structural, not verbal. The median Real-SWE instruction runs 1,742 characters, shorter than FrontierCode at 2,056 and DeepSWE at 1,975. But the median reference solution edits 11 files, against 6 for both of those benchmarks. Short brief, wide blast radius. That is what production work looks like.

Six of the ten tasks scored below 15% resolution. Tax jurisdiction: 3.1%. Linearizable scan: 4.7%. Meanwhile a multi-region sweep hit 67.2% and API keys and environments hit 65.6%. The spread is the finding. Agents clear contained, well-bounded changes and fall apart when the change touches business rules held in someone's head.

The failures are judgment failures

Specific grouped every failed run by behavior. The categories are not syntax errors. They are unverified assumption — building on a guess instead of checking the workspace; missed requirement; integration error; regression; wrong file. Grok 4.6 missed a stated requirement in 67.2% of its failed runs. Kimi K3 in 53.8%. GPT-5.6 Sol built on unverified assumptions in 43.3% of its failures. Gemini 3.8 Flash produced integration errors in 49.1% of its.

More compute does not fix this. Failure rates barely moved with time spent: 71.4% of rollouts under ten minutes failed, versus 73.4% of the longer ones. Cost per rollout ranged from $2.50 to $6.96, and the cheapest configuration outscored three more expensive ones. Gemini 3.8 Flash reached 31.2% at $2.50 while GLM 5.3 reached 28.8% at $5.12.

The consensus reading of a 38.8% top score is that agents are not ready and the job is safe. Wrong on both halves. Look at what does clear: bounded, verifiable, single-context tasks — which is the majority of the ticket queue in most organisations. What survives is the work that requires knowing which unstated rule applies, which system will break, and who to ask. That is not coding. That is context and accountability, and it is exactly the layer that does not appear in any repository.

Your Next Move

Sort your own queue by verifiability. Take last month's tickets and mark each one as either fully specified by its written brief or dependent on knowledge that lives outside the codebase. The first pile is being absorbed now. Your standing depends on the size of the second.

Write down the undocumented rules. The exemptions, the sandbox-versus-production conventions, the reasons a service behaves oddly. Models fail on missed requirements more than anything else because nobody wrote the requirements down. Being the person who documents them makes you the person who reviews the agent output.

Own the verification step. Specific injects verifiers at grading time; in your organisation, that role is a human. Volunteer to define what "done" means for agent-generated changes before someone above you assigns it as a junior task.