What a SWE-bench score actually measures

Model launches lead with SWE-bench numbers, and in 2026 the top ones sit in the mid-90s. That reads as "AI fixes 95% of software bugs." It doesn't mean that. This post covers what the benchmark does, why the version most people quote was retired, and why its replacement, SWE-bench Pro, has run into the same problem.

Disclosure: ARC sells legacy modernization, the work these benchmarks are often used to market. We have not published a SWE-bench score. Every number below links to its source.

The task

Each SWE-bench task is a real GitHub issue from a popular open-source Python project, such as Django or scikit-learn, paired with the pull request that fixed it (Jimenez et al., 2023). The agent gets the repository as it was just before the fix, plus the issue text, and has to produce a patch.

A grading harness applies the patch in a clean container and runs two lists of tests the agent never saw. The task counts only if every one passes. That design has two consequences.

• A correct fix can fail. The hidden tests were written by one maintainer for one solution. If they check an error message or function name the issue never mentions, a different correct fix fails.
• A wrong fix can pass. When researchers strengthened the tests on SWE-bench Verified, 15.7% of previously passing patches were flagged as erroneous, and 24.4% of leaderboard rankings changed (UTBoost, ACL 2025). Adversarially strengthened tests cut the top score from 78.8% to about 62% (SWE-ABS, ICML 2026).

The tasks are small

"SWE-bench" usually means Verified, a 500-task subset that OpenAI had human engineers screen for fairness in 2024. They also estimated how long each fix would take an experienced engineer.

91% of the tasks are under an hour of work, in 12 well-known Python libraries, from a written bug report (OpenAI, 2024). That is a real skill, and a narrow one. There's no ambiguous requirement, no database migration, no front end, and no code the model hasn't already read in training.

The agent program changes the score

A score belongs to a model running inside a particular agent program, which researchers call the scaffold. Vals.ai ran the same models through a minimal 100-line agent and through commercial coding agents. Claude Opus 4.8 scored 88.6% in the minimal agent and 85.8% in Claude Code. GPT 5.5 scored 82.6% and 76.4% in Codex (Vals.ai). The model didn't change; the program around it did.

On 500 tasks, one run also has a margin of error of about 3 percentage points either way. A 2-point lead from a single run is a tie.

Why Verified was retired

OpenAI stopped reporting Verified in February 2026 (OpenAI). It audited 138 tasks its o3 model failed to solve consistently, and found that at least 59.4% had tests that reject correct solutions. It also concluded that recent gains reflected how much models had seen the benchmark in training. Vals.ai archived its leaderboard in September with the top score at 97.0%, noting that performance had saturated.

SWE-bench Pro, and the same problem again

OpenAI pointed the industry at SWE-bench Pro, from Scale AI. Pro is harder by design (Scale AI, 2025): fixes average 107 lines across 4.1 files, it covers Python, Go, JavaScript and TypeScript, and it uses copyleft and privately purchased code to keep tasks out of training data.

One experiment in the Pro paper is worth knowing. With the human-written requirements and interface spec removed, GPT-5 fell from 25.9% to 8.4%. The models didn't get worse at engineering. They lost the detail they needed to match what the hidden tests expected.

In July 2026 OpenAI reviewed all 731 public Pro tasks. Its human reviewers flagged 249, or 34.1%, for flaws: tests demanding details the prompt never mentions, prompts missing requirements the tests check, tests too weak to catch incomplete fixes, and prompts that contradict their tests (OpenAI, July 2026; summary via NeuronFeed). An independent review of 100 tasks in February found the same pattern (Gabor).

With roughly a third of tasks flawed, a public-set score in the high 80s needs evidence of how it was produced. Independent and self-reported results have split: the best entry on Scale's own leaderboard is 61.5%, while the best self-reported figure is 89.9%. Scale's board doesn't accept closed-source agents, so vendor agent scores never appear there.

What a score is good for

Below about 70% on Verified, a model is a real warning sign. Results from a standard minimal agent, run by an independent party, are useful for comparing models with each other. What no SWE-bench variant measures is moving a whole legacy system to a new stack with its business rules intact. That work spans hundreds of thousands of lines and requirements nobody wrote down.

FAQ

Is a 95% SWE-bench score fake? Usually not. It's a real measurement of a narrow, saturated task. The mistake is reading it as general engineering ability.

Which number should I trust most? An independent run in a standard agent, such as the bash-only leaderboard. For Pro, the commercial set, which Scale runs on code nobody can train on.

Does ARC publish a SWE-bench score? No. SWE-bench tasks are bug fixes in open-source libraries. We would rather measure the work we do: whole-system parity on real legacy code.

Sources

All accessed 2026-10-30.

• Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?", ICLR 2024. https://arxiv.org/abs/2310.06770
• OpenAI, "Introducing SWE-bench Verified", 2024-08-13. https://openai.com/index/introducing-swe-bench-verified/
• OpenAI, "Why we no longer evaluate SWE-bench Verified", 2026-02-23. https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/
• OpenAI, "Separating signal from noise in coding evaluations", 2026-07-08. https://openai.com/index/separating-signal-from-noise-coding-evaluations/
• Scale AI, "SWE-Bench Pro", 2025-09. https://arxiv.org/html/2509.16941
• Scale AI, SWE-Bench Pro public leaderboard. https://labs.scale.com/leaderboard/swe_bench_pro_public
• Vals.ai, SWE-bench Verified leaderboard (archived), 2026-09-01. https://www.vals.ai/benchmarks/swebench
• Yu et al., "UTBoost", ACL 2025. https://arxiv.org/html/2506.09289v1
• "SWE-ABS", ICML 2026. https://arxiv.org/abs/2603.00520
• Jonathan Gabor, "SWE-Bench Pro is even worse", 2026-02-24. https://www.lesswrong.com/posts/nAMhbz5sfpcynjPP5/swe-bench-pro-is-even-worse
• LLM Stats, Claude Opus 5.5 launch summary, 2026-09. https://llm-stats.com/blog/research/claude-opus-5-5-launch