OpenAI Audits SWE-Bench Pro, Retracts Leading Coding-Eval Recommendation
On July 8, 2026, OpenAI published an audit finding roughly 30% of SWE-Bench Pro tasks broken—hidden requirements, contradictory instructions, overly strict tests, or incomplete grading—and retracted its prior recommendation that the research community treat SWE-Bench Pro as a leading coding evaluation.
TLDR
OpenAI on July 8, 2026 published “Separating signal from noise in coding evaluations,” reporting that a large share of SWE-Bench Pro tasks no longer reliably measure frontier coding ability. Model-based investigator agents plus five independent experienced software engineers supported the analysis. OpenAI estimates ~30% of Pro tasks are broken and retracts its earlier recommendation that labs treat SWE-Bench Pro as a primary coding eval—months after it had steered the field away from contaminated SWE-bench Verified toward Pro.
What the audit found
From OpenAI’s primary post and same-day X thread:
- Headline: SWE-Bench Pro “no longer reliably measures frontier coding capability.”
- Broken-task estimate: ~30% of tasks; automated pipeline flagged 200/27.4% broken; human annotation campaign flagged 249/34.1%.
- Failure modes: Correct solutions fail due to hidden requirements, contradictory instructions, overly strict tests, or incomplete grading criteria.
- Method: Model-based investigator agents at scale, plus independent review by five experienced software engineers.
- Policy change: Retract prior recommendation that the research community use SWE-Bench Pro as a leading coding eval; call for harder, fairer, more trustworthy benchmarks as coding models improve.
Context: In February 2026, OpenAI had already stopped leaning on SWE-bench Verified over contamination and mismeasurement, pushing labs toward Pro. The July 8 audit closes that chapter: the successor benchmark is itself materially flawed under current scrutiny.
Why this story matters
Vendor leaderboards and marketing slides still hang on SWE-Bench-style resolve rates. If ~1 in 3 Pro tasks are broken, score gaps between GPT, Claude, Grok, and Gemini can be noise as much as capability. OpenAI’s public retraction forces the industry to rebuild coding eval infrastructure just as agentic coding models (Grok 4.5, Fable 5, GPT-5.6) are racing on those same charts.
Sources
- OpenAI: “Separating signal from noise in coding evaluations” (openai.com/index/separating-signal-from-noise-coding-evaluations/, July 8, 2026). Primary.
- OpenAI X thread (July 8, 2026) summarizing 30% broken-task finding and retraction.
- Prior context: OpenAI Feb 2026 move off SWE-bench Verified toward Pro.
Featured Image Alt Text
SWE-Bench Pro leaderboard with a cracked audit stamp and 30% broken-tasks callout.
Tags
OpenAI, SWE-Bench Pro, Benchmarks, Coding Agents, Evaluation, Research