OpenAI pulled the plug on SWE-bench Verified earlier this year after finding contamination and design issues. Their replacement recommendation: SWE-Bench Pro. That one just failed its own audit.
In a new writeup, OpenAI’s research team reports that roughly 30% of SWE-Bench Pro’s 731 tasks are broken. The evaluation — designed to test agentic coding on realistic, longer-horizon tasks — has flaws severe enough that OpenAI is now retracting their earlier endorsement.
“Given the issues uncovered in this analysis, we retract our earlier recommendation to adopt SWE-Bench Pro.”
What’s actually broken
The audit combined AI investigator agents and human reviewers (five engineers per flagged task). They identified 200–249 broken tasks depending on method. Four failure patterns dominated:
- Overly strict tests — enforcing specific implementation details not mentioned in the prompt
- Underspecified prompts — missing requirements that hidden tests know about, but the model can’t reasonably infer
- Low-coverage tests — incomplete fixes pass because the tests don’t fully check the feature
- Misleading prompts — pointing models toward the wrong behavior entirely
The structural cause: these benchmarks are built from real GitHub pull requests. Human-to-human PR collaboration doesn’t produce clean, isolated tasks. Tests written to validate a specific contributor’s PR aren’t the same as tests designed to measure model capability.
The interesting meta-layer
OpenAI used Codex-based investigator agents to run this audit — inspecting repo history, executing tests, analysing failure traces at scale.
“Evaluation flaws are easier to detect now than they would have been even a short time ago. As model capabilities improve, we can use those models to inspect prompts, tests, patches, and edge cases with much greater depth and consistency.”
The benchmarks used to measure model progress are now being audited by those same models. That loop is new. It matters.
What to do
- Citing SWE-Bench Pro numbers? Add a ~30% asterisk. Score gains may be noise, not signal.
- Building your own evals? Source tasks from engineers designing tests for models — not repurposing human PRs.
- Following leaderboards? Score inflation on compromised benchmarks has been the story for 18 months. This is a data point, not an outlier.
- Waiting for the next replacement? OpenAI is calling for new benchmarks “built by experienced software developers specifically to test model capabilities.” Nothing announced yet on what they’ll use internally.
The coding eval landscape is in a rough patch. The community’s best tools for measuring agentic coding progress keep failing quality checks — and the replacement cycle is accelerating. Worth watching whether Scale AI (who runs SWE-Bench Pro) responds with a revised dataset.
Source: Separating signal from noise in coding evaluations — OpenAI
✏️ Drafted with KewBot (AI), edited and approved by Drew.
답글 남기기