We measured whether AI-written code is messier than human-written code, across 500 real bugs where a human fix already exists.
Thirteen of the fourteen models we tested raised a function’s complexity or added dead code more often than the person who fixed the same bug. One tied. None did better.
We measured with pyscn, our own Python analyzer. It reports cyclomatic complexity, dead code, duplicate code, class coupling and cohesion, and module dependencies. This used two of them, complexity and dead code, for reasons in the next section. Every result set is committed, so the numbers can be checked.
The bugs and the human fixes come from SWE-bench: 500 closed issues from open-source Python projects, each keeping the repository as it was before the fix along with the commit that closed it. The model patches are public leaderboard submissions, which we downloaded. No model was run for this.
For every bug and every patch we measured the touched files, applied the patch, measured again, and took the difference. Then we did the same with the human’s commit for that bug. “+3 complexity” means nothing without something to compare it against.
The measurement
In full, for each (issue, patch) pair:
git checkout base_commit
analyze the files the patch touches -> before
git apply the patch
analyze the same files -> after
delta = after - before
Enter fullscreen mode Exit fullscreen mode
Only complexity and dead code count. Both come from the control-flow graph of one function in one file, so they stay exact when you analyze part of a repository. Coupling, cohesion and clone detection are not exact on a subset: they undercount whatever the missing files would have contributed, so the score would end up depending on which files the patch happened to touch.
A patch regresses if it raised some function’s complexity, added a function pyscn rates high-risk, or added dead code. One extra branch in one function trips it. The human patch is measured identically and trips it on 24% of instances.
Two rules keep it honest.
Only files present on both sides get measured. Agents leave scratch files behind, and a file with no before-state inflates every delta it lands in. Deletion is the same bug with the sign flipped: remove a file and its entire complexity books as an improvement.
Only patches that passed their tests count. That isn’t a filter chosen to flatter the result: without it the case against the models is stronger, not weaker, because the sample roughly doubles and eleven models clear Bonferroni instead of six. It’s that scoring a patch which didn’t fix anything answers a question nobody asked, and broken code measures cleaner than working code, so the ungated numbers aren’t trustworthy in either direction.
The result
model n model worse human worse difference p devstral-small 211 79 44 +16.6pt <0.001 gpt-5 303 115 76 +12.9pt <0.001 glm-4.6 331 118 79 +11.8pt <0.001 deepseek-v3 209 69 46 +11.0pt 0.001 Nemotron-CORTEXA 339 113 79 +10.0pt <0.001 kimi-k2 301 106 76 +10.0pt <0.001 o4-mini 220 68 51 +7.7pt 0.009 o3 288 87 66 +7.3pt 0.008 claude-4-opus 325 98 77 +6.5pt 0.011 qwen3-coder-480b 264 79 62 +6.4pt 0.033 claude-sonnet-4 311 93 75 +5.8pt 0.027 claude-opus-4.5 334 106 89 +5.1pt 0.019 gemini-2.5-pro 260 67 59 +3.1pt 0.312 qwen2.5-coder-32b 42 4 4 +0.0pt 1.000Thirteen worse, one tied, none better. A model no different from the human would land on either side by coin flip, so thirteen in a row is p = 0.00024.
Twelve of the fourteen clear p < 0.05 individually. Six survive Bonferroni correction across fourteen comparisons. On django alone it was three and two; most of what the other eleven repositories bought was statistical power, not a different answer.
Isolate the instances where the two sides disagreed, one regressing and the other not. There are 741. The model is the one that regressed in 530.
The ordering isn’t capability. The largest gap belongs to devstral-small, near the bottom of the leaderboard. Second is gpt-5, near the top. Both run on OpenHands. Among the seven models that ran through an identical framework there is no trend at all: o3 and o4-mini sit above claude-sonnet-4 and qwen3-coder-480b.
Scratch files
While an agent works on a bug it writes throwaway scripts: one to reproduce the bug, one to check the fix took. reproduce_bug.py. check_url_parts.py. final_verification.py. The final patch is a git diff, so any of those still sitting in the working tree ship with the fix. The human commits only the fix.
We counted, for each solved bug, how many files the model’s patch creates that the human’s does not. The human averages 0.00. Models go as high as 5.00.
The obvious objection is that this belongs to the agent framework wrapped around the model rather than to the model itself. It doesn’t. Seven of these submissions ran through an identical framework, prompt and retry policy, with the model as the only thing that differed:
model (identical framework) scratch files per solved bug claude-sonnet-4 3.53 claude-4-opus 3.50 qwen3-coder-480b 3.28 qwen2.5-coder-32b 0.67 gemini-2.5-pro 0.65 o3 0.23 o4-mini 0.19Eighteen-fold, with everything but the model held fixed. The same split shows up under a different framework: gpt-5 leaves 0.07 and claude-opus-4.5 leaves 0.00, while kimi-k2 leaves 3.84.
SWE-bench scores whether the tests pass. Three spare files per bug fail no test, so none of this costs a submission anything on the leaderboard. It still lands in the repository.
Limits
Python only. Every SWE-bench Verified instance is Python, all twelve repositories are Python, and pyscn is a Python analyzer. Nothing here says anything about how these models write Go, TypeScript or Rust. Whether the finding survives a language with a different idiom for control flow is an open question, and the reason SWE-bench Multilingual is the obvious next target.
The submissions are old. These are leaderboard entries, scored when they landed. The newest, claude-opus-4.5, is from November 2025; gpt-5 and deepseek-v3 are from August 2025, and Nemotron-CORTEXA from May. Eight to fourteen months, in a field that does not stand still. No model released this year is in here, and nothing in this data forecasts one. Rerunning it needs either new leaderboard submissions or running the models ourselves, which is a different and much more expensive project.
The human side isn’t one person and isn’t one-shot. Each patch is whatever landed on that issue, sometimes from a core maintainer and often from an outside contributor. It landed through a pull request, which means review, revision and CI before anyone merged it. The model patch is a single attempt with none of that. Comparing them says something about what ends up in a repository, not about what a person types on the first try.
“Worse” is narrow: cyclomatic complexity, high-risk functions, dead code. Not naming, not whether the fix sits in the right layer.
The 530-to-211 count is descriptive. Every model is scored against the same human baseline over overlapping instances, so those pairs aren’t independent. The sign test across models, and the per-model McNemar tests, are the claims.
Difficulty isn’t controlled. A model that solves more instances is being scored on harder ones, and harder issues take bigger patches. Which is why every comparison here is paired against the human on the same instances rather than compared across models.
One more, and it’s why these aren’t the numbers we published first. pyscn’s --min-complexity defaults to 5, and it filters the reported function list, not the display. At the default, query.py reports 29 functions out of 105. Worse, a function simplified from 6 to 4 drops out of the report and reads as a deleted function, while one that grows from 3 to 8 appears from nowhere with no before value to compare against. Filed as pyscn#696. If you ever subtract two static-analysis reports, check what the analyzer left out of them first.
Running it yourself
The benchmark lives in polyscan/benchmarks/patch-quality. No model APIs are called. Every patch is an already-published leaderboard submission, so a full run costs nothing but disk and time. Zero dependencies, stdlib only.
All twelve result sets are committed, so the tables above come back without running anything:
python3 report.py results/*.jsonl.gz --resolved-only
Enter fullscreen mode Exit fullscreen mode
analyzer.py exposes a single run(). Pointing this at jscan and SWE-bench Multilingual is a matter of implementing it; nothing in the benchmark’s own code is Python-specific.
I went in expecting better models to write tidier patches. They don’t. The two largest gaps belong to models at opposite ends of the leaderboard, and the seven-model controlled group shows no trend at all. What’s there instead is flatter and more uniform: on the issues they solve, measured against whoever solved the same issue first, every one of them leaves a little more behind.
Issues and ideas: github.com/ludo-technologies/polyscan.