IBM’s Benchpress pins the seeds that build its circuits and passes no seed to the code that compiles them. On one circuit, over 400 seeds per version, the real change between two Qiskit releases is +5.37%. The suite’s own three-run protocol reports it as a ≥10% regression 24.4% of the time. Twenty runs per version — about forty hours of compute — still leaves 3.7%.
A benchmark suite is an instrument. You point it at two versions of a compiler and it tells you whether the new one got worse. Somebody then merges or reverts a pull request on the strength of that answer.
So the question I care about is not “is this compiler slower”. It is how often does the instrument give an answer that disagrees with its own long-run behaviour. I call that finite-sample decision risk: the probability that a verdict computed from k runs per version disagrees with the verdict implied by the mean of the same measurement.
Where the randomness comes from
Qiskit’s transpiler is stochastic. Its routing pass uses randomness, and generate_preset_pass_manager takes a seed_transpiler argument to pin it.
At the pinned revision I measured, Benchpress’s Qiskit gym does not pass it. Circuit construction seeds are fixed — seed=12345, fourteen occurrences. The compilation seed is not. So every gate count the suite reports is one draw from a distribution nobody measured.
One detail matters for honesty here, and I got it wrong in an earlier draft: this is not universal across the suite. The BQSKit gym in the same repository does seed its compiler, with seed=0. seed_transpiler is a Qiskit-specific API the other gyms cannot call. The claim is about the Qiskit gym, and only about it.
What one circuit does
bv_n140 is a Bernstein–Vazirani circuit that Qiskit issue #14402 names directly. Mapped to a heavy-hex lattice, going from Qiskit 1.4.3 to 2.0.0, measured at 400 seeds per version across 21 operating-system processes:
The true change is nowhere near the threshold. The verdict crosses it a quarter of the time anyway.
I did not trust that number when I first got it, because all 200 of the original seeds came from one contiguous block inside one process — so any per-process state, hash randomisation, allocator layout, a cached RNG, was held constant. I drew a disjoint set of 200 seeds spanning 5.9 million to 1.08 billion, ran them across ten fresh processes with differing PYTHONHASHSEED, and got 22.6% [16.4, 30.6] against 26.3% [18.5, 34.2] from the contiguous set. Each point falls inside the other’s interval.
Running it more does not fix it
The obvious response is “then run it more times”. I measured exactly how much that buys, pooling all 400 seeds:
runs per version 1 3 5 8 10 20 false-positive rate 34.58% 24.38% 18.56% 12.94% 10.36% 3.74%Twenty runs per version is on the order of forty hours of compute, at the issue’s own stated “about two hours” per suite run. It still leaves 3.74%. One seed_transpiler argument removes the sampling variance at a single run.
Taking the minimum of three runs instead of the mean — a common instinct — makes it worse, not better. All eight circuits I tested still err, and every rate roughly doubles: bv_n280 goes from 17.3% to 28.4%.
The number in the issue
Issue #14402 reports +46.1% for that circuit. A single three-run comparison of the same circuit, under the same protocol, returns anywhere from −10.5% to +100.0%, with a 95% range of [+9.3%, +57.2%].
+46.1% sits at the 88th percentile of that distribution. The same range extends below the +10% threshold. The reported figure is not wrong — it is one draw, reported as though it were a measurement.
The part I pre-registered
One circuit is an anecdote. So before I looked at any result, I wrote down the selection rule, the analysis, the endpoint and the labels, and committed them. Then I collected the data. The commit ordering is checkable:
a36a34a 2026-09-03 23:51:08 pre-registration
fbc573d 2026-09-03 23:52:38 analysis code
30224a0 2026-09-04 09:13:07 raw data
Enter fullscreen mode Exit fullscreen mode
The rule selected every qasmbench-large circuit whose twelve-seed heavy-hex runtime was under ten seconds — 39 circuits, 200 seeds per version per arm.
The pre-registered endpoint: 12 of 26 eligible circuits have a decision-error rate whose interval excludes zero. That is 46.2%, Wilson 95% [28.8%, 64.5%]. Seven of 26 sit at 5% or worse, four at 10% or worse.
What I withdrew
That 46.2% is the number I trust least in this work, and I want to say why rather than let someone find it.
The 26 circuits are not 26 independent observations. They come from eleven algorithm families, and within a family the results are nearly all-hit or all-miss. A cluster bootstrap over families gives [17.4%, 81.0%] — against the Wilson interval’s [28.8%, 64.5%]. The effective sample size is far below 26, and no interval I can compute is informative about a population of circuits.
So the suite-level rate is withdrawn as a headline. The 12/26 stands as a descriptive count of these 26 circuits and nothing more.
Three other claims went the same way, and all four withdrawals are still in the repository with the evidence that defeated them. One of them was mine: I had reported a correlation of ρ = +0.876 between “compilation is stochastic” and risk, presented as a discovered mechanism. Circuits whose two arms are constant have zero risk by arithmetic, so a correlation computed over a sample containing them is measuring a definition, not a finding. Simulated data containing no compiler at all reproduces the statistic.
What went wrong in my own instrument
The paper’s whole argument is that measurement apparatus should be checked rather than trusted, so it would be poor form not to turn that on myself.
After three rounds of adversarial review and a sixteen-phase red team — all of which attacked the claims — I had an independent audit read the ~4,600 lines of Python that nobody had ever read. It found six defects. None changed a published number. The worst one was this:
My numeric inventory recomputed every reported figure and compared it to a recorded value. But the recorded value was itself a function call evaluated when the registry was built, and the recomputation called the same function on the same file. It compared f() with f(). It could not fail. I proved it by falsifying every value in a summary file — tripling them and adding 40 percentage points — and the checker still reported “41/41 reproduce”.
The values are frozen literals now, twenty-nine of forty-one re-derive from raw measurements rather than from a summary, and re-running that same falsification produces twelve failures. A checker that cannot fail is worse than no checker, because it buys confidence it has not earned.
What this does and does not say
It does not say Qiskit’s transpiler got worse, or that Benchpress is a bad suite. It measures one property of one benchmarking protocol: that an unseeded stochastic compiler, sampled three times, produces regression verdicts that disagree with its own long-run behaviour at rates that are not small.
Limits, stated plainly: one SDK, one version pair, one machine. The +10% threshold is mine — Benchpress defines none and the issue states no formal cut. The estimate of the true change is a plug-in from 200 seeds, not an external criterion. And seed_transpiler is not a universal remedy: it removes false positives on the circuits I tested, but was worse on three of the four circuits showing false negatives.
Everything is in the record
The paper, the complete analysis code, and all 41,790 raw per-seed measurements are archived with a DOI: 10.5281/zenodo.22310060. verify.py re-runs the toolchain pin check, the test suite, the numeric inventory, the replication artifact and a proof of one withdrawn claim, in under a minute.
The full write-up is on my site: The Compiler Got 5% Slower. The Benchmark Called It a 10% Regression a Quarter of the Time.
If you maintain a benchmark that drives accept/reject decisions on a stochastic system, the cheap version of this check is: run the same comparison twenty times without changing anything, and look at the spread of verdicts rather than the spread of values. If the verdict moves, the number of runs is part of your instrument, and it belongs in the write-up.