The Quota Said 100% Used. Actual Demand Was 13%.

Originally published on hexisteme notes.

On July 31st, the accelerator quota API on one of my Kaggle accounts came back at 108,511.735 seconds used against 108,000 allowed — 100.47%. The TPU line read 0 of 72,000. Four competitions were active on that account, and all four had a submission go out within the same 25-minute window that morning. The obvious read: one shared pool, several competitors, and whichever session submits first eats the week. So I built a priority table — rank 1, rank 2, rank 3, do-not-run — with a rationale and a stated falsifier next to each row.

The table had ranks. It did not have a single number in it. That was the first warning sign, and I didn’t catch it until afterward.

The table with no numbers

A priority ranking answers “who goes first.” It doesn’t answer “what does a turn cost.” I had ordered four competitions by importance without ever asking what a session from each of them actually spent against the pool I was rationing. That’s a strange thing to skip, in hindsight — you can’t allocate a budget you haven’t priced. The absence of any unit-price number in a document whose entire job was allocation was itself the tell that nobody had measured anything yet. I just didn’t read it that way at the time; I read a 100.47% line and reached straight for a rationing policy.

Three measurements

Before shipping the table, I went back and actually measured what was being charged to the pool. Three things fell apart at once.

Rescoring doesn’t touch personal quota. Code competitions on this platform re-run your submitted notebook on the host side to score it. I had been assuming that re-run was billed against my personal quota. One of the four competitions had already checked: personal quota read the identical 108,511.735s before and after a full scoring cycle, bit for bit. A second competition confirmed it independently. So on my account, in late July 2026, the only thing that spends personal quota is a new commit — and I want to state that with exactly the scope I measured it at, not one inch wider. It is a reading from one account in one window, not a claim about how the platform bills everyone. The whole point of this essay is that I hadn’t checked before; overgeneralizing what I did check would be the same mistake wearing a lab coat. Either way, the allocation table’s entire unit — “how many submissions can we afford” — was pricing something that, on this account, cost nothing.

A “GPU” label bills wall clock whether or not a GPU is used. A kernel-log audit on one competition turned up 1,043 seconds of runtime with zero occurrences of the string device: GPU anywhere in the log. The training path was hardcoded to CPU — torch.device('cpu'), map_location='cpu' — and the gradient-boosted model defaulted to CPU too. But the kernel’s metadata still said machine_shape: "Gpu", and the quota meter charged the full 1,043 seconds against the accelerator pool anyway, for a run that never touched an accelerator. Flip that one field to enable_gpu:false and, on this account, CPU-only re-runs are unmetered — no quota concept applies to them at all.

The billing multiplier isn’t 1x. A separate, direct measurement: 1.5 wall-clock hours of work consumed 2.92 hours of quota — a 1.94x multiplier, scaled to the accelerator count on that machine shape. So the week’s usable budget wasn’t “108,000 seconds” in any intuitive sense. Divided by the multiplier, it was closer to 30 quota-hours, which is roughly 15.5 hours of actual wall-clock work. A smaller number than the raw quota figure suggests, and one I’d never converted before.

Once I had those three, I could finally build an actual unit-price table instead of a priority table:

Operation Quota cost Smoke commit ≈0.4h Local probe ≈2.4h Local eval (120-task) ≈10.2h Submission rescoring 0 (competition compute, not personal quota) CPU-only kernel 0 (unmetered)

Two of the five rows in the thing I was supposedly rationing cost nothing at all.

Correction, 2026-08-08. A reader pointed out that this table carries no falsifier and no measured-on date, unlike the priority table it replaced, and that in three months it would be the same confident unchecked artifact with numbers in it. I went to add those two columns. The re-run broke three of the five rows.

Smoke commit is not ≈0.4h; it is 1.45h (5,211s). The measurement refuting it sits in a document timestamped the same morning I wrote the source notes for this post. The post went out six days later, and nothing compared the two. Local probe was never measured at all: the source called it an expectation, and the qualifier didn’t survive the trip into a table of measured unit prices. Local eval-120 later came in around 5.2h against the ≈10.2h above, still unreconciled — the two may not even be measuring the same operation, which is its own problem. Rescoring = 0 holds as a finding, but the citation behind it was misattributed: the file I pointed at doesn’t contain the number, and the exact before-and-after comparison described earlier in this post isn’t in any file I can find. Most likely it was a live API call inside a session and only the rounded value was ever written down — but I can’t confirm that either. CPU-only = 0 holds, and is the only row that was measured the way this table implies.

The multiplier above is also not the constant I stated. My own logs already had it moving from 0.485 to 0.869 per GPU-hour with the machine shape unchanged — a 1.79x swing — because the dominant variable is occupancy, and short runs are dominated by startup that pins every accelerator for the duration.

An essay arguing that you should price the operations before you ration them shipped a price table that had never been re-checked against its own sources. The full accounting is in the comment thread below.

Recomputed: real demand was 13% of the pool

With unit prices in hand, I profiled what each competition on the account was actually running. Four were active; a fifth was scaffolded but hadn’t started work yet.

Competition Stack Actual demand A program-synthesis competition (LLM + test-time training) PyTorch + Unsloth (Triton/CUDA) + custom CUDA kernels 11,520s An agent-security competition Pure-Python search. Zero ML-framework imports Commit time only (~0) A tabular geoscience competition (sequential estimation) numpy/pandas + a CPU-only auxiliary model 0 A game-agent competition ctypes C tree search + LightGBM (CPU). Zero torch/tf/jax imports 0 Not yet started Scaffold only 0

Sum it: roughly 11,520 of 108,000 seconds, about 10.7%. Padding every estimate generously, the ceiling is 13%. One competition, out of five on the account, was doing anything that touched an accelerator at all.

So the 100.47% reading wasn’t four competitions fighting over a shared pool. It was one competition’s mislabeled kernel burning wall clock against an 87%-empty pool. Had I shipped the priority table as written, it would have spent real effort rationing a pool that was almost entirely free, while the actual cause — a checkbox that said “Gpu” on a run that never used one — sat there unfixed, because the table never asked what anything cost.

Worth being precise about what went wrong, because it isn’t the usual false alarm. The 100.47% was not a misreading, a broken dashboard, or a contaminated log. Those seconds were charged, and the meter reported them correctly. The number was true; the denominator of judgment behind it was missing. I compared a real usage figure against a total demand nobody had ever priced, and filled the gap with a story about contention. A metric can be perfectly accurate and still support a conclusion it does not license.

The same error, three times, three different answers

This wasn’t the first time in this project that the resource carrying the name of the bottleneck turned out not to be the actual bottleneck. It’s the third, and each time the correct answer was different:

When Believed scarce Actually scarce How it was found Early on Submission slots (94 of them) Quota — a submission is 25x cheaper in quota, and its result 14x more precise in sigma, than a local eval run Measuring unit price directly July 31 Quota Commits — rescoring leaves personal quota bit-identical before and after Comparing quota before/after a scoring cycle A separate competition GPU time Per-decision CPU budget — 0.3–30s used against a 600s ceiling, 1,340x of it sitting unused Instrumenting actual per-decision usage

Correction, 2026-08-08. Both ratios in the first row were already refuted before this post went out: 25x cheaper is 7.0x, and 14x more precise in sigma is about 2.3x. The direction survives — a submission still wins on both axes — but both magnitudes were badly overstated. And 7.0x is itself computed against the ≈10.2h figure corrected above, so if that settles nearer 5.2h it falls again, to roughly 3.6x. I’m not citing it as final either.

Three different names for “the thing we’re rationing.” Three different real answers underneath. That third row stings the most: while I was carefully protecting GPU time, that same competition was sitting on 99.93% of an already-allocated CPU decision budget, unused. The scarcity I was managing and the slack I was ignoring were right next to each other.

The pattern across all three rows is identical: treat the name of a resource as the bottleneck, and never price the operations charged against it. It’s a related failure to one I’ve written about before — a quality gate whose own measurement noise turned out to exceed the thing it was guarding — but it’s a different mistake: that one measured the wrong way, this one never measured at all before deciding how to ration. And because the answer changed each time, “what’s scarce this time” has to be asked fresh at every phase of a project. Carrying forward last cycle’s answer as if it were a constant is exactly how the next misdiagnosis gets seeded.

The subagent made the same mistake one level up

I parallelized the profiling work across five subagents, one per competition. One of them reported the quota required as 140,000 seconds — a number larger than the entire weekly pool, on its own. The cause: it had double-counted submission rescoring as personal-quota spend, the same wrong assumption that started this whole investigation. The real figure was 11,520 seconds. Its report was off by 12x.

The part worth sitting with is where the correct number was already written down: in that same competition’s own report, which the subagent cited as evidence in its output. It read the file. It quoted from the file. It just never applied what the file said to its own arithmetic. This isn’t a story about a model being unreliable in some general sense — it’s the same failure I’d just made myself, one level up: citing a source doesn’t guarantee the citation gets used, and a measurement you delegate needs to be checked against its own primary source before it goes into a summary, not trusted because it came with a citation attached.

The part that transfers

None of this is really about Kaggle GPU quota. It’s about any metered resource you’ve decided is scarce before checking what’s actually charged against it — a cloud bill, CI minutes, an API rate limit, database connections, seat licenses. The rules that came out of this:

1. Before you believe “X is scarce,” measure the unit price of the operations you’ve attributed to X. The resource’s name doesn’t set the bottleneck; its measured price does.
2. Don’t build the allocation policy before you’ve measured the price. A rationing table built on an unmeasured cost doesn’t fix the underlying bug — it makes the bug permanent, because a policy that exists reads as a problem that’s already being managed. Concretely, on a cloud bill: before you write the rule that caps how many staging environments each team may run, price one staging deploy, one CI job, and one flaky-test rerun against the invoice line that scared you. If it turns out the line is dominated by an idle managed cluster nobody scheduled work onto, the cap you were about to write would have taxed the wrong people and left the cluster running.
3. Check whether a path that looks free is actually being billed. And check the reverse just as hard: a path that looks expensive might already be free.
4. Re-ask “what’s scarce this time” at every phase, on every project. The answer moves. Treating last time’s answer as a constant is how the next misdiagnosis gets seeded.
5. A measurement you delegate goes into a summary only after you’ve checked it against its own source — a citation is not the same claim as a correct conclusion drawn from it.

Where this breaks

An essay telling people to measure before they ration owes its own falsifiers:

• If any of the concurrent competitions turns on real GPU training — an offline model, say — the “13% of the pool” figure stops holding. The unit-price table stays valid; only the allocation recalculation changes.
• The billing behavior itself — that rescoring left personal quota untouched — is platform policy, measured on one account in one window. Platform policy can change without notice. That bit-identical check needs to be re-run periodically, not treated as a fact fixed at measurement time — otherwise this essay becomes the source of the next misdiagnosis instead of the fix for the last one.
• The 1.94x wall-clock multiplier is a function of the accelerator count on that specific machine shape. It doesn’t transfer to a different shape without re-measuring. (Correction, 2026-08-08: this falsifier was too narrow. A shape change is not the only thing that invalidates the multiplier — holding the shape fixed and changing run length alone swings it 1.79x, because occupancy is the dominant variable. See the correction above.)

More notes at hexisteme.github.io/notes.