Sub-Agent Metrics Are Not Comparable to Main-Thread Metrics

Originally published on hexisteme notes.

I run a small fleet of coding agents on one machine. Every thread ends up in a log, and a measurement pipeline reads those logs into a database, attributing each turn to the model that produced it. After a few thousand threads I had the table people keep asking for: seven model-versions, five behavioural metrics, real workload rather than a benchmark.

Then I printed one cross-tab I had been skipping, and most of that table stopped meaning what I thought it meant.

The cross-tab was role × model. In this fleet a model runs in one of two roles. It is either the long interactive main thread I drive by hand, or a short one-shot sub-agent that a main thread spawns, runs once, and discards. Same model. Same weights. Two completely different jobs.

Role turned out to move the numbers by up to 135x, and the role mix is wildly different for each model. Those two facts together are enough to make a pooled comparison manufacture a large gap that exists in neither stratum.

A note on labels: model names are replaced with letters on purpose. The point of this note is that these numbers are not a model ranking, and printing the names invites exactly that misreading. Every figure is real, from one snapshot of one fleet.

The role gap is an order-of-magnitude thing

Median output tokens per thread, same model, split by role:

model main n main median sub-agent n sub-agent median main ÷ sub A 200 478,238 2,761 6,212 77x C 88 427,838 156 8,784 49x D 11 185,534 122 1,371 135x G 21 128,415 18 11,358 11x B 20 16,432 1,338 14,157 1.2x F 15 4,194 6 51,562 0.08x

The behavioural metrics are worse than lopsided — in one stratum they are flat:

metric main-thread medians sub-agent medians same-file re-edit rate 0.40 / 0.44 / 0.50 / 0.53 (four models) exactly 0 for six of seven error-recovery sequences 1 / 2 / 2 / 2 (four models) exactly 0 for six of seven validation runs 0 for six of seven 0 for all seven

The “(four models)” qualifier is load-bearing, so let me not hide behind it: the other three model-versions sit at a median of 0 in main too — two genuinely, one because its main cell holds 2 rows and is marked not comparable. Main is where between-model signal can live, not where it always does.

Still, the pattern is not a subtle covariate. A main thread iterates: read, edit, re-edit the same file, hit a failure, recover, run a check. A sub-agent is fire-and-forget — it does its one job and exits, so it rarely touches the same file twice and rarely has a failure to recover from. The metric is structurally near-zero there.

Which means: for re-edit rate and recovery count the sub-agent stratum carries no between-model signal at all — the median is a constant. All the signal lives in the main-thread stratum, 7% of my rows.

The mix is different for every model

Here is the cross-tab I should have printed on day one:

model main rows sub-agent rows main share E 2 132 1.5% B 20 1,338 1.5% A 200 2,761 6.8% D 11 122 8.3% C 88 156 36.1% G 21 18 53.8% F 15 6 71.4%

Across the whole attributed corpus, 4,533 of 4,890 rows — 92.7% — are sub-agent rows. So a pooled number is mostly a description of sub-agents. But how mostly ranges from 1.5% main to 71.4% main, a 48-fold spread in composition.

The reason is not random sampling — it is the delegation policy. My orchestration rules send mechanical fan-out work to cheaper tiers, so those models accumulate sub-agent rows by the thousand; models I drive by hand accumulate main rows. The edge counts show it directly: model A spawned 1,788 sub-agents that were also A and 730 that were B; model C spawned 393 that were B. Role is assigned by the same policy that assigns the model: the confounder is baked into the architecture, not introduced by chance.

Anything with an orchestration layer has this shape — retry tiers, canary vs steady-state traffic, batch vs interactive queues, free vs paid users. The router picks both which variant handles a request and what kind of request it is.

The pooled number can reverse

Pool A and C across roles, weighted by row counts — exactly what GROUP BY model gives you:

A:  (200 × 684,639  +  2,761 × 13,691) / 2,961  =  59,010
C:  ( 88 × 565,678  +    156 × 13,913) /   244  = 212,910

Enter fullscreen mode Exit fullscreen mode

Pooled, C burns 3.6x the output tokens of A — the headline you would ship.

Now look inside each stratum:

stratum A mean C mean C ÷ A main 684,639 565,678 0.83x sub-agent 13,691 13,913 1.02x

Within each stratum the two models are close. C is lower than A on main threads (0.83x) and 1.6% higher on sub-agent runs — a near-tie that happens to lean the same way as the pooled number, just nowhere near its size. Neither ratio comes close to 3.6x. The 3.6x is manufactured by the weights: main threads carry 40–50x the mean output of a sub-agent run, and 36.1% of C’s rows are main threads against A’s 6.8%, so C’s average is dragged toward the expensive stratum and A’s is not.

Worth being precise about the name. This is not textbook Simpson’s paradox, which needs every stratum to point one way while the pooled number points the other; here one stratum reverses and the other is a near-tie. Arguably that is the more dangerous shape — there is no meaningful within-stratum effect in either direction, and pooling still produced a 3.6x headline out of nothing. Call it an amalgamation effect: a difference in composition, amplified into an apparent difference in the metric.

The reason to walk it out is that it never announces itself. Nobody sets out to pool — pooling is what the obvious query does. The number arrives looking like a model comparison and is in fact a weighted average with different weights per model. What you measured was your dispatcher.

Stratifying is step one, not the answer

Splitting by role fixes the mix problem. It opens two more.

The metric may not mean the same thing in each stratum. My completion proxy — a heuristic that reads whether a thread finished from the shape of its last logged line — collapsed on sub-agent threads, where one model had 92 of 99 threads ending on a tool-result line purely as a harness logging convention. I unpacked that particular failure in an earlier note; what matters here is what it implies for stratification.

It implies stratification is not a repair. A metric whose value is set by logging convention in one stratum is not a noisier measurement there — it is a different measurement, and averaging it separately does not make it comparable. So the question after “did I stratify?” is “does this metric measure the same construct in every stratum?” Construct validity is per-stratum, not per-metric. Where the answer is no, drop that stratum for that metric and say so in the output.

A quieter version of the same problem: the unit can change between metrics. My behavioural metrics are per-turn attribution rows; the completion proxy is thread-level, with a censoring rule that treats threads cut right after a user turn as missing rather than failed. Model A has 200 main attribution rows and 154 usable main threads for the proxy — two “n”s in the same report, meaning different things. Label the unit next to every count, or someone will divide one by the other.

The role gap is not a constant you can adjust away. It is 77x for A, 135x for D, 11x for G, 1.2x for B — and for two thin cells the sign flips outright. If role were a fixed multiplier you could divide it out and carry on. It is not; it interacts with the model. Partly that is because a stratum holding 1.5% of a model’s rows is not a random 1.5% — whatever rare circumstance put that model in that role is also selecting the kind of work it did there. Stratification buys you a list of licensed comparisons. It does not buy you a correction factor.

A [0, 0] confidence interval is not precision

Validation-run counts, main threads only. Every model’s median is 0. The means are not: A 4.11, C 2.38, G 1.67, B 0.60. I ran percentile bootstrap on all six model pairs. Every one came back with a median difference of 0 and a 95% interval of [0, 0].

A ~7x spread in means, reported as exactly zero with zero width. Both are arithmetically correct: the median really is 0 in each cell and in nearly every resample, so the bootstrap distribution is a spike and the percentile interval collapses onto it. That is tie degeneracy, not precision — and read as confidence it launders a tie into a finding, in the most persuasive possible format.

The part I had not anticipated is how hard this is to guard against. My pipeline already skipped a pair when both cells had median 0 and an IQR of 0–0. Necessary and insufficient: A’s IQR here is 0–4 and B’s is 0–0, so the pair sails through the filter and degenerates anyway. The right condition is not “both inputs look constant” but “the statistic is constant across resamples,” which you can only check on the resample distribution itself. Report the share of tied values next to any median-based interval, and refuse to publish a zero-width interval you cannot explain.

What I run before any fleet comparison now

1. Print the stratum × treatment cross-tab first, before a single metric. If the composition differs across treatments, every pooled number is a mix effect until proven otherwise.
2. Set the comparability threshold in advance. Mine is n ≥ 5 to appear at all, n ≥ 20 on both sides to earn an interval; thinner cells print as NOT_COMPARABLE rather than as a small number.
3. Pre-register cross-stratum comparison as forbidden — a written prohibition in the spec, enforced by the pair filter, not a caution someone relitigates at 1am.
4. Ask per stratum whether the metric measures the same construct. If a stratum’s value is set by logging convention, drop that stratum for that metric and record why.
5. Never read a degenerate interval as a result. Zero width on tied data is coverage failure.
6. When pooled and stratified disagree, publish both and name the mix that separates them. The disagreement is the finding.

What this still cannot tell you

Routing was never randomized, so none of this is causal — the model each thread got was chosen by policy, entangled with task difficulty, project and week. Everything here is association under a fixed dispatch policy, and the role finding does not rescue it.

Role is also not the only stratum. Split model A’s main threads by project and the re-edit median runs 0.34 / 0.45 / 0.53 / 0.70 across four projects — a 2x spread inside a single model-and-role cell, while the same model’s sub-agent value is 0 in every project. And the re-edit metric cannot tell healthy iteration from thrash; it counts both.

Two more limits. The sessions in which I built and audited this pipeline are logged like any other work, so the observer stands inside the frame; the next iteration gets an explicit exclusion stratum. And this is one snapshot, one operator, one harness — the constants will not transfer. The procedure does: cross-tab first, threshold in advance, construct validity per stratum, no faith in narrow intervals over tied data.

The single sentence I would keep: before you compare treatments, check whether they are running in the same role, in the same proportion. In an agent fleet the answer is almost always no, and nothing downstream survives that going unasked.

More notes at hexisteme.github.io/notes.