My Release Gate Passed. The Model It Shipped Answered 'Neutral' To Everything.

I trained a model that scored 4 percentage points better than the one I ship. It’s gone. I
deleted it from the roadmap yesterday.

That’s not the interesting part. The interesting part is how I found out it was broken — because
it passed every check I had, and then shipped a file that answered neutral to every question you
could ask it.

The setup

I maintain semantix, a local NLI judge for
compliance validation. The shipped model, nli-popia-v2, is an 82M-param cross-encoder fine-tuned
on clauses of South Africa’s POPIA. It runs as INT8 ONNX on CPU — ~15–50 ms depending on text length, no API key, no data leaving
the machine. Determinism is the whole point — same input, same score, every run, because “rerun that
validation” is a question a regulator actually asks.

Its model card has said this for months:

82M-param base. A larger base would likely improve in-domain F1. v2 retained the small base
for ONNX deployability (~79 MB quantized).

So: obvious next move. Try a bigger base. deberta-v3-base — 184M params, 2.25× larger. Train it,
measure it, ship it.

The bigger model won

It trained clean on an L4 for about fifteen cents. Against the pinned holdouts:

holdout v2 (shipped) v3 (new) delta v1 clauses (7) 0.7465 0.7850 +3.9pp v2 clauses (3) 0.8621 0.9619 +10.0pp

The hypothesis in the model card was right. Bigger base, better F1, on both holdouts, same eval
script, same pinned data. The release gate passed. I published the repo.

Everything above is true and every number is real. The model I shipped was still broken.

The file that shipped

Someone loaded the actual artifact — onnx/model_quint8_avx2.onnx, the file the library downloads —
and ran it against the canonical failing example from my own demo:

“Our signup flow presents a pre-ticked marketing opt-in checkbox that customers must untick if
they don’t want newsletters.”
vs. “The responsible party is obtaining explicit, informed, voluntary consent…”

A pre-ticked box is the textbook contradiction of freely-given consent. v1 knows this:

contradiction entailment neutral v1 0.9513 0.0092 0.0395 v3 0.0168 0.0216 0.9616

v3 said neutral, hard. My first instinct was the worst possible one: label ordering. I’d shipped
exactly that bug once before — reading probs[2] (neutral) where probs[1] (entailment) belonged —
and it produced a negative correlation in a benchmark I nearly published as a finding. I wrote
that one up too.

So we tested it properly: score the artifact on 24 stratified pairs from its own bundled eval set,
8 per class, under all six possible index→label mappings.

  0:contradiction 1:entailment    2:neutral        macroF1=0.1667  <-- the declared mapping
  0:contradiction 1:neutral       2:entailment     macroF1=0.1667
  0:entailment    1:contradiction 2:neutral        macroF1=0.1667
  0:entailment    1:neutral       2:contradiction  macroF1=0.1667
  0:neutral       1:contradiction 2:entailment     macroF1=0.1667
  0:neutral       1:entailment    2:contradiction  macroF1=0.1667

  pred index counts: {0: 0, 1: 0, 2: 24}

Enter fullscreen mode Exit fullscreen mode

Six identical numbers. That flatness is the diagnosis.

0.1667 is exactly what macro-F1 degenerates to when a model emits one class for everything on a
3-way balanced set: one class scores F1 0.5, the other two score 0, and 0.5/3 = 0.1667. Permuting
the labels of a constant just renames the class it always guesses. The table is flat because there
was no signal to recover.

It wasn’t label ordering. It was a dead model wearing a live model’s config.

Why the gate didn’t catch it

Here’s the uncomfortable bit.

My release gate was real. It enforced macro-F1 deltas and per-clause non-regression, in CI, on every
push. It scored 0.7850. That number was never wrong.

It scored the PyTorch weights.

The pipeline is: train → gate → export to ONNX → quantize to INT8 → upload. The gate sat
before the two steps that produce the file anyone actually downloads. Export and quantization are
transformations, and I had zero checks after them. So the gate could pass — honestly, correctly —
on a model whose shipped artifact was a constant.

I had built a validation you couldn’t reproduce from the thing being validated. Which, given that my
entire product exists because “the LLM said it was fine” isn’t a defensible answer, is a special
kind of embarrassing.

The actual cause

deberta-v3‘s disentangled attention is sensitive to per-tensor INT8 dynamic quantization. The
activation ranges across its attention projections don’t survive a single scale factor per tensor —
the logits collapse and the model saturates to one class.

v1 and v2 are roberta. Roberta quantizes cleanly. I’d never hit this because I’d never shipped a
DeBERTa. The moment I changed the base, a step I had never needed to think about became the step
that broke everything.

The fix for the collapse is one argument:

AutoQuantizationConfig.avx2(is_static=False, per_channel=True)

Enter fullscreen mode Exit fullscreen mode

Per-channel scales instead of per-tensor. That stopped the constant predictor dead.

And then the real answer

With per-channel quantization, v3 works. It predicts real classes. And here’s what it scores:

v3 @ fp32 v3 @ INT8 cost v1 clauses 0.7850 0.5760 −21pp v2 clauses 0.9619 0.5840 −38pp

Quantized v3 (0.576) is far worse than quantized v2 (0.7465), the model it was supposed to
replace. The new artifact gate — the one that scores the shipped file — blocked the upload
automatically. First real outing, and it earned itself.

So the honest summary of the whole experiment:

v3’s advantage exists only at fp32, at 704 MB. Under the INT8 quantization my product requires,
v3 is worse than what I already ship. deberta-v3-base is a bad fit for a quantized-CPU-judge
line. Not a bug. A property.

That old line in the v2 model card — “v2 retained the small base for ONNX deployability” — was a
hunch when I wrote it. Now it’s a measured result. The bigger base really is better, and it’s
unusable under the constraint that actually binds. I’m glad I know that for a reason instead of a
feeling.

What actually changed

Two things, both now on master and shipped in 0.2.3:

1. The gate scores the artifact. It loads the quantized ONNX through onnxruntime, scores it on
the same holdout, and asserts macro-F1 within tolerance of the PyTorch gate.

2. It counts distinct predictions. This is the cheap one:

def onnx_macro_f1(onnx_path, tokenizer, eval_rows, batch_size=16) -> tuple[float, int]:
    """Macro F1 for a quantized ONNX artifact + number of distinct predictions."""

Enter fullscreen mode Exit fullscreen mode

len(set(preds)) > 1. One line. It catches a constant predictor in seconds. Everything above —
the permutation sweep, the config archaeology, the tokenizer forensics — would have been unnecessary
if that assertion had existed from the start.

The thing worth taking

Your release gate probably validates a model. Ask whether it validates the file you upload.

If there’s any transformation between them — export, quantization, conversion, pruning, a format
change — then your gate and your artifact are two different objects, and the gate is measuring the
one nobody downloads. Mine passed with an honest 0.785 while shipping something that couldn’t tell
consent from a contradiction.

A model can pass its own release gate and ship dead. Mine did.

Postscript, in the interest of not overclaiming: v3 also never beat v1 on v1’s own 7-clause
holdout — 0.785 against v1’s 0.813. v1 is a 7-clause specialist; v2 and v3 are 10-clause
generalists, and broader coverage costs in-domain accuracy. The bigger base bought back about 4 of
the ~7 points v2 gave up. It didn’t repay the debt. That’s in the v2 model card too, and it should
travel with any sentence that starts “the newer model is better.”

semantix is MIT, the models are Apache-2.0, and every holdout mentioned here is pinned and
downloadable, so you can disagree with me from the data:
github.com/labrat-akhona/semantix-ai