Quality went from 0.81 to 0.78. Someone had edited a prompt that week. Obvious culprit, obvious investigation.
Nobody had measured that re-running the same prompt scores 0.77-0.84 across seeds.
0.78 was never a regression. It was Tuesday.
The number was real. The comparison was not, because nobody measured the instrument before trusting it.
So now I do this in order, and the order is the whole point:
Calibrate the judge. Can it separate a known-good answer from a known-bad one? A judge returning 3/4 for everything gives you a rock-steady dashboard that would stay green if the agent returned Lorem Ipsum.
Measure the noise floor. Run each case across several seeds. That spread is the resolution of your instrument.
Then gate. A delta smaller than the noise floor is not a small regression. It is no information at all.
A gate that fires on noise gets marked flaky and gets continue-on-error added within a month. Then you have no gate.
How many of your eval numbers have a measured error bar?
Calibrate the judge, measure the noise floor, then gate in that order.
Github Repo: https://lnkd.in/dbfwtsM6