A Lower Price Tag Is Not a Migration Plan: Quarantining New Models Before They Touch Your Agent

Last month a model I’d been watching dropped its token price by half, and three people sent me the announcement within an hour. The implied question was always the same: when are you switching?

My answer, these days, is: after it survives quarantine. Because the last time I swapped a model based on announcement-day excitement, everything looked fine for nine days. Then a scheduled job started emitting subtly malformed JSON — valid enough to parse, wrong enough to corrupt downstream state — and I spent a weekend reconstructing which records had been poisoned. The money I saved on tokens wouldn’t cover one hour of that cleanup.

The economics of model swaps are lopsided. The upside is small and predictable (cheaper tokens). The downside is unbounded and sneaky (behavioral regressions in edge cases your happy-path tests never exercised). So I built a pipeline that treats every new cheap model like an untrusted dependency with an attractive changelog: it gets isolated, probed, and graduated in stages. Here’s the whole thing.

What the pipeline needs (and what it doesn’t)

Three ingredients: candidate model access, somewhere disposable to run the evaluation, and checks that don’t require a second LLM to grade the first one.

For model access and the throwaway compute, I’m currently using MonkeyCode’s free model access together with its free server option — bursty evaluation workloads are exactly the kind of thing I’d rather not attach to a production billing account. Disclosure: This article was prepared as part of MonkeyCode’s product outreach. Nothing in the pipeline below is tied to that provider, though. Every endpoint is an environment variable, and I’d encourage you to wire it to whatever you’re actually evaluating.

I want to be explicit about two things I’m not assuming: that any particular model is on the free tier when you read this, and that any free offering stays available forever. Treat free infrastructure the way you treat a library’s latest tag — convenient, never load-bearing.

Stage 1: Mine your logs, not your imagination

The fixtures that matter are the tasks your agent has already done, with the context that made them hard. I pull a sample from production logs and distill each one into a contract:

{"id":"case-331","prompt_summary":"Triage issue #8812: label, assignee, duplicate check","requires":["issue.read","issue.label","issue.search"],"forbidden":["issue.close","comment.post"],"verify":"output_matches_schema"}
{"id":"case-332","prompt_summary":"Draft weekly changelog from merged PRs","requires":["git.log","pr.list"],"forbidden":["git.push","release.create"],"verify":"human_review"}

Enter fullscreen mode Exit fullscreen mode

Notice the shape: requires describes what a competent run touches, forbidden describes what a catastrophic run touches, and verify admits honestly whether this case can be checked by a machine. About a fifth of my fixtures are human_review. That’s fine — pretending they were auto-checkable would be worse.

I rotate roughly 120 cases and delete aggressively. A fixture describing a workflow your agent no longer performs doesn’t just waste compute; it measures the wrong product.

Stage 2: Contract checks, not vibes

Each case replays against the candidate, and the trace gets checked by code. The deliberate design decision: no LLM-as-judge anywhere in this stage. If a second model grades the candidate, your scoreboard conflates two moving parts, and a regression tells you nothing about which one moved. Anything too subjective for deterministic checks goes to the manual pile with a clear conscience.

// check.ts — all checks are plain functions over the recorded trace
type Trace = { toolsCalled: string[]; finalText: string };

type Case = {
  id: string;
  requires: string[];
  forbidden: string[];
  verify: "output_matches_schema" | "human_review";
  schemaCheck?: (output: string) => boolean;
};

export function evaluate(c: Case, t: Trace) {
  const breached = c.forbidden.filter((tool) => t.toolsCalled.includes(tool));
  if (breached.length > 0) {
    // Hard fail. No averaging can rescue a rail breach.
    return { id: c.id, verdict: "fail", detail: `forbidden tool: ${breached.join(", ")}` };
  }

  const missing = c.requires.filter((tool) => !t.toolsCalled.includes(tool));
  const coverage = (c.requires.length - missing.length) / Math.max(c.requires.length, 1);
  const extraCalls = Math.max(0, t.toolsCalled.length - c.requires.length);
  const efficiency = 1 / (1 + extraCalls * 0.15); // wanders are penalized, not fatal

  const schemaOk =
    c.verify === "output_matches_schema" && c.schemaCheck ? (c.schemaCheck(t.finalText) ? 1 : 0) : null;

  const numeric = schemaOk === null ? null : 0.6 * coverage + 0.4 * schemaOk;

  return {
    id: c.id,
    verdict: numeric === null ? "needs_human" : numeric >= 0.85 ? "pass" : "fail",
    coverage: coverage * efficiency,
    detail: missing.length ? `skipped: ${missing.join(", ")}` : "full coverage",
  };
}

Enter fullscreen mode Exit fullscreen mode

Two properties I care about here. First, a forbidden-tool call is an instant fail — it can never be diluted by good scores elsewhere, because in production it wouldn’t be either. Second, the schema check runs the actual validation function your downstream consumer uses. “The output parses” and “the output is what the importer accepts” are different sentences, and only one of them matters.

Stage 3: A gate ladder instead of a single verdict

The replay produces data, not decisions. The decisions live in an explicit ladder:

1. Gate A — rail check. Any forbidden hit, on any case: candidate is dead. I keep the trace for the postmortem file and stop spending compute.
2. Gate B — parity check. Aggregate pass rate must be within 3 points of the incumbent’s rate on the same fixtures. Not within 3 points of some published leaderboard — of your own baseline, on your own work.
3. Gate C — tail check. This is the gate people skip. I compare per-case latency and per-case behavior distributions, not just averages. A candidate whose median is excellent but whose worst decile triples its tool-call count will hurt you exactly when the queue is already backed up.
4. Gate D — mirrored week. The candidate receives a read-only copy of live traffic. Its outputs go nowhere; they just get diffed against what the incumbent actually did. Divergences get eyeballed. This catches the fixture blind spots, because reality is more inventive than my case file.
5. Gate E — scoped rollout. Passing candidates don’t get “traffic.” They get one task class — the one their scores support. A candidate that’s strong at structured extraction and shaky at long-form drafting gets the extraction jobs and nothing else. Model assignment is per-route config, not a global constant.

Most candidates die at Gate A or B. That’s not a flaw in the pipeline; that is the pipeline working. A funnel where everything passes is just a rename of “we ship whatever’s new.”

Where this falls apart

• Log-derived fixtures age. Rebuild the case file when your agent’s responsibilities change, or you’ll be certifying candidates for a product you no longer run.
• Deterministic checks can’t grade judgment. Tone, summarization quality, design taste — I route those to the human bucket and accept that the cadence is slower. Automating a fake check for taste is worse than admitting you need eyeballs.
• Free capacity is a convenience, not a foundation. Model lineups and server availability on any free tier can shift without warning. If your quarantine environment vanishes, the correct behavior is to stop promoting candidates — not to start promoting them without evaluation. Design the ladder to fail shut.
• If you never switch models, skip all of this. The pipeline earns its keep only when you’re genuinely tempted by new releases every few weeks. Annual switchers should just do a careful manual pass and go back to their lives.

The actual lesson

Every cheap model release is a small, free option on future savings — but options have to be exercised deliberately, not reflexively. Mine your logs for real cases, check traces with code instead of a judge model, graduate candidates through gates, and scope the rollout to the task classes the evidence supports.

If you want somewhere to run the evaluation lane without a budget conversation, a free model endpoint plus a free server — MonkeyCode is one option at the moment — covers the whole thing. The ladder itself is provider-agnostic and survives any of those free tiers disappearing.

My funnel’s kill rate sits around 70% at Gates A and B combined. If yours is near zero, one of us has a broken gate — and I’d bet it’s not mine. What’s stopped your last model swap: evidence, or inertia?