The Week After the Eval: A Cost-Aware Routing Harness for New Model Drops

A new model drops roughly every other week now, and my timeline fills up with the same two claims: “it’s cheaper” and “it’s better.” I’ve written before about how I gate new models with a small self-written eval deck instead of trusting the hype. This post is about the week after that eval — the part nobody blogs about, where you actually have to decide which model gets which traffic without lighting your budget on fire.

The short version: a single winner-take-all model choice is usually wrong, and the fix is a tiny routing harness with per-task costs, not a bigger eval.

The problem with “the new model is my default now”

After my eval deck blesses a new release, the temptation is to flip a config variable and move on. Every time I’ve done that, one of three things happened within a month:

1. The cheap model wasn’t cheap on my workload. Sticker price per token means little if the model needs twice the retries, longer prompts, or produces output my downstream parser rejects 12% of the time.
2. The strong model was wasted on easy tasks. Renaming variables and writing docstrings don’t need the frontier model. Paying frontier prices for them is a quiet leak.
3. A silent regression appeared in one task category — usually the one with the fewest eval cases — and I noticed only because a coworker did.

So the real question isn’t “is the new model good?” It’s “for each task type I actually run, which model gives the lowest cost per accepted result?” That’s a routing question, and it deserves its own artifact.

The artifact: a 60-line routing logger

The harness below is deliberately boring. It runs your task through a candidate model, checks the result against whatever acceptance test you already have (a unit test, a JSON schema, a linter, a diff review), and logs one line of JSONL per attempt. After a few days of real traffic, you have evidence instead of vibes.

# router.py — log cost-per-accepted-result per (task_type, model)
import json, time, hashlib, pathlib

LOG = pathlib.Path("routing_log.jsonl")

# Fill these in from each provider's CURRENT pricing page.
# Prices change; do not hardcode numbers you haven't verified this month.
PRICE_PER_1K_TOKENS = {
    "frontier-model": {"in": None, "out": None},   # verify before use
    "budget-model":   {"in": None, "out": None},   # verify before use
}

def acceptance_check(task_type: str, output: str) -> bool:
    """Your real checks go here. Examples:
    - 'sql':      run the query against a scratch DB
    - 'code':     run pytest on the generated file
    - 'json':     validate against the schema
    - 'summary':  length + keyword-presence heuristic (weakest check!)
    """
    raise NotImplementedError

def run_and_log(task_type: str, model: str, prompt: str, call_fn):
    t0 = time.time()
    resp = call_fn(model, prompt)          # your existing API wrapper
    elapsed = time.time() - t0
    accepted = acceptance_check(task_type, resp["text"])
    record = {
        "ts": int(t0),
        "task_type": task_type,
        "model": model,
        "prompt_hash": hashlib.sha256(prompt.encode()).hexdigest()[:12],
        "in_tokens": resp["usage"]["in"],
        "out_tokens": resp["usage"]["out"],
        "latency_s": round(elapsed, 2),
        "accepted": accepted,
    }
    with LOG.open("a") as f:
        f.write(json.dumps(record) + "\n")
    return resp["text"] if accepted else None

Enter fullscreen mode Exit fullscreen mode

Then a weekly rollup:

# rollup.py — cost per ACCEPTED result, not per call
import json, collections

stats = collections.defaultdict(lambda: {"calls": 0, "accepted": 0, "in_t": 0, "out_t": 0})
for line in open("routing_log.jsonl"):
    r = json.loads(line)
    k = (r["task_type"], r["model"])
    s = stats[k]
    s["calls"] += 1
    s["accepted"] += r["accepted"]
    s["in_t"] += r["in_tokens"]
    s["out_t"] += r["out_tokens"]

for (task, model), s in sorted(stats.items()):
    rate = s["accepted"] / s["calls"]
    # cost = (in_t/1000)*price_in + (out_t/1000)*price_out — plug in verified prices
    print(f"{task:12s} {model:18s} accept={rate:.0%} calls={s['calls']}")

Enter fullscreen mode Exit fullscreen mode

The metric that matters is cost per accepted result: raw token cost divided by acceptance rate. A model that’s 5x cheaper per token but passes your checks half as often is only 2.5x cheaper in reality — and that’s before you count the engineering time of babysitting retries.

The decision matrix I actually use

After two or three weeks of logs, I fill in this table per task type:

Task type Frontier accept rate Budget accept rate Verdict Boilerplate codegen / renames ~high ~high Budget model, always SQL with schema constraints high medium Budget, retry once, escalate on failure Novel debugging / root-cause high low Frontier only JSON transforms with schema test ~high ~high Budget — the test does the gating Free-text summaries ??? ??? Neither — my acceptance check is too weak to trust

The last row is the honest one. If you can’t write an acceptance check for a task type, your routing decision for it is superstition. Fix the check first or route conservatively.

Escalation routing, in one sentence

For the middle row of that table, the pattern is: try the budget model first, and if the acceptance check fails, escalate to the stronger model with the failure message included in the prompt. In my logs this catches most of the gap at a fraction of always-frontier cost, because the escalation prompt is unusually informative — “here’s what a weaker attempt got wrong” is great context.

Where free access fits

The expensive part of this workflow isn’t production traffic — it’s the exploration phase, where you’re re-running eval decks and shadow-routing candidates before committing. That’s where I’ve been using MonkeyCode: it offers free access to a set of models plus a free server option, which maps neatly onto the “run the harness against candidates without a billing page open in another tab” phase.

Disclosure: This article was prepared as part of MonkeyCode’s product outreach.

Concretely, my loop when a new release lands is: point call_fn at the candidate via MonkeyCode’s free model access, replay last month’s logged prompts (the prompt_hash column makes this a clean shadow test on your own distribution rather than a generic benchmark), then read the rollup. The free server option covers the always-on piece — the rollup cron and the small routing sidecar — so the whole gate costs nothing to leave running between releases, which matters because model gating is a recurring chore, not a one-time project. If you want to try the same loop, their docs describe how to point an OpenAI-compatible client at it; the harness above doesn’t care which backend call_fn wraps.

Two caveats I’d hold onto: free tiers can change in scope, rate, or availability, so treat them as an evaluation convenience rather than production infrastructure; and confirm which models are actually available there this week before designing around any specific one — the landscape moves fast enough that I now verify model names and pricing on official pages the day I make a routing change, not the day I read about them.

Limitations, and who shouldn’t bother

• Small samples lie. A task type with 15 logged calls has no accept rate, it has a coin flip. I wait for a few hundred calls per cell before acting on the matrix.
• Acceptance checks are the whole game. For tasks where “correct” is subjective (design review, prose), this harness degenerates into measuring which model you like. That’s fine to know, but call it preference, not quality.
• Latency-sensitive paths (interactive autocomplete, user-facing chat) may not tolerate try-then-escalate. Route those directly.
• If you run one model on one task type fifty times a month, the routing infrastructure costs more attention than it saves. The single-winner default is fine for you — just re-run your eval deck when something new drops.
• Don’t route on this week’s headlines. Any claim about a brand-new release — speed, price, “beats X on Y” — deserves a look at the provider’s own docs and a run through your deck before it influences traffic.

The eval deck tells you whether a model can do your work. The routing log tells you whether it should. You need both, and the second one is cheaper to build than the first.