Cheap Model First, Strong Model on Failure: Building an Auditable Two-Tier LLM Pipeline

There’s a pattern I see every release cycle: a new budget-friendly model ships, the discourse explodes with hot takes, and within 48 hours half my feed has declared it a drop-in replacement for everything. The claim might even be true. But here’s what nobody posting those takes can tell you: whether it’s true for your specific workload. And most of the time, that’s the only question that matters.

A healthier mental model: stop treating model selection as a one-time shopping decision and start treating it as a runtime policy. Route work to the inexpensive option by default, check the output with something that isn’t a model, and only pay for the heavyweight option when the check fails. Below is a working implementation of that policy, plus the measurement discipline that turns it from a hunch into an auditable system.

Two lanes, not two opinions

My configuration is deliberately boring:

• Lane A (try first): the current low-cost or free option. I’m intentionally not naming a specific checkpoint in this article — model names decay faster than the code below does, and you should be re-benchmarking against whatever shipped this month anyway.
• Lane B (fallback): the pricier, more capable model, invoked only when Lane A’s output fails an objective check.

On the cost side, one note: I iterate on this pipeline using MonkeyCode, which at the time of writing provides free model access along with a free server option, so experimentation doesn’t rack up a bill. Disclosure: This article was prepared as part of MonkeyCode’s product outreach. The pipeline itself is provider-agnostic — anything speaking the OpenAI-compatible chat API drops in, so treat endpoints as configuration, not commitment.

The core rule: escalation needs a non-model referee

The single most important design decision: never let a model decide whether its own output (or a peer’s) was good enough. LLMs report confidence regardless of correctness. Instead, escalate only when an external, deterministic check fails — a test suite, a parser, a schema validator, a diff against expected output.

Here’s a compact implementation. Standard library plus requests, fully rerunnable, and every routing decision gets written to a JSONL audit log:

# lane_router.py
import hashlib, json, subprocess, time
from dataclasses import dataclass, asdict

import requests

@dataclass
class RouteRecord:
    job_id: str
    lane: str            # "A" or "B"
    fell_back: bool
    check_ok: bool
    seconds: float
    prompt_fingerprint: str

def chat(endpoint: str, model: str, prompt: str) -> str:
    resp = requests.post(
        f"{endpoint}/v1/chat/completions",
        json={
            "model": model,
            "messages": [{"role": "user", "content": prompt}],
            "temperature": 0.2,
        },
        timeout=120,
    )
    resp.raise_for_status()
    return resp.json()["choices"][0]["message"]["content"]

def objective_check(job: dict, candidate: str) -> bool:
    """Deterministic validation only. No LLM-as-judge allowed here."""
    mode = job["check"]
    if mode == "pytest":
        path = job["write_to"]
        with open(path, "w") as fh:
            fh.write(strip_fences(candidate))
        run = subprocess.run(job["command"], shell=True,
                             capture_output=True, timeout=90)
        return run.returncode == 0
    if mode == "valid_json":
        try:
            parsed = json.loads(strip_fences(candidate))
        except json.JSONDecodeError:
            return False
        required = job.get("required_keys", [])
        return all(k in parsed for k in required)
    if mode == "regex":
        import re
        return re.fullmatch(job["pattern"], candidate.strip()) is not None
    raise ValueError(f"unsupported check: {mode}")

def strip_fences(text: str) -> str:
    """Pull code out of a markdown fence if present; else return as-is."""
    if "```

" not in text:
        return text
    block = text.split("

```")[1]
    lines = block.splitlines()
    if lines and lines[0].strip().isalpha():  # language tag line
        lines = lines[1:]
    return "\n".join(lines)

def route(job: dict, lanes: list[dict], log_path: str = "routes.jsonl") -> RouteRecord:
    fp = hashlib.sha256(job["prompt"].encode()).hexdigest()[:12]
    fell_back = False
    for idx, lane in enumerate(lanes):
        start = time.time()
        candidate = chat(lane["endpoint"], lane["model"], job["prompt"])
        elapsed = round(time.time() - start, 2)
        passed = objective_check(job, candidate)
        if passed or idx == len(lanes) - 1:
            record = RouteRecord(
                job_id=job["id"],
                lane=lane["label"],
                fell_back=fell_back,
                check_ok=passed,
                seconds=elapsed,
                prompt_fingerprint=fp,
            )
            with open(log_path, "a") as fh:
                fh.write(json.dumps(asdict(record)) + "\n")
            return record
        fell_back = True

Enter fullscreen mode Exit fullscreen mode

Wiring it up:

lanes = [
    {"label": "A", "endpoint": "https://lane-a-endpoint", "model": "current-budget-model"},
    {"label": "B", "endpoint": "https://lane-b-endpoint", "model": "premium-model"},
]

job = {
    "id": "csv-to-json-migration-014",
    "prompt": (
        "Convert the transformation in migrate.py so it emits newline-delimited JSON. "
        "Return only the complete updated file inside a code fence."
    ),
    "check": "pytest",
    "write_to": "migrate.py",
    "command": "python -m pytest tests/test_migrate.py -q",
}

print(route(job, lanes))

Enter fullscreen mode Exit fullscreen mode

The audit log is the actual deliverable

The router code is maybe a weekend of effort. The compounding value lives in routes.jsonl. After a few weeks of real traffic you can compute things that are otherwise pure speculation:

• Fallback rate per job family. Bucket records by job type and look at fell_back. If data-formatting jobs pass on Lane A 92% of the time, Lane A is a rational default there. If multi-file refactors fail 60% of the time, Lane A is a false economy for that family — you’re paying for a doomed first attempt plus added latency on most calls.
• Effective cost per successful task. One Lane A call at $X that succeeds beats one Lane B call at $5X. A Lane A call that fails and triggers Lane B costs $6X. Do the arithmetic per category; the answer is frequently different per category, which is the entire point of routing instead of picking one model globally.
• Regression testing new releases. When the next checkpoint drops, point Lane A at it and replay your logged job corpus (keep the prompts, or a sanitized corpus, alongside the log). Comparing fallback rates across model versions on your jobs beats any public leaderboard for your decision.

Operating rules I’d insist on:

1. The referee is deterministic, full stop. The moment you add “ask a second LLM to grade it,” you’ve rebuilt the confidence problem with extra latency. Tests, parsers, schemas, regexes, exact-match — nothing that hallucinates.
2. Fingerprint prompts, don’t necessarily store them. A SHA-256 prefix lets you dedupe and correlate without persisting potentially sensitive content in the log.
3. Two lanes maximum. A three-tier cascade reads like clever engineering and behaves like a latency generator with a bigger debug surface. If Lane B fails the check, that’s a human’s problem, not a third model’s.

Where this breaks down

• Jobs without an oracle. This approach is only as strong as its checks. Code with tests, structured extraction, format conversions — great. Open-ended prose, architectural judgment, “summarize this for an exec” — there is no deterministic referee, and routing those on vibes is precisely the guesswork this pipeline exists to eliminate.
• Synchronous, latency-bound paths. A failed Lane A attempt plus a Lane B retry can roughly double wall-clock time. Keep the router in background workers, batch jobs, and async pipelines until you have p99 numbers proving otherwise.
• Assuming free stays free. Free model access and free server options are statements about the present, not contracts. The endpoints-as-config design above is your insurance policy; don’t hardcode a provider’s generosity into your architecture.
• Tiny samples. Fifteen routed jobs prove nothing. I don’t trust a fallback-rate number until a job family has roughly 50+ records behind it, and even then I look at the distribution, not just the mean.

If most of your workload is unverifiable generation, or everything you run is on a hard latency budget, honestly — skip the router. Picking the strong model outright is the simpler and more correct engineering decision in that world.

A concrete starting point

Pull your last ~50 real prompts, sort them into “has a deterministic check” versus “doesn’t,” and push the checkable subset through the two-lane setup for a week. If you want the measurement phase to cost nothing, MonkeyCode’s free model access and free server option work fine as Lane A and host while you gather data — and since the log format is provider-neutral, whatever you learn transfers when you point the lanes elsewhere.

The question “is the cheap model good enough?” has an answer, and it’s already sitting in your prompt history. Measure it; don’t outsource the decision to launch-week sentiment.