A New Cheap Model Dropped. Here's the 2-Hour Canary Test I Run Before Touching It

Every few weeks a new coding model launches with a price tag that makes the incumbent look embarrassing, and my feed fills up with people rewriting their configs on day one. I’ve been burned by this twice: once a “drop-in replacement” silently stopped emitting valid unified diffs, and once a cheaper model passed all my prompts but tripled the retry rate on long files, which erased the savings.

So now I don’t evaluate new models on vibes or on leaderboard screenshots. I run a small canary suite built from my own repository history before the candidate model gets anywhere near real work. This post is that harness: the task extraction script, the runner, the decision table, and the honest limits of the approach.

This builds on a personal scorecard I’ve written about before, but the goal here is different — not “rank models in general,” but “answer one narrow question: is this specific cheap model safe to route my traffic to?”

Step 1: Mine your own git history for tasks

Benchmarks test what benchmark authors care about. Your git history tests what you care about. I pull completed tasks straight from commit metadata:

#!/usr/bin/env bash
# extract_tasks.sh — build canary tasks from your own repo history.
# Each task = the state before a real commit + the human-written
# commit message as the instruction. The real diff becomes the reference.
set -euo pipefail

REPO="$1"
OUT="canary_tasks"
N="${2:-30}"

mkdir -p "$OUT"

cd "$REPO"
git log --format='%H' -n "$((N + 1))" -- 'src/**' | tac > /tmp/canary_commits.txt

i=0
while read -r commit; do
  parent=$(git rev-parse "$commit^" 2>/dev/null) || continue
  # Skip merge commits and giant diffs — they make noisy tasks.
  [ "$(git cat-file -p "$commit" | grep -c '^parent')" -eq 1 ] || continue
  files_changed=$(git diff --name-only "$parent" "$commit" | wc -l)
  [ "$files_changed" -le 4 ] || continue

  msg=$(git log -1 --format='%s%n%n%b' "$commit")
  diff=$(git diff "$parent" "$commit")

  mkdir -p "../../$OUT/task_$i"
  echo "$msg"  > "../../$OUT/task_$i/instruction.md"
  echo "$diff" > "../../$OUT/task_$i/reference.diff"
  git archive "$parent" | tar -x -C "../../$OUT/task_$i" 2>/dev/null || true
  i=$((i + 1))
done < /tmp/canary_commits.txt

echo "Extracted $i tasks into $OUT"

Enter fullscreen mode Exit fullscreen mode

Thirty small, single-purpose commits give you a far more honest test set than thirty hand-written prompts, because you already know these tasks are representative — they’re literally work your project needed.

One rule: never run this against a repo with secrets in history. Strip anything sensitive first, same as you would before giving any agent access to a codebase.

Step 2: The runner and the three failure classes

The runner sends each task to the candidate model, applies the result, and classifies the outcome. I only track three classes — more granularity than this stops being actionable:

Class Meaning Verdict impact Pass Tests (if any) pass; diff touches only expected files Counts toward adoption Soft fail Runs, but wrong behavior, or edits files it wasn’t asked to Tolerable below ~15% Hard fail Invalid output format, refuses, loops, deletes unrelated code Any hard fail pattern = investigate before adopting

# grade.py — deliberately simple. Example code; adapt to your stack.
import json, subprocess, sys
from pathlib import Path

def grade(task_dir: Path) -> str:
    ref = (task_dir / "reference.diff").read_text()
    out = task_dir / "candidate.diff"
    if not out.exists() or not out.read_text().strip():
        return "hard_fail"          # empty / unparseable output
    apply = subprocess.run(
        ["git", "apply", "--check", str(out)], cwd=task_dir,
        capture_output=True)
    if apply.returncode != 0:
        return "hard_fail"          # malformed diff
    ref_files = set(l.split()[1] for l in ref.splitlines()
                    if l.startswith("+++ b/"))
    out_files = set(l.split()[1] for l in out.read_text().splitlines()
                    if l.startswith("+++ b/"))
    if not out_files <= ref_files | {f.replace('b/', 'a/') for f in ref_files}:
        return "soft_fail"          # scope creep
    return "pass"                   # shape is right; eyeball diffs after

results = {p.name: grade(p) for p in Path(sys.argv[1]).iterdir() if p.is_dir()}
print(json.dumps(results, indent=2))

Enter fullscreen mode Exit fullscreen mode

This grading is intentionally shallow — it checks shape and scope, not semantic correctness. Semantic checking is step 3, and it’s manual on purpose.

Step 3: Read ten diffs, not a dashboard

Aggregate scores hide the failures that matter. After the run, I open the candidate’s diff next to the reference diff for ten randomly chosen tasks and ask one question: would I have caught this in code review, or would it have shipped? Silently-dropped error handling and “helpfully” reformatted files both look fine in a pass-rate number and are exactly what a cheaper model tends to introduce. If I find either pattern twice in ten samples, the model doesn’t get routing traffic regardless of its score.

Where the free tier fits

This harness is cheap to run but not free — thirty tasks times N candidate models adds up, and re-running it every time a new model drops is precisely the cost that makes people skip evaluation and YOLO their configs instead.

Disclosure: This article was prepared as part of MonkeyCode’s product outreach.

This is where I’ve been pointing the harness at MonkeyCode: its free model access covers the candidate side of the comparison for supported models, and the free server option means the runner itself — extraction, grading, result storage — runs without me paying for a box or tying up my laptop. The practical effect is that “evaluate the new hyped model before adopting” stops being a cost decision and becomes a habit.

Two honest caveats. First, “free” tiers change — treat this as a way to run the evaluation now, not a permanent subsidy, and keep the harness provider-agnostic (mine just reads an env var for the endpoint) so you can move it in an afternoon. Second, free access shapes which models you can canary this way; if the model you actually want to test isn’t available there, run a smaller task set against a paid endpoint rather than skipping evaluation entirely.

The routing decision table

Once a model passes the canary, I still don’t route everything to it. The output of the whole exercise is a routing table, not a winner:

Task type Route to Why Boilerplate, renames, mechanical refactors Cheapest passing model Hard fails are caught by git apply + CI Bug fixes with a failing test in hand Cheapest passing model Test is the oracle New feature code, no tests yet Stronger model No oracle → soft fails ship silently Anything touching auth, billing, migrations Stronger model + mandatory review Cost of a miss dwarfs token savings Long-context work (large files, many files) Whatever won that specific canary subset Long-context regression is the most common cheap-model failure I’ve seen

Limitations, and who shouldn’t do this

• Thirty commits is a vibe with structure, not statistics. It catches gross regressions and format breakage. It will not catch a 3% quality decline, and it can’t compare two good models.
• Your history is not your future. If the new work you’re planning (a new language, a new service shape) isn’t represented in the commit window, the canary tells you nothing about it. Extract tasks from the repos you’re actually about to work in.
• Reference diffs aren’t ground truth. The human commit might itself have been mediocre. You’re testing “can it do work shaped like my work,” not “is it correct.”
• Don’t bother if your volume is tiny. If you run a few agent tasks a week, the evaluation overhead exceeds any plausible savings — just pay for the model you trust. This pays off when you’re routing real traffic or when your team is about to standardize on a model.
• Never canary on a repo with secrets in its history, and keep the candidate model’s sandbox as locked down as you’d keep any agent’s.

The harness above is the whole thing — extraction script, grader, table. If you want somewhere zero-cost to run your first pass, MonkeyCode’s free server and free model access are a reasonable starting point; the scripts don’t care where they run.

The broader point: a model launch is a marketing event, and your git history is the only benchmark that knows what your work actually looks like. Two hours of canary testing is cheaper than one silent diff corruption.