Last week I turned my data audit into a build step: a check that runs before anything else and fails the build when the database and any static copy of my travel site’s legal-status data disagree. It ended the era of the site contradicting itself.
It did nothing about the site agreeing with itself on something false.
That’s not a hypothetical. The most expensive error the whole project found was a country whose law changed in January while every copy on my site — database, data files, search index — kept saying the old thing in perfect unison. Internal consistency was the camouflage. No diff between my own sources could ever have caught it, because every internal source was equally behind the world.
A build gate proves agreement. Agreement is not truth. Something has to look outside.
You can’t diff against the world, but you can sample it
The naive version of “look outside” is another audit — a human session checking primary sources jurisdiction by jurisdiction. I’ve done three of those now, and I know exactly what they’re worth: they’re correct the day they ship and they decay from that morning on. Laws don’t change on my audit schedule.
So the outside check became what the inside check became: a scheduled job. Once a week, a script asks a web-connected model — one that searches and cites, not one answering from training memory — for the current legal status of about fourteen jurisdictions, and compares each answer to the corresponding database row.
Fourteen, not all 271, because the selection is doing the real work:
- A hot list is checked every single run: the highest-traffic pages plus the jurisdictions with active legislative motion — the places where being a month stale costs the most.
- Everything else sits on a rotating cursor: eight per run, round-robin, so every row on the site gets sampled roughly twice a year without any run costing more than a few cents. The whole thing runs on about seven cents a week.
Two rules were non-negotiable, both inherited from the build gate:
It never writes. A model’s answer is not data — it’s a tripwire. When the model and the database disagree, the script sends me a flag that says, in so many words: verify this against primary sources, do not trust it. Every fabricated-but-plausible fact I’ve written about in this series was an AI-generated value that skipped exactly this step. The pipeline is model → flag → human → primary sources → fix. The model never touches the last three.
A run that can’t check anything is a failure, not a quiet pass. If the database is unreachable or every query errors, the job exits nonzero and my monitoring treats it like any other broken cron. “Couldn’t verify” reported as success is the most expensive bug a verification step can have — same rule, new layer.
Each flag also alerts exactly once per claimed value. A definitional disagreement I’ve decided not to act on shouldn’t renag me weekly; an alert channel that repeats itself gets muted, and a muted tripwire is decoration.
First run: four alerts, three of them mine
The first live run checked thirteen jurisdictions and raised four flags. Three of them said the same alarming thing: states my database grades as medical — real, functioning patient programs — were, according to the model, illegal.
All three flags were wrong. And the model hadn’t failed. My question had.
I’d phrased the prompt the way I think about the site: “what does an ordinary adult visitor face here?” For a medical-only state, the honest answer to that question is prohibition — a visitor has no local patient card and no way to get one. The model answered the question I asked, correctly. My database answers a different question: what is the jurisdiction’s regime. Medical program: exists. Same territory, same statutes, two defensible grades — because grading needs a rubric, and my dataset and my prompt were using different ones.
This is the part I’d generalize furthest: a comparison is only as strong as the rubric both sides share. Last month I resolved twenty-six “conflicts” between two of my own data stores by discovering they meant different things by the same word, and the fix was writing the definition down at the top of the data files. The first run of the sampler was the same bug at a new boundary — this time between the dataset and the question being asked about it. The fix was the same too: the prompt now carries the dataset’s own written definition, verbatim rules included, so the model grades with the rubric the data was graded by. Re-ran the three states: all came back medical, matching. The flags dissolved.
A sensor doesn’t measure the world. It measures the world through its question. Calibrate the question first, or every reading is an artifact.
The fourth flag was real — and it beat me at my own rule
One flag survived the prompt fix. A small New England state sat in my database as medical. The model said decriminalized, and cited its reason: the state removed jail for small-amount possession back in 2017 — a civil violation, a hundred-dollar fine, no cell.
Here’s what makes this one uncomfortable. My dataset’s written definition includes a tiebreaker for exactly this situation: when a jurisdiction is both medical and decriminalized, the status that describes possession by an ordinary person wins. I wrote that rule. I applied it across an entire continent’s worth of countries a week earlier. And I had still left this state graded by its program instead of by its possession law — the value predated the rule, and nothing had forced it through.
The model wasn’t smarter than the dataset. It was more consistent with the dataset’s own rules than I had been. That’s what a good tripwire does: it doesn’t need to out-think you, it just needs to apply your stated policy without your habits.
The flag went through the full pipeline: verified against primary sources (the 2017 statute is current law; the fine schedule is unchanged), then fixed in the database and the static file in the same commit — because the build gate from last week fails if the two move separately. The page’s headline counts recomputed themselves at build time, since they derive from the data. One wrong prompt and one wrong row, both found and fixed on day one, for the price of a coffee’s loose change per year.
What I’d carry to any dataset that describes the world
- Consistency checks certify agreement; something else must certify truth. A fact wrong in every copy at once is invisible to every internal comparison you will ever write.
- Sample the world on a schedule, weighted by cost of staleness. Hot set every run, full rotation over months. You don’t need to check everything weekly — you need everything checked eventually and the expensive things checked often.
- A web-connected model is a tripwire, not an oracle. Its disagreement is a signal to open primary sources. Give it no write path — not because it’s usually wrong, but because “usually” is doing unpriced work in that sentence.
- The prompt must carry the dataset’s rubric. If your data grades by a written definition, the question you ask the model must include that definition, or you’ll spend your alert budget on vocabulary mismatches.
- Expect the first flags to audit the sensor. Three of my four were the question’s fault. That’s not a failed launch; that’s what launching a measurement instrument looks like.
- Alert once per finding. A check that nags gets muted, and a muted check is worse than none, because you believe you have coverage.
This series has been one long inventory of places the same fact can be wrong: four data copies, then the prose, then the agreement between copies, and now the gap between all of them and the world. Each layer’s check certifies something the next layer can’t see. The build gate catches my copies lying to each other, minutes after it happens. The sampler catches all of them being wrong together — and its first real catch was a fact my own rulebook said I should have already fixed.