The gate couldn't read my JavaScript, so I removed the numbers from it

Originally published on hexisteme notes.

I run a publishing gate that won’t let a page go live unless every number on it traces back to a declared piece of evidence. A draft carries an editorial.json file listing evidence entries, each tagged evidence:<i>, and one of five machine gates, numeral_gate, strips a page’s HTML down to plain text, pulls every number out of that text, and blocks the draft if even one number has no matching entry. All five gates have to pass before a human ever sees the draft to sign off on it, and the verdict is permanent per draft — edit the body, and you start over from a clean dossier.

A calculator that needs numbers the gate can’t see

I hit the edge of that design building a payment-fee calculator page with a separate site-builder tool I run: type in a revenue figure, get back the fee and the amount you actually take home. A calculator like that needs JavaScript, and the JavaScript needs to know the fee rates. The gate, though, cannot read inside a <script> tag. Whatever turns a page’s HTML into checkable text either drags the script body along as text — a false positive, flagging numbers that were never really claims — or drops it entirely, a false negative on exactly the thing the gate exists to catch.

Two ways to lose

Two fixes come to mind immediately, and both are bad.

The first is to teach the gate to parse JavaScript. That means putting a JS parser inside the gate, which means the gate now owns a second language runtime — every bundler, minifier, and syntax change downstream becomes its maintenance debt from then on. Worse, it’s a fight where the losing condition belongs to whoever edits the code next: twist the JavaScript slightly and the parser loses.

The second is to exempt <script> tags from the check entirely. That one is more dangerous than it looks, and I know because I’d closed a version of exactly this mistake earlier the same morning. numeral_gate carries an exemption for alphanumeric identifiers — CSS values like 20px, say — so they don’t get flagged as unsourced numbers. British pence notation, 20p, was matching that same exemption and riding straight through the check, untouched. A convenience exemption had quietly become a laundering channel. Exempting <script> wholesale would have reopened that same defect as a much bigger hole.

Making the JavaScript have nothing to read

So I took a third path: instead of teaching the gate to read the JavaScript, I made sure there was nothing in the JavaScript worth reading. The fee rate travels like this:

evidence:<i> in editorial.json
  -> <tr data-calc-key="checkout" data-percent="3.49" data-fixed="0.49">
    -> JavaScript reads only the DOM's data-* attributes

Enter fullscreen mode Exit fullscreen mode

The rate rides into the page as a data-* attribute on a table row. That row is HTML, which the gate can read, and its cell value is already rendered from a cited evidence entry. The JavaScript itself never sees a fee rate — it reads a value that’s already sitting on screen, already proven, and does arithmetic on it. The gate ends up able to verify where this calculator’s rates come from without interpreting a single line of JavaScript.

Five invariants, five enforcement points

A direction like that isn’t a guarantee by itself. It only becomes a rule once every part of it has an enforcement point attached, and this one has five.

• Rates enter only through an evidence reference. validate_calculator_evidence checks that every rate in a calculator row points at a declared entry; write a literal number directly into the row and the build fails.
• Every numeric literal inside any <script> has to belong to the set {0, 1, 2, 100}. This is a new gate, SCRIPT_NUMERIC_LITERAL, and it’s the one actually carrying the weight of the design. It isn’t a parser — it’s a whitelist. 0, 1, and 2 cover indices and sign; 100 covers percentage division. A domain number structurally cannot fit inside that set. Adding it meant bumping the gate’s schema from v4 to v5.
• Calculator output has to equal the declared derived value. A small test harness runs eight scenarios and checks for an exact string match against what the gate computed as the derived value — rounding, currency symbol, and digit count all have to agree.
• Script and style content never becomes body text in the first place. extract_body_text, the function that pulls a page’s body text for checking, removes both elements wholesale before it ever strips a tag, so there’s nothing left for the gate — or anything downstream of it — to accidentally read out of a script block.
• The fact-check sheet a human signs shows the evidence the calculator cites. Whoever signs off sees the rate’s provenance on the same screen as the number it produced.

A comment that predicted its own failure

There’s a coda to that fourth invariant worth telling on its own. extract_body_text already had a docstring warning about exactly this situation, before the calculator page existed. It said, in effect: content inside <script> and <style> isn’t visible on screen, but this function drags it along as text anyway; since nothing in this package puts either one into a page body yet, that’s fine for now, but the day it does, fix this function first.

That day arrived. The calculator was the first page this package had ever put a <script> into.

The lesson isn’t that the comment was well written. A comment that names its own precondition is a tripwire — but a tripwire only works if someone actually walks over it. That docstring sat there doing nothing for months, and it would have done nothing this time either if I hadn’t happened to read that function. Without code that detects the precondition breaking, a comment like that isn’t an alarm. It’s a note left behind, read only after the fact.

Skip is not a check

The third invariant’s harness needs a JavaScript runtime to execute its eight scenarios. As first written, if that runtime wasn’t on the machine, the test called pytest.skip.

That’s not a check. On any machine that’s missing the runtime — including CI — the check just quietly disappears, and the color stays green. There’s no way, from the outside, to tell “this passed” apart from “this never ran.”

I changed it to pytest.fail. No runtime now means a red light, and someone has to either install it or make a conscious decision to turn the check off — silence stops being an option. One older test in the same file kept its skip, with a docstring explaining why that one is different. To confirm the fix actually worked, I pointed the process’s PATH at an empty directory to genuinely hide the runtime, and watched the test report FAILED.

The gate made me delete something I thought was finished

The gate also did something I hadn’t asked it to do. I decided to cut a numeric claim from the body text — a line about a $14.99 item — and just deleted the sentence. warrant_gate blocked the draft anyway: evidence was still attached to that claim, and now nothing referenced it. An evidence item had gone orphaned.

The tempting fix was to keep the evidence alive by hanging a footnote off some other sentence in the prose. Instead I checked the code first. editorial.sections[].html gets rendered exactly as written, and the only path that attaches an evidence marker runs through structured calculator and table cells — never through prose. There was no way to fake having one. So I deleted the claim and its evidence together, and the evidence count dropped from 13 to 12. The gate pushed me toward the honest fix instead of the convenient one.

Proving the gate isn’t decoration

None of this tells you whether the new gate actually catches anything. Passing it proves nothing on its own — you have to break it on purpose.

So I did, twice. I injected the literal rate 3.49 straight into the JavaScript, and SCRIPT_NUMERIC_LITERAL blocked it. Then I deleted the script-removal logic from extract_body_text, and the corresponding test went red. Both times I reproduced the failure myself rather than trusting a worker’s self-report that it worked.

What generalizes

When there’s a region a verifier can’t look inside — a JS bundle, a binary, generated code, a third-party embed — the fix isn’t to extend the verifier’s reach into that region. It’s to restructure things so that region never holds a claim that needs verifying in the first place. Move the claims onto a surface you can verify, and let the opaque region be nothing more than a pure function of that surface.

There’s one question that tells you whether this is worth doing: can this opaque region hold a wrong value at all? If the answer is yes, a better parser isn’t the end of the road — check first whether you can make a wrong value structurally impossible instead.

The same shape shows up elsewhere: deriving a config from code instead of parsing the config file to check it, emitting structured events instead of running logs through a regular expression, linting the input that feeds a generator instead of linting what the generator produces.

What it cost

On the value side: the test tier the deploy gate runs went from 407 to 438. Fifteen pieces are live under this regime. The local build and the live page are byte-identical, and the calculator’s eight scenarios match string-for-string.

The costs are worth stating just as plainly. The calculator cannot do anything outside its declared derived values — supporting one more scenario means adding evidence and a derived-value declaration first, not just editing JavaScript. I traded flexibility for provability, on purpose. Accessibility debt is still sitting there, too: the calculator form doesn’t have its label/for and aria wiring yet. I deferred that deliberately this round and wrote it down as the first item of the next one — what this gate proves is where a number came from, not whether the page is a good page. And on the project’s own internal naturalness scale, the style gate scored the calculator draft a 2 out of 5 and blocked it, which took three rounds to clear before it finished at 4 out of 5. Discipline has a time cost: my budget for this unit of work was 12 turns, and I actually spent 26. Most of that overrun is fair to book as quality, since it was the gate doing its job — but some of it was plainly a coefficient error in my own verification script, not the gate’s doing.

Where this stops being true

If the opaque region genuinely has to hold a domain number — a calculation that depends on a value only knowable client-side, say — this inversion doesn’t apply, and verification needs to move to runtime instead, checking the output rather than trying to prove the source is clean.

If the {0, 1, 2, 100} whitelist turns out to be too narrow in practice and people keep finding workarounds around it, that’s the point where pushing every constant out to the DOM costs more than it’s worth. Three or more accumulated workaround attempts is the trigger to reconsider the design.

And if SCRIPT_NUMERIC_LITERAL goes six months without catching a single real defect, that’s worth noticing too — a gate that’s only ever been shown alive by a mutation test might just be a gate that costs upkeep and catches nothing.

Three sentences worth keeping

1. When a verifier can’t read a region, don’t extend its reach into that region — change the region until there’s nothing left in it worth reading.
2. A convenience exemption written into a check has a way of turning into a laundering channel later — the 20px exemption is exactly what let 20p through.
3. skip is not a check. If “never ran” and “passed” show up as the same color, that check doesn’t exist.

More notes at hexisteme.github.io/notes.