Stop letting your AI guess selectors.

Your AI writes E2E tests that look right and fail. I built a map so it stops guessing.

AI agents are surprisingly bad at writing end-to-end tests for real apps. Not demo apps – real ones. The kind with zero data-testid attributes, three i18n locales, forms that load their defaults asynchronously, and selectors that only make sense if you wrote the frontend yourself.

I watched agents do this at work: guess a selector, fail, guess again, loop, then finally produce a test that looks completely reasonable and fails on the first run. My favorite failure mode: the agent wrote getByRole('tab', ...) for tabs that are – of course – plain buttons with click handlers. The test was well-written. It was also fiction.

The root cause is simple: the agent has never seen your app. Runtime AI-navigation tools (browser-use, Stagehand, Playwright MCP) explore it live, but they have no persistent memory – every task starts from zero, and exploration burns tokens on every run.

So I built terra-incognita, an open-source “app knowledge compiler”. The idea:

1. A human records navigation sessions in a real browser – you just click through your app once.
2. A deterministic distiller (no LLM, no API key) compiles those sessions into compact artifacts: an app map, an element dictionary with ranked selectors, a testability report, and a generated skill file.
3. The agent consumes the map when writing tests: prefer mapped selectors, never invent locators.

It is not a test recorder like codegen, not a runtime agent, not session replay. It compiles human ground truth into knowledge that any agent can read.

Does it help? Depends how big a mess your app is.

I benchmarked it with an A/B harness: same agent, same tasks, same turn budget – with and without the map. Beyond “does the test pass”, I scored acceptance (how many acceptance criteria the test actually asserts) and strict (passes AND covers everything), because pass rates alone let narrow, under-asserting tests through.

On a realistically messy production SPA (zero test-ids, three locales, async forms):

• A small model went from 0/8 to 7/8 strict with the map. Without it, its tests looked right and guessed wrong.
• A mid-size model went from 29% to 54% strict; acceptance-criteria coverage rose from 43% to 78%.
• The result I find most interesting: the small model with the map beat a ~7x more expensive model without it on every quality metric. A map instead of a bigger model.

And the honest flip side: on an app that is already well-testable – stable roles, test-ids everywhere – the map produced zero lift. There is nothing to fix. If that is your app, you do not need this tool. Maps are for wild territory.

(These are small pilots – 1-3 runs per task, tasks written by me. Directional evidence, not a published benchmark. A methodology write-up is coming.)

The part I did not expect: honesty needs a linter

The generated skill tells the agent: if a flow is not on the map, say so instead of inventing selectors. Some models follow that rule 100% of the time. Others ignore it routinely and fabricate plausible-looking locators anyway.

Prompt rules are advisory. So the tool ships a mechanical gate: terra lint checks every locator in your test files against the recorded element dictionary and fails the build on anything that is not traceable to a real, observed element. The model’s discipline stops mattering; the gate does not care whether it felt like following instructions today.

Try it

The whole pipeline is deterministic and runs offline – record, distill, generate, lint. MIT licensed:

github.com/andrzej-stepien/terra-incognita

terra record http://localhost:3000 --name checkout-flow
terra distill
terra generate-skill --target claude-code
terra lint tests/**/*.spec.ts

Enter fullscreen mode Exit fullscreen mode

If your app is a well-lit city grid, keep walking. If it is terra incognita – give your agent a map and tell me what breaks.