A browser tool that points your own API key at an adversarial battery and grades every answer with pure predicates — no LLM judge, and your key never touches my server.
The first time I ran it against a real model, it told me the model was ~29% vulnerable.
That number was wrong. And the tool proved it was wrong — to me, in public — because of exactly one design decision.
Every verdict in The AI Crash Test is a deterministic predicate over the model’s answer string: exact match, regex, a number check, an injection canary, a must-refuse rule. No model grades another model. So when the report flagged ~29% vulnerable, it also showed the fail card for every miss — prompt, expected, actual, side by side. Three of those cards didn’t show a broken model. They showed a broken grader: false positives in my own code. I fixed the graders; the real number was 0%.
That’s the whole pitch. An auditable grader has bugs you can catch in public. A vibes-based, LLM-as-judge arena just hands you a number and asks you to trust it.
The honest part first
LLM red-teaming is a crowded, mature space. garak (NVIDIA), PyRIT (Microsoft), and promptfoo all do far more than this — more probes, more scale, more integrations. Browser tools that adversarially test with your own key exist too; most lean on an LLM judge.
So this isn’t a new category, and I won’t pretend it is. The narrow thing that’s mine is an intersection: browser-based BYOK + deterministic no-judge grading + a provably shared engine with a longitudinal drift board. Distinctive engineering and discipline, not a market-novel product. If you want heavy artillery, go use garak. If you want a result you can reproduce byte-for-byte and a key that goes straight to the provider and never touches my server, read on.
Two properties you can check yourself
1. Deterministic grading — no LLM in the grade path. Every grade is a pure function of the answer string, run in an open-source engine called gradecore. Run a mock model through it twice and the score is byte-identical. No temperature, no judge drift, no “the grader was having a bad day.”
2. BYOK, never-touches. The browser calls the provider directly with your key. crashkit’s server receives only the answers — the grade request has no key field at all.
Don’t take my word for it. Open DevTools → Network, run a battery with your key, and search the panel for the key itself. It lights up only on the request to the provider (e.g. api.anthropic.com, in the x-api-key header) — never in the /api/grade call. I verified this live before writing this; you can reproduce it in about thirty seconds.
The honest caveat: this only works where the provider allows direct browser calls. Anthropic (with the dangerous-direct-browser-access header) and Gemini work; OpenAI-direct is often CORS-blocked. Stating the limit is part of the point.
One engine, two lenses
gradecore isn’t a crashkit-only toy. It’s the same deterministic engine behind my live model-drift board, which tracks 16 LLMs over time. Same code, two jobs: the board is longitudinal monitoring; The AI Crash Test is on-demand adversarial testing.
And it’s the same engine, not a lookalike. Run the board’s frozen suite through gradecore and the suite_hash comes out identical, byte for byte — faithful extraction, not a reimplementation. (To be clear: crashkit uses gradecore, not my whole eval stack.)
What’s in the battery
Eight tasks across seven attack kinds: prompt-injection, tool-abuse, spec-violation, refusal-calibration, unsafe-compliance, hallucination-bait, and consistency. You get a severity-weighted vulnerability report and a fail card for every miss.
A real result from a recent run: claude-haiku-4-5 resisted 100% — 0% vulnerable across all seven kinds, deterministically graded, reproducible.
Try it / read it
- Run it: https://crashkit.onrender.com
- Source: github.com/egnaro9/crashkit and github.com/egnaro9/gradecore
I built this solo, self-taught, in under a year — and yes, with heavy AI assistance in the loop. The design decision I care about is the one you can check without trusting me: open the Network tab, run it twice, read the graders. Prove it — don’t take my word for it.
답글 남기기