Prompts rot silently. I swap a model, tweak a system prompt, add a tool, and everything still runs. No exception is thrown, no test goes red, the JSON still parses. The output is just quietly worse, and I usually find out from a user rather than from CI. Unit tests are the wrong instrument here because there is nothing to catch: the failure mode is not a crash, it is a drop in quality.
So I built evalgate, a small TypeScript tool that treats prompt and agent quality like a build artifact. You write a declarative eval suite, evalgate runs it, scores it, stores a baseline, and on every pull request it re-runs the suite, computes the quality delta against the base branch, and fails the build when the score regresses. Then it posts the delta table as a PR comment.
The core idea
The important design decision is what question CI is allowed to ask. “Is this prompt good?” is subjective and unwinnable in an automated gate. “Is this worse than it was on main?” is objective and answerable. evalgate is built around that second question. You capture a baseline once, and from then on every change is judged as a delta against it, not against some absolute notion of goodness.
The second decision was that the whole thing has to run with zero API keys. evalgate ships a deterministic mock provider, so you can run a suite, save a baseline, compare runs, and execute the full test suite completely offline. The project itself has 67 tests and none of them touch the network. Every feature has to work in mock mode before it counts as done.
How it works
A suite is a YAML (or JSON) file that lives in version control next to the code it checks. Each case has an input, an expected reference value, and one or more scorers. Here is a minimal one:
name: my-agent
provider: mock # works with no API key
threshold: 0.9 # mean score required to pass
cases:
- id: greeting
input:
prompt: |
Reply with the standard greeting.
exactly: Hi there! How can I help you today?
expected: "Hi there! How can I help you today?"
scorers:
- type: exact-match
- type: latency
budgetMs: 500
Enter fullscreen mode Exit fullscreen mode
A case passes when every one of its scorers passes, and its numeric score is the weighted mean of the individual scorer scores. There are 10 scorers in the catalog, covering the range of things you actually want to assert about model output:
-
exact-match,regex,contains, andnot-containsfor string-level checks (contains gives partial credit across multiple substrings). -
json-schemafor structured output, so you can assert the model returns valid JSON matching a schema. -
embedding-similarityfor “close enough in meaning” via cosine similarity. -
llm-judgeandrubricfor the softer, criteria-based judgments. -
latencyandcostfor budget gates, so a change that makes the agent slow or expensive can also fail the gate.
Two of those scorers are pluggable and ship with deterministic offline fallbacks, which is what keeps the mock-first rule honest. embedding-similarity uses the provider’s embed() if it has one, and otherwise falls back to a stable local bag-of-hashed-words embedding. llm-judge calls a real provider and parses a {score, reason} JSON reply, but on the mock provider it computes a reproducible word-overlap score instead. So a suite that uses judges and embeddings still runs identically on every machine with no keys.
The workflow is three commands. You run a suite, save a baseline, then compare later runs against it:
npx @royalpinto007/evalgate run suite.eval.yaml
npx @royalpinto007/evalgate baseline suite.eval.yaml --out baseline.json
npx @royalpinto007/evalgate compare suite.eval.yaml --base baseline.json --tolerance 0.01
Enter fullscreen mode Exit fullscreen mode
compare exits non-zero when any case regresses beyond the tolerance, and that non-zero exit is what fails the CI job. The tolerance matters because model output is not perfectly stable; you usually want a small allowed drift before something counts as a real regression.
The part that makes it feel like CI rather than a script is the GitHub Action. On each pull request it re-runs the suite and upserts a single comment, editing its own comment in place instead of stacking a new one on every push. A regression renders like this:
### evalgate: support-agent
FAIL - Quality regressed. 5 case(s) got worse.
Overall score: 94.2% (base) -> 60.1% (head) = -34.2pp
| Case | Base | Head | Delta | Change |
| ------------------- | ------ | ----- | -------- | ------ |
| refund-intent-json | 100.0% | 0.0% | -100.0pp | down |
| order-id-format | 100.0% | 0.0% | -100.0pp | down |
| greeting-exact | 100.0% | 66.7% | -33.3pp | down |
| judge-helpfulness | 73.6% | 69.1% | -4.5pp | down |
| paraphrase-quality | 86.0% | 84.6% | -1.3pp | down |
Enter fullscreen mode Exit fullscreen mode
You get the overall movement in percentage points and a per-case breakdown of exactly what got worse, right in the review, before the merge. Under the hood the same logic is exposed as a library, so loadSuite, runSuite, compareRuns, and renderCompareMarkdown are all importable if you want to wire evalgate into something other than the Action, and both scorers and providers are registrable so you can add your own.
An honest limitation
evalgate tells you the score moved; it does not tell you whether the new score is the correct one. If your prompt genuinely got better and the reference expectations are now stale, evalgate will still flag a delta, and it is on you to update the baseline. The gate is a change detector, not an oracle. That is also true for the softer scorers: llm-judge and embedding-similarity are only as trustworthy as the judge model and the criteria you write, so a green suite built on weak criteria is a false sense of safety. I lean on the exact, regex, and schema scorers for anything I want to be strict about, and treat the model-based scores as directional signal rather than ground truth.
Closing
The thing I wanted was simple: make prompt quality something a pull request can fail on, the same way a broken type or a failing test does. evalgate does that with a declarative suite, a baseline plus delta engine, 10 scorers, and a GitHub Action that comments the regression inline, all runnable offline through a deterministic mock provider.
답글 남기기