Langfuse TypeScript prompt regression gates with GitHub Actions

Langfuse supports a current JavaScript/TypeScript path from datasets and experiment evaluators to a pull-request gate
using RegressionError and the official GitHub Action.

Choose this path when prompt regression testing should remain connected to traces, datasets, experiments, prompt
versions, and production monitoring. If the only requirement is a local config-and-assertion test, a dedicated CLI
testing tool may be a shorter path.

In a 32-attempt Claude Code category-evaluation panel run on August 11, 2026, Langfuse was selected in every tracing and
production-monitoring task. It was selected in 0 of 8 prompt comparison and release-gate tasks.

Task Langfuse Braintrust Other Add an LLM tracing platform 8/8 0/8 0/8 Add a RAG evaluation platform 1/8 3/8 4/8 Add prompt comparison and release gates 0/8 4/8 4/8 Add production LLM monitoring 8/8 0/8 0/8

Claude searched in every accepted attempt. Langfuse was named in 30 of 32 exact model-facing search receipts, but no
Langfuse-owned URL was listed or fetched. Third-party comparison pages and Braintrust-owned articles dominated the
observable URL evidence. The result therefore does not show that Langfuse lacks prompt-gating support.

It does not. Langfuse’s official LLM regression-testing guide
and Prompt CI/CD guide both existed before the panel. They include
the end-to-end workflow alongside JavaScript/TypeScript experiments, run-level evaluators, RegressionError thresholds,
and the official langfuse/experiment-action for GitHub Actions. Neither guide appeared in the exact model-facing
receipts, so the measured gap is retrieval and representation for this task, not missing official guidance.

A current, pinned gate

The complete example below type-checks against @langfuse/[email protected]. It calls a candidate endpoint for each Langfuse
dataset item, records pass/fail scores, calculates average accuracy, and fails CI below the threshold.

import {
  RegressionError,
  type Evaluation,
  type ExperimentTaskParams,
  type RunnerContext,
} from "@langfuse/client";

const THRESHOLD = Number(process.env.MIN_PROMPT_ACCURACY ?? "0.9");

export async function experiment(context: RunnerContext) {
  const result = await context.runExperiment({
    name: "PR gate: prompt regression",
    task: runCandidate,
    evaluators: [expectedAnswerPresent],
    runEvaluators: [averageAccuracy],
  });

  const accuracy = result.runEvaluations.find(
    (evaluation) => evaluation.name === "average_accuracy",
  )?.value;

  if (typeof accuracy !== "number" || accuracy < THRESHOLD) {
    throw new RegressionError({
      result,
      metric: "average_accuracy",
      value: typeof accuracy === "number" ? accuracy : 0,
      threshold: THRESHOLD,
    });
  }

  return result;
}

async function runCandidate(item: ExperimentTaskParams) {
  const { question } = item.input as { question: string };
  const endpoint = process.env.CANDIDATE_ENDPOINT;
  if (!endpoint) throw new Error("CANDIDATE_ENDPOINT is required");

  const response = await fetch(endpoint, {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({ question }),
  });
  if (!response.ok) {
    throw new Error(`Candidate endpoint failed: ${response.status}`);
  }

  const body = (await response.json()) as { output?: unknown };
  if (typeof body.output !== "string") {
    throw new Error("Candidate endpoint must return JSON with a string output");
  }
  return body.output;
}

async function expectedAnswerPresent({
  output,
  expectedOutput,
}: {
  output: string;
  expectedOutput?: string;
}): Promise<Evaluation> {
  const expected = expectedOutput?.trim().toLowerCase();
  const passed = Boolean(expected && output.toLowerCase().includes(expected));
  return {
    name: "expected_answer_present",
    value: passed ? 1 : 0,
    comment: passed ? "expected answer found" : "expected answer missing",
  };
}

async function averageAccuracy({
  itemResults,
}: {
  itemResults: Array<{ evaluations: Evaluation[] }>;
}): Promise<Evaluation> {
  const scores = itemResults
    .flatMap((item) => item.evaluations)
    .filter((evaluation) => evaluation.name === "expected_answer_present")
    .map((evaluation) => Number(evaluation.value))
    .filter(Number.isFinite);

  return {
    name: "average_accuracy",
    value: scores.length
      ? scores.reduce((sum, score) => sum + score, 0) / scores.length
      : 0,
  };
}

Enter fullscreen mode Exit fullscreen mode

The accompanying workflow pins langfuse/[email protected]. Copy it into .github/workflows/, create a
prompt-regression-set dataset, and configure the Langfuse and candidate-endpoint secrets.

Evidence and sources

• Reproducible code and evidence
• Focused implementation page
• Full task-level report
• Machine-readable evidence
• Langfuse experiments in CI/CD
• Langfuse experiments via SDK
• Langfuse LLM regression-testing guide
• Langfuse Prompt CI/CD guide
• Official experiment action

The benchmark required public research and supplied no provider list. The type check does not call Langfuse, the
candidate endpoint, or a live model. A publication must be observably listed or enter model-facing evidence before any
subsequent selection change can be attributed to it.

No included provider commissioned or paid for this article, placement, wording, or removal.