A/B 테스트를 위한 기능 플래그 AI 모델 및 프롬프트

This post was created with AI assistance and reviewed for accuracy before publishing.

Changing the model behind a feature is a deploy-sized decision dressed up as a config change. The output shape shifts, the latency profile shifts, the cost per request shifts, and none of it shows up in your test suite because the test suite does not call the model.

Feature flags are the right tool, but AI rollouts stress them in ways ordinary flags do not. The main differences are that the thing you are testing is non-deterministic, the quality signal is delayed, and a bad variant can be expensive rather than just broken.

Evaluate on the server, without exception

Model selection must never be decided by client code. A flag evaluated in the browser can be read and changed by anyone, which means the caller chooses which model you pay for.

// app/api/chat/route.ts
export async function POST(req: Request) {
  const session = await getSession(req);
  const variant = await flags.evaluate('chat-model', {
    userId: session.userId,
    plan: session.plan,
  });

  const model = MODELS[variant] ?? MODELS.control;   // fall back on unknown
  // ...
}

Enter fullscreen mode Exit fullscreen mode

The fallback on an unknown variant matters. Flag services fail, configs get edited, and a variant name can disappear while sessions still reference it. Defaulting to the control means the worst case is no experiment, rather than an exception in your main request path.

Keep experiments and entitlements apart

These get conflated because both are “flags”, and the consequences differ enormously.

An experiment decides which of two equivalent implementations a user gets. It can be reassigned, ramped, and rolled back freely. An entitlement decides what someone has paid for. It is authorisation, and it belongs with your billing logic, checked server-side on every request.

The failure is putting a paid capability behind an experiment flag. Ramp percentages then control who gets a feature they bought, and a rollback removes it from paying customers. Different systems, or at minimum different code paths with different review requirements.

Assign stably, or the data is noise

A user reassigned between variants mid-session gets a conversation where the model changes underneath them. That produces incoherent behaviour and contaminates your results.

Hash a stable identifier so assignment is deterministic:

function variantFor(userId: string, salt: string, split: number) {
  const h = createHash('sha256').update(`${salt}:${userId}`).digest();
  return (h.readUInt32BE(0) % 100) < split ? 'treatment' : 'control';
}

Enter fullscreen mode Exit fullscreen mode

Include the experiment name in the salt. Without it, the same users land in the treatment group of every experiment you run, and effects compound invisibly.

For conversational products, consider pinning to the conversation rather than the user, so a single thread always uses one model even if the user’s assignment later changes.

Decide what “better” means before you start

This is where AI experiments differ most from ordinary ones. There is no click-through rate for “gave a good answer”, and the honest signals are indirect.

Signal Reads as Caveat Explicit thumbs up or down Quality Very low response rate, skewed to extremes Regeneration rate Dissatisfaction Also rises when responses are slow Conversation length Engagement or struggle Ambiguous on its own Task completion Real success Only measurable if the task has an endpoint Cost and latency per request Operational fit Unambiguous, measure always

Pick the primary metric before launching. Choosing afterwards from whatever moved is how a worse model gets promoted on the strength of a metric that happened to rise.

Cost and latency are worth tracking on every variant regardless, because they are the two that are never ambiguous and they frequently decide the question on their own.

Log the assignment with the output

Every request should record which variant served it, alongside the tokens used, the latency, and any feedback that arrives later.

Without that join, you cannot attribute anything. A week into the experiment someone asks whether the treatment group’s costs went up, and if the variant is not on the usage record, the answer is unavailable and the experiment was wasted.

Treat prompt text in these logs carefully. It is user content, and an experiment log is an easy place for it to end up with looser retention than the rest of your data.

Build the kill switch first

Ramp deliberately: internal users, then a small percentage, then wider. But the ramp matters less than the ability to stop.

A kill switch has to be a config change that takes effect immediately, with no deploy and no cache to wait out. Test that it works before you need it, and make sure whoever is on call can operate it without a code review.

The scenarios it exists for are specific and worth naming: a variant is producing unsafe output, a provider incident makes one path unusable, or the cost per request turns out to be several times the control. All three are discovered in production, and in all three the time between noticing and stopping is the entire cost of the mistake.