멀티 테넌트 AI 채팅: 4단계로 하드 코딩된 구성에서 BYOK까지

Two tenants, two AI providers, two prompts. Sounds simple, and on day one it is. That’s the trap. It stays simple right up until customer number two sends their first “quick question,” and eighteen months later you’re running a small distributed system to answer it. Here’s the honest version of that slide, four stages, each one caused by a real human typing a real request into Slack.

Day 1: it just works (he says, foolishly) 😅

Tenant A wants OpenAI. Tenant B wants Claude. Both want their own system prompt. The obvious first version: one config object, one row per tenant. What could possibly go wrong. (Everything. Everything could go wrong. But not yet.)

┌─────────────┐     ┌──────────────────┐     ┌─────────────┐
│  Request    │ →   │  TENANT_CONFIG   │ →   │  Provider   │
│  (tenantId) │     │  (hardcoded obj) │     │  SDK call   │
└─────────────┘     └──────────────────┘     └─────────────┘

Enter fullscreen mode Exit fullscreen mode

const TENANT_AI_CONFIG = {
  tenantA: { provider: 'openai', model: 'gpt-5', prompt: 'You are terse and technical.' },
  tenantB: { provider: 'anthropic', model: 'claude-sonnet-5', prompt: 'You are friendly. Antworte auf Deutsch.' },
} as const

async function handleChat(tenantId: string, userMessage: string) {
  const config = TENANT_AI_CONFIG[tenantId]
  const client = config.provider === 'openai' ? openai : anthropic
  return client.chat(config.model, config.prompt, userMessage)
}

Enter fullscreen mode Exit fullscreen mode

Ships in an afternoon. Two tenants, two rows, demo goes great, everyone claps 👏. Put this moment in a frame, it’s the calmest the codebase will ever be.

Evolution 1: Tenant B wants their own twist

A week in (a week, we didn’t even get a full sprint), tenant B messages: “can we change the prompt ourselves, without waiting for a deploy?” Fair ask, they know their users, we don’t, and also nobody wants to be the on-call engineer who gets paged to edit a string literal. A hardcoded object can’t answer that, it needs a rebuild to change a comma.

┌─────────────┐     ┌──────────────────┐     ┌─────────────┐
│  Request    │ →   │  tenant_settings │ →   │  Provider   │
│  (tenantId) │     │  (DB row, admin  │     │  SDK call   │
│             │     │   editable)      │     │             │
└─────────────┘     └──────────────────┘     └─────────────┘

Enter fullscreen mode Exit fullscreen mode

Config moves from a code constant to a table the tenant’s own admin UI can write to.

async function getAiConfig(tenantId: string) {
  const row = await ctx.db.tenantSettings.findOne({ tenantId })
  return row.aiConfig // { provider, model, prompt }
}

Enter fullscreen mode Exit fullscreen mode

Same shape as before, different source. handleChat doesn’t change at all, it has no idea any of this happened, which is exactly the point of putting the lookup behind one function.

Evolution 2: “wait, who changed the prompt?” 🕵️

Self-service is great until it isn’t: tenant B’s prompt quietly changed last Tuesday, their bot started answering in pirate-speak for reasons nobody can reconstruct, support gets a ticket, and the honest answer is “we have no idea, the database doesn’t remember either.” A plain DB row just gets overwritten, the past has no representation, it’s Ctrl+Z with no undo history.

┌──────────────┐     ┌────────────────┐     ┌──────────────────┐
│  Admin edits │ →   │  ConfigChanged │ →   │  current config  │
│  the prompt  │     │  event (who,   │     │  = fold(events)  │
│              │     │  when, diff)   │     │                  │
└──────────────┘     └────────────────┘     └──────────────────┘

Enter fullscreen mode Exit fullscreen mode

The config becomes event-sourced instead of a mutable row: every change is an event, the current value is a projection over them.

async function updateAiConfig(tenantId: string, patch: Partial<AiConfig>, actor: string) {
  await ctx.emit('AiConfigChanged', { tenantId, patch, actor, at: ctx.now() })
}

async function getAiConfig(tenantId: string): Promise<AiConfig> {
  const events = await ctx.db.events.find({ tenantId, type: 'AiConfigChanged' })
  return events.reduce((cfg, e) => ({ ...cfg, ...e.patch }), DEFAULT_AI_CONFIG)
}

Enter fullscreen mode Exit fullscreen mode

Now “who changed it and when” is a query, not a seance 🔮. handleChat still hasn’t changed, it just calls getAiConfig, blissfully unaware it’s now talking to an event log instead of a table.

Evolution 3: BYOK and a usage cap 💸

A bigger tenant shows up, the kind that gets its own Slack channel, with two demands: they want to use their own OpenAI key (cost control, their own rate limits, their own finance team breathing down their neck), and they want a hard cap on monthly spend so an over-caffeinated intern’s script can’t turn into a five-figure invoice.

┌──────────────┐     ┌───────────────────────┐     ┌──────────────┐
│  Request     │ →   │  config.apiKey?       │ →   │  usage < cap?│
│              │     │  (BYOK, encrypted)    │     │  → call      │
│              │     │  else our shared key  │     │  → else 429  │
└──────────────┘     └───────────────────────┘     └──────────────┘

Enter fullscreen mode Exit fullscreen mode

async function callProvider(tenantId: string, userMessage: string) {
  const config = await getAiConfig(tenantId)
  const usage = await getMonthlyUsage(tenantId)
  if (config.usageCap && usage >= config.usageCap) {
    throw new UsageCapExceeded(tenantId)
  }

  const apiKey = config.byokApiKey ?? process.env.SHARED_API_KEY // BYOK overrides shared key
  const client = getClient(config.provider, apiKey)
  const reply = await client.chat(config.model, config.prompt, userMessage)

  await recordUsage(tenantId, reply.usage.totalTokens)
  return reply
}

Enter fullscreen mode Exit fullscreen mode

Two additive fields on the same config, byokApiKey and usageCap, and one counter check before the call. No new architecture, no rewrite of the first three stages, no “sorry, we need a full quarter to redesign this.”

The pattern behind the pattern 🧵

Every stage kept getAiConfig(tenantId) → { provider, model, prompt, ... } as the seam. Storage changed underneath it four times (constant, DB row, event-sourced projection, projection with encrypted secrets) and the call site never noticed, never cared, never even asked. That’s the actual lesson: don’t design the multi-tenant AI system upfront, design one seam that can absorb whatever the next Slack message throws at it.

Skipped on purpose: provider fallback, streaming, per-model cost tables. Add those when a tenant actually asks, same as everything above. If a tenant asks for a fifth provider before you’ve read this sentence, that’s not a counterexample, that’s Tuesday. 🙃