TL;DR
Route narrow, testable prompts to a small model first, fall back to a large model only on an explicit quality signal, and move delay-tolerant work into a batch lane. The cheapest architecture for a SaaS app is the one that minimizes cost per accepted result while still meeting its latency, quality, and US or EU data-handling SLOs.
Treat every route as an operational policy, not a clever prompt.
Measure accepted work.
How should a SaaS app reduce its LLM API bill with prompt routing?
I start by splitting traffic into request classes. Support-intent classification, field extraction, document summarization, and an open-ended assistant have different failure costs, even if the finance export puts them on the same LLM API line item. For each class, I write down the output contract, latency objective, maximum input size, acceptable fallback rate, and consequence of a wrong answer. Only then do I test a small model against a large model on held-out examples that resemble actual tenant traffic.
The routing rule should be boring enough to explain during an incident. A small model goes first when the response can be checked locally: valid JSON, an allowed label, required fields, or a bounded score. The large model is a fallback when that check fails or when the request was classified as high risk before inference. Don’t use model confidence as the only gate unless it has been calibrated on the same workload; a confident invalid result is still invalid.
No silent escalation.
This is the buy-versus-build table I use in roadmap reviews:
Option Good fit On-call load Lock-in and capacity trade-off One managed model Low or changing volume; a small platform team Lowest initial serving ownership Simple, but every request takes the same capability path Managed small/large routing Stable request classes with measurable acceptance rules Policy, quota, and evaluation ownership Provider behavior belongs behind an application interface Self-hosted inference Predictable sustained load and an ML operations team Accelerator capacity, rollout, patching, and saturation are yours More control, with a much larger failure surfaceI wouldn’t add a router merely because two models exist. When volume is low, prompts change weekly, or nobody owns the evaluation set, stick with one managed route and collect measurements first. The catch is that routing creates a second production system: policy versions, telemetry, evaluation data, and rollback all need owners.
Make the fallback path explicit in code
The application should own the model interface and the validation rule. That keeps a provider client replaceable and makes the policy testable without spending tokens. Although the original service asking this question may be Node.js, the operational contract is language-independent; all production examples here are Go because that’s the stack I carry on call.
package routing
import (
"context"
"errors"
"strings"
)
type Client interface {
Complete(ctx context.Context, model, prompt string) (string, error)
}
type Result struct {
Text string
ModelClass string
Fallback bool
}
func acceptedLabel(text string) bool {
switch strings.TrimSpace(text) {
case "billing", "security", "support":
return true
default:
return false
}
}
func Classify(ctx context.Context, client Client, prompt string) (Result, error) {
if len(prompt) == 0 || len(prompt) > 1200 {
return Result{}, errors.New("prompt is outside the classifier contract")
}
text, err := client.Complete(ctx, "small-model", prompt)
if err == nil && acceptedLabel(text) {
return Result{Text: text, ModelClass: "small"}, nil
}
text, err = client.Complete(ctx, "large-model", prompt)
if err != nil || !acceptedLabel(text) {
return Result{}, errors.New("no model produced an accepted label")
}
return Result{Text: text, ModelClass: "large", Fallback: true}, nil
}
Enter fullscreen mode Exit fullscreen mode
Notice what the code doesn’t do: retry blindly, parse a friendly paragraph, or hide escalation inside the provider client. In production I would emit the request class, policy version, selected model class, validation outcome, fallback reason, latency, and input/output token counts when available. Prompt bodies stay out of default logs because tenant text may carry personal or confidential data.
Test that boundary.
One cold-start incident fixed this lesson in my head: under real traffic, p99 reached 8.4 seconds for 17 minutes, while our synthetic checks stayed green because their steady cadence kept the relevant path warm. The model call shared the budget with connection setup, queueing, and retries — a cheap first hop that damages the tail SLO isn’t cheap in any useful sense.
Put delay-tolerant LLM work on a batch lane
Batch processing belongs behind a durable queue, not in a loop hanging off the web tier. Backfills, nightly enrichment, evaluation runs, and scheduled summaries can tolerate a completion window; interactive chat and blocking form validation usually can’t. Separating those lanes lets the synchronous service protect its latency budget while workers pace demand against configured capacity.
Each batch item needs an idempotency key, tenant and region policy, prompt version, model class, attempt count, and durable terminal state. A worker should claim bounded work, write the result atomically, and retry only errors declared retryable by the client contract. Poison items go to a review queue rather than cycling forever. This sounds like ordinary job processing because it is — LLM calls don’t suspend queueing theory.
Capacity planning starts with arrival rate, tokens per item, acceptable completion window, and measured service time. I reserve headroom for retries and replays, then cap worker concurrency so a batch import cannot consume the interactive route’s quota. Cost dashboards should divide spend by accepted outputs for each request class; raw token cost hides schema failures, duplicate work, and fallback amplification.
Keep the boundary sharp.
Batch isn’t suitable when a customer is waiting on the same request, and self-hosting isn’t suitable when the team lacks accelerator capacity planning and inference on-call experience. Conversely, a stable, high-volume offline workload may justify evaluating self-hosted inference because utilization can be planned. I’m not sure where that crossover lands for your traffic; your mileage may vary with model size, utilization, staffing, and the quality target. The decision should come from a load test and an ownership review, not a spreadsheet cell containing an optimistic utilization percentage.
For a Node.js application, the web process can enqueue the same policy envelope and a Go worker can consume it, provided the message schema is versioned and both sides agree on idempotency. The language boundary is less important than preserving the request contract.
Verify US and EU policy, deploy gradually, and roll back cleanly
Region selection is a data-governance decision before it is a latency tweak. For US and EU tenants, record the approved processing region in tenant policy, pass only the minimum required content to the model path, define retention expectations, and have security and legal owners verify the provider contract. An endpoint label alone is not evidence for the complete data flow. If residency or transfer requirements can’t be demonstrated, keep that workload on an approved path even when another route scores better in a quality test.
Before release, replay a held-out set through the current and proposed policies. Compare acceptance rate, schema failures, fallback rate, input and output tokens per accepted result, and latency percentiles. Slice the results by prompt class, tenant region, and input-size band; an average can conceal a long-context route that consumes the error budget. Embeddings can help group similar inputs for evaluation and retrieval workflows, but they don’t replace labeled acceptance criteria for the generated answer.
Watch the tail.
Deploy behind a versioned flag to a small tenant cohort. The stop conditions belong in the change plan: validation failures above the class threshold, excessive fallback, or a tail-latency breach should pin that class to the known-good route. Rollback means changing policy, not shipping application code under pressure. Pause affected batch consumers, preserve their durable items, switch the synchronous class, and verify recovery from telemetry before resuming queued work.
My final go/no-go review is blunt: who owns the evaluation corpus, who receives the page, what is the capacity ceiling, how quickly can we reverse the policy, and has somebody other than the policy author exercised that reversal while the queue contains real-shaped test items and dashboards are being watched? If those answers are vague, optimization waits. A single-model design is the right choice when its simplicity protects the SLO better than the projected savings from routing; an extra inference tier has to earn its operational footprint.
답글 남기기