LLM costs scale linearly with usage. A system processing 10,000 requests a day at $0.01 per request costs $100 daily — $365 a year. At enterprise scale, that’s over $10,000.
Cost optimization isn’t about cutting corners. It’s about spending tokens where they matter.
Every token you waste is a token you could have spent on a better answer.
Token budgeting
The simplest way to control costs is to set limits. Per session, per task, or per day.
Strategy 1: Per-Session Budgets
Per-session budgets are straightforward:
class SessionBudget:
def __init__(self, budget_tokens: int = 10000):
self.budget = budget_tokens
self.used = 0
def allocate(self, tokens: int) -> bool:
if self.used + tokens <= self.budget:
self.used += tokens
return True
return False
def remaining(self) -> int:
return self.budget - self.used
Enter fullscreen mode Exit fullscreen mode
Strategy 2: Per-Task Budgets
Per-task budgets are more useful. Different tasks need different amounts of context:
task_budgets:
classify:
max_tokens: 100
model: qwen2.5-1.5b
summarize:
max_tokens: 500
model: qwen2.5-7b
code_review:
max_tokens: 2000
model: qwen2.5-coder-7b
reason:
max_tokens: 4000
model: qwen2.5-32b
Enter fullscreen mode Exit fullscreen mode
Strategy 3: Adaptive Budgets
Adaptive budgets adjust based on what actually happens. If classification tasks consistently use 80 tokens, stop allocating 100:
class AdaptiveBudget:
def __init__(self):
self.task_history = {}
def allocate(self, task_type: str) -> int:
if task_type in self.task_history:
return int(self.task_history[task_type] * 1.5)
return 1000
def record(self, task_type: str, tokens_used: int):
if task_type not in self.task_history:
self.task_history[task_type] = tokens_used
else:
self.task_history[task_type] = (
0.9 * self.task_history[task_type] + 0.1 * tokens_used
)
Enter fullscreen mode Exit fullscreen mode
The exponential moving average (0.9 weight) means recent usage matters more than history. Adjust the weight based on how volatile your workloads are.
API vs local inference
Local inference is cheaper at scale. The break-even depends on your hardware and API rates.
Model API ($/M tokens) Local cost/hour Break-even GPT-4o $2.50 / $10.00 — N/A Claude Sonnet 4 $3.00 / $15.00 — N/A Qwen2.5-72B $0.50 / $2.00 ~$0.50 ~4 hours/day Qwen2.5-32B $0.30 / $1.20 ~$0.20 ~2 hours/day Qwen2.5-7B $0.10 / $0.40 ~$0.05 ~1 hour/dayThe hardware math:
Hardware Upfront Monthly electricity Break-even vs API RTX 3090 (used) $600 $15 ~4 months RTX 4090 $1,500 $20 ~6 months RTX 5080 $1,000 $18 ~5 months DGX Spark $2,000 $30 ~8 monthsAt moderate usage — an hour or more per day — local inference pays for itself. At high usage, the savings are dramatic. The catch is upfront capital. A RTX 5080 is $1,000. An API bill you can pause. Hardware you can’t.
Fallback strategies
When your preferred model is too expensive or too slow, fall back to something cheaper. The key is knowing when quality is “good enough.”
Strategy 1: Quality-Based Fallback
Quality-based fallback tries models until the output meets a threshold:
class QualityFallback:
def __init__(self, quality_threshold: float = 0.8):
self.threshold = quality_threshold
self.models = [
{"model": "claude-sonnet-4", "cost": 0.015},
{"model": "qwen2.5-72b", "cost": 0.002},
{"model": "qwen2.5-32b", "cost": 0.001},
{"model": "qwen2.5-7b", "cost": 0.0004},
]
def route(self, prompt: str) -> str:
for model_config in self.models:
result = self.call_model(model_config["model"], prompt)
if self.evaluate_quality(result) >= self.threshold:
return result
return self.call_model(self.models[0]["model"], prompt)
Enter fullscreen mode Exit fullscreen mode
The problem is evaluation itself. How do you measure quality without calling another model? Some systems use a small classifier. Others use heuristic checks — length, structure, keyword presence. None of these are perfect.
Strategy 2: Latency-Based Fallback
Latency-based fallback is simpler. Route to the fastest model that meets your time budget:
class LatencyFallback:
def __init__(self, max_latency: float = 5.0):
self.max_latency = max_latency
self.models = [
{"model": "qwen2.5-1.5b", "latency": 0.5},
{"model": "qwen2.5-7b", "latency": 2.0},
{"model": "qwen2.5-32b", "latency": 10.0},
{"model": "claude-sonnet-4", "latency": 5.0},
]
def route(self, prompt: str) -> str:
for model_config in sorted(self.models, key=lambda x: x["latency"]):
if model_config["latency"] <= self.max_latency:
return self.call_model(model_config["model"], prompt)
return self.call_model(self.models[0]["model"], prompt)
Enter fullscreen mode Exit fullscreen mode
Caching
Caching is the most underrated cost optimization. Identical prompts happen more often than you think — classification requests, FAQ-style queries, repeated tool calls.
Strategy 1: Prompt Caching
Exact prompt caching is simple:
import hashlib
class PromptCache:
def __init__(self, max_size: int = 1000):
self.cache = {}
self.max_size = max_size
def get(self, prompt: str) -> str | None:
key = hashlib.sha256(prompt.encode()).hexdigest()
return self.cache.get(key)
def set(self, prompt: str, response: str):
key = hashlib.sha256(prompt.encode()).hexdigest()
if len(self.cache) >= self.max_size:
self.cache.pop(next(iter(self.cache)))
self.cache[key] = response
Enter fullscreen mode Exit fullscreen mode
Strategy 2: Semantic Caching
Semantic caching is more useful. It catches prompts that are different but mean the same thing:
from sentence_transformers import SentenceTransformer
class SemanticCache:
def __init__(self, similarity_threshold: float = 0.95):
self.model = SentenceTransformer('all-MiniLM-L6-v2')
self.cache = {}
self.threshold = similarity_threshold
def get(self, prompt: str) -> str | None:
prompt_embedding = self.model.encode([prompt])[0]
for cached_prompt, cached_response in self.cache.items():
cached_embedding = self.model.encode([cached_prompt])[0]
similarity = self.cosine_similarity(
prompt_embedding, cached_embedding
)
if similarity >= self.threshold:
return cached_response
return None
def set(self, prompt: str, response: str):
self.cache[prompt] = response
Enter fullscreen mode Exit fullscreen mode
The threshold matters. 0.95 is aggressive — only very similar prompts match. 0.85 is more forgiving but risks returning wrong answers. Measure your miss rate and adjust.
Response caching for common queries is worth it too. If users ask “what’s the weather” or “what time is it” repeatedly, cache the pattern, not just the exact prompt:
class ResponseCache:
def __init__(self):
self.common_queries = {
"what is the weather": "Check weather API",
"what is the time": "Check system time",
"who is the president": "Check current president",
}
def get(self, query: str) -> str | None:
query_lower = query.lower()
for common_query, response in self.common_queries.items():
if common_query in query_lower:
return response
return None
Enter fullscreen mode Exit fullscreen mode
This isn’t sophisticated, but it works. Common queries are common for a reason.
When optimization helps
Optimization matters when you’re processing high volumes, running mixed workloads, or paying API costs that add up.
It doesn’t matter when you’re prototyping, using a single model, or processing low volumes. The complexity of budgeting, fallback, and caching isn’t worth it for a system that makes 100 requests a day.
Get the basic flow working first. Add optimization when the bill comes in.
Tradeoffs
Strategy Cost Quality Complexity No optimization Highest Consistent Lowest Token budgeting Moderate Variable Medium Fallback models Low-Medium Variable Medium Caching Lowest High (for cache hits) Medium Hybrid Optimized Optimized HighestProduction systems usually run hybrid. Budget per session, fall back on quality or latency, cache what you can. The complexity is real, but so are the savings.
Related
- Model Routing Strategies — capability-based, cost-aware, latency-aware routing
- LLM Guardrails in Practice — input validation, output filtering, safety
- Multi-Model System Design — architecture for multiple models
- LLM Architecture — system design pillar: routing, cost, guardrails, and orchestration
답글 남기기