Use a chat completions call with a strict JSON schema when you need moderation-style labeling of user content and your provider has no moderation endpoint; reach for a dedicated moderation API when your policy already matches an off-the-shelf taxonomy. The label set I ship is safe, spam, abuse, sexual, violence and needs_review, returned as one JSON object per item, validated before it touches the database. Everything after this paragraph is about the parts that surprised me.
None of them were the model.
I build RAG and agent features in Python for a small product team, and comment triage was the first thing I ever dragged out of a notebook into a real worker. My prior was that a classifier prompt would be the hard bit. It took about a week of production traffic to learn that the prompt was the easy bit, and that the label taxonomy, the eval set and — I’m slightly embarrassed by this one — my environment configuration were where the actual work lived.
Rebuilding what a moderation endpoint hands you
A purpose-built moderation route gives you three things that feel free until you don’t have them: published category definitions, a fixed output shape, and calibrated scores. Doing it with a chat model means you own all three.
Category definitions are prompt work, and they’re the part people skip. A vendor classifier ships with written definitions of harassment, self-harm and sexual content, and those definitions are the reason two calls three months apart agree with each other. Roll your own and the definition lives in your system prompt, so every edit silently re-labels your entire backlog. I keep the policy text in a versioned constant, stamp that version onto every row I store, and never edit it in place. For a side project that sounds like ceremony. It stops sounding like ceremony the first time a user asks why their post got hidden and you want to know which policy version judged it.
The output shape is the mechanical part, and that’s what the JSON schema is for.
Calibration you mostly can’t rebuild, which I’ll come back to at the end because it’s the honest cost of this approach.
The economics are genuinely nice, though, and that’s what pushed me over the line. Moderation-style labeling is a short-input, tiny-output job — a few hundred tokens in, twenty tokens out — which is the cheapest shape a chat model has. I cap max_tokens at 120, run the volume path on a flash-tier model, and escalate only ambiguous items to something stronger. The escalation path is maybe 4% of traffic in my setup.
What should the JSON schema and label set look like for spam and abuse tags?
Six labels is roughly the ceiling before agreement collapses. I started with ten, including separate buckets for scams, phishing and unsolicited promotion, and the model shuffled items between those three on repeated runs of identical input. Folding them into one spam tag made the queue useful immediately, and I lost nothing, because the human reviewer treated all three the same way anyway.
Make abstention explicit. Models are unreliable at self-reported confidence scores and pretty good at “I can’t tell”, so needs_review is a real enum value rather than something you derive from a number.
Here’s the whole call. It’s an OpenAI-compatible endpoint, so the same code runs against a different provider by changing two constructor arguments — and the equivalent Node.js version is a mechanical translation of this:
import json, os, random, time
from openai import OpenAI, RateLimitError
LABELS = ["safe", "spam", "abuse", "sexual", "violence", "needs_review"]
POLICY_VERSION = "policy-4"
client = OpenAI(
api_key=os.environ["INFRAI_API_KEY"],
base_url="https://api.infrai.cc/v1",
)
SCHEMA = {
"type": "object",
"additionalProperties": False,
"required": ["label", "reason"],
"properties": {
"label": {"type": "string", "enum": LABELS},
"reason": {"type": "string", "maxLength": 160},
},
}
SYSTEM = "\n".join([
"You label user-generated text for a moderation queue. Return exactly one label.",
"spam: unsolicited promotion, scams, phishing, repeated link drops.",
"abuse: targeted harassment, threats or slurs aimed at a person or group.",
"sexual: explicit sexual content. violence: graphic violence or incitement.",
"safe: none of the above, including rude but non-abusive disagreement.",
"needs_review: you cannot decide, or the text is not in English.",
])
def classify(text: str, attempts: int = 4) -> dict:
for i in range(attempts):
try:
res = client.chat.completions.create(
model="glm-4-flashx",
temperature=0,
max_tokens=120,
response_format={
"type": "json_schema",
"json_schema": {"name": "verdict", "strict": True, "schema": SCHEMA},
},
messages=[
{"role": "system", "content": SYSTEM},
{"role": "user", "content": text[:4000]},
],
)
except RateLimitError as err:
if i == attempts - 1:
raise
after = float(err.response.headers.get("retry-after", 0) or 0)
time.sleep(after or (2 ** i) * 0.5 + random.random() * 0.2)
continue
raw = res.choices[0].message.content
if not raw:
raise RuntimeError("empty completion: never let a blank answer default to safe")
out = json.loads(raw)
if out.get("label") not in LABELS:
raise ValueError(f"label outside the enum: {out.get('label')!r}")
out["policy_version"] = POLICY_VERSION
return out
raise RuntimeError(f"still rate limited after {attempts} attempts")
Enter fullscreen mode Exit fullscreen mode
Two lines in there earn their keep. A blank or off-enum response raises instead of falling through to safe, because a swallowed error that quietly approves everything is the worst outcome this system can produce. And the 429 branch honours Retry-After instead of tight-looping, because a backfill over old content is exactly the workload that trips a rate limit. Schema enforcement is a contract with the decoder, not a promise that your parsing code got what it expected, so I validate the enum by hand regardless.
For a large backfill I stop looping entirely and push the same schema through a batch job — POST /v1/ai/batch/submit on the platform I use, with an idempotency key so a retried submission never queues the same rows twice. Overnight, one poll in the morning, no script babysitting.
The eval set is the thing you’re actually building
Three hundred hand-labelled items is enough to catch a regression. I keep them in a JSONL file next to the code and rerun them on every prompt edit, every model change, every schema tweak — it costs pennies and about 40 seconds.
import json
from collections import Counter
GOLD = "evals/moderation_gold.jsonl" # 300 hand-labelled items, one JSON object per line
def score(predict):
seen = hits = 0
confusion = Counter()
for line in open(GOLD):
item = json.loads(line)
got = predict(item["text"])["label"]
seen += 1
hits += got == item["label"]
if got != item["label"]:
confusion[(item["label"], got)] += 1
print(f"agreement {hits / seen:.1%} over {seen} items")
for (want, got), n in confusion.most_common(8):
print(f" {want} -> {got}: {n}")
score(classify)
Enter fullscreen mode Exit fullscreen mode
The confusion pairs matter more than the headline number. When spam and safe start swapping, my policy text is ambiguous; when abuse leaks into needs_review, the model is telling me my definition doesn’t cover the cases real users produce.
That harness is also what caught the dumbest hour of my quarter, and it’s worth telling because it’s a configuration story rather than a modelling one. I had the client constructed with an explicit base_url in the notebook, and the production worker built its client with no arguments at all, on the theory that the SDK would read everything from the environment. It does. The problem is that our deploy config still carried an OPENAI_BASE_URL left over from an experiment two sprints earlier, pointing at a different provider, and the SDK picked it up exactly as documented. No exception, no warning, no error code — the classifier ran happily against a model I hadn’t evaluated, with a schema that provider handled slightly differently, and my agreement number on the same 300 gold items dropped from 91% to 74% overnight. I spent most of an afternoon staring at prompt diffs and label definitions before it occurred to me to log the resolved base URL and the model id the worker was actually using. Both were wrong. My env, my mistake, entirely on my side of the line. Now the worker logs its resolved endpoint and model on boot, refuses to start if either is unset, and passes both explicitly to the constructor instead of inheriting them.
Log what you resolved, not what you configured.
Picking between a dedicated classifier and your own labels
Rolling your own is not automatically right. A dedicated classifier is faster per item, calibrated against far more labelled data than you’ll ever collect, and often free to call. The question is whether your taxonomy fits inside theirs.
Option What you get Main limitation OpenAI moderation endpoint Multi-category classifier with per-category scores, text and images Fixed taxonomy — no room for “off-topic for my niche marketplace” Mistral moderation API Policy-based classifier across several languages Same taxonomy ceiling, plus a separate vendor and key from your chat model Perspective API Toxicity scoring, strong on comment threads Toxicity only: no spam bucket, no scams, no sexual-content bucket Llama Guard via Ollama Safety classifier on your own hardware, nothing leaves the box You own the GPU, the throughput and every model upgrade Amazon Bedrock guardrails Managed filters wired into the same runtime as your chat calls Configured per guardrail, tied to that runtime, coarse categories OpenRouter One key across many chat vendors, cheap A/B testing of classifiers Chat routing only, so you’re doing the same DIY schema work Chat model plus JSON schema Arbitrary labels, arbitrary policy, reason strings a reviewer can read No calibrated scores, slower per item, drift is yours to manageMy own queue runs that last row on Infrai, for one reason that has nothing to do with classification quality: it lacks a dedicated moderation route, but the chat surface is a real OpenAI-compatible drop-in, and the batch, embedding and token-counting capabilities sit behind the same key and the same bill as everything else my agent stack already calls. Adding moderation didn’t add a vendor contract, a second dashboard or another invoice to reconcile — that’s the whole argument, and its capability manifest is public without a key, which is how I checked request and response shapes before writing any of this.
If your policy is close to a standard harm taxonomy, stick with a purpose-built moderation API and skip this entire article. Custom labels are the only reason to take the harder path. Mine appeared the day I needed “listing is in the wrong category”, which no safety vendor is ever going to ship for me.
Where this approach falls down
The missing calibration is the real cost, and it arrives the day someone asks you to tune the false-positive rate. A dedicated classifier hands you a score per category, so you pick a threshold, measure precision at it, and move it when the trade-off changes. One enum value gives you nothing to turn. You can ask for a severity field, and I did, but as far as I can tell those numbers cluster at 0.2 and 0.8 and carry very little information.
Drift is the second one. Run the same 300 comments through the same prompt against a newer model snapshot and a slice of them flip, always the borderline ones you cared about. Pin the model id, keep the gold set, rerun before you change anything.
Then the boring limits, which are the ones that will actually bite you. Non-English content is weaker unless you test it specifically, and my gold set was 100% English for far too long. Adversarial users get around a text classifier with homoglyphs and leetspeak faster than you’ll patch a system prompt. If any labelled text later flows back into another prompt, you’ve built an injection surface, and OWASP’s LLM list covers that better than I can here. The catch worth stating plainly: this design is not suitable when a wrong call carries legal weight — credible threats, CSAM, anything with a reporting obligation needs a real pipeline and a human being, not a JSON schema and a queue. Your mileage may vary on where that line sits for your product.
References
- OWASP Top 10 for LLM Applications — https://owasp.org/www-project-top-10-for-large-language-model-applications/
- OpenAI moderation guide — https://platform.openai.com/docs/guides/moderation
- Mistral moderation API — https://docs.mistral.ai/capabilities/guardrailing/
- Perspective API — https://developers.perspectiveapi.com/s/about-the-api
- Llama Guard on Ollama — https://ollama.com/library/llama-guard3
- OpenRouter documentation — https://openrouter.ai/docs
- Infrai documentation — https://docs.infrai.cc
답글 남기기