Fine-tuning MuRIL for Multilingual Citizen Grievance Classification (with all the bugs I hit)

Built a text classifier that routes Indian citizen grievances (Hindi / Hinglish / English, code-mixed) into 14 government department categories, using a fine-tuned google/muril-base-cased. Final honest holdout metrics: 87.3% accuracy, 0.869 macro F1. Full writeup of the bugs I hit below, because the debugging was more instructive than the happy path.

Dataset: https://www.kaggle.com/datasets/abhisheksingh016/citizen-grievance-dataset
Notebook: https://www.kaggle.com/code/abhisheksingh016/citizen-grieveances/
Model: https://huggingface.co/pyofpython/citizen-grievance-classifier-muril

The problem

Citizen grievances in India get typed in whatever language/script mix is natural to the writer — Devanagari Hindi, Romanized Hindi (“Hinglish”), English, or all three in one sentence. Classic TF-IDF-style models can’t bridge that: “पानी” and “paani” share zero characters, so a lexical model treats them as unrelated tokens even though they mean the same thing.

Bug #1: The eval that lied to me

First pass: 100% accuracy with a TF-IDF + ensemble (NB, LogReg, SVM, RandomForest, XGBoost, soft-voted). Too good. Turned out my train/test split let the exact same sentence templates appear on both sides — the model was matching template fingerprints, not learning language.

Fix: built a second evaluation set from templates held out entirely from training — genuinely unseen phrasing. Re-ran: accuracy dropped to 20–31%. That’s the real number for a lexical model on code-mixed text.

Switching to MuRIL

google/muril-base-cased is pretrained on 17 Indian languages and their transliterated forms, so it embeds “पानी” and “paani” close together in vector space — solving the exact gap that broke the TF-IDF approach.

from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("google/muril-base-cased")
model = AutoModelForSequenceClassification.from_pretrained(
"google/muril-base-cased", num_labels=14
)

Bug #2: Shared RNG silently breaking unrelated categories

When I added more training templates to a few categories, completely unrelated categories (that I never touched) started scoring near-zero F1. Root cause: one global random instance shared across all category generation — adding templates upstream shifted the RNG’s internal state, which changed which templates got selected as holdout for categories downstream in the dict.

Fix: give every category its own independent RNG, seeded from a hash of its name:

def category_rng(category):
seed = sum(ord(c) for c in category) * 7919 + len(category) * 104729
return random.Random(seed)

Now editing category A can never again perturb category B. Verified with a reproducibility check — identical MD5 hash across 3 repeated generation runs.

Bug #3: Real keyword bleed between categories

A Sanitation & Garbage template about sewage overflow happened to mention “paani” (water) — pulling it toward the Water Supply category. Similarly, an Employment template mentioned “ration” in passing. Both fixed by rewording to remove the borrowed vocabulary. Confirmed the fix worked: Water Supply and Sanitation both went from mediocre to 1.00 F1 in the next training run.

Bug #4: Kaggle disk quota crash mid-training

RuntimeError: [enforce fail at inline_container.cc:668]

Not a code bug — Kaggle’s /kaggle/working has a ~20GB quota, and save_strategy=”epoch” with no limit was writing a full model + optimizer checkpoint every epoch. Filled the disk by epoch 2.

Fix:

TrainingArguments(
...
load_best_model_at_end=True,
metric_for_best_model="macro_f1",
save_total_limit=1, # only keep the best checkpoint
)

Final results

Evaluated on the held-out, unseen-phrasing set (the honest one):

Metric Score
Accuracy: 87.3%
Macro F1: 0.869

Most categories land near-perfect (Water Supply, Sanitation, Healthcare, Land Records all ≥0.98 F1). The weak spot is Corruption & Bribery (F1 0.40) — genuinely ambiguous, since corruption complaints usually reference another department (a bribe-taking ration dealer, a bribe-taking clerk), not a labeling error.

Inference

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
tokenizer = AutoTokenizer.from_pretrained("pyofpython/citizen-grievance-classifier-muril")
model = AutoModelForSequenceClassification.from_pretrained("pyofpython/citizen-grievance-classifier-muril")
text = "Hamare mohalle mein 5 din se paani nahi aa raha hai."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=96)
logits = model(**inputs).logits
print(model.config.id2label[logits.argmax(-1).item()])#-> Water Supply

Would love feedback from anyone doing NLP on code-mixed/low-resource languages — especially if you’ve found a cleaner way to handle the “corruption spans every department” labeling problem.