We Asked 330 Models a Question in Korean. Half of Them Answered in the Wrong Alphabet.

We graded 330 language models on Korean across seven axes. Before any of that, a four-line function
threw out a third of the answers.

That function turned out to be the most useful thing in the harness.

The check

def contamination(text: str) -> dict:
    total = len(text.strip())
    if not total:
        return {"hangul_ratio": 0.0, "hanja": 0, "kana": 0, "broken": True}
    h  = len(HANGUL.findall(text))
    hj = len(HANJA.findall(text))
    kn = len(KANA.findall(text))
    return {"hangul_ratio": round(h / total, 3), "hanja": hj, "kana": kn,
            # not Korean enough, or another script leaked in
            "broken": h / total < 0.25 or hj > 3 or kn > 0}

Enter fullscreen mode Exit fullscreen mode

Three conditions, all countable, no model in the loop:

• less than 25% Hangul — the answer is not really in Korean
• more than 3 Hanja — Chinese characters bleeding in
• any kana at all — Japanese characters bleeding in

If broken, the answer is an F and we never pay a judge to read it.

What it caught

Answers auto-failed 766 of 2,304 — 33.2% Models with at least one 171 of 330 — 51.8% Models that failed all seven 54 — 16.4%

Fifty-four models did not produce one clean Korean answer out of seven attempts.

The failure is not evenly spread across tasks:

Task Models auto-failed Honorifics 142 Terminology 125 Korean institutions 113 Summarisation 105 Naturalness 99 Register 93 Format compliance 89

The hardest linguistic task is also where script discipline collapses first.

Why a mechanical gate and not the judge

An LLM judge will happily grade a Chinese-Korean hybrid on its register. It will produce a
plausible sentence about tone while the answer is not in the requested language at all. The grade
comes back looking like data.

So we kept a rule for this build of the harness: anything countable is counted, not judged.
Script mix is countable. Tone is not. The judge only sees answers that already passed the count.

This also cuts cost — 766 fewer judge calls — but that is a side effect. The point is that the two
failure modes are different and must not be averaged into one grade.

The harness refuses to start if it cannot tell good from bad

Before measuring anything, the script runs a fixed set of control answers with known expected
grades, and aborts if it does not score them perfectly:

rate = hits / len(CONTROLS)
print("채점기 정확도 %d/%d = %.0f%%" % (hits, len(CONTROLS), rate * 100))
#      ^ "judge accuracy" — verbatim from our source
if rate < 1.0:
    return False        # do not begin the real measurement

Enter fullscreen mode Exit fullscreen mode

An A-expected control must come back A. An F-expected control must come back F. If the judge cannot
separate the two on cases we already know the answer to, the numbers it produces on unknown cases
are not worth having.

Two more things we do to the judge, both cheap:

• The judge never sees the model name. Self-preference is real and free to remove.
• The rubric says length is irrelevant, because judges reward long answers if you let them.

The grades, once the surviving answers are graded

330 models, 8 bands:

A+++   5      B+   55      D   29
A++    2      B    58      F  111
A+    18      C    52

Enter fullscreen mode Exit fullscreen mode

7.6% reach any A band. 33.6% are F.

Per-axis A rates, and this is the part that surprised us:

Axis A rate Honorifics 8.5% Knowledge of Korean institutions 9.4% Terminology 31.2% Format compliance 47.6% Summary accuracy 49.2% Freedom from translationese 53.1% Register 53.3%

Two axes sit at roughly a tenth of the field while the rest cluster near half.

Vendor averages

Mean score, 0–3, for vendors with at least eight models measured:

Vendor n Mean anthropic 14 2.29 mistralai 18 2.07 openai 51 1.96 qwen 51 1.48 google 27 1.48 deepseek 15 1.33 meta-llama 8 1.16 nvidia 10 0.93 minimax 8 0.70 z-ai 15 0.36

One prompt set, one judge, one run. Read it as a ranking of this measurement, not a verdict on the
vendors.

Recency and size predict nothing

Five models hold a perfect 3.00:

openai/gpt-5.4
openai/gpt-5.4-mini
openai/gpt-4o-2024-05-13
openai/gpt-3.5-turbo-16k      ← 2023
google/gemini-3.1-flash-image

Enter fullscreen mode Exit fullscreen mode

A 2023 model sits with the 2026 flagships. And the 28 models that earn an A on honorifics include
gpt-4o-mini, qwen-2.5-72b-instruct, llama-3.1-70b-instruct — mid-size models, not the tops of
anyone’s leaderboard.

If you are picking a model for a Korean-facing product, release date, parameter count and English
benchmark position are not proxies. They carry no signal on this axis.

Run it yourself

Open API, no key:

curl .../api/korean    # every model, every axis, with the judge's reason string
curl .../api/summary   # grade distribution, judge id, what the run cost

Enter fullscreen mode Exit fullscreen mode

https://huggingface.co/spaces/ginigen-ai/open-router-leaderboard

426 models, 330 graded, refreshed daily. The judge id and the per-answer reason strings are in the
payload, so you can disagree with a specific grade rather than with the average.

The portable part

Whatever language you ship into, write the countable check first and let it fail answers before your
judge sees them. Ours is four lines and it disqualified a third of the corpus.

A judge asked to grade an answer that is in the wrong script will still return a grade. That grade
will look exactly like the real ones in your CSV.

Methodology: https://huggingface.co/blog/ginigen-ai/openrouter-leaderboard