I had a ranking problem that looked trivial and turned out to have the wrong shape.
Contexto is a daily word game that scores your guess by semantic distance to a hidden answer, reporting a rank from 1 to whatever. Players want good opening words. The obvious approach is to take an archive, count how often each word lands near the answer, and publish the top ten.
I did that over 1,421 puzzles, and the result was useless in an instructive way.
The top ten were nearly one word
Rank every word by how often it lands inside the top 100 and you get potato, tomato, onion and their neighbours. Three separate picks by the metric. One pick in practice, because they all fire on the same food days and stay quiet on the same non-food days.
The metric scored each word in isolation. What a player actually needs is a small set whose members fail on different puzzles. That is not a ranking problem at all. It is set cover.
Greedy set cover
Treat each of the 1,421 puzzles as an item, and each candidate word as covering every puzzle where it lands at or inside the threshold. Repeatedly take the word covering the most not-yet-covered puzzles. Ties break alphabetically, which keeps it deterministic.
Pick Word Newly covered Cumulative Share 1 one 191 191 13.4% 2 salad 131 322 22.7% 3 park 98 420 29.6% 4 blue 89 509 35.8% 5 hand 72 581 40.9% 6 box 64 645 45.4% 7 frog 58 703 49.5% 8 work 58 761 53.6% 9 water 53 814 57.3% 10 interest 46 860 60.5%Ten words, at least one inside the top 100 on 60.5% of puzzles. The spread across a verb, a food, a place, a colour, a body part, a container and an animal was not a design choice. The algorithm produced it, because after salad is picked, every other food word covers almost nothing new.
This is the standard greedy approximation for a problem that is NP-hard in general, so it is not a proven optimum. It is deterministic and re-runs identically, which was enough for the use I had.
The measurement problem underneath
Contexto publishes a top 500. A word outside it has no observed rank. It is not missing data, it is right censored: the true value exists and is known to be worse than 500.
That distinction decides the whole analysis. If you average only the observed ranks, a word that lands 3rd on two days out of 1,421 outranks a word that lands 200th on a thousand of them. The first word is noise. The second is the one you want.
So there is no mean rank anywhere in the results. Everything is coverage counts and medians taken over the puzzles where a word appears, with the appearance rate reported alongside so the two are read together.
One gate before pooling anything
Pooling four years of puzzles assumes the scoring model did not change partway through. If the embedding were retrained mid-archive, early and late puzzles would not be comparable and every pooled number would be quietly wrong.
The check: within a single year, split the puzzles into two halves by alternating id and compare each half’s 50 highest-coverage words by Jaccard overlap. That is the baseline, what the metric reads when the model is definitely unchanged. Then compare the same-parity halves of consecutive years, so both sides are built from the same number of puzzles.
Within-year baseline came in at a median of 0.754. The lowest cross-year comparison was 0.493, above the threshold derived from the baseline spread. It passed, and it is worth being precise about what passing means: it shows the set of broadly-near words is stable. It cannot rule out a retraining that preserved which words are broadly near while reshuffling fine-grained ranks. So the results are reported at coverage granularity, which is what the probe supports, and not at single-rank precision.
Testing the advice that already exists
A widely cited guide recommends opening with person, place, thing, idea, concept, then food, occupation, animal, event, then city, nature, home, work, technology. Measured the same way, counting a puzzle once if any word in the set reaches the threshold:
Set Words Reaches top 100 All fourteen recommended 14 39.5% Set cover above 10 60.5%The guide is right about the idea. Spreading openers across categories instead of hunting one magic word is the same conclusion set cover reaches independently. The specific words are the problem: person, place, thing, idea and concept are abstract category labels, and an abstract word sits at a middling distance from nearly everything, so it rarely lands close and rarely lands far. occupation appears in the top 500 on 0.8% of puzzles.
Takeaway
When a top-N list feels unsatisfying, check whether the items are independent. If picking the second-best gains you almost nothing over the best, you are ranking when you should be covering, and the fix is a different algorithm rather than a better metric.
Full tables, the drift probe, and the answer-frequency statistics are in the original post. The scripts that produce every figure are deterministic, so the numbers can be re-derived rather than taken on trust.