I graded my own ML option forecasts. Here's the Brier score.

In May I promised — out loud, on the internet, where people can screenshot you — that I’d grade Helium’s published prob_itm forecasts when the June 2026 AAPL contracts expired.

June 26 came. The contracts died. So I graded them.

This is not a victory lap. Mean Brier loss on n=2 is 0.3846. A coin flip that always says 0.5 scores 0.25 when the world resolves to 0 or 1. We did worse than a coin flip. That is information.

What we froze in May

Contract Helium prob_itm Market-ish implied AAPL $310C 2026-06-26 0.42 ~0.50 AAPL $295P 2026-06-26 0.23 ~0.24

What the stock actually did

AAPL NASDAQ close on 2026-06-26: $283.78

• $310 call → OTM → realized ITM = 0 → Brier = (0.42 − 0)² = 0.1764
• $295 put → ITM → realized ITM = 1 → Brier = (0.23 − 1)² = 0.5929

Honest read (short)

1. The call forecast was closer to reality than the market’s ~0.50 shrug — but “closer” is not “good.” Any non-zero probability on an OTM finish still costs you.
2. The put is the bruise. Probabilities looked aligned pre-expiry; then the underlying fell hard enough that both market and model had understated the ITM risk. Helium’s number was far from 1, so the Brier loss is ugly.
3. n=2. This is a promise kept, not a calibration paper. If you want to treat it as science, wait for more expiries on the honesty board.

Reproduce

git clone https://github.com/connerlambden/helium-mcp-cookbook
cd helium-mcp-cookbook
python calibration/grade_june_expiry.py

Enter fullscreen mode Exit fullscreen mode

Frozen inputs live in calibration/june_2026_forecasts.json.

Why bother publishing a bad score

Because the alternative is the usual ML options blog post: a backtest, a vibe, a screenshot of a green P&L. Publishing prob_itm with a fixed expiry is a tiny discipline. Grading it after the fact is the second half of that discipline.

If you log your own forecasts with recipe 02, you can run the same scorer when your contracts expire. I’ll keep adding rows.

—

Also adjacent, if you’re here for news rather than options: we published a free 212×37 news-outlet framing corpus with an explorer. Different animal. Same “put numbers where people can grade them” instinct.