Most model comparisons try to answer a question that is too broad: “Which model is best?” We wanted a smaller, reproducible question instead:
What happens when eight current Chinese AI models receive the same coding task through the same agent?
We used Pi as the coding agent and ran one isolated JavaScript task across these models:
- GLM-5.3
- GLM-5.3-Flash
- Kimi K3
- Qwen3.8-Max
- Qwen3.8-Flash
- DeepSeek-V4-Flash
- DeepSeek-V4-Pro
- MiniMax-M3
Method
Each run started from its own copy of the fixture. The test directory was kept unchanged, the work directory was checked for unexpected files, and raw run evidence was stored separately from the task workspace. The same Pi provider configuration and task contract were used for every model.
This is deliberately a narrow test. It does not measure architecture work, long-horizon debugging, frontend judgment, or performance on a real production repository.
What we measured
All eight models completed the task successfully. Across the complete run we recorded:
- 8/8 passing runs
- 45 Pi requests
- 94,502 tokens
- $0.037618 production-audited billed cost
The public page includes the model-by-model table, runtime and token measurements, the Pi configuration, methodology notes, and a downloadable JSON file:
View the full benchmark and data
Why publish a small benchmark?
A small benchmark cannot tell you which model is generally better. It can still answer useful operational questions:
- Can every model finish the same concrete agent task?
- How many agent requests does the run take?
- How many tokens are consumed?
- Does the billed amount match the evidence in production logs?
- Can someone else inspect the setup rather than trusting a screenshot?
For this task, the answer to the first question was yes for all eight models. The differences are in the detailed run data, not a winner label.
A note on cost
The total above is the audited amount recorded for these eight runs, not a forecast for arbitrary coding work. Agent cost depends heavily on task length, retries, context growth, and tool behavior. A real repository can be much more expensive than this small fixture.
Reproducing it
The benchmark page includes the Pi setup and downloadable structured results. If you repeat it, keep the task, tests, agent version, model IDs, timeout, and evidence rules fixed. Otherwise you are comparing different experiments.
Disclosure: I operate Vancine, the OpenAI-compatible API used for these runs. The page is published as product evidence, and the result should not be read as a general model ranking.
I would especially value feedback on the harness and on what the next coding-agent task should test.