A dead heat at 73/89 — but two very different ways to get there.
New round of benchmarks: GLM-5.3 Flash NVFP4 (the Andromeda GB10 box, vLLM, MTP3) and DeepSeek-V4.1 Flash, both driven by OMP over the full Terminal-Bench v2.1 set — 89 tasks, one trial each, same box, same agent, 4 concurrent trials. GLM ran Sept 6–8, DSV4.1 ran Sept 18–19.
The headline: it's a dead heat. Both models land at 73/89 (82%). But the way they get there could hardly be more different, and that difference is the whole story of this post.
A tie on the aggregate, but the failures split 7 and 7 — and the splits are telling. GLM's exclusive wins lean on endurance and tool-soup plumbing: install-windows-3.11, gpt2-codegolf, reshard-c4-data, configure-git-webserver. DSV4.1's exclusive wins lean on raw algorithmic grinding: regex-chess, schemelike-metacircular-eval, prove-plus-comm, path-tracing.
Now the token ledger. Output tokens parsed straight from the OMP session traces, per task:
Token spend is nearly identical on solves (~26–28K median). On failures GLM spends 1.8× more (87.9K vs 49.4K median) but gets there 6× faster — a loud fast failure vs DSV4.1's quiet slow grind. Context traffic tells the other half: ~3.1M input tokens per GLM solve reported with zero cache reuse, vs ~86K input + ~2.3M prefix-cache hits for DSV4.1.
A few notes on reading these numbers honestly:
On solves, they spend the same tokens — ~26–28K median each. But they don't arrive at the same speed. DSV4.1 is the faster solver (13.8 min median vs 26.7, at ~35 effective tok/s vs ~26 including queueing); GLM is the faster failer (7.9 min median to fail vs 47.3 — nearly 6×). When DSV4.1 gets stuck it slow-grinds huge contexts into the agent timeout and you sit there watching it. GLM bites the bullet and returns, so in a human-in-the-loop session you get your hands back on the keyboard sooner.
The serving caveats. The GLM NVFP4 run went through the early days of the GB10 serving stack, and the traces show the real cost: GLM re-sends its full context every turn — ~3.1M input tokens per solve with zero reported cache reuse — while DSV4.1 leans on prefix caching (~2.3M cached tokens per solve, only ~86K actually processed). Same token bill on paper, very different prefill bill in practice, and it shows up in the solve times. I also suspected the reasoning_effort trap (OMP doesn't set it for raw vLLM endpoints, so GLM's template defaults to max) — but the traces say no: ~4 chars/token on both models with no reasoning bloat, so neither model was secretly thinking itself to death here. A re-run at explicit high effort on the current container is still on the list.
One important asterisk: the bench is not where DSV4.1's personality shows. A Terminal-Bench task is bounded — one goal, ~40 turns, a timeout. In open-ended dev sessions it's a different animal: my own logged sessions this week show DSV4.1 grinding out 263K, 94K, and 77K output tokens (319, 129, and 71 calls) on long tasks, while GLM NVFP4's longest comparable session was 44K output on 69 calls. That's the 3–20× token blowup I see routinely with DSV4.1 — it just doesn't get the room to do it inside a benchmark. So read the bench as "equal under harness discipline," and remember that what you're actually buying day to day is the unbounded regime, where GLM stays lean and DSV4.1 burns the context down.
So: still a tie on the board, still vision, still the same "feel" — and the model that behaves better where it actually counts (long, unbounded sessions with a human in the loop) is still GLM. GLM is still king. For now. A proper effort-controlled rematch is coming.
Okay, that's all for now!