A New Model Type Emerges

We're going to be a bit all over the map here, first some new benchmarks with GLM 5.3 Flash NVFP4, then on to some new benchmarks using turn-based strategy games, then finally on the topic of Jev and general decision models.

Since the last post, I was relegated back to using 3 RTX Pro 6Ks instead of 4, due to one dying. This meant I could not run native precision GLM 5.3 Flash and instead needed to run NVFP4. I figured, while I was there, might as well benchmark it:

Aaaaand somehow the NVFP4 GLM 5.3 Flash model became the new #1 model out of everything benchmarked locally so far. A very surprising result to say the least! I think the "norm" these days is to run terminal bench 3 times and then average the scores, but I am mostly just after a general feel for the model as well as use the benchmark as a way to measure other metrics like speeds, time to solution/failure...etc, but this was still quite shocking.

Then, Deepseek v4.1 Flash also came out, and this was proving to be a very popular model among developers. I did try using it on some APIs a few times, but to be honest I felt like this model was not enjoyable. The amount of thinking, rethinking, doing, redoing, rethinking...etc was just too obnoxious. But still, I did benchmark it too, and it also scored the same as GLM 5.3 Flash, which is interesting to say the least!

Despite slightly different results in what they passed/failed, I would say these models are likely similarly intelligent, but just like DSV4F, there's more to it than the final score, it's how did we get there? Deepseek V4.1 Flash might be fast, but it uses 3-15X more tokens on real problems that I've tested compared with GLM 5.3 Flash. I see truly no upside to using DSV4.1F.

Using GLM 5.3 Flash, I never find myself wanting to pull out GPT-6 Astra, except in 1 case of needing to fix a broken windows install on a machine that also had a linux enviornment that I cared about. I pulled out Astra out of caution there. For truly sensitive things like operating NAS commands or something, I would still use either of them as an advisor instead of letting them run themselves.

To help visualize my personal findings with GLM 5.3 Flash vs DSV4.1 Flash, here's a real example of me trying to use both to make a YouTube thumbnail for a video. It's a relatively simple image editing type of task and I find this to be an indicative example of my overall experience using GLM 5.3 Flash and DSV4.1 Flash:

To actually complete the task: DeepSeek V4.1 Flash uses: 115K tokens and takes 25 minutes. GLM 5.3 Flash uses 21K tokens and takes 1.5 minutes. DeepSeek repeatedly rethinks and re-does its work. In real work, a lot of time is spent actually doing a task, as opposed to pure token gen, so the deepseek behavior is just substantially punishing, both in more tokens used but also in other wall clock times.

Ask for more variance/options/tweaking, another 2:22, bringing total to 31K tokens

I can imagine many people will assert that the end result from DeepSeek resulted in more/better options. This is obviously an opinion, and I do think I would like multiple versions to pick from and iterate on. If you wanted that originally though, you could just say that in your original prompt. This would both make GLM 5.3 Flash likely just do that, and it MAY notify DSV4.1 Flash to stop earlier and not do its own iteration.

Regardless, asking GLM 5.3 Flash to give us more options and make some tweaks, costs us another 2:22 and brings our token total to 31K.

While subjective, I also preferred the thumbnails DeepSeek had around the 5 minute and 10 minute marks, but DeepSeek chose to continue iteration instead, totally redoing the work and erasing the better options.

I just personally find GLM 5.3 Flash to be a faster and better experience overall. I know many people still love DSV4.1 Flash, and if you've tried GLM 5.3 Flash and still think DSV4.1 F is better, you're ofc wrong, but that's okay!

Turn-based strategy game benchmarks

Lately, traditional benchmarks like Terminal Bench v2.1 are seemingly doing a poorer and poorer job of helping us to understand the differences of various models. Sure, deepseek v4.1 flash and glm 5.3 flash are scoring the same on terminal bench v2.1, but IMO they are vastly different models. GLM 5.3 Flash and DSV4.1 Flash also score awfully close to a model like GPT-6 Astra, and it's hard to really quantify how different they really are. I began looking into some of the older turn-based strategy games to help maybe show this visually. All of these models are good at generating words and code, but there's a substantial delta in the planning and strategy capabilities when working on projects, and I wondered if maybe this translates well to games.

I think it does.

First, we have some GLM 5.3 Flash vs DSV4.1 Flash, just to add more insult to injury:

Once again, we can see how heavily DSV4.1 Flash just wants to slurp up tokens. 6 games isn't enough to call this fairly, but it's still an interesting result just how much better GLM 5.3 Flash appears to be so far. I can only currently run 1 model locally at the moment, so the rest is happening over API. I am not sure just how much money I want to put into each benchmark this way. I may also wind up hosting a 2nd model on the GB10s or something like this potentially. Haven't decided yet. I have many other strategy games I'd like to consider too.

Okay, now some games against GPT 5.6 Sol. Same engine, same 96×96 maps, same replay-verified harness, just swapping the opponent. GLM went 12 for 12 against it, and honestly it wasn't close: GLM spent about 71k tokens across all twelve games while sol burned through 4.4 million trying to keep up. Every single game ends the same way. I found this result particularly interesting. I am actually inclined to believe this is a result of GPT 5.6 Sol being a passive model, it does not want to be aggressive. While the game is a loss, it's still an interesting model characteristic that has emerged.

And then there's GPT-6-astra...where GPT 5.6 Sol was a pacifist, Astra is quite aggressive. In all games, besides just Halite, Astra is dominant, and it's been very interesting to see the stark difference between 5.6 Sol and Astra! Still though, GLM does get a couple wins. I really would like to do many more games against Astra, but they get quite costly.

General Purpose Decision Models. Jev and OpenJev

Right as I was developing the strategy game benchmarks, a new challenger showed up: Jev. Jev is a new style of model meant for classification, essentially. You can probably just call it a decision model. These models do not generate tokens auto regressively and instead give a near-instant response to a decision. But, unlikely historical classifiers, these are generally intelligent and general purpose, ready to be applied immediately to a task. Currently, they lack vision, but this will likely be coming soon. An obvious usecase here is with robotics, but I have not currently been able to produce a compelling example where using Jev is better than just using whatever vision-enabled model you currently use. But, in the case of strategy games and any workflow that mimics it, the power really shines.

I feel like these decision models are where LLMs were pre-chatGPT. You can tell there's *something* here. It's unclear yet exactly where they'll fit in, but they have clear and obvious value, so I'll show at least some examples. First though, a bit about how they work. We dont know precisely how Jev itself works, it's a closed model, but we do know how OpenJev/SemIf works.

How OpenJev/SemIf actually works

SemIf is an independent open-source project (it was formerly named OpenJev, and it is not affiliated with Jev or TypeSafe). It reproduces Jev's interface pattern using open models. In our deployment that model is Qwen3.5-4B (BF16, pinned revision) sitting in 8 GB of VRAM. It never generates anything: no tokens are ever decoded. You send a state, a question, and 2–16 described options. The prompt is built so that the next token after the prompt is the "letter" of an option, and the model does exactly one forward pass. The code then reads the logits of the option tokens directly off the final layer and softmaxes just those. The probability distribution over the option tokens is the decision.

A few things matter here. First, the options menu is yours, not the model's. It can only choose from what you list, so illegal outputs are structurally impossible. Second, the numbers are not confidence: a 0.75 is not "75% likely to be right." What is usable is the margin. Across our test fixture, wrong answers averaged a max-probability around 0.70 while correct ones averaged 0.87, so the working pattern is: execute when max-prob ≥ 0.9, escalate borderline decisions to a bigger model. Third, the speed difference is structural: on a 42-row real-operations fixture, SemIf scored 90.5% at ~0.06 s per decision while a full GLM free-form answer scored 66.7% at ~2 s, roughly 30× slower and less accurate on the same decisions. Agent loops can also amortize prefill: when many decisions share one long state (one game turn with 21 independent questions, for example), the state is prefilled once and the question suffixes branch off that shared prefill. Five questions take ~134 ms total, and the 21-question case is the same order of magnitude.

I repeatedly found that we could decrease token spend/compute, speed up overall tasks significantly, AND increase performance overall just by using Jev/SemIf. In my limited testing, I actually found SemIf to perform better than the official Jev, which is very interesting. Here are some of those results:

Decision models in production: the Halite II games

The same harness that ran the head-to-head benchmarks above also ran decision-model matchups on Halite II. Three of them are worth playing. All replays are fully client-side; each cell shows final ships, damage, tokens burned, and mean turn time per side, and the token bars fill as the game progresses.

As you can see, Jev is capable of winning, but not reliably. Almost certainly because it's not really capable of being a good planner/strategist.

Adding Jev to a planning model, however, we get some interesting results. In this case, we retain GLM 5.3 Flash as the planner/strategist, but use Jev for individual ship decisions.

The addition of Jev is 13x faster on average, 44% "cheaper" if we're going off inference API prices, and performs better.

Around this time, "OpenJev" was released (now SemIf), and I decided to try that:

Interestingly enough, the OpenJev/SemIf Hybrid does *better* than the Jev one.

Why not just do this with a vision model?

Real tasks have pixels, so the obvious question is whether this works with a vision model. Mechanically, nothing about the readout is text-only. A vision-language model is still a causal transformer emitting next-token logits at the end, so reading option-token logits off a VLM is just as possible as off Qwen3.5-4B. What changes is the cost around that forward pass:

• The prefill bill. SemIf's speed comes from a tiny prompt: ~119 text tokens through a 4B model in one cheap pass. An image becomes hundreds to thousands of vision tokens that must pass through the vision encoder and the LLM prefill on every single decision. On our own serving stack, per-frame GLM vision calls take whole seconds, not 60 milliseconds, and the decision models we're benchmarking (Jev at ~0.4 s flat per call regardless of question count) only make sense if per-decision cost stays flat. A vision-encoder prefill per decision breaks that.
• Scale. The VLMs good enough to be worth consulting are big. GLM-5.3 Flash is a 320B-parameter MoE served across four GPUs; you do not keep one resident per decision loop the way a 4B model fits in 8 GB.
• Logit access. Hosted APIs generally do not hand you raw logits at all, so this technique requires self-hosting the model, which compounds problem 2.

I tested this directly. GLM-5.3-Flash (NVFP4, self-hosted on our vLLM box) on the SemIf repo's 144-row authored classification fixture, same rows, same balanced-accuracy metric the project publishes:

While you can disable the thinking, GLM 5.3 Flash is just seems to be a model that wants to reason and think. It makes sense, it was tuned and a lot of effort went into this. The 4B Qwen model just seems to be a better fit. In time, I expect someone to train a model specifically for SemIf, especially also with vision. Looking forward to seeing that! I was also able to get vision with GLM 5.3 Flash + SemIf working, but GLM 5.3 Flash desperately wants to wax poetic about images first, so we really do need a model that isn't quite tuned like this. Maybe even a fine-tune is all we need, but I was hoping to not need to run multiple models.

Okay, that's all for now!