The Right Harness Is All You Need
GLM-5.2 FP8 · IQ4_NL · Q2_K_XL · 8bit-KV · 3.25 BPW (EXL3) · 3.25 BPW OMP (EXL3) · DeepSeek-V4-Flash · DeepSeek-V4-Flash 0731 · DeepSeek-V4-Flash 0731 OMP · MiniMax-M3 · Qwen3.6 27B · Laguna S 2.1 · DSpark
The model roster keeps growing. Since the last post, I've added Qwen 3.6 27B NVFP4,
Poolside Laguna S 2.1 NVFP4 to the lineup —, most interestingly, a very specific
community quant of GLM 5.2 that's been the standout of this round:
willfalco/GLM-5.2-EXL3-TR3-3.25bpw, and finally, an update to Deepseek V4 Flash, called 0731.
I originally had this post written as "all roads lead back to GLM 5.2." Despite Poolside's Laguna S 2.1 benchmarks and Deepseek's DSV4F-0731 benchmarks, I was simply unable to replicate them. The main questionmark around both were that they were benchmarked around the makers' very specific harnesses. DSV4F-0731 was claiming a better-than-GLM5.2 benchmark list, which I found very tough to believe and simply could not replicate. I almost tossed these models into the bin for good, which was sad to me because I did over one BILLION tokens locally on DSV4F-preview. I loved that model! But, when benchmarking it, and using it, I found it to seriously under-perform even DSV4F-preview!
It wasn't all bad though, we got our PCIe switch and other hardware working:
I was writing about model makers overfitting to their harnesses and overfitting those harnesses to benchmarks, but then I just kept hearing the name: OMP. Yet another harness on the block that people were yapping about and I decided to give it a shot with my new love: GLM 5.2 3.25 BPW EXL3 (quite the mouthful).
And man, it sure felt good using OMP with GLM 5.2. OMP definitely helps to smooth out many issues with longer-horizon tasking with agents/LLMs. It was repeatedly one-shotting my requests! But it was using a LOT of time, turns, and tokens to do it, as compared to my personal coding harness, minion. So then I wondered, well, let's try DSV4F-0731. People still wont shut up about it online, it's gotta be somewhat decent but it is definitely being hyped by a lot of people who never said a peep about DSV4F preview.
Dang, it felt good in OMP. I was still one-shotting my problems and a TON faster. GLM 5.2 3.25 BPW runs ~70-80 tok/sec for me. DSV4F-0731 runs 300-400 tok/sec. But DSV4F-0731 wasn't immediately running fast for me, there was some issue that was causing TP=4 to be quite slow. I started off by having DSV4F-0731 first fix my tok/sec, prefill speeds, and issuess that were causing TTFT (time to first token) to be quite delayed per turn. I just could tell it wasn't snappy like DSV4F-preview was and it's basically the same model so this was confusing. OMP and DSV4F-0731 solved this, it was mostly just a backend issue, but even GPT 5.6 Sol couldn't figure it out fwiw.
Well... shoot. That means I really better benchmark DSV4F-0731 with OMP huh. After doing that, I realized I needed to bench GLM 5.2 3.25 BPW too on OMP. So I did that too.
A bit about this model:
| Weights | KV format | KLD |
|---|---|---|
| NF3 | Dynamic NVFP4 + RoPE8 | 0.139036 ± 0.002010 (n=3) |
| NF3 | Standard FP8 + BF16 RoPE | 0.1263 † |
| EXL3 3.0-bpw | Dynamic NVFP4 + RoPE8 | 0.119525 |
| EXL3 3.0-bpw | Standard FP8 + BF16 RoPE | 0.102508 |
| EXL3 3.25-bpw | Dynamic NVFP4 + RoPE8 | 0.095971 |
| EXL3 3.25-bpw | Standard FP8 + BF16 RoPE | 0.087711 |
| EXL3 3.36-bpw | Dynamic NVFP4 + RoPE8 | 0.077767 |
| EXL3 3.36-bpw | Standard FP8 + BF16 RoPE | 0.068458 |
The 3.25 BPW variant in the table above is the one driving our current top score. It's
willfalco/GLM-5.2-EXL3-TR3-3.25bpw
on Hugging Face. There's also a slightly-larger sibling,
willfalco/GLM-5.2-EXL3-TR3-3.36bpw,
which is on the roadmap but not something I'm rushing to test just yet. No question the KLD is better overall for the 3.36, but
the 3.25 was cherrypicked for coding specifically. I believe the 3.36 is focused more on overall KLD. I am certain both are great though and I *am* curious about 3.36 too! Just only so much time and I prefer to run models both thru Terminal Bench V2.1 but also actually drive as a main model before giving my thoughts. As I write this, there's a 3.40 BPW, 3.5 BPW and probably others too FYI.
The EXL3 is short for EXLlamaV3. It's a quantization format targeted for a specific CUDA inference engine. EXL3 also allows for mixed precision.
TR3 is tree based quantization, specific to EXL3.
3.25 BPW is 3.25 bits per weight, on average. Typically, more precision = better, but also slower. These particular models are highly optimized to have more bits for the things we care about and less for the things we don't. But this is all obviously a "taste" type of thing about what you'd pick.
Okay whatever, what're the results?!
The 3.25 BPW EXL3 GLM 5.2 comes in at #1 now, and under the OMP harness it does even better: 73% overall (65/89) versus 69% (61/89) on the stock harness, which is very solid. This further drives home my personal suspicion that GLM 5.2 "FP8" on OpenRouter was actually NVFP4. It's also very impressive just how good DSV4F-preview DSpark is/was, especially given it's speed as a model without needing any big harness to carry any weight. I've since run 0731 through the same OMP harness and added it to the charts below: 49% (44/89) on the stock harness, well below the preview DSV4F, but on OMP it jumps to 71.9% (64/89), edging out the preview! I do see lots of people like the 0731 version, and on this harness at least, the numbers back them up. That's a VERY solid score. Especially being basically neck and neck w/ GLM 5.2. With a simple harness, there really is no comparison between them, but with a fully-featured harness, suddenly, they're almost identical in performance!
I am currently seeing ~80 tok/sec with GLM 5.2, a far cry from the 300+ from DSV4F-DSpark-preview. I am not totally confident that I'll stick with GLM 5.2 over DSpark, especially given concurrency, but I am a big fan of GLM 5.2's intelligence, which seems to show through very nicely on this 3.25 BPW variant.
tokens/sec · decoded speed
Measured at 1 concurrent chat session — both hosted the same way (vLLM, TP=4). DSV4F is ~4.5× faster on decode: the whole game for interactive feel.
solved · what OMP buys you
OMP lifts DSV4F-0731 by +20 tasks (49.4% → 71.9%) and GLM 3.25 BPW by +4 tasks (68.5% → 73.0%) — the 0731 that looked like a dud on the stock harness is a different model under its right harness.
cost · normalized to GLM 5.2 stock = 1.0×
Same 75 tok/s both GLM runs, so tokens track time 1:1. Per solved task: OMP takes 2.2× the time of GLM 5.2 stock → 2.3× the tokens per win. One attempt per task (latest trial, pass or fail, retries excluded): the totals nearly converge because GLM 5.2 stock spent 65% of its attempt budget failing — 28 unsolved tasks at ~54m each (1505 · task-min vs 824 solving). Both panels are the same 89 shots per model; the first prices successes, the second prices attempting the whole benchmark. Absolute numbers assume ~50% of wall time is decode; the × ratios don’t depend on that split.
I found this to be very interesting overall. We don't actually see too many additional tokens from GLM 5.2 w/ OMP, but the time per solution is over double. I think a lot of this has to do with prefills probably, and maybe other factors. It definitely feels way slower in OMP, so maybe it's also more turns? Not sure here.
The bottom line is, it really does appear like with certain harnesses, DSV4F-0731 really is just as good as GLM 5.2 ~4bit. I went from thinking Deepseek was being very dishonest in their benchmark scores to running a test that I think *is* fair, using OMP for both, and finding out that yeah, it really can be just as good. At least on that one bench with this one harness.
I really do find that fascinating that OMP lifted DSV4F 0731 up so much, but only just barely moved GLM 5.2. Obviously DSV4F had a way more room to go up, but...still. Test time compute and harnesses obviously play a large role but it's interesting to see it illustrated objectively.
I will personally continue to run both DSV4F 0731 and GLM 5.2 through OMP and see if I really do feel like DSV4F is just as good. I have been extremely impressed by GLM 5.2's performance in OMP, other than it really is quite a bit slow. You absolutely must be doing many things at once or you're going to get bored and distracted.
Besides the model choices and testing, a lot of work went into actually achieving 4x RTX Pro 6K cards in any decently reliable and stable way. I ended up using a PCIe switch, MCIO cables, a retimer, and PCIe adapters for the GPUs.
The Gen 5 PCIe switch topology
The goal is to get all four GPUs talking to each other over PCIe Gen 5 peer-to-peer (P2P), regardless of what generation the host motherboard happens to be. That's what the PCIe switch and retimer are for, they let the cards form a fast private network that the host CPU only has to reach once. One caveat: this "motherboard generation doesn't matter" point holds for a single switch. If you start chaining multiple switches together, those switches communicate through the host, so the inter-switch link runs at whatever generation the motherboard's slot is and that's where a Gen 3 or Gen 4 board would actually start to bottleneck you.
Reading the diagram
The host motherboard contributes one PCIe slot per switch, and each switch gives 4 cards. That slot can be Gen 3, Gen 4, or Gen 5, it genuinely doesn't matter for the P2P performance of 4 cards, because the host is not in the hot path. The CPU only needs to reach the switch once to set things up; after that the GPUs talk directly to each other through the switch and never bounce off the host. The one exception is if you chain multiple switches: switch-to-switch traffic goes through the host, so then the slot generation does become the bottleneck for that link.
Plugged into that single slot is a retimer, bridging down from the motherboard slot. PCIe Gen 5 runs at 32 GT/s, and at that frequency a signal degrades quickly over distance, so the retimer conditions and re-drives the signal so it survives the cable run intact. From the retimer, two MCIO cables carry the signal into the input side of a PCIe Gen 5 switch. MCIO is just the physical cabling standard. It's a high-density, high-speed interconnect cabling that carries PCIe lanes over a cable instead of board traces. The speed of that retimer-to-switch link depends on what the connection to the retimer is actually negotiated at, not necessarily Gen 5.
The switch is the heart of it. On its output side there are more MCIO connectors, fanning out to the four GPUs, with two MCIO cables per GPU. Because a GPU's physical slot is PCIe, not MCIO, each GPU sits behind an MCIO-to-PCIe x16 adapter that translates the cable back into a standard x16 slot the card can plug into. The important part: every link from the switch out to the cards, and every card-to-card path through the switch, is Gen 5. So even if the motherboard is only Gen 3 or Gen 4, the four GPUs are speaking to each other at full Gen 5 P2P, unless you have something like two switches and now you need those 2 switches to talk to eachother through the motherboard.
ReBAR and Above 4G Decoding
There are two BIOS settings you really want, or everything's nerfed. The problem is many motherboard BIOS are lacking one or both.
ReBAR (Resizable BAR) lets the CPU map a GPU's entire VRAM as one big contiguous addressable block, instead of the legacy approach of peeking through a tiny 256 MB aperture window. For normal single-GPU work that's an optimization; for P2P between multiple GPUs it's close to a requirement. Without ReBAR, large memory-mapped transfers and GPU-to-GPU copies get chopped up to fit through that 256 MB window and bandwidth collapses. With it, the CPU can hand the GPUs each other's full memory maps and let them transfer directly.
Above 4G Decoding is the other one. By default, legacy PCIe addressing only assigns device memory below the 4 GB mark. Every modern GPU has far more than 4 GB of memory, and the switch and its address space sit well above that line too, so without 64-bit (above-4G) address decoding enabled the BIOS won't even map the devices correctly and the topology won't come up. Both settings have to be ON.
Get those two right, and the generation of the host slot stops mattering as much, other than for things like speed of model loading. The cards form their own Gen 5 fabric through the switch and the host is just a spectator that set up the initial mappings.
My personal struggle here was finding a motherboard that I wanted to "dedicate to the cause" and that also had these BIOS settings. I started off trying to use an older Puget Systems machine, because it has quite a few PCIe Gen 4 x16 slots, a terabyte of RAM, and just plain made a lot of sense to dedicate to being the AI server. But this machine totally crapped its pants when I plugged 3 and 4 GPUs into the switch. 2 was fine. And, interestingly, this is the machine I was running 3x RTX Pro 6K in directly. But 3 in the switch was too much. I got stuck in a boot loop and ended up reseating the CPU to no avail and then later deciding it was probably the BAR and discovered there was no available BIOS for the supermicro board. But also simply could not update the bios to the latest version, probably because it was many years old and the leap I was asking it to make was too high, but supermicro doesn't seem to make older BIOS available.
Then I tried to use a Dell T2 tower, because that machine has a Gen 5 slot and it's definitely nice to have gen 5 model loading speeds when you're testing and tinkering with lots of models. But, for some reason, despite that machine having ReBAR and Above 4G decoding, the switch would just immediately cut off as soon as I booted into an OS. I did want to try degrading that slot to PCIe 4 or 3, but this isn't an option in that Dell proprietary BIOS. Not sure what else to check.
Then I tried an older dual CPU PCIe 3.0 machine from Boxx, similar story, no ReBAR and a supermicro motherboard with no good support. It would at least load everything, but performance was nerfed. Then I tried an older workstation, it wouldn't even post with the switch.
Then I decided to dedicate my current (at the time) work machine. That's a PCIe gen 4 board, with ReBAR and Above 4G decoding in BIOS. It also had the added benefit of keeping the switch on when the OS loaded. So I ended up dedicating my workstation to the cause, and taking the Dell T2 tower as my new main work computer. Loaded up the GPU mini rack into the full size server rack, and have been enjoying GLM 5.2 3.25 && DSV4F-0731 mostly since.
I am sure that's the last of epic model releases and hardware changes and now I can just sit back and enjoy my models!