A Datacenter at Home

Both memory pools sit in one coherent address space. The CPU can touch HBM directly, and the GPU can dereference host pointers through UVA and read LPDDR5X from inside a kernel, no staging buffers or copies. This is what lets dsv4.1 flash keep expert weights and the Engram n-gram tables in Grace RAM and read them in place from device code.

Update: As of the time of publishing this too, DeepSeek just released an Oct 17th version of DSV4.1F (1017), and it looks to be legitimately the best open source model overall now.

DeepSeek-V4.1 Flash was the first model I brought up at full scale on this box. It is a big mixture-of-experts model: 384 routed experts per layer across 40 layers, plus two Engram n-gram tables it reads like an addressable memory. None of that fits in 252 GiB of HBM.

Instead, we're using a recipe from EL8 from github.

This model is a full DGX Station experience, where the GB300 holds 230GB worth of model weights, KV cache, and 285 experts. The RTX Pro 6K holds 99 of the cold experts, and then the Grace LPDDR5X RAM holds the 254GB ngram table along with 139GB of page cache.

The results for this recipe are very nice, and definitely would make me suggest to anyone buying a DGX Station that they definitely put an RTX Pro 6K in there too.

decode throughput vs concurrency

Based on these findings, I'd say the sweet spot is serving an average of 16 concurrent requests. 32, 64, and 128 are all possible, but you get no further aggregate token gain.

Besides being fast tok/sec decode, prefill is boringly consistent, but also insanely fast:

These prefill speeds are exceptional. As high as 60K+ tok/sec, where normally I see 10-15K on 4x RTX Pro 6Ks.

Here, we can see that actually on concurrency 1, we are slightly faster locally, but then after 2-4 or more, the DGX Station begins to perform better, but the cold prefills are again staggeringly different.

And how hot did all of this get? The whole sweep carries a 2-second telemetry loop giving us 2,005 samples across the GB300, the RTX sidecar, and the Grace CPU. Peak 75 °C on the GB300 is...impressive to say the least. That's cold for a 1000+W GPU.

Okay, so it's clear the DGX Station with an RTX Pro 6K is quite enjoyable, but what about a model that runs purely on the HBM of the GB300 instead?

The QAD checkpoint is local-inference-lab's GLM-5.3-Flash-NVFP4-MXFP8-CSF-QAD, revision dec48abd, 179 GB across 36 shards. CSF, the third name in the list, is the container: a lossless compression format that packs the routed experts' block scales so tightly that every original byte comes back out, and we re-inflate all 36,288 of them on the CPU at load. QAD is quantization-aware distillation.

The sales pitch of QAD is performance closer to full BF16 than NVFP4, and faster.

Lately, GLM 5.3 Flash has been my favorite model, but the latest Deepseek V4.1 Flash 1017 model is probably my new favorite. That said, it doesn't appear that the GLM 5.3 Flash architecture benefits much from this machine.

The interesting thing I've found so far though is even models that aren't stellar on this machine, they're still just as good as on Pro 6Ks, but when a model works well on the GB300/DGX Station, it really flies. Let's check out a model that performs literally the best I've ever seen any model scale out in concurrency next.

Nemotron 3.5 Super VL is a 120B-parameter hybrid MoE with 12B active per token: 512 routed experts, 22 of them active, alternating Mamba, MoE and attention blocks. NVIDIA's EA NVFP4 checkpoint fits on one GB300 at TP1/PP1 with MTP3, FP8 KV. Room for 28M KV pool. It is a vision-language model and good at agentic coding tasks. But the real incredible part is how well this model scales to extreme concurrency.

On the GB300, codegen aggregate decode goes from 524 tok/s at C1 to 7,282 tok/s at C64. That is 13.9× the throughput for 64× the concurrency, so the median stream only falls from 538 to 130 tok/s. The model is not free at high concurrency, but it is remarkably close to linear and I have just never seen a model scale this well. Most models tend to scale out decently to 16ish concurrents, but this model just keeps giving on the GB300.

Prefill is also lighting fast:

This is not the smartest model of the open source models, but the ability to work on agent swarms is exceptionally unique and this is no slouch of a model either. I plan to dig in deeper into using agent swarms with this model, and I am excited to see some of the next models from NVIDIA and if any other model provider ends up offering something like this one that scales in this way.

Obviously, compute's getting expensive and we're all sort of adjusting to a new normal of pricing, but that compute is getting expensive because it's also getting a heck of a lot more useful, especially with local models that you can run and achieve much faster performance than via APIs, protect IP, and not deal with random outages out of your control. Since having the ability to run the models shown here, I've moved almost entirely off API/frontier models for my own dev work.

With that in mind, the DGX station is a really cool offering that I would definitely recommend for professional/enterprise use. I do like my custom 4x RTX Pro 6K build, it is cheaper, but it's wires and cables going everywhere, takes up more space than this, draws more power, generates more heat, and makes more noise from fans dissipating that heat.

I will be certainly very sad to send this incredible machine back home!