DeepSeek Does It Again

DeepSeek shipped the 1017 checkpoint for their 4.1 Flash model on October 17. In the last post I called it as likely the best open model, but as usual I wanted to spend a bit more time with it. If you've followed along my frontier at home journey, then you know I was a big fan of DSV4F, especially when DSpark came along, because of how crazy fast the decode speeds became. But then, when GLM 5.3 Flash came around, I felt the GLM flash model was simply better and that DSV4F 0731 and the new 4.1 Flash were both extreme token abusers that took 5-15x more time and tokens to do the same work as GLM 5.3 Flash.

But, with the DSV4.1F-1017 variant, Deepseek really has nailed the balance perfectly. This model still does use a lot of tokens, still more than GLM 5.3 Flash, but I think the tradeoff is worth it. This is the first local model that feels basically identical to the frontier models, so much so that I really haven't even thought about Astra since this model came out. I did try to use Astra to solve one AI model conversion for the specific hardware (GB300) that DSV4.1F-1017 was struggling to solve, only to find that Astra too was struggling just as much and honestly both models made the same progress. I still think Astra is likely a better overall model, but Astra might be nerfed for AI work, similar to Claude's content limitations (and all the more reason to just use local models).

First, the benchmarks and where DSV4.1F-1017 sits:

DSV4.1F-1017 lands at 82/89 (92%), the highest score in this chart, ahead of GLM-5.3 Flash NVFP4 and the pre-1017 DSV4.1 Flash at 73/89 (82%).

Notably, that's both really good and we're running out of headroom to continue comparing models at this point on Terminal Bench v2.1! I have always just wanted this to be a reference check, and I will still use Terminal Bench v2.1 probably for smaller/faster models, but now I've also added Terminal Bench 4.0 to the mix, testing DSV4.1F-1017 and GLM 5.3 Flash so far, which will be the new baseline:

On TB4.0, GLM-5.3 Flash leads at 27/63 (43%) while DSV4.1F-1017 lands at 21/63 (33%), which is a result I didn't quite expect. I am not entirely certain what to make of this difference, I still feel as though DSV4.1F-1017 is the superior model, but I think it's still of note just how impressive GLM 5.3F really is. Also these are just 2 similar benchmarks of course.

Now, on to model speeds:

There are often many variables involved with these models running depending on hardware and software. One really good place to check and compare your performance is: Local Inference Lab's benchmarks. You can see what numbers people hit with various configurations as well as run the exact same bench on your hardware to see if you can figure out how to get a bit more perf.

So again, for a lot cheaper, you can make GLM 5.3 Flash go a lot faster, and it's a model that uses way fewer tokens, so GLM 5.3 Flash is still a very strong model that I wouldn't hesitate to use, but I do still think DSV4.1F-1017 pulls ahead in my real usage. Where DSV4.1F used to just overthink and re-do work, DSV4.1F-1017 just simply keeps working until the job is done, and in this way it feels very much like the closed source frontier models where you can give it a task and trust that it will get it done with very little back-and-forth.

The areas that I actually use opensource models in with real work are: robotics/simulation, AI model inference/serving, and lately 3D modeling mainly around 3D printing. DSV4.1F is a very intelligent-visually model, capable of doing 3D modeling for you, controlling robots with vision, and doing a lot of AI work. Recently, I was trying to do convert a specific quant of DSV4.1F-1017 to run on the GB300 completely in HBM, and this had DSV4.1F struggling to get it done, so I tried Astra for a bit and found it was actually worse than DSV4.1F-1017 at the same problem! This could be due to both OAI and Anthropic trying to limit people using their models to improve AI and do research like this, but the only time I ever felt like reaching for a frontier model, that model turned out to not be useful, for what it's worth

On that note of a DSV4.1F 1017 model that could fit entirely on the HBM of the Asus DGX station/GB300, I also wanted to explore a quant for DSV4.1F-1017 that is specced to fit on just 2 RTX Pro 6K GPUs. I recently got a Dell T4 with 2 RTX Pro 6K MaxQ GPUs, and I was curious how the quantized model might do. Lately, I've begun to think quantizing is not the way to go, but if there's a quant from Brandon Music, I have to at least try it!

Before running any benchmarks, I switched by local model to the DSV4.1F-1017-TP2 TrellisMX K2 Mix to see if I could notice any difference in capability. I DID notice some repetition in thinking. At no point did the model ever get stuck in a real loop, and the repeating thoughts were new/novel and useful, but they all started with things like "Need maybe ..." and then some more text. There might be a tower of 20+ of these. I think the only downside here was it might be using more tokens than necessary in thinking, but it was never an actual result degredation for me (at least not so far).

Very shocked by the result here, but that's what it is. I even thought maybe I was running a TP4 maybe, but I had no machine setup to even fit the native DSV4.1F-1017 model at the time. I was fully expecting to have to do more explanation and defending about how benchmarks aren't the only measurement and don't measure retaining the same human in the loop, but dang, it scored just about the same super high TB2.1 score. Very impressive. This makes me want to spend even more time to make this model fly on the GB300!

The main question I had with this test was if it would be better than GLM 5.3 Flash NVFP4, which also runs on just 2 RTX Pro 6Ks, and I think I have my answer. I also have felt in my own usage that it is likely better than GLM 5.3 Flash.

If you are interested in actually running this model, you should definitely check out the Local Inference Lab's benchmarks to see what #s other people get with what configs. I have so far been unsuccessful to get the DSV4.1F quantized for TP2 variant to run well on the Asus DGX Station. It fits, but it's just not significantly faster than 2 rtx pro 6ks. I know there's a lot of perf being left on the table, but I am not yet sure how to unlock it.

There are also ways to overclock the RTX Pro 6Ks to achieve even faster decoden in some configurations and in some cases the TP4 DSV4.1F-1017 codegen can hit speeds of 700+ tok/sec, which is insane.

I used to focus on codegen a lot in my comparisons, but I am starting to think maybe that's not the correct way of thinking since even if you're doing mainly coding, you're still actually having a model do mostly regular thinking/planning...etc, so I'll potentially stop referencing decode speeds as codegen into the future, but I haven't fully decided that yet.