by Chris DePuy / September 12, 2026
On September 10, DeepSeek released DeepSeek-V4.1-Flash, a multimodal mixture-of-experts model with a 552-billion-parameter backbone and a context window of one million tokens. The launch kits for NVIDIA’s desk-side DGX Spark machines arrived the same day: a four-machine stack on vLLM was serving requests by 9:17 Eastern time, roughly seven hours after the weights became available, and a three-machine stack on SGLang published its measured tables overnight. I think the useful story is not the benchmark sheet but the fitting: what made a 500-billion-class model usable on desk hardware was an accounting change, moving roughly 200 gigabytes of conditional memory out of unified memory and onto local disk.
DeepSeek: the model card, and the KV story inside it
The model card describes a 40-layer causal encoder-decoder, 20 encoder layers and 20 decoder layers, that activates 8 billion parameters per token during prefill and 16 billion during decode, drawing on 384 routed experts per layer with six active. Conditional memory is the headline trick: an Engram table of 196 billion parameters sparsely accessed by token-based lookup, and the component that does not fit — more about that below. Compressed Sparse Attention 2 shares key and indexer state across layers, and with FP4 key-value caching the card accounts the global footprint at 890 bytes per token, roughly a quarter of DeepSeek-V4-Flash and some 437 times smaller than the first generation. The model was trained from scratch on 45 trillion multimodal tokens, with the context extended to one million at 34 trillion, and the weights carry an MIT license. On the agentic scoreboard the card lists Terminal-Bench 2.1 at 90.6 and DeepSWE v1.1 at 74.2. I read the 890-bytes-per-token line as the number that matters on this class of box: at full context that footprint is a little under 0.9 GB of the pool, and the SGLang recipe’s own ledger — which replicates the latent across tensor-parallel ranks — prices it at 1,670.75 bytes per token per rank, about 1.7 GB for a request at the model’s maximum length.
The four-Spark vLLM stack: Engram goes to disk
The problem, stated in the recipe, is arithmetic. The checkpoint occupies 510 GB on disk, of which the two Engram n-gram tables are 203 GB, and the engine would keep those tables in host memory — which, on a unified-memory machine, is the GPU’s memory too. The first patch keeps the tables in the safetensors files and reads the needed rows on demand from NVMe. With Engram on disk and 81.6 GiB of weights per box, four Sparks serve a 1,070,168-token pool at 300K context, and a separate boot proved the full one-million-token path. Measured single-stream decode after the first token ran at 73.8 tokens per second on code and 24.4 on prose; six streams reached a 131.9 tokens-per-second mean aggregate across the eight prompt categories, with peaks of 225.5 on code and 259.9 on counting; cold prefill ran 902 to 1,539 tokens per second, and a 93,335-token prompt completed its prefill in 78 seconds.
The recipe’s own boot-by-boot fix ladder reads like a cost curve, which is why I would cite this stack over the raw release: 5.1 tokens per second in eager mode without speculation; 19.5 to 22.1 with DSpark enabled; 41.5 once CUDA graphs could be captured after the Engram lookup was moved out of the forward pass; 60.8 after a silent GPU clock latch on two of the boxes was cleared by unplugging them for half a minute, which a reboot does not fix; and 84.9 to 92.2 once each worker kept its own Engram rows on local NVMe instead of reading them across the network. The speculative drafter accepted a mean of 3.57 draft tokens per verification step, with a range of 1.88 to 5.92 — content decides. The maintainers are candid about the caveats, among them a GPU slow-state phase on the GB10 that can move a single benchmark cell by up to about half again. Since September 11 the default configuration has been the EXL3 lane, in which a 3.5-bits-per-weight quantization of the routed experts trades expert precision for a 3,304,863-token pool: eleven full 300K-context requests at once, against 3.57 on the release checkpoint.
The three-Spark SGLang stack: why three, and not two
Mia AI Lab’s recipe opens with a division table that settles the hardware question before the software one. Of the 476 GiB checkpoint, 189 GiB is Engram, which the kit moves to each node’s NVMe; of the remainder, a two-rank layout would need 145 GiB of resident weights per GPU and does not fit the 121.7 GiB of unified memory in each GB10 module, while a three-rank layout brings it to roughly 101 GiB and does. The three boxes form a ConnectX-7 RoCE triangle, and the published measurements put prose decode at 37.9 tokens per second in a single stream, 248 milliseconds to first token, and about 82 milliseconds per speculative step; four concurrent streams reach a 78.6 tokens-per-second aggregate, 20.9 per stream. The most consequential repairs are memory-side rather than compute-side: reining in the NCCL connection buffers, from 4.7 GiB down to 0.14 GiB pinned per node, turned a head node that had less than 1 GiB free while serving into one with about 6 GiB, and the recipe states that greedy decoding is deterministic run to run — an unusual and, on this stack, valuable property. A full boot takes 12 to 13 minutes, the bulk of it reading the checkpoint. Tensor parallelism by three forces padding the model’s attention heads from 64 to 96 and its groups from 8 to 12, so rank two’s attention shard is all padding; the maintainers also warn that PyTorch’s expandable segments return NaN logits on any prefill longer than 64 query tokens, so the knob stays off. The four-machine profile — a 4,000,000-token pool at the model’s full context — has been validated for configuration only; the README states plainly that it has not been booted, so I would not quote its figures beside the measured three.
An independent run on the same four boxes
A community operator posted its own stream of measurements running the four-Spark vLLM stack at six concurrent coding sessions: 170.73 tokens per second aggregate, with 1,200 output tokens completed in 7.028 seconds. Its warm single-stream figures — 77.61 tokens per second on code, 74.08 on mathematics, 48.75 on reasoning, 33.02 on prose — sit comfortably above the recipe’s own run, and a counting stream peaked at 291.64. Tool calling and vision were live, and a 127,480-token retrieval test passed. The measurement credits the ninth generation of the recipe, which is worth noting: day-one numbers on this tier improve by the hour, and I treat single runs as directional rather than as settled.
| Configuration | Engine and quantization | Streams | Aggregate tok/s | Single-stream highlight | Source |
| 4× Spark TP4, release checkpoint | vLLM, MXFP4 experts + FP8 dense | 1 | 38.0 mean | 73.8 code, 92.2 counting | GitHub recipe |
| 4× Spark TP4, release checkpoint | vLLM, MXFP4 + DSpark | 6 | 131.9 mean; 225.5 code | — | GitHub recipe |
| 4× Spark TP4, EXL3 lane | vLLM, EXL3 3.5 bpw experts | 6 | 141.2 mean | 3,304,863-token KV pool | GitHub recipe |
| 3× Spark TP3 | SGLang, native MXFP4 + DSpark | 1 | 37.9 | 37.9 prose, TTFT 248 ms | GitHub recipe |
| 3× Spark TP3 | SGLang, native MXFP4 + DSpark | 4 | 78.6 | 20.9 per stream | GitHub recipe |
| 4× Spark TP4, community re-run | vLLM, MXFP4 + DSpark | 6 | 170.73 coding; 291.64 counting | 77.61 warm code | community benchmark |
What the day-one numbers say
Read together, the three sets point one way. The quantization choice now moves the key-value pool by a factor of three — the EXL3 lane triples the release checkpoint’s pool while the per-step time holds nearly constant — and the memory repairs, not the kernels, bought the fivefold improvement from eager mode to the served configuration. I believe the deskside tier has crossed into the 500-billion-parameter class not by adding hardware but by subtracting from the memory bill: with the conditional-memory table on disk and the cache at 890 bytes a token, a fleet of three or four desktop boxes is now a credible endpoint for a frontier-class open-weight agent, and the gap between day one and day thirty of tuning closes by the hour. As firms such as 650 Group have been tracking in their AI infrastructure research, the constraint on this class of hardware has moved from what fits to how much of the box the serving stack can be persuaded to hand back — and on this release the serving stack, not the silicon, did the work.
| Configuration | Memory | Lead stack | Performance | Best for |
| 4× Spark, release checkpoint | 1.07M-token pool, 81.6 GiB weights per box | vLLM + DSpark | 73.8 tok/s code, single stream | Daily driving, tools and vision |
| 4× Spark, EXL3 lane | 3.3M-token pool | vLLM + DSpark | 141.2 tok/s at six streams | Long-context contention, many users |
| 3× Spark, native quantization | 750K-token pool, 101 GiB weights per rank | SGLang + DSpark | 78.6 tok/s at four streams | Three-box fleet, deterministic serving |
| Community re-run | 300K configured context | vLLM + DSpark | 170.73 tok/s coding, six streams | Coding agents, directional reference only |
Several clusters in the same week of notes remain queued for their own posts: a second round of single-Spark Qwen3.8-27B speedups, in which a calibrated NVFP4 drafter and a concurrency-tuned build each claim roughly a quarter more throughput on the same GB10 hardware; the v1.5.0 release of the two-Spark GLM-5.3-Flash EXL3 kit, published alongside an eight-change update to its sibling that cuts maximum prompt wait under contention from about 120 seconds to 15; the arrival of official NVIDIA NVFP4 weights for GLM-5.3-Flash, some 28 GB larger than the community EXL3 build, and the top score that class posted on a community benchmark on a single Spark; and the still-open items from last week’s queue — the second instalment of the heterogeneous prefill-and-decode split and the two-Spark NVFP4 pool comparison.