Qwen3.8-Flash-Next on One DGX Spark: NVIDIA’s Official NVFP4 Weights Hit 44-49 tok/s, and Scaling Buys Context, Not Speed

by Chris DePuy / September 7, 2026

Two weeks ago, Qwen3.8-Flash-Next was a day-zero curiosity: NVIDIA’s official NVFP4 quantization of the model landed on Hugging Face on August 31, and the community was still proving it could run at all. This past week it stopped being a curiosity. Three independent reference deployments now serve the model from a single 128 GB DGX Spark at 41 to 49 tokens per second single-stream, with measured 260K-context sweep data, 8-stream concurrency sweeps, and clicking working recipes. I think the more interesting finding is what the numbers say about scaling: for this model, a second or third Spark does not buy throughput. It buys context.

The Checkpoint: What NVIDIA Actually Quantized

The NVIDIA NVFP4 checkpoint is not a uniform quantization. Model Optimizer v0.46.0 quantizes the routed MoE expert layers to W4A4 NVFP4 with MSE-calibrated scales, keeps attention, shared experts, and other main-model layers in BF16, stores the MTP routed experts as 128×128 block-scaled FP8, and keeps the n-gram embedding table at per-tensor FP8. The result is about 2.7x smaller than BF16 — 124 GB on disk, measured — with no fine-tuning. On NVIDIA’s own evaluation it stays within about a point of the FP8 release in both directions: GPQA Diamond 91.5 vs 92.0, HLE 35.4 vs 34.7, Terminal-Bench 2.1 82.9 vs 83.3, MMMU Pro 78.3 vs 77.1. The underlying model is the Qwen4-architecture preview: 125B parameters with 6B active, plus a 51B n-gram embedding and a 4B MTP head; 48 layers, 512 experts with 10 routed plus 1 shared active; hybrid Gated DeltaNet and Qwen Sparse Attention; 262,144-token native context. It takes text, image, and video input.

Why It Fits One Box: The PLE Table Never Loads

The trick that makes the whole thing fit in 128 GB is structural, not cleverness. The 51B n-gram (PLE) embedding table is a pure lookup — roughly 16 rows per token, never a matmul — and in the NVIDIA checkpoint that table is 47.68 GiB of FP8. The tonyd2wild reference deployment keeps it on NVMe and reads the rows on demand inside the model state’s prepare_inputs step into a fixed GPU buffer, so decode still runs as CUDA graphs. About 76 GiB of real weights stay resident, and the rest of the pool becomes KV cache: 1,027,392 tokens at gpu-memory-utilization 0.80, or roughly four concurrent full-262K requests. The MiaAI Lab recipe packs the table to about 27 GB and memory-maps it with MADV_RANDOM, which took disk reads per decoded token from 1,366 KiB down to 57 KiB — a 24x reduction — and measures 992,584 tokens of KV pool on the equivalent profile. The blazux PLE-mmap route is the base several of these build on.

One Spark, Three Recipes, One Story

The numbers differ by launch configuration, which is exactly what the three reference sets show.

I think the most useful single result is the context sweep from redp314’s recipe: decode holds 37 to 44 tok/s from a 327-token prompt all the way to 258,790 tokens, holding steady instead of sagging at depth. The architecture explains it — only part of the layer stack pays for sequence length, and the recurrent layers move no more state as the prompt grows. The trade-off is cold time-to-first-token at the longest rung — 158 seconds at 259K — and the honesty is the repo’s own: the hybrid fp8-side-layer conversion is worth 10-20%, MTP 3 is worth another 12-15% over MTP 2, and MTP 4 buys nothing. The tonyd2wild deployment shows the same shape on NVIDIA’s own weights with a reduced-vocabulary draft (65,536 token ids; the target verifies the full vocabulary, so output is exact): 43.9 tok/s median across a 40-prompt harness with 0.30 s time-to-first-token, 29.0 tok/s prose, 49.3 tok/s JSON, and 68.8 tok/s aggregate at six streams, with the p99 per-token latency staying within 10% of p50 at six streams. The MiaAI Lab recipe tops the single-stream number at 48.7 tok/s prose and 162.9 tok/s aggregate at eight streams (a run with max_num_seqs raised to 8, above the value the shipped config sets; the author flags that single change), with prefill at 8K up 24.7% to 2,200 tok/s after its optimization pass. Two caveats are documented rather than hidden: the MiaAI profile leaves only about 14-16 GiB of host memory headroom against a 6 GiB watchdog floor — the KV budget was cut from 22 to 20 after three lost servers on September 4 — and decode-under-prefill is where the 2,048-token chunk width hurts most, with a p50 chunk gap of about 1,057 ms when a 64K prompt lands against two live streams versus 78 ms on a quiet server. The redp314 recipe’s boot-time story is worth its own line: 8-13 minutes normally, about 300 seconds with a fast loader that is still flagged experimental after OOM-killing the container twice in five boots.

Two Sparks and Four Sparks: What Scaling Actually Buys

The two-Spark TP2 comparison is the counter-intuitive part. Doubling the box nearly sextuples the KV pool — 995,129 to 5,874,061 tokens — and improves every per-reply metric: median 32.5 to 35.8 tok/s, prose 21.7 to 24.1, time-to-first-token 300 to 250 ms, cold prefill at 28K up 27%. But aggregate throughput at six streams moves just 5%, because on one box the drafter already saturates the batch. A separate SPEED profile on two Sparks reaches 53.7 tok/s median and 97.9 tok/s aggregate at six streams with 180 ms first token, and the four-Spark CONTEXT profile holds 40.5 tok/s median, 87.4 tok/s aggregate, and a 9.09M-token KV pool — 34 full 262K-context jobs in flight, at the cost of expert-parallel mode being required because FlashInfer’s CUTLASS NVFP4 kernels only split the MoE intermediate two ways. The recommendation in the repo is the conclusion: one Spark stays the default; multi-Spark lanes exist for people who run long concurrent agent sessions. Same-day signal: two independent projects and the MiaAI dual-Spark recipe separately fixed the same two Model Optimizer loader gaps on the same day the reference deployment did — draft-local layer index and the 128×128 block-scaled FP8 experts — with no code shared between the three.

Why This Matters

The model’s own numbers say this is a real agentic tool: SWE-bench Pro 62.5, DeepSWE 1.1 58.7, CoWorkBench 73.9, Vision2Web 64.0 — above DeepSeek-V4-Flash-0731 on the first two. A community web-development chart posted over the weekend places Qwen3.8-Flash about level with the top closed frontier models on frontend work and above DeepSeek V4 Flash/Pro and the GLM 5.x family, though I would treat that as community-reported rather than a verified benchmark. The operational takeaway for the desk-side tier is that the 128 GB question has been answered with a recipe: the n-gram table lives on NVMe, the weights stay resident, and a single GB10 serves a 180B-stated-parameter multimodal model with enough KV pool for a fleet of long-running agents — at a fraction of the per-token cost of the API tier. As firms such as 650 Group have been tracking in their AI infrastructure research, the desk-side inference tier is now defined less by what fits than by which serving stack and offload strategy a buyer inherits with the box.

A separate batch of .md notes on heterogeneous prefill/decode (a 284B DeepSeek-V4-Flash split across two DGX Sparks for prefill and an Apple M3 Ultra for decode, with asymmetric cache priming over plain Ethernet) is pending a dedicated post, as is the agent shared-memory / second-brain cluster and the Hermes agent-platform mega-update notes.