For local inferencing, here is a setup that has proven to be quite stable and fast. It’s the 2x DGX Spark running DeepSeek V4 Flash. It keeps the model running locally and, I’d say, is near Frontier, and doesn’t get lost often when using agents. It’s pretty good at planning, too.
The reviews quote ~40 tok/s for DeepSeek V4 Flash on a dual DGX Spark. That number misses the one lever that actually moves: speculative decoding.
Fraser Price and DeepSeek-AI’s DeepSpec project built DSpark — a speculative-decode build of V4 Flash. (Note: DSpark is a decoding method, not NVIDIA hardware — the name is confusing.)
The Numbers (2x GB10, TP=2)
All numbers from:
| Build | Single-stream | Notes |
|---|---|---|
| Stock FP8 (baseline) | ~40-45 tok/s | What reviews quote [1] |
| DSpark (code) | ~60-67 tok/s | +50% over baseline [1], [2] |
| DSpark (mixed content) | ~40 tok/s | Lower acceptance rate [1] |
| DSpark @ 6 concurrent, 1M ctx | ~182 tok/s aggregate | ~30 tok/s per stream [1] |
| DSpark @ 16 concurrent, 200K ctx | ~315 tok/s aggregate | Static profile [1] |
Our Setup: Two Desk-Side Units, Production Since Mid-July
We have been running this exact 2x DGX Spark configuration as a daily-driver inference cluster since mid-July. Our deployment uses two GB10 units connected via ConnectX-7 RoCE fabric on a shared docker compose orchestration, serving DeepSeek-V4-Flash-DSpark with TP=2 across both GPUs. The model is mounted from /cache/huggingface/fraserprice/DeepSeek-V4-Flash-DSpark and the cluster serves requests at 192.168.1.177:8888/v1 with the model name deepseek-v4-flash-dspark.
Our config:
- KV cache:
nvfp4_ds_mla, 256-block size, 0.85 GPU memory utilization. Our current KV pool is ~2.83M tokens from ~19 GiB of available cache memory — roughly 2.1x denser than earlier Stage-C builds. - Context: 262K
max_model_lenper request (configurable up to 1M without a restart — the model handles variable-length context),max_num_seqs=2,max_num_batched_tokens=8192. - Speculative decoding: DSpark with 3 draft tokens, probabilistic sampling. The
VLLM_DSPARK_GPU_REJECTED_CONTEXT_MASK=1flag passes rejected draft candidates’ context to the target model instead of discarding it — this is the optimization that unlocks the 60-67 tok/s range rather than the ~40 tok/s baseline. - MoE backend:
flashinfer_b12x(native MXFP4) — without this flag, the model silently falls back to Marlin (W4A16), halving throughput. - Networking: RoCE fabric with dedicated fabric IPs (192.168.200.2/24 head, 192.168.200.3/24 worker) via interface
rocep1s0f0. - Image: Anemll’s
dspark-vllm-gx10:0.1.1Docker image, which bundles vLLM 0.25.1 with a tight 584-byte packed sparse-MLA envelope and the b12x native-MXFP4 MoE backend. - Cold boot: ~10-15 minutes including kernel compilation for the CUDA graphs on first launch.
In practice, we see single-stream code and structured output at 60-67 tok/s with the DSpark speculative decode path active. The acceptance rate degrades under multi-turn coding sessions (the draft acceptance distribution’s tau drops from ~4.3 to ~2.8 as context grows to ~80K), so real coding-session throughput between agent turns is lower than the best-case single-turn number. Mixed content is slower (~40 tok/s).
Cold boot time is the main operational friction — the model reload takes ~15 minutes with kernel compilation after any config change. Pre-warming the Docker image and caching CUDA kernels on a persistent volume would cut this substantially, but we have not done that yet.
Quick Takeaways
The 1M context path is verified. With NVFP4 KV cache (nvfp4_ds_mla), 1M-token capability is achievable on a ~2M-token KV pool. The decode cycle stays context-independent (~70 ms per token), so throughput stays flat across 0.5K to 65K context. However, draft acceptance decay under realistic multi-turn coding context means real coding-session throughput will be lower than the best-case single-turn number.
Caveat: Concurrent long-context usage still hits CUDA device-side asserts. Not OOM — it crashes under concurrency. This is a correctness edge in the kernel. Works cleanly on code and structured output. The NVIDIA Developer Forum recipe (tonyd615) and LLMRequirements.com validation both confirm the same 60-67 tok/s range for single-stream.
Concurrency Detail
With a concurrency patch (credited to Keys/drowzeys), DSpark now serves multiple streams at once — it used to be single-stream only. The ~2M KV pool is shared and allocated on demand, not reserved per-request. So 6 concurrent streams at 1M context ceiling does not reserve 6x1M: max_model_len is a per-request ceiling, max_num_seqs is a concurrency cap. The real limit is sum(live tokens) <= ~2M. Agent turns rarely hit 1M, so 6 normal streams share the pool cleanly.
What This Means
A pair of $9,500-per-unit desk-side boxes (2x DGX Spark) running a 284B-parameter MoE model at 60-67 tok/s on code, with verified 1M context capability via NVFP4 KV cache. We have run the 2x262K context as a daily-driver agent inference cluster since mid-July — this is a different class of self-hosted inference than the early reviews suggested. The recipe uses vLLM with the DSpark speculative decode checkpoint on two GB10 nodes connected via ConnectX-7 RoCE. Operationally, the main friction is the ~15-minute cold boot time after config changes; the steady-state performance is production-usable for agent workloads that value data privacy and latency over cloud-scale throughput.
Sources: NVIDIA Developer Forum — tonyd615, LLMRequirements.com, HuggingFace — DeepSeek-V4-Flash-DSpark. Tested on 2x DGX Spark (GB10), ConnectX-7 RoCE, vLLM 0.25.1 (Anemll dspark-vllm-gx10:0.1.1) with DSpark checkpoint. Deployment config: 262K context, nvfp4_ds_mla KV cache, flashinfer_b12x MoE backend, 3-token DSpark speculative decode. Credits: Fraser Price (DSpark model), Rafael Caricio (vLLM integration), Keys/drowzeys (concurrency patch), MiaAI-Lab (2-node launch config, 1M validation).