Blog

This week, a wave of open-source recipes and community benchmarks clarified what the deskside AI inference market can do in mid-2026 — specifically, what fits on a single or paired NVIDIA DGX Spark, and where the GB300-based DGX Station picks up later this year. As firms such as 650 Group have been tracking in their AI inference research, the number of usable, public configurations for running capable models on consumer-class hardware has reached a level we haven’t seen before.

Source: NVIDIA Newsroom

The DGX Spark is a $4,000 deskside system powered by NVIDIA’s GB10 Grace Blackwell Superchip (140W SoC TDP, 240W system power supply, 128 GB unified memory). The product has been shipping since early 2026 through NVIDIA and channel partners, and the open-source community has now had enough time with the hardware to produce reproducible, benchmarked recipes. This is what they’ve shown.

Single DGX Spark: What Fits, and How Fast

The best benchmark we found came from a community-run suite called Spark-Bench, which evaluates models across 74 scenarios in 11 domains with deterministic grading — no LLM-judge, just unit tests, JSON schema validation, executable code tests, and pixel-based visual analysis. It tracks TrueScore (quality-weighted throughput) alongside raw tok/s.

On a single Spark, the top-ranked model as of this week is Qwopus 27B (AWQ) at TrueScore 89.1, followed by Agents A1 (NVFP4) at 88.1, and Qwen 3.6-35B in various configurations (NVFP4 with DFlash, MTP-3, or UD-Q8) scoring 86.6–87.5. For reference, Grok 4.5 via cloud API scored 82.6 on the same benchmark (with low-effort settings). The Spark’s ~128 GB unified memory (≈121 GiB usable) accommodates 27B–35B parameter models comfortably at NVFP4 precision, which is the format NVIDIA has optimized for Blackwell hardware.

The most striking throughput data point came from independent benchmarks of Qwen 3.6-35B (a 35B-A3B MoE model) running on a single Spark. The results showed 64 concurrent users served at over 700 tok/s aggregate, with nvidia-smi reporting the GPU drawing approximately 38W during sustained inference. The full benchmark suite is on GitHub, covering 57 scenarios across 10 domains with partial-credit grading and K=2 repetition per run. Total system power at the wall under similar loads typically ranges from 60–90W, with peaks under 200W (the system ships with a 240W USB-PD power supply). For enterprise edge deployments and campus AI networking use cases where a device needs to serve dozens of users without dedicated cooling, these are the first real numbers we’ve seen.

Unsloth released new dynamic NVFP4 quantizations of Qwen 3.6 on July 10 that run 2.5x faster than NVIDIA’s reference NVFP4 quants, using a full W4A4 matrix multiply path rather than W4A16. The 27B variant fits in 24 GB VRAM (single-Spark territory), and the 35B-A3B “Fast” variant reached 17,561 tok/s on a B200-class GPU. The models are on Hugging Face, and the release thread from Unsloth’s co-founder — an ex-NVIDIA engineer who built the company’s custom Triton kernels — includes a 2h42m technical walkthrough.

Two DGX Sparks: Chaining for Larger Models

Two DGX Sparks at roughly $8,000 total can serve DeepSeek-V4-Flash with a 1M token context at 60+ tok/s. The recipe is on GitHub, including an abliterated (uncensored) variant that achieves 55–60 tok/s on the same hardware. For organizations that cannot send sensitive legal, financial, or medical data to cloud APIs, this price point is worth evaluating.

A second double-Spark config runs GLM 5.2, demonstrated by a community contributor earlier this week.

The Configuration Tuning Story That Matters More Than Specs

The most instructive result we saw this week is how much performance is left on the table with default vLLM settings. A community contributor published the full DGX Spark recipe for Nemotron Puzzle 75B-A9B at NVFP4 precision on GitHub. At stock configuration, a single Spark delivered roughly 40 tok/s for a single stream, with max_num_seqs=1 meaning four parallel requests were serialized — one at a time, the rest queued. After changing max_num_seqs to 4 and enabling prefix caching, the same hardware achieved 75 tok/s aggregate across four parallel streams.

The honest assessment from the recipe’s README: “the big win is concurrency + prefix cache, not a free jump in solo tok/s.” The system’s MTP-3 speculative decoding acceptance was already healthy at ~75% draft tokens. What the config change unlocked was multi-stream capacity. This matters for agent-based workloads where multiple simultaneous outputs (tool calls, sub-tasks) are the norm, not a burst scenario.

Infrastructure Layer Updates

NVIDIA’s ModelOpt 0.45.0 release continues the trend of bringing enterprise-scale models down to consumer hardware through automated quantization and optimization pipelines. The company has published a detailed comparison of NVFP4 versus MXFP4 4-bit formats running GPT-OSS-120B on Blackwell, covering quantization accuracy, throughput, and hardware compatibility — a useful reference for anyone choosing a quantization target for production deployment.

On the inference engine side, DFlash speculative decoding was recently merged into llama.cpp (PR #22105, merged 2 weeks ago). DFlash produces an entire block of candidate tokens in a single draft forward pass, unlike EAGLE3’s autoregressive one-by-one draft. Benchmarks on Qwen3.6-27B show 3.2x–4.9x decode speedups on structured coding tasks, with the PR authors reporting up to 8x on ideal Qwen3 configurations. The Spark-Bench leaderboard already lists a DFlash-k10 variant of Qwen 3.6-35B at TrueScore 87.2, confirming it works on the GB10 hardware.

Where This Goes: DGX Station GB300

Source: NVIDIA Newsroom

In Q4 2026, NVIDIA’s GB300 Grace Blackwell Ultra Desktop Superchip enters the picture through partner systems branded as DGX Station. The spec sheet: 748 GB of coherent memory (252 GB HBM3e GPU at 7.1 TB/s + 496 GB LPDDR5X CPU at 396 GB/s), 20 petaFLOPS FP4, 72-core Grace CPU, 1,600W power delivery, and support for models up to 1 trillion parameters. Systems are orderable now from ASUS, Dell, GIGABYTE, HP, MSI, and Supermicro, with pricing from partners like Bizon starting at $108,159 and shipping in the September–November range. The full press release is on NVIDIA’s news site.

The implication is straightforward: the recipes being built and shared for GB10 this week will be directly applicable to GB300 deployments this year, running models 10–30x larger on the same desk footprint.