by Chris DePuy / August 12, 2026
NVIDIA this week released Nemotron 3.5 Lightning, an open 30B MoE model with just 3B active parameters, and I think it is the clearest signal yet that the deskside inference market is splitting into two distinct tiers. The model is built for the high-volume execution layer of long-running AI agents — the tool calls, validation, formatting, and subagent work that an agent burns through constantly — rather than for the deliberate reasoning that dominates the 1T-class research tier. NVIDIA frames it as delivering up to 4x faster output and roughly 30% faster agentic task completion than comparable 25-35B open models.
The Architecture: 3B Active, Sparse Attention
The design point is deliberately compressed. Lightning is a hybrid interleaved Mamba-2 plus MoE plus Attention model with 30B total parameters and 3B active per token, 52 layers deep and 128 experts with 6 active plus a shared one. The notable choice is the sparse attention profile: only six of the 52 layers are full attention, which is what lets the model carry a ~18M-token KV cache and a realistic 1M-token context on a single GB10. The shipped checkpoint is the NVFP4 (W4A16) quantization.
The DGX Spark Numbers
Because the model is small-active and heavily optimized for one class of work, the community benchmarks on a single DGX Spark are striking. One community contributor built a pinned SM121-optimized vLLM runtime with native NVFP4 and baked-in multi-token prediction (K=1) and measured 376.3 tok/s aggregate at six concurrent users, 90.95 tok/s single-stream decode, and a 92.3 ms time-to-first-token. NVIDIA’s own developer forum testing shows ~78.5 output tok/s target-only, rising to 90.7 tok/s with DSpark speculative decoding at roughly a 53% acceptance rate. An independent review measured 79.6 tok/s without speculative decoding, 115.75 tok/s with DSpark single-stream, and 421.85 tok/s total across eight concurrent streams, with roughly 5,400 tok/s prefill. The accuracy side holds up for the class: SWE-bench Verified up to 52.8% and PinchBench productivity in the mid-80s.
vLLM v0.27.0 Lands the Ecosystem
The software stack converged the same week. vLLM v0.27.0 shipped full-stack support for the 2.8T Kimi K3, added Qwen3.5 dense and MoE variants, and brought a broad set of DeepSeek-V4 sequence-parallelism and kernel optimizations, while SGLang added day-0 support for Nemotron Lightning with MTP, DFlash, and DSpark paths. The result is that the execution layer can now be served by multiple engines rather than a single vendor’s reference.
What This Means
I think the deskside market is rapidly bifurcating. On one side there is the reasoning tier — the 975B Inkling, 1.6T DeepSeek V4 Pro, and the 2.8T Kimi K3 models that cost multiple Spark nodes and justify dozens of tokens per second. On the other is the execution tier, where a 3B-active hybrid model sustains hundreds of tokens per second in aggregate on one box because the workload is high-volume and low-difficulty. NVIDIA is betting that both tiers belong on the desk and has paired Lightning with NeMo Switchyard, its model-routing layer, so an agent can fan a request out to the right model for the job rather than paying frontier pricing for every call. As firms such as 650 Group have been tracking in their AI infrastructure research, the question for 2026 is no longer whether a given model fits on a desk; it is which models belong on that desk at all — and NVIDIA has now made an explicit architectural argument that the agent execution layer belongs there, as a distinct and far cheaper tier than the research models above it.