Tag: KV Cache
-
Google Puts TPUv7 Through an Outside Agentic Benchmark
by Chris DePuy, 650 Group / October 25, 2026 Google’s first TPUv7 submission to an outside agentic-inference benchmark landed this week. The operator’s October 14, 2026 announcement reports FP8 results served through vLLM at 1.67x better performance per dollar than NVIDIA’s GB300 NVL72 at FP4 the operator’s announcement thread. The submission is the first submitted…
-
DeepSeek Open-Sources DeepInfer, NVIDIA Answers Quietly With Flare
by Chris DePuy, 650 Group / October 22, 2026 In the early hours of October 20, Pacific time, DeepSeek announced DeepInfer, its in-house inference framework, as fully open source under the Apache 2.0 license. DeepInfer is the stack that powers DeepSeek-V4.1-Flash at scale in the company’s own production services. The four-post thread laid out the…
-
The Spark Fleet Learns to Run Itself
by Chris DePuy, 650 Group / October 21, 2026 The last month of desk-side serving stories were about engines. Athena put two frontier models on one 128 GB box at 262K context with a 141,519-token conversation restored from disk in 2.1 seconds; Imprint turned that same restore trick into a standalone context-cache service with a…
-
Five Spark Clusters, Two Kinds of Proof: TP5 GLM-5.3 Runs as Beam Sets the Efficiency Bar
by Chris DePuy, 650 Group / October 6, 2026 Five DGX Sparks now carry the full 753-billion-parameter GLM-5.3: an operator reported the first working five-board tensor-parallel build on Monday, with 1.5 million tokens of KV cache, prefill above 2,000 tokens per second, and 76 tokens per second single-stream — rising to 98–113 tokens per second…
-
New TensorFold LLM Model Serving Software Dramatically Improves Decode Speed on Macs and Sparks
by Chris DePuy, 650 Group / September 27, 2026 Two repositories shipped from one maintainer this week and made the same promise in two different places. On September 25 and 26, a developer publishing under the name Ash Hart released TensorFold, a local inference engine for Apple Silicon and NVIDIA GPUs, and Imprint, a tool…
-
NVIDIA’s Official NVFP4 Checkpoints for DeepSeek-V4.1-Flash and GLM-5.3-Flash: Four DGX Sparks Read the Receipts
by Chris DePuy / September 22, 2026 NVIDIA: what the conversion actually changes The DeepSeek checkpoint converts only the ordinary routed experts — projections w1, w2 and w3 across 384 experts in 40 layers — from the source MXFP4 format to NVFP4 weights and activations at group size 16, per the model card; attention, shared…
-
DeepSeek-V4.1-Flash on Four DGX Sparks: The Profile Day One Only Configured Now Boots, Serves and Passes a Million-Token Needle Test
by Chris DePuy / September 14, 2026 Late Sunday evening Pacific time, Mia AI Lab pushed a four-rank serving profile for DeepSeek-V4.1-Flash to its DGX Spark recipe, the same lab whose September 13 post on X put the model on four DGX Sparks and invited readers to run the stack themselves. The significance is not…
-
GLM-5.3-Flash on a Single DGX Spark: Three Community Quantizations Fit Where the Vendor’s Does Not
by Chris DePuy / September 13, 2026 On September 9, NVIDIA published an official NVFP4 quantization of Z.ai’s GLM-5.3-Flash. Its Model Optimizer recipe leaves the key-value cache at FP8 and compresses only the linear layers inside the transformer blocks — the sparse-MoE shared experts and the dense MLP — from 16 bits down to 4,…
-
Day One on Three and Four Sparks: Community Recipes Put DeepSeek-V4.1-Flash on the Desk
by Chris DePuy / September 12, 2026 On September 10, DeepSeek released DeepSeek-V4.1-Flash, a multimodal mixture-of-experts model with a 552-billion-parameter backbone and a context window of one million tokens. The launch kits for NVIDIA’s desk-side DGX Spark machines arrived the same day: a four-machine stack on vLLM was serving requests by 9:17 Eastern time, roughly…