NVIDIA separates synthetic inference gains from a real test set

NVIDIA reports 42.5% higher decode throughput for Ling-3.0-flash on DGX Spark across all 1,319 GSM8K questions at batch size one. It combined specialized expert kernels, FP8 output-layer weights and speculative decoding. A synthetic workload showed a much larger 185.1% gain; the article explains why repetitive outputs exaggerate benefits and describes a configuration mismatch that can make the combination slower.

NVIDIA Technical Blog · Oct 27, 2026

Why it matters

Inference engineers get both an optimization recipe and a warning to validate configuration on realistic workloads.

More context

The techniques reduce weight traffic per step and the number of decoding steps. When combining FP8 output weights with speculative decoding, the draft model must share the full output-layer module. The article links a Ling deployment cookbook; use it to verify every optimization is active.

Keep in mind

These are NVIDIA’s measurements for one model, device and workload. GSM8K accuracy was 96.89% before and 96.82% after; the source reports no statistically detectable change, not identical outputs or universal accuracy preservation.

Original read