ReplaySSM speeds decoding by buffering inputs instead of rewriting state
NVIDIA and Princeton’s Dao AI Lab describe ReplaySSM, which represents a state-space model’s memory as a checkpoint plus recent inputs. Most decoding steps read the checkpoint without rewriting the full state; periodic flushes incorporate the buffered inputs. Across four tested hybrid models, the authors report 1.20–1.47 times faster standard decoding steps at batch size 256. Their batch-size-one tests showed no speedup, making workload scale central to the result.
NVIDIA and Dao AI Lab · Oct 12, 2026
Why it matters
Reducing memory writes can improve a bandwidth-limited inference path even when the mathematical model stays the same.
More context
The experiments use one H100 or B300 per model, FP32 state and a buffer window of 16. Code is available as a vLLM fork, with some kernels already upstream and others only in the research checkout. Try the documented benchmark configuration before generalizing the gains.
Keep in mind
Results depend on model family, hardware and batch size. Standard decoding, speculative throughput and quantized-state accuracy are separate measurements; kernel speedups are not universal application speedups.
Original read