DeepInfer opens DeepSeek’s inference engine, with serving components still missing

DeepSeek released DeepInfer, an inference framework for its V4.1 Flash model, with optimized kernels, asynchronous batch scheduling and draft-token verification. The public code supports offline inference on NVIDIA SM100 GPUs and Huawei Ascend 950 accelerators. It is not a complete production service: the repository explicitly excludes a key-value cache system, service disaggregation and replica load balancing. Infrastructure teams should distinguish the engine from the surrounding serving stack.

DeepSeek

Why it matters

Open inference kernels and scheduling code are useful to study without assuming the repository includes every component behind a hosted model.

More context

The Apache-2.0 project requires Linux, GCC 13 or newer and Python 3.12, plus CUDA 13 or CANN 9.2 for its respective hardware. Try the documented offline example only on a compatible system. Its exact first public release date was not established.

Keep in mind

Published performance measurements use four SM100 GPUs for prefill and 64 for decoding. They do not establish single-GPU performance or a complete service’s latency, reliability or operating cost.

Original read