DeepSeek just open-sourced its inference engine
Points and comments are a snapshot, not live.
DeepSeek has open-sourced DeepInfer, the high-performance inference engine behind its production services.
DeepInfer supports the DeepSeek-V4.1-Flash model and includes optimized kernels (FlashMLA, DeepGEMM, DeepEP), parallelism strategies (Attention DP, MoE EP), and features like DSpark speculative decoding, SWA bounded replay, asynchronous scheduling, batch-invariant inference, and structured outputs. It runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Performance benchmarks show 155 trillion tokens/day per cluster and $38/card/hour off-peak revenue. Built on vLLM with inspiration from SGLang and TensorRT-LLM.
License: Apache 2.0.
What commenters are saying
Commenters highlight the profitability metrics from the article, with some arguing inference is a highly profitable business (70%+ net margins on rented infra) and that open-source models threaten to compress margins over time. Others counter that inference is an undifferentiated commodity that will eventually face supermarket-like margins, and that ongoing training costs are often excluded from profit calculations. A few users share personal experience: one finds DeepSeek 4.1 Flash comparable to Opus 5.5 for their workflow, while another reports the open-weight model still requires more iterations than Claude Sonnet 5.5.
A specific correction notes the revenue figure of $38/card/hour does not include cost of capital, depreciation, or training overhead.