DeepSeek Open-Sources DeepInfer, the Engine Behind Its AI Service
DeepSeek has open-sourced DeepInfer, the inference engine behind its own AI service. Here's what's inside, what's missing and who can actually run it.
DeepSeek has open-sourced DeepInfer, the inference framework that runs its own large-scale production services, under the Apache 2.0 licence. It went live on Tuesday, October 20, and quickly became the day's biggest developer talking point.
Here's what you need to know:
DeepInfer is the engine DeepSeek uses to serve DeepSeek-V4.1-Flash, and for now that's the only model it supports.
It runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.
The release covers offline inference only. A KV cache system, prefill/decode disaggregation and load balancing aren't included.
DeepSeek says inference code for future DeepSeek models will land in the same repo.
The GitHub repo passed 13,000 stars and 1,200 forks in under a day.
What DeepInfer ships with today, and what you still have to build yourself.
What Happened
DeepSeek announced the release on X at 09:30 UTC (15:00 IST) on October 20. "Introducing DeepInfer, our in-house inference framework, now fully open source!" the company wrote in DeepSeek's announcement on X, adding that it's "sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable."
The code sits in the DeepInfer GitHub repository, which describes it as "the high-performance inference framework that powers DeepSeek's large-scale production services." The package bundles DeepSeek's own kernel libraries, including FlashMLA, DeepGEMM and DeepEP, plus custom TileLang kernels.
On top of those kernels, it adds attention data parallelism, MoE expert parallelism with load balancing, and a speculative decoding system DeepSeek calls DSpark. DSpark drafts tokens ahead and uses confidence estimates to decide when to verify them. There's also asynchronous batch scheduling, deterministic sampling and structured outputs that constrain generation to a regex, JSON schema or EBNF grammar.
DeepSeek is upfront about the gaps. The README says a KV cache system, prefill/decode disaggregation and load balancing across data-parallel replicas are "required for a complete inference service" but "not included in this release."
Its published performance charts were measured on 4 SM100 GPUs for prefill and 64 for decode. Those are DeepSeek's own numbers: The Next Web reports that nobody outside the company has independently tested them yet.
The README credits vLLM as the original foundation, with inspiration from SGLang and NVIDIA's TensorRT-LLM. The vLLM project responded on X that it was "grateful" for the credit and will "keep improving with ideas from DeepInfer."
Why It Matters
AI labs usually guard their serving stacks closely, because inference is where much of the running cost lives. As Bloomberg reported, opening it up could make DeepSeek's technology cheaper and easier for businesses to run.
Cloud and hosting providers that already serve DeepSeek models can now copy the techniques DeepSeek itself uses. That puts pressure on rivals that sell access to closed models, and on inference startups whose edge was squeezing more tokens out of the same GPUs.
The hardware list matters too. Supporting Huawei's Ascend 950 next to NVIDIA's SM100 parts gives teams in China a serious open serving stack that doesn't depend on NVIDIA. It fits the wider chip contest we covered in the AI chip war of 2026.
The timing is pointed. The Next Web notes the release landed a day after the US ordered civilian federal agencies to remove DeepSeek products from their systems. Apache 2.0 code is hard to ban in practice, but government contractors will still need to check their own rules before touching it.
What It Means for Developers
DeepInfer is not a pip install and go. It's a reference implementation for teams that run serious GPU fleets.
DeepInfer's prerequisites and first run, per DeepSeek's README.
Check your hardware first. Only NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs are listed. You'll also need Linux, GCC 13 or newer, Python 3.12, and CUDA 13 or CANN 9.2.
Expect to build the serving layer yourself. You'll have to bring your own KV cache, request routing and prefill/decode split, or keep using vLLM or SGLang for online serving.
Treat the benchmarks as claims. Reproduce them with the repo's sweep script on your own traffic before you plan capacity around them.
Study the tricks even if you never deploy it. Batch-invariant inference and deterministic sampling are useful ideas for anyone debugging flaky LLM outputs or building reproducible evals.
Use structured outputs for agents. Regex, JSON schema and EBNF constraints are built in, which matters if you're wiring V4.1-Flash into tool-calling pipelines like the DeepSeek search API we covered this week.
Watch upstream projects. vLLM has already said it will borrow ideas from DeepInfer, so many of the gains may reach your existing stack without a migration.
What's Next
DeepSeek says inference code for future DeepSeek models and ongoing optimisations will land in the same repo, per a follow-up post on X. That makes DeepInfer the place to watch for early hints about DeepSeek's next model.
The big open questions are the missing pieces. DeepSeek hasn't said whether it will release its KV cache system or its disaggregated serving setup. It also hasn't said whether other GPU generations, such as NVIDIA's Hopper chips, will be supported.
Expect independent benchmarks against vLLM and SGLang within days. Those numbers will show whether DeepInfer's gains hold up outside DeepSeek's own clusters.
FAQ
What is DeepSeek DeepInfer?
DeepInfer is the inference framework DeepSeek uses to run its models in production. It's now open source under the Apache 2.0 licence and currently supports DeepSeek-V4.1-Flash.
Can I run DeepInfer on my own GPU?
Only if you have supported hardware. The README lists NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, and DeepSeek's own benchmarks used 4 GPUs for prefill and 64 for decode.
Does DeepInfer replace vLLM or SGLang?
Not yet. This release handles offline inference and leaves out the KV cache, prefill/decode disaggregation and load balancing a full online service needs.
Sources
DeepInfer repository and README, DeepSeek on GitHub
DeepSeek's DeepInfer announcement on X, DeepSeek
DeepSeek Unveils Open-Source Tool to Help Cut AI Operating Costs, Bloomberg
DeepSeek open-sources DeepInfer, the engine that runs its AI models, The Next Web
Follow TheTechLane
Keep learning AI, development, and modern developer technologies with TheTechLane. Learn • Build • Share Follow @thetechlanehq for more developer guides, tutorials, and practical technology content.