Gergely Orosz · 31 min read · AI & Product
What is inference engineering? Deepdive
Read the original on newsletter.pragmaticengineer.com →
The article explains inference engineering as the set of techniques for optimizing the deployment phase of LLMs after training, highlighting its growing importance with open models. It covers why inference matters, the hardware/software stack involved, when to invest, and five key approaches to speed up inference, along with essential metrics and concepts.
Key Takeaways
- Inference engineering focuses on post-training deployment and becomes crucial as open models become more capable and widely used.
- Invest in inference optimization when scaling product usage and gaps remain with off-the-shelf solutions.
- Five core techniques to accelerate inference: quantization, speculative decoding, caching, parallelism, and disaggregation.
- A practical stack and practices include datacenter GPUs, CUDA, Dynamo, PyTorch/vLLM, autoscaling with Kubernetes, and potential multi-cloud setups.
- Key metrics and concepts to track include TTFT (time to first token), TPS (tokens per second), ITL (intertoken latency), KV cache, and the prefill/decode phases.
Excerpt
Two years ago, we learned about how LLMs work at a high level from the ChatGPT team , and today, almost all software engineers use large language models (LLMs) in our day-to-day work. The most visible part of using an LLM is inference ; when an existing model takes an input (prompt) and generates an output, one token at a time. So, with AI models and AI agents everywhere across the tech industry in 2026, that means so is inference. And now, inference engineering is becoming more widespread, too, as open LLM models grow more capable. This is because with…