What is Inference?

Inference is the deployment-time counterpart to training. While training involves learning model parameters from data over hours or days, inference applies those learned parameters to make predictions on new inputs, typically in milliseconds to seconds. The engineering challenges of inference differ fundamentally from training and have spawned their own subfield.

Inference optimization techniques include model quantization (reducing numerical precision from 32-bit to 8-bit or 4-bit), pruning (removing unnecessary weights), knowledge distillation (training smaller models to mimic larger ones), and compilation (optimizing model graphs for specific hardware). These techniques can reduce model size and latency by 2-10x with minimal accuracy loss.

For LLMs, inference involves unique challenges. Autoregressive generation produces one token at a time, creating sequential dependencies that limit parallelization. KV-cache management stores intermediate attention computations to avoid redundant computation across tokens. Batching strategies group multiple requests to improve hardware utilization. Speculative decoding uses a smaller draft model to propose multiple tokens that the larger model verifies in parallel.

Infrastructure for serving AI models has become its own domain. Frameworks like vLLM, TensorRT-LLM, and Triton Inference Server optimize throughput and latency. Serverless inference platforms handle scaling automatically. Edge deployment brings models closer to users for lower latency. The cost of inference, particularly for LLMs, is a major factor in AI product economics and drives ongoing optimization research.