Inference
Model serving and inference for AI systems: inference engines, serving stacks, batching, optimization, and cost-per-token governance.
Core Concepts
| Concept |
Description |
| Model Server |
Loads a model and exposes it via HTTP/gRPC for inference requests |
| Batching |
Combining multiple requests into one forward pass to improve throughput |
| Continuous Batching |
Dynamically adding/removing requests from a batch as they complete (vLLM, SGLang) |
| PagedAttention |
Memory management for KV caches that reduces fragmentation and waste |
| Speculative Decoding |
Using a small draft model to speculate tokens, verified by a large target model |
| Quantization |
Reducing model weight precision (FP16, INT8, INT4, AWQ, GPTQ) to cut memory and latency |
| Cost per Token |
The primary efficiency metric: dollars (or energy) per generated token |
Decision Guidance
| Situation |
Recommended Approach |
| Low latency, single user |
Streaming inference with a small model on GPU |
| High throughput, batch workloads |
Continuous batching + PagedAttention engine (vLLM/SGLang) |
| Cost-sensitive at scale |
Quantized model + speculative decoding |
| Edge / on-device |
Small language model, quantized, local runtime |
| Multi-tenant, many models |
Model multiplexing / scheduling on shared GPUs |
Anti-Patterns
| Anti-Pattern |
Fix |
| Serving a 70B model on a single consumer GPU |
Offload to CPU, quantize aggressively, or use a smaller model |
| No batching under load |
Enable continuous batching; measure throughput before tuning |
| Ignoring KV-cache memory |
Use PagedAttention; set max_model_len and gpu_memory_utilization |
| Deploying without cost-per-token tracking |
Instrument token counts and dollar cost per request |
- Inference Optimization — vLLM, SGLang, TensorRT-LLM, speculative decoding
- Small Language Models — SLMs, distillation, quantization, edge
- Model Routing — Cascade, mixture-of-agents, dynamic selection
- MLOps — Deployment and rollout for inference endpoints
- Observability — Monitoring inference latency, cost, error rates
Sources
- NVIDIA Triton Inference Server, vLLM, SGLang, TensorRT-LLM, PagedAttention paper, vLLM blog, AWS AI Inferentia, Google TPU docs, Hugging Face Text Generation Inference.*
