# Inference

Model serving and inference for AI systems: inference engines, serving stacks, batching, optimization, and cost-per-token governance.

## Core Concepts

| Concept | Description |
|---|---|
| **Model Server** | Loads a model and exposes it via HTTP/gRPC for inference requests |
| **Batching** | Combining multiple requests into one forward pass to improve throughput |
| **Continuous Batching** | Dynamically adding/removing requests from a batch as they complete (vLLM, SGLang) |
| **PagedAttention** | Memory management for KV caches that reduces fragmentation and waste |
| **Speculative Decoding** | Using a small draft model to speculate tokens, verified by a large target model |
| **Quantization** | Reducing model weight precision (FP16, INT8, INT4, AWQ, GPTQ) to cut memory and latency |
| **Cost per Token** | The primary efficiency metric: dollars (or energy) per generated token |

## Decision Guidance

| Situation | Recommended Approach |
|---|---|
| Low latency, single user | Streaming inference with a small model on GPU |
| High throughput, batch workloads | Continuous batching + PagedAttention engine (vLLM/SGLang) |
| Cost-sensitive at scale | Quantized model + speculative decoding |
| Edge / on-device | Small language model, quantized, local runtime |
| Multi-tenant, many models | Model multiplexing / scheduling on shared GPUs |

## Anti-Patterns

| Anti-Pattern | Fix |
|---|---|
| Serving a 70B model on a single consumer GPU | Offload to CPU, quantize aggressively, or use a smaller model |
| No batching under load | Enable continuous batching; measure throughput before tuning |
| Ignoring KV-cache memory | Use PagedAttention; set max_model_len and gpu_memory_utilization |
| Deploying without cost-per-token tracking | Instrument token counts and dollar cost per request |

## Related Notes

- [[AgenticEngineering/ModelArchitectures/Inference-Optimization.md]] — vLLM, SGLang, TensorRT-LLM, speculative decoding
- [[AgenticEngineering/ModelArchitectures/Small-Language-Models.md]] — SLMs, distillation, quantization, edge
- [[AgenticEngineering/ModelArchitectures/Model-Routing.md]] — Cascade, mixture-of-agents, dynamic selection
- [[MLOps/]] — Deployment and rollout for inference endpoints
- [[Observability/]] — Monitoring inference latency, cost, error rates

## Sources

* NVIDIA Triton Inference Server, vLLM, SGLang, TensorRT-LLM, PagedAttention paper, vLLM blog, AWS AI Inferentia, Google TPU docs, Hugging Face Text Generation Inference.*