# Inference Engines

Model serving infrastructure for AI systems: vLLM, SGLang, TensorRT-LLM, NVIDIA Triton, and Hugging Face TGI — how they work, when to choose each, and how to deploy them.

## Core Concepts

| Concept | Description |
|---|---|
| **Inference Engine** | Runtime that loads model weights and executes forward passes, exposing HTTP/gRPC endpoints |
| **KV Cache** | Key-value cache storing prior token activations to avoid recomputation during autoregressive generation |
| **PagedAttention** | Attention algorithm that manages KV cache in non-contiguous memory blocks, reducing fragmentation (vLLM) |
| **Continuous Batching** | Iterative batching that adds new requests to a running batch as others finish, maximizing GPU utilization |
| **Speculative Decoding** | A small draft model proposes tokens in bulk; a large target model verifies them in parallel, speeding up generation |
| **Tensor Parallelism** | Splitting model layers across multiple GPUs within a single node to serve models too large for one GPU |
| **Pipeline Parallelism** | Splitting model layers across GPUs arranged in a pipeline; micro-batches flow through stages |

## Engine Comparison

| Engine | Best For | Key Strength | Limitation |
|---|---|---|---|
| **vLLM** | General-purpose LLM serving | PagedAttention, continuous batching, OpenAI-compatible API | Python overhead; single-node focus |
| **SGLang** | Complex workloads (RAG, multi-step) | RadixAttention for prefix caching, structured generation | Younger ecosystem |
| **TensorRT-LLM** | NVIDIA GPU optimization | Maximum throughput on NVIDIA hardware, production hardening | NVIDIA-only, more complex setup |
| **NVIDIA Triton** | Multi-model, multi-framework | Unified inference server for LLMs + traditional ML | More configuration overhead |
| **Hugging Face TGI** | Quick start, Hugging Face ecosystem | Easy deployment, built-in quantization support | Less tuning control than vLLM |

## Decision Guidance

| Situation | Recommended Engine | Why |
|---|---|---|
| Fastest time to production, OpenAI-compatible API | vLLM | Mature, well-documented, drop-in OpenAI replacement |
| RAG with long shared prefixes | SGLang | RadixAttention caches common prompt prefixes across requests |
| Maximum throughput on NVIDIA H100/A100 | TensorRT-LLM | Hardware-specific optimizations, quantization-aware |
| Serving LLMs alongside classical ML models | NVIDIA Triton | Single server for all model types, model ensembles |
| Hugging Face models, minimal infra work | TGI | `docker run` with model ID, built-in quantization |

## Anti-Patterns

| Anti-Pattern | Fix |
|---|---|
| Loading full FP16 model on a single consumer GPU | Quantize (INT8/INT4), offload to CPU, or use a smaller model |
| No batching under concurrent load | Enable continuous batching; measure throughput at target QPS |
| Ignoring KV-cache memory pressure | Set `gpu_memory_utilization` and `max_model_len`; monitor OOM kills |
| Deploying without token-level cost tracking | Instrument prompt + completion token counts; alert on budget burn |
| Running inference on CPU for latency-sensitive workloads | Move to GPU; use a quantized small model if GPU unavailable |

## Related Notes

- [[Inference/Batching-And-Quantization.md]] — Continuous batching, PagedAttention, quantization techniques
- [[AgenticEngineering/ModelArchitectures/Inference-Optimization.md]] — vLLM, SGLang, TensorRT-LLM, speculative decoding
- [[AgenticEngineering/ModelArchitectures/Model-Routing.md]] — Cascade routing, mixture-of-agents, dynamic model selection
- [[MLOps/]] — Deployment and rollout for inference endpoints
- [[Observability/]] — Monitoring inference latency, cost per token, error rates
- [[Platform/]] — Internal model-serving platform for AI teams

## Deployment Example: vLLM

```python
# Serve a model with vLLM OpenAI-compatible API
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    tensor_parallel_size=2,          # GPUs per node
    gpu_memory_utilization=0.90,     # Fraction of GPU memory for KV cache
    max_model_len=4096,              # Max sequence length
    enable_prefix_caching=True,      # Cache common prefixes
)

sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=512,
)

outputs = llm.generate(
    ["Explain quantum computing in simple terms.", "Write a Python function to sort a list."],
    sampling_params,
)
for output in outputs:
    print(output.outputs[0].text)
```

```yaml
# Docker Compose for vLLM serving
services:
  vllm:
    image: vllm/vllm-openai:latest
    ports:
      - "8000:8000"
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    environment:
      - CUDA_VISIBLE_DEVICES=0,1
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 2
              capabilities: [gpu]
    command: >
      --model meta-llama/Llama-3.1-8B-Instruct
      --tensor-parallel-size 2
      --gpu-memory-utilization 0.90
      --enable-prefix-caching
      --max-model-len 4096
```

## Sources

* vLLM paper (Kwon et al.), vLLM documentation, SGLang paper (Wang et al.), TensorRT-LLM documentation, NVIDIA Triton Inference Server, Hugging Face Text Generation Inference, PagedAttention, Continuous Batching research.*