Inference Engines

Model serving infrastructure for AI systems: vLLM, SGLang, TensorRT-LLM, NVIDIA Triton, and Hugging Face TGI — how they work, when to choose each, and how to deploy them.

Core Concepts

Concept	Description
Inference Engine	Runtime that loads model weights and executes forward passes, exposing HTTP/gRPC endpoints
KV Cache	Key-value cache storing prior token activations to avoid recomputation during autoregressive generation
PagedAttention	Attention algorithm that manages KV cache in non-contiguous memory blocks, reducing fragmentation (vLLM)
Continuous Batching	Iterative batching that adds new requests to a running batch as others finish, maximizing GPU utilization
Speculative Decoding	A small draft model proposes tokens in bulk; a large target model verifies them in parallel, speeding up generation
Tensor Parallelism	Splitting model layers across multiple GPUs within a single node to serve models too large for one GPU
Pipeline Parallelism	Splitting model layers across GPUs arranged in a pipeline; micro-batches flow through stages

Engine Comparison

Engine	Best For	Key Strength	Limitation
vLLM	General-purpose LLM serving	PagedAttention, continuous batching, OpenAI-compatible API	Python overhead; single-node focus
SGLang	Complex workloads (RAG, multi-step)	RadixAttention for prefix caching, structured generation	Younger ecosystem
TensorRT-LLM	NVIDIA GPU optimization	Maximum throughput on NVIDIA hardware, production hardening	NVIDIA-only, more complex setup
NVIDIA Triton	Multi-model, multi-framework	Unified inference server for LLMs + traditional ML	More configuration overhead
Hugging Face TGI	Quick start, Hugging Face ecosystem	Easy deployment, built-in quantization support	Less tuning control than vLLM

Decision Guidance

Situation	Recommended Engine	Why
Fastest time to production, OpenAI-compatible API	vLLM	Mature, well-documented, drop-in OpenAI replacement
RAG with long shared prefixes	SGLang	RadixAttention caches common prompt prefixes across requests
Maximum throughput on NVIDIA H100/A100	TensorRT-LLM	Hardware-specific optimizations, quantization-aware
Serving LLMs alongside classical ML models	NVIDIA Triton	Single server for all model types, model ensembles
Hugging Face models, minimal infra work	TGI	`docker run` with model ID, built-in quantization

Anti-Patterns

Anti-Pattern	Fix
Loading full FP16 model on a single consumer GPU	Quantize (INT8/INT4), offload to CPU, or use a smaller model
No batching under concurrent load	Enable continuous batching; measure throughput at target QPS
Ignoring KV-cache memory pressure	Set `gpu_memory_utilization` and `max_model_len`; monitor OOM kills
Deploying without token-level cost tracking	Instrument prompt + completion token counts; alert on budget burn
Running inference on CPU for latency-sensitive workloads	Move to GPU; use a quantized small model if GPU unavailable

Batching And Quantization — Continuous batching, PagedAttention, quantization techniques
Inference Optimization — vLLM, SGLang, TensorRT-LLM, speculative decoding
Model Routing — Cascade routing, mixture-of-agents, dynamic model selection
MLOps — Deployment and rollout for inference endpoints
Observability — Monitoring inference latency, cost per token, error rates
Platform — Internal model-serving platform for AI teams

Deployment Example: vLLM

# Serve a model with vLLM OpenAI-compatible API
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    tensor_parallel_size=2,          # GPUs per node
    gpu_memory_utilization=0.90,     # Fraction of GPU memory for KV cache
    max_model_len=4096,              # Max sequence length
    enable_prefix_caching=True,      # Cache common prefixes
)

sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=512,
)

outputs = llm.generate(
    ["Explain quantum computing in simple terms.", "Write a Python function to sort a list."],
    sampling_params,
)
for output in outputs:
    print(output.outputs[0].text)

# Docker Compose for vLLM serving
services:
  vllm:
    image: vllm/vllm-openai:latest
    ports:
      - "8000:8000"
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    environment:
      - CUDA_VISIBLE_DEVICES=0,1
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 2
              capabilities: [gpu]
    command: >
      --model meta-llama/Llama-3.1-8B-Instruct
      --tensor-parallel-size 2
      --gpu-memory-utilization 0.90
      --enable-prefix-caching
      --max-model-len 4096

Sources

vLLM paper (Kwon et al.), vLLM documentation, SGLang paper (Wang et al.), TensorRT-LLM documentation, NVIDIA Triton Inference Server, Hugging Face Text Generation Inference, PagedAttention, Continuous Batching research.*

From the Systems Engineering research vault · CC BY 4.0 · all notes · raw markdown