Batching and Quantization

Continuous batching, PagedAttention, and quantization — the three levers that determine inference throughput, latency, and cost per token.

Continuous Batching

Traditional batching waits for all requests in a batch to complete before starting the next batch. Continuous batching (iterative batching) adds new requests to a running batch as others finish, keeping GPUs saturated.

graph TD
    subgraph Traditional Batching
        A[Batch 1: Req 1,2,3] --> B[Wait for all to finish]
        B --> C[Batch 2: Req 4,5,6]
        C --> D[Idle GPU during tail latency]
    end

    subgraph Continuous Batching
        E[Running Batch: Req 1,2,3] --> F[Req 1 finishes]
        F --> G[Add Req 4 to batch]
        G --> H[Req 2 finishes]
        H --> I[Add Req 5 to batch]
        I --> J[GPU stays saturated]
    end

    style D fill:#ffcdd2
    style J fill:#e8f5e9

PagedAttention

PagedAttention (vLLM) manages the KV cache like virtual memory: non-contiguous blocks, reducing fragmentation and waste.

# Conceptual PagedAttention block table
block_table = {
    "request_1": [block_10, block_25, block_41],   # non-contiguous
    "request_2": [block_11, block_26],              # shares block 10's content if prefix match
    "request_3": [block_12, block_27, block_42],
}

# With prefix caching, shared prefixes map to same blocks
# "What is the capital of" -> blocks [10, 25]
# Req 1: "What is the capital of France?" -> [10, 25, 41]
# Req 2: "What is the capital of Germany?" -> [10, 25, 26]

Quantization Deep Dive

Post-Training Quantization (PTQ)

Quantize a pre-trained model without retraining.

Quantization-Aware Training (QAT)

Simulate quantization during training so the model adapts to lower precision.

Decision Guidance

Anti-Patterns

Related Notes

• Inference Engines — vLLM, SGLang, TensorRT-LLM engine comparison
• Quantization For Edge — Edge-specific quantization techniques
• Inference Optimization — vLLM, SGLang, speculative decoding
• Observability — Monitoring throughput, cost per token, KV-cache usage

vLLM Batching Configuration

from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    tensor_parallel_size=8,
    gpu_memory_utilization=0.95,   # Higher = larger batches, more OOM risk
    max_model_len=8192,
    enable_prefix_caching=True,    # Critical for RAG / multi-turn
    swap_space=16,                 # CPU swap space for KV cache overflow (GiB)
    max_num_seqs=256,              # Max sequences in continuous batch
    max_num_batched_tokens=4096,   # Max tokens per batch iteration
)

Quantization Quality Validation

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

def evaluate_quantization(original_model_path, quantized_model_path, test_prompts):
    tokenizer = AutoTokenizer.from_pretrained(original_model_path)

    original = AutoModelForCausalLM.from_pretrained(original_model_path)
    quantized = AutoModelForCausalLM.from_pretrained(quantized_model_path)

    results = []
    for prompt in test_prompts:
        inputs = tokenizer(prompt, return_tensors="pt")

        with torch.no_grad():
            orig_out = original.generate(**inputs, max_new_tokens=50)
            quant_out = quantized.generate(**inputs, max_new_tokens=50)

        orig_text = tokenizer.decode(orig_out[0], skip_special_tokens=True)
        quant_text = tokenizer.decode(quant_out[0], skip_special_tokens=True)

        results.append({
            "prompt": prompt,
            "original": orig_text,
            "quantized": quant_text,
            "match": orig_text == quant_text,
        })

    return results

Sources

• vLLM PagedAttention paper (Kwon et al.), SGLang RadixAttention paper (Wang et al.), AWQ (Lin et al.), GPTQ (Frantar et al.), QLoRA (Dettmers et al.), GGUF/llama.cpp, NVIDIA TensorRT-LLM.*