# Batching and Quantization

Continuous batching, PagedAttention, and quantization — the three levers that determine inference throughput, latency, and cost per token.

## Continuous Batching

Traditional batching waits for all requests in a batch to complete before starting the next batch. Continuous batching (iterative batching) adds new requests to a running batch as others finish, keeping GPUs saturated.

```mermaid
graph TD
    subgraph Traditional Batching
        A[Batch 1: Req 1,2,3] --> B[Wait for all to finish]
        B --> C[Batch 2: Req 4,5,6]
        C --> D[Idle GPU during tail latency]
    end
    
    subgraph Continuous Batching
        E[Running Batch: Req 1,2,3] --> F[Req 1 finishes]
        F --> G[Add Req 4 to batch]
        G --> H[Req 2 finishes]
        H --> I[Add Req 5 to batch]
        I --> J[GPU stays saturated]
    end
    
    style D fill:#ffcdd2
    style J fill:#e8f5e9
```

| Metric | Traditional Batching | Continuous Batching |
|---|---|---|
| GPU utilization | Variable (idle during tail) | Consistently high |
| Throughput | Lower | Higher |
| Latency (p99) | Higher (head-of-line blocking) | Lower |
| Implementation complexity | Simple | Complex (PagedAttention required) |

## PagedAttention

PagedAttention (vLLM) manages the KV cache like virtual memory: non-contiguous blocks, reducing fragmentation and waste.

```python
# Conceptual PagedAttention block table
block_table = {
    "request_1": [block_10, block_25, block_41],   # non-contiguous
    "request_2": [block_11, block_26],              # shares block 10's content if prefix match
    "request_3": [block_12, block_27, block_42],
}

# With prefix caching, shared prefixes map to same blocks
# "What is the capital of" -> blocks [10, 25]
# Req 1: "What is the capital of France?" -> [10, 25, 41]
# Req 2: "What is the capital of Germany?" -> [10, 25, 26]
```

| Benefit | Impact |
|---|---|
| Reduced KV-cache waste | 2–4x more sequences per GPU |
| Larger max batch size | Higher throughput at same latency |
| Prefix caching | Shared prompts computed once |

## Quantization Deep Dive

### Post-Training Quantization (PTQ)

Quantize a pre-trained model without retraining.

| Method | Calibration | Quality | Speed | Use Case |
|---|---|---|---|---|
| **INT8 (static)** | Small dataset | Good | Fast | General edge deployment |
| **INT8 (dynamic)** | None | Good | Fast | Activations vary widely |
| **INT4 (AWQ)** | Calibration set | Very good | Medium | LLMs, production edge |
| **INT4 (GPTQ)** | Calibration set | Very good | Medium | LLMs, one-shot |
| **INT4 (GGUF)** | None | Good | Fast | llama.cpp, local inference |

### Quantization-Aware Training (QAT)

Simulate quantization during training so the model adapts to lower precision.

| Method | When to Use | Cost | Quality |
|---|---|---|---|
| **QLoRA** | Fine-tuning on limited GPU | Low | Near full-precision |
| **LoftQ** | Combined quantization + LoRA | Medium | Good |
| **QAT (full)** | Maximum quality at low precision | High | Best |

## Decision Guidance

| Situation | Recommended Approach |
|---|---|
| Maximize throughput on GPU cluster | Continuous batching + PagedAttention (vLLM/SGLang) |
| Shared prefixes across requests (RAG, chat) | Enable prefix caching (RadixAttention/PagedAttention) |
| Deploy on edge / mobile | INT4 quantization (GGUF/AWQ) + small model |
| Quality-critical, some headroom | INT8 quantization |
| Fine-tuning on limited hardware | QLORA (4-bit) |

## Anti-Patterns

| Anti-Pattern | Fix |
|---|---|
| No batching under load | Enable continuous batching; measure throughput at target QPS |
| Ignoring KV-cache memory | Set `gpu_memory_utilization` and `max_model_len`; monitor OOM |
| Quantizing without calibration | Use representative dataset for AWQ/GPTQ; validate quality after |
| Full precision on memory-constrained device | INT4 quantization + memory-mapped loading |
| No prefix caching for repetitive prompts | Enable prefix caching; measure cache hit rate |

## Related Notes

- [[Inference/Inference-Engines.md]] — vLLM, SGLang, TensorRT-LLM engine comparison
- [[EdgeAI/Quantization-For-Edge.md]] — Edge-specific quantization techniques
- [[AgenticEngineering/ModelArchitectures/Inference-Optimization.md]] — vLLM, SGLang, speculative decoding
- [[Observability/]] — Monitoring throughput, cost per token, KV-cache usage

## vLLM Batching Configuration

```python
from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    tensor_parallel_size=8,
    gpu_memory_utilization=0.95,   # Higher = larger batches, more OOM risk
    max_model_len=8192,
    enable_prefix_caching=True,    # Critical for RAG / multi-turn
    swap_space=16,                 # CPU swap space for KV cache overflow (GiB)
    max_num_seqs=256,              # Max sequences in continuous batch
    max_num_batched_tokens=4096,   # Max tokens per batch iteration
)
```

## Quantization Quality Validation

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

def evaluate_quantization(original_model_path, quantized_model_path, test_prompts):
    tokenizer = AutoTokenizer.from_pretrained(original_model_path)
    
    original = AutoModelForCausalLM.from_pretrained(original_model_path)
    quantized = AutoModelForCausalLM.from_pretrained(quantized_model_path)
    
    results = []
    for prompt in test_prompts:
        inputs = tokenizer(prompt, return_tensors="pt")
        
        with torch.no_grad():
            orig_out = original.generate(**inputs, max_new_tokens=50)
            quant_out = quantized.generate(**inputs, max_new_tokens=50)
        
        orig_text = tokenizer.decode(orig_out[0], skip_special_tokens=True)
        quant_text = tokenizer.decode(quant_out[0], skip_special_tokens=True)
        
        results.append({
            "prompt": prompt,
            "original": orig_text,
            "quantized": quant_text,
            "match": orig_text == quant_text,
        })
    
    return results
```

## Sources

* vLLM PagedAttention paper (Kwon et al.), SGLang RadixAttention paper (Wang et al.), AWQ (Lin et al.), GPTQ (Frantar et al.), QLoRA (Dettmers et al.), GGUF/llama.cpp, NVIDIA TensorRT-LLM.*