# LLM Fundamentals

Core concepts underlying modern Large Language Models. Essential grounding for agent engineering.

---

## Transformer Architecture

### Attention Mechanism
```
Attention(Q, K, V) = softmax(QK^T / √d_k) V
```

| Component | Role |
|---|---|
| **Query (Q)** | What we're looking for |
| **Key (K)** | What we're matching against |
| **Value (V)** | What we're extracting |
| **Scale (√d_k)** | Prevents gradient vanishing for large d_k |

### Multi-Head Attention
- Multiple attention heads in parallel
- Each head learns different relational patterns
- Concatenated + projected: `MultiHead(Q,K,V) = Concat(head_1...head_h)W^O`

### Transformer Block
```
Input → LayerNorm → MultiHeadAttention → Residual
    → LayerNorm → FFN (GeLU/SwiGLU) → Residual → Output
```

**Key variants**:
- **Pre-LN** (standard): LayerNorm before attention/FFN
- **Post-LN** (original): LayerNorm after residual
- **Parallel**: Attention + FFN in parallel (PaLM, GPT-NeoX)

---

## Training Pipeline

### 1. Pretraining
**Objective**: Next-token prediction (causal LM)
```
L = -Σ log P(x_t | x_<t; θ)
```

**Data**: Trillions of tokens (CommonCrawl, code, books, Wikipedia)
- **Deduplication** critical for quality
- **Data mixing ratios** heavily tuned
- **Curriculum learning** (easy → hard) sometimes used

**Compute scaling laws** (Kaplan et al., Chinchilla):
```
L(N, D) ≈ E + A/N^α + B/D^β
```
- Optimal: `N ∝ D^0.5` (Chinchilla) vs `N ∝ D^0.3` (Kaplan)
- **Compute-optimal**: Train smaller models longer on more data

### 2. Supervised Fine-Tuning (SFT)
**Objective**: Instruction following
- High-quality (prompt, response) pairs
- **Formats**: Alpaca, ShareGPT, ChatML, system+user+assistant
- **Data quality > quantity** (LIMA: 1k examples sufficient)

### 3. Preference Alignment
| Method | Mechanism | Pros | Cons |
|---|---|---|---|
| **RLHF/PPO** | Reward model + RL | Strong alignment | Complex, unstable |
| **DPO** | Direct preference optimization | Simple, stable | Requires paired prefs |
| **KTO** | Kahneman-Tversky optimization | Only needs (x, y, label) | Newer, less proven |
| **ORPO** | Odds ratio preference optimization | No reference model | Recent |
| **RLAIF** | AI feedback (constitutional) | Scalable | Depends on critic quality |

### 4. Continued Pretraining / Domain Adaptation
- Additional pretraining on domain corpus (code, legal, medical)
- Often + SFT on domain instructions
- **Catastrophic forgetting** mitigated by replay, low LR

---

## Inference Mechanics

### Autoregressive Generation
```
for t in 1..T:
    logits = model(x_<t)
    x_t = sample(logits / temperature)
```

### Sampling Strategies
| Strategy | Description | Use Case |
|---|---|---|
| **Greedy** | argmax | Deterministic, debugging |
| **Temperature** | Softmax(logits/T) | Creative (T>1), focused (T<1) |
| **Top-k** | Keep top k tokens | Balance diversity/quality |
| **Top-p (nucleus)** | Keep minimal set with cumprob ≥ p | Dynamic, popular default |
| **Typical** | Tokens near expected information content | Coherent, less repetitive |

### KV Cache
- **Problem**: O(T²) attention recomputation
- **Solution**: Cache K, V for past tokens
- **Memory**: 2 × layers × heads × head_dim × seq_len × bytes
- **Optimizations**: Quantization (KV-8bit), sliding window, paging (vLLM)

### Speculative Decoding
```
Draft model (small) → generates k tokens
Target model (large) → verifies in parallel
Accept/reject → if reject, fallback to target
```
- **Speedup**: 2-3× typical
- **Draft models**: Smaller distilled, n-gram, MLP predictor

---

## Model Families & Architectures

### Dense Transformers
| Model | Architecture | Context | Notable |
|---|---|---|---|
| **GPT-4/4o** | Dense (MoE rumored) | 128k/1M | Multimodal, strong reasoning |
| **Claude 3/3.5** | Dense | 200k/1M | Constitutional AI, long context |
| **Llama 3/3.1** | Dense (GQA) | 8k/128k | Open weights, strong base |
| **Qwen 2/2.5** | Dense (GQA) | 32k/128k | Multilingual, code strong |
| **Nemotron 3/4** | Dense | 4k/128k | NVIDIA, alignment focus |

### Mixture of Experts (MoE)
| Model | Experts | Active | Notes |
|---|---|---|---|
| **Mixtral 8x7B** | 8 | 2 | 47B total, 13B active |
| **Mixtral 8x22B** | 8 | 2 | 141B total, 39B active |
| **DeepSeek-V2/V3** | 160/256 | 6/8 | MLA, auxiliary-loss-free |
| **Qwen-MoE** | 64 | 4 | Open weights |
| **Nemotron-4-340B** | ? | ? | NVIDIA, synthetic data |

### Specialized Architectures
| Architecture | Innovation | Examples |
|---|---|---|
| **MLA (Multi-head Latent Attention)** | Compressed KV cache | DeepSeek-V2/V3 |
| **GQA (Grouped Query Attention)** | Fewer KV heads | Llama 3, Qwen, Gemma |
| **MQA (Multi-Query Attention)** | Single KV head | PaLM, Falcon |
| **Sliding Window** | Local attention | Mistral, LongLoRA |
| **Linear Attention / RNN hybrids** | O(N) inference | RWKV, Mamba, RetNet, Griffin |

---

## Scaling Laws & Trade-offs

### Chinchilla Scaling (Compute-Optimal)
| Compute (FLOPs) | Params (N) | Tokens (D) | Example |
|---|---|---|---|
| 10^18 | 1.3B | 200B | Small |
| 10^21 | 7B | 1.4T | Medium |
| 10^24 | 70B | 14T | Large |
| 10^25 | 340B | 60T | Frontier |

### Inference Cost Model
```
Cost ∝ (Prefill: O(N × L²) + Decode: O(N × L × T))
```
- **Prefill**: Quadratic in context (attention)
- **Decode**: Linear in context (KV cache)
- **Optimization**: Reduce N (distillation, quantization), reduce L (compression), reduce T (speculative)

### Quantization
| Method | Bits | Quality Loss | Speedup |
|---|---|---|---|
| **FP16/BF16** | 16 | None | 1× (baseline) |
| **INT8** | 8 | Minimal | ~1.5× |
| **GPTQ/AWQ** | 4 | Low | ~2-3× |
| **GGUF/llama.cpp** | 2-8 | Variable | CPU-friendly |
| **FP8** | 8 (E4M3/E5M2) | Low | H100 native |

---

## Multimodal Foundations

### Vision-Language
- **Architecture**: Vision encoder (ViT/CLIP) + projector + LLM
- **Training**: Image-text pairs → alignment → instruction tuning
- **Examples**: GPT-4o, Claude 3.5, LLaVA, Qwen-VL, InternVL

### Audio/Speech
- **Architecture**: Audio encoder (Whisper-style) + projector + LLM
- **Streaming**: Chunked processing, voice activity detection
- **Examples**: GPT-4o Audio, Qwen-Audio, Moshi, Ultravox

### Unified Multimodal
- **Native**: Single transformer from pixels/tokens/audio (Chameleon, Transfusion)
- **Tokenization**: Discrete tokens for all modalities (VQ-VAE, DAC)

---

## Key References

| Topic | Essential Reading |
|---|---|
| **Transformer** | "Attention Is All You Need" (Vaswani et al., 2017) |
| **Scaling Laws** | "Scaling Laws for Neural Language Models" (Kaplan et al., 2020) |
| **Chinchilla** | "Training Compute-Optimal Large Language Models" (Hoffmann et al., 2022) |
| **RLHF** | "Training Language Models to Follow Instructions" (Ouyang et al., 2022) |
| **DPO** | "Direct Preference Optimization" (Rafailov et al., 2023) |
| **MoE** | "GShard: Scaling Giant Models" (Lepikhin et al., 2020) |
| **DeepSeek MoE** | "DeepSeek-V2: A Strong, Economical Mixture-of-Experts" (2024) |
| **Speculative Decoding** | "Accelerating LLM Inference" (Leviathan et al., 2023) |
| **vLLM** | "Efficient Memory Management for LLM Serving" (Kwon et al., 2023) |

---

## Related Notes

- [[Foundations/Agent-Definitions.md]] — Agents built on these foundations
- [[ModelArchitectures/Mixture-of-Experts.md]] — MoE deep dive
- [[ModelArchitectures/Inference-Optimization.md]] — Serving, KV cache, speculative
- [[ModelArchitectures/Reasoning-Models.md]] — CoT, ToT, self-consistency
- [[ContextEngineering/Context-Window-Management.md]] — Long context strategies

---

*Sources: Original papers, LLM training books (Sebastian Raschka, Huyen Chip), framework docs (vLLM, SGLang, TensorRT-LLM), scaling law replications, Chinchilla/Hoffmann et al., DeepSeek technical reports.*