LLM Fundamentals

Core concepts underlying modern Large Language Models. Essential grounding for agent engineering.

Transformer Architecture

Attention Mechanism

Attention(Q, K, V) = softmax(QK^T / √d_k) V

Multi-Head Attention

• Multiple attention heads in parallel
• Each head learns different relational patterns
• Concatenated + projected: MultiHead(Q,K,V) = Concat(head_1...head_h)W^O

Transformer Block

Input → LayerNorm → MultiHeadAttention → Residual
    → LayerNorm → FFN (GeLU/SwiGLU) → Residual → Output

Key variants: - Pre-LN (standard): LayerNorm before attention/FFN - Post-LN (original): LayerNorm after residual - Parallel: Attention + FFN in parallel (PaLM, GPT-NeoX)

Training Pipeline

1. Pretraining

Objective: Next-token prediction (causal LM)

L = -Σ log P(x_t | x_<t; θ)

Data: Trillions of tokens (CommonCrawl, code, books, Wikipedia) - Deduplication critical for quality - Data mixing ratios heavily tuned - Curriculum learning (easy → hard) sometimes used

Compute scaling laws (Kaplan et al., Chinchilla):

L(N, D) ≈ E + A/N^α + B/D^β

• Optimal: N ∝ D^0.5 (Chinchilla) vs N ∝ D^0.3 (Kaplan)
• Compute-optimal: Train smaller models longer on more data

2. Supervised Fine-Tuning (SFT)

Objective: Instruction following - High-quality (prompt, response) pairs - Formats: Alpaca, ShareGPT, ChatML, system+user+assistant - Data quality > quantity (LIMA: 1k examples sufficient)

3. Preference Alignment

4. Continued Pretraining / Domain Adaptation

• Additional pretraining on domain corpus (code, legal, medical)
• Often + SFT on domain instructions
• Catastrophic forgetting mitigated by replay, low LR

Inference Mechanics

Autoregressive Generation

for t in 1..T:
    logits = model(x_<t)
    x_t = sample(logits / temperature)

Sampling Strategies

KV Cache

• Problem: O(T²) attention recomputation
• Solution: Cache K, V for past tokens
• Memory: 2 × layers × heads × head_dim × seq_len × bytes
• Optimizations: Quantization (KV-8bit), sliding window, paging (vLLM)

Speculative Decoding

Draft model (small) → generates k tokens
Target model (large) → verifies in parallel
Accept/reject → if reject, fallback to target

• Speedup: 2-3× typical
• Draft models: Smaller distilled, n-gram, MLP predictor

Model Families & Architectures

Dense Transformers

Mixture of Experts (MoE)

Specialized Architectures

Scaling Laws & Trade-offs

Chinchilla Scaling (Compute-Optimal)

Inference Cost Model

Cost ∝ (Prefill: O(N × L²) + Decode: O(N × L × T))

• Prefill: Quadratic in context (attention)
• Decode: Linear in context (KV cache)
• Optimization: Reduce N (distillation, quantization), reduce L (compression), reduce T (speculative)

Quantization

Multimodal Foundations

Vision-Language

• Architecture: Vision encoder (ViT/CLIP) + projector + LLM
• Training: Image-text pairs → alignment → instruction tuning
• Examples: GPT-4o, Claude 3.5, LLaVA, Qwen-VL, InternVL

Audio/Speech

• Architecture: Audio encoder (Whisper-style) + projector + LLM
• Streaming: Chunked processing, voice activity detection
• Examples: GPT-4o Audio, Qwen-Audio, Moshi, Ultravox

Unified Multimodal

• Native: Single transformer from pixels/tokens/audio (Chameleon, Transfusion)
• Tokenization: Discrete tokens for all modalities (VQ-VAE, DAC)

Key References

Related Notes

• Agent Definitions — Agents built on these foundations
• Mixture of Experts — MoE deep dive
• Inference Optimization — Serving, KV cache, speculative
• Reasoning Models — CoT, ToT, self-consistency
• Context Window Management — Long context strategies

Sources: Original papers, LLM training books (Sebastian Raschka, Huyen Chip), framework docs (vLLM, SGLang, TensorRT-LLM), scaling law replications, Chinchilla/Hoffmann et al., DeepSeek technical reports.