# Quantization for Edge

Quantization techniques for deploying AI models on edge and on-device hardware: INT8, INT4, AWQ, GPTQ, GGUF, and runtime selection.

## Quantization Techniques

| Technique | Precision | Memory Reduction | Quality Loss | Best For |
|---|---|---|---|---|
| **FP16** | 16-bit | ~2x vs FP32 | Minimal | GPU inference, quality-critical |
| **INT8** | 8-bit | ~4x vs FP32 | Small | Edge GPUs, mobile NPUs |
| **INT4** | 4-bit | ~8x vs FP32 | Moderate | On-device, tight memory |
| **AWQ** | 4-bit (activation-aware) | ~8x | Low | LLMs, preserves salient weights |
| **GPTQ** | 4-bit (gradient-based) | ~8x | Low | LLMs, one-shot calibration |
| **GGUF** | 4-bit/8-bit (file format) | ~8x | Low | llama.cpp, local/edge inference |
| **QLoRA** | 4-bit (training) | ~8x | Minimal | Fine-tuning on edge hardware |

## Decision Guidance

| Situation | Recommended Approach |
|---|---|
| Privacy-sensitive data, no cloud | Local inference with quantized SLM (1B–7B) |
| Intermittent connectivity | On-device model with cloud fallback |
| Tight latency budget | Quantized SLM locally (no network hop) |
| Limited device memory | INT4 quantization + memory-mapped loading |
| Quality-critical, edge GPU available | INT8 or FP16 on device |

## Anti-Patterns

| Anti-Pattern | Fix |
|---|---|
| Running a 70B model on-device | Use a 1B–3B SLM; offload complex tasks to cloud |
| Sending every prompt to the cloud | Local inference first; cloud only for complex tasks |
| Ignoring thermal throttling | Benchmark on target device; account for sustained load |
| Full-precision weights on mobile | Quantize to INT4/INT8; use GGUF or Core ML |
| No calibration data | Use representative dataset for AWQ/GPTQ calibration |

## Related Notes

- [[EdgeAI/Local-Inference-Runtimes.md]] — llama.cpp, MLX, ONNX Runtime, Core ML
- [[AgenticEngineering/ModelArchitectures/Small-Language-Models.md]] — SLMs, distillation, quantization, edge
- [[AgenticEngineering/ModelArchitectures/Inference-Optimization.md]] — Quantization, speculative decoding
- [[Inference/]] — Model serving and inference engines
- [[Reliability/]] — Guardrails and failover for local models

## Quantization Example: AWQ

```python
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "meta-llama/Llama-3.1-8B-Instruct"
quant_path = "meta-llama/Llama-3.1-8B-Instruct-AWQ-INT4"

# Load model
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

# Configure quantization
quant_config = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMM",
}

# Quantize using calibration dataset
model.quantize(tokenizer, quant_config=quant_config, calib_data=calibration_dataset)

# Save quantized model
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

# Load and run quantized model
model = AutoAWQForCausalLM.from_quantized(quant_path)
```

## Quantization Example: llama.cpp (GGUF)

```bash
# Convert and quantize a model for local edge inference
python convert-hf-to-gguf.py \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --outtype f16 \
    --outfile llama-3.1-8b-f16.gguf

# Quantize to INT4 (Q4_K_M = balanced quality/size)
./llama-quantize llama-3.1-8b-f16.gguf llama-3.1-8b-Q4_K_M.gguf Q4_K_M

# Run locally on CPU/edge
./llama-server \
    -m llama-3.1-8b-Q4_K_M.gguf \
    -c 4096 \
    -t 4 \
    --host 0.0.0.0 \
    --port 8080
```

## Sources

* llama.cpp, MLX, ONNX Runtime, Core ML, TensorFlow Lite, GGUF, AWQ (Lin et al.), GPTQ (Frantar et al.), QLoRA (Dettmers et al.), NVIDIA Jetson, Apple Silicon ML, Qualcomm AI Hub.*