Quantization for Edge
Quantization techniques for deploying AI models on edge and on-device hardware: INT8, INT4, AWQ, GPTQ, GGUF, and runtime selection.
Quantization Techniques
| Technique |
Precision |
Memory Reduction |
Quality Loss |
Best For |
| FP16 |
16-bit |
~2x vs FP32 |
Minimal |
GPU inference, quality-critical |
| INT8 |
8-bit |
~4x vs FP32 |
Small |
Edge GPUs, mobile NPUs |
| INT4 |
4-bit |
~8x vs FP32 |
Moderate |
On-device, tight memory |
| AWQ |
4-bit (activation-aware) |
~8x |
Low |
LLMs, preserves salient weights |
| GPTQ |
4-bit (gradient-based) |
~8x |
Low |
LLMs, one-shot calibration |
| GGUF |
4-bit/8-bit (file format) |
~8x |
Low |
llama.cpp, local/edge inference |
| QLoRA |
4-bit (training) |
~8x |
Minimal |
Fine-tuning on edge hardware |
Decision Guidance
| Situation |
Recommended Approach |
| Privacy-sensitive data, no cloud |
Local inference with quantized SLM (1B–7B) |
| Intermittent connectivity |
On-device model with cloud fallback |
| Tight latency budget |
Quantized SLM locally (no network hop) |
| Limited device memory |
INT4 quantization + memory-mapped loading |
| Quality-critical, edge GPU available |
INT8 or FP16 on device |
Anti-Patterns
| Anti-Pattern |
Fix |
| Running a 70B model on-device |
Use a 1B–3B SLM; offload complex tasks to cloud |
| Sending every prompt to the cloud |
Local inference first; cloud only for complex tasks |
| Ignoring thermal throttling |
Benchmark on target device; account for sustained load |
| Full-precision weights on mobile |
Quantize to INT4/INT8; use GGUF or Core ML |
| No calibration data |
Use representative dataset for AWQ/GPTQ calibration |
- Local Inference Runtimes — llama.cpp, MLX, ONNX Runtime, Core ML
- Small Language Models — SLMs, distillation, quantization, edge
- Inference Optimization — Quantization, speculative decoding
- Inference — Model serving and inference engines
- Reliability — Guardrails and failover for local models
Quantization Example: AWQ
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
model_path = "meta-llama/Llama-3.1-8B-Instruct"
quant_path = "meta-llama/Llama-3.1-8B-Instruct-AWQ-INT4"
# Load model
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)
# Configure quantization
quant_config = {
"zero_point": True,
"q_group_size": 128,
"w_bit": 4,
"version": "GEMM",
}
# Quantize using calibration dataset
model.quantize(tokenizer, quant_config=quant_config, calib_data=calibration_dataset)
# Save quantized model
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
# Load and run quantized model
model = AutoAWQForCausalLM.from_quantized(quant_path)
Quantization Example: llama.cpp (GGUF)
# Convert and quantize a model for local edge inference
python convert-hf-to-gguf.py \
--model meta-llama/Llama-3.1-8B-Instruct \
--outtype f16 \
--outfile llama-3.1-8b-f16.gguf
# Quantize to INT4 (Q4_K_M = balanced quality/size)
./llama-quantize llama-3.1-8b-f16.gguf llama-3.1-8b-Q4_K_M.gguf Q4_K_M
# Run locally on CPU/edge
./llama-server \
-m llama-3.1-8b-Q4_K_M.gguf \
-c 4096 \
-t 4 \
--host 0.0.0.0 \
--port 8080
Sources
- llama.cpp, MLX, ONNX Runtime, Core ML, TensorFlow Lite, GGUF, AWQ (Lin et al.), GPTQ (Frantar et al.), QLoRA (Dettmers et al.), NVIDIA Jetson, Apple Silicon ML, Qualcomm AI Hub.*
