EdgeAI
On-device AI: quantization, small language models, local inference, and edge deployment.
Core Concepts
| Concept |
Description |
| Small Language Model (SLM) |
Compact models (1B–7B params) optimized for edge/on-device |
| Quantization |
Reducing weight precision (INT8, INT4, AWQ, GPTQ, GGUF) for memory and speed |
| Local Inference |
Running models on-device without cloud round-trips |
| ONNX Runtime |
Cross-platform inference engine for edge devices |
| Core ML / TensorFlow Lite |
Mobile-native model runtimes (iOS/Android) |
| Memory-Mapped Models |
Loading model weights via mmap to reduce resident memory |
Decision Guidance
| Situation |
Recommended Approach |
| Privacy-sensitive data |
Local inference; never send prompts to the cloud |
| Intermittent connectivity |
On-device model with cloud fallback when online |
| Tight latency budget |
Quantized SLM running locally (no network hop) |
| Limited device memory |
Aggressive quantization (INT4) + memory-mapped loading |
Anti-Patterns
| Anti-Pattern |
Fix |
| Running a 70B model on-device |
Use a 1B–3B SLM; offload heavy tasks to cloud |
| Sending every prompt to the cloud |
Local inference first; cloud only for complex tasks |
| Ignoring thermal/throttling |
Benchmark on target device; account for sustained load |
| Full-precision weights on mobile |
Quantize to INT4/INT8; use GGUF or Core ML |
- Small Language Models — SLMs, distillation, quantization, edge
- Inference Optimization — Quantization, speculative decoding
- Inference — Model serving and inference engines
- Reliability — Guardrails and failover for local models
Sources
- Llama.cpp, MLX, ONNX Runtime, Core ML, TensorFlow Lite, GGUF, AWQ, GPTQ, NVIDIA Jetstein, Apple Silicon ML, Qualcomm AI Hub.*
