# EdgeAI

On-device AI: quantization, small language models, local inference, and edge deployment.

## Core Concepts

| Concept | Description |
|---|---|
| **Small Language Model (SLM)** | Compact models (1B–7B params) optimized for edge/on-device |
| **Quantization** | Reducing weight precision (INT8, INT4, AWQ, GPTQ, GGUF) for memory and speed |
| **Local Inference** | Running models on-device without cloud round-trips |
| **ONNX Runtime** | Cross-platform inference engine for edge devices |
| **Core ML / TensorFlow Lite** | Mobile-native model runtimes (iOS/Android) |
| **Memory-Mapped Models** | Loading model weights via mmap to reduce resident memory |

## Decision Guidance

| Situation | Recommended Approach |
|---|---|
| Privacy-sensitive data | Local inference; never send prompts to the cloud |
| Intermittent connectivity | On-device model with cloud fallback when online |
| Tight latency budget | Quantized SLM running locally (no network hop) |
| Limited device memory | Aggressive quantization (INT4) + memory-mapped loading |

## Anti-Patterns

| Anti-Pattern | Fix |
|---|---|
| Running a 70B model on-device | Use a 1B–3B SLM; offload heavy tasks to cloud |
| Sending every prompt to the cloud | Local inference first; cloud only for complex tasks |
| Ignoring thermal/throttling | Benchmark on target device; account for sustained load |
| Full-precision weights on mobile | Quantize to INT4/INT8; use GGUF or Core ML |

## Related Notes

- [[AgenticEngineering/ModelArchitectures/Small-Language-Models.md]] — SLMs, distillation, quantization, edge
- [[AgenticEngineering/ModelArchitectures/Inference-Optimization.md]] — Quantization, speculative decoding
- [[Inference/]] — Model serving and inference engines
- [[Reliability/]] — Guardrails and failover for local models

## Sources

* Llama.cpp, MLX, ONNX Runtime, Core ML, TensorFlow Lite, GGUF, AWQ, GPTQ, NVIDIA Jetstein, Apple Silicon ML, Qualcomm AI Hub.*