# Local Inference Runtimes

On-device and edge inference runtimes: llama.cpp, MLX, ONNX Runtime, Core ML, TensorFlow Lite — when to use each and how to deploy.

## Runtime Comparison

| Runtime | Platform | Model Format | Best For | GPU/NPU Support |
|---|---|---|---|---|
| **llama.cpp** | Linux, macOS, Windows, Android | GGUF | Local LLM inference, CPU/GPU | CUDA, Metal, Vulkan, BLAS |
| **MLX** | Apple Silicon (macOS, iOS) | MLX arrays | On-device LLMs on Mac/iPhone | Apple Neural Engine, Metal |
| **ONNX Runtime** | Cross-platform | ONNX | General ML models, edge devices | CUDA, TensorRT, OpenVINO, ACL |
| **Core ML** | iOS, macOS | .mlmodel | iOS apps, Apple devices | Neural Engine, GPU |
| **TensorFlow Lite** | Android, iOS, Linux | .tflite | Mobile ML, microcontrollers | GPU, NNAPI, Edge TPU |
| **ExecuTorch** | Android, iOS | .pte | PyTorch models on mobile | Vulkan, CPU |

## Decision Guidance

| Situation | Recommended Runtime |
|---|---|
| Local LLM on laptop/desktop | llama.cpp (GGUF) |
| iOS app with on-device LLM | MLX or Core ML |
| Android app with ML model | TensorFlow Lite or ExecuTorch |
| Cross-platform edge device | ONNX Runtime |
| PyTorch model on mobile | ExecuTorch |

## Anti-Patterns

| Anti-Pattern | Fix |
|---|---|
| Running 70B model on-device | Use 1B–3B SLM; offload complex tasks to cloud |
| Full-precision weights on mobile | Quantize to INT4/INT8; use GGUF/Core ML |
| Ignoring thermal throttling | Benchmark on target device; account for sustained load |
| No fallback when on-device fails | Cloud fallback for complex requests |
| Sending every prompt to cloud | Local inference first; cloud only for complex tasks |

## Related Notes

- [[EdgeAI/Quantization-For-Edge.md]] — Quantization techniques for edge
- [[AgenticEngineering/ModelArchitectures/Small-Language-Models.md]] — SLMs, distillation, quantization
- [[AgenticEngineering/ModelArchitectures/Inference-Optimization.md]] — Quantization, speculative decoding
- [[Inference/]] — Model serving and inference engines
- [[Reliability/]] — Guardrails and failover for local models

## llama.cpp Deployment

```bash
# Build with GPU support (CUDA)
cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release

# Download and quantize a model
wget https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct-gguf/resolve/main/llama-3.1-8b-instruct-Q4_K_M.gguf

# Run a local server
./llama-server \
    -m llama-3.1-8b-instruct-Q4_K_M.gguf \
    -c 4096 \
    -ngl 35 \
    -t 8 \
    --host 0.0.0.0 \
    --port 8080 \
    --embedding

# Query via OpenAI-compatible API
curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "llama-3.1-8b",
        "messages": [{"role": "user", "content": "Hello"}],
        "max_tokens": 100
    }'
```

## MLX (Apple Silicon) Example

```python
import mlx.core as mx
from mlx_lm import load, generate

# Load model on Apple Silicon
model, tokenizer = load("mlx-community/Llama-3.1-8B-Instruct-4bit")

# Generate
prompt = "Explain quantum computing in simple terms."
response = generate(
    model,
    tokenizer,
    prompt=prompt,
    verbose=True,
    max_tokens=512,
    temp=0.7,
)

print(response)
```

## ONNX Runtime Example

```python
import onnxruntime as ort
from transformers import AutoTokenizer

# Load quantized ONNX model
session = ort.InferenceSession(
    "model-quantized.onnx",
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)

tokenizer = AutoTokenizer.from_pretrained("model-folder")

def infer(text):
    inputs = tokenizer(text, return_tensors="np", padding=True, truncation=True)
    ort_inputs = {
        "input_ids": inputs["input_ids"].astype("int64"),
        "attention_mask": inputs["attention_mask"].astype("int64"),
    }
    outputs = session.run(None, ort_inputs)
    return outputs

result = infer("Hello, how are you?")
```

## Core ML (iOS) Example

```python
# Convert and quantize for iOS
import coremltools as ct
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")

# Convert to Core ML
mlmodel = ct.convert(
    model,
    source="pytorch",
    convert_to="mlprogram",
    compute_units=ct.ComputeUnit.ALL,
)

# Quantize to INT4
mlmodel_int4 = ct.quantize_weights(mlmodel, nbits=4)

# Save for iOS
mlmodel_int4.save("Llama-3.2-1B-INT4.mlpackage")
```

```swift
// Swift - load and run Core ML model on iOS
import CoreML

let model = try Llama3_2_1B_INT4(configuration: MLModelConfiguration())
let input = Llama3_2_1B_INT4Input(prompt: "Hello, how are you?")
let output = try model.prediction(input: input)
print(output.response)
```

## Sources

* llama.cpp (ggerganov), MLX (Apple MLX team), ONNX Runtime (Microsoft), Core ML (Apple), TensorFlow Lite (Google), ExecuTorch (Meta PyTorch), GGUF, Apple Neural Engine, Qualcomm AI Hub, NVIDIA Jetson.*