Local Inference Runtimes

On-device and edge inference runtimes: llama.cpp, MLX, ONNX Runtime, Core ML, TensorFlow Lite — when to use each and how to deploy.

Runtime Comparison

Runtime	Platform	Model Format	Best For	GPU/NPU Support
llama.cpp	Linux, macOS, Windows, Android	GGUF	Local LLM inference, CPU/GPU	CUDA, Metal, Vulkan, BLAS
MLX	Apple Silicon (macOS, iOS)	MLX arrays	On-device LLMs on Mac/iPhone	Apple Neural Engine, Metal
ONNX Runtime	Cross-platform	ONNX	General ML models, edge devices	CUDA, TensorRT, OpenVINO, ACL
Core ML	iOS, macOS	.mlmodel	iOS apps, Apple devices	Neural Engine, GPU
TensorFlow Lite	Android, iOS, Linux	.tflite	Mobile ML, microcontrollers	GPU, NNAPI, Edge TPU
ExecuTorch	Android, iOS	.pte	PyTorch models on mobile	Vulkan, CPU

Decision Guidance

Situation	Recommended Runtime
Local LLM on laptop/desktop	llama.cpp (GGUF)
iOS app with on-device LLM	MLX or Core ML
Android app with ML model	TensorFlow Lite or ExecuTorch
Cross-platform edge device	ONNX Runtime
PyTorch model on mobile	ExecuTorch

Anti-Patterns

Anti-Pattern	Fix
Running 70B model on-device	Use 1B–3B SLM; offload complex tasks to cloud
Full-precision weights on mobile	Quantize to INT4/INT8; use GGUF/Core ML
Ignoring thermal throttling	Benchmark on target device; account for sustained load
No fallback when on-device fails	Cloud fallback for complex requests
Sending every prompt to cloud	Local inference first; cloud only for complex tasks

Quantization For Edge — Quantization techniques for edge
Small Language Models — SLMs, distillation, quantization
Inference Optimization — Quantization, speculative decoding
Inference — Model serving and inference engines
Reliability — Guardrails and failover for local models

llama.cpp Deployment

# Build with GPU support (CUDA)
cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release

# Download and quantize a model
wget https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct-gguf/resolve/main/llama-3.1-8b-instruct-Q4_K_M.gguf

# Run a local server
./llama-server \
    -m llama-3.1-8b-instruct-Q4_K_M.gguf \
    -c 4096 \
    -ngl 35 \
    -t 8 \
    --host 0.0.0.0 \
    --port 8080 \
    --embedding

# Query via OpenAI-compatible API
curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "llama-3.1-8b",
        "messages": [{"role": "user", "content": "Hello"}],
        "max_tokens": 100
    }'

MLX (Apple Silicon) Example

import mlx.core as mx
from mlx_lm import load, generate

# Load model on Apple Silicon
model, tokenizer = load("mlx-community/Llama-3.1-8B-Instruct-4bit")

# Generate
prompt = "Explain quantum computing in simple terms."
response = generate(
    model,
    tokenizer,
    prompt=prompt,
    verbose=True,
    max_tokens=512,
    temp=0.7,
)

print(response)

ONNX Runtime Example

import onnxruntime as ort
from transformers import AutoTokenizer

# Load quantized ONNX model
session = ort.InferenceSession(
    "model-quantized.onnx",
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)

tokenizer = AutoTokenizer.from_pretrained("model-folder")

def infer(text):
    inputs = tokenizer(text, return_tensors="np", padding=True, truncation=True)
    ort_inputs = {
        "input_ids": inputs["input_ids"].astype("int64"),
        "attention_mask": inputs["attention_mask"].astype("int64"),
    }
    outputs = session.run(None, ort_inputs)
    return outputs

result = infer("Hello, how are you?")

Core ML (iOS) Example

# Convert and quantize for iOS
import coremltools as ct
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")

# Convert to Core ML
mlmodel = ct.convert(
    model,
    source="pytorch",
    convert_to="mlprogram",
    compute_units=ct.ComputeUnit.ALL,
)

# Quantize to INT4
mlmodel_int4 = ct.quantize_weights(mlmodel, nbits=4)

# Save for iOS
mlmodel_int4.save("Llama-3.2-1B-INT4.mlpackage")

// Swift - load and run Core ML model on iOS
import CoreML

let model = try Llama3_2_1B_INT4(configuration: MLModelConfiguration())
let input = Llama3_2_1B_INT4Input(prompt: "Hello, how are you?")
let output = try model.prediction(input: input)
print(output.response)

Sources

llama.cpp (ggerganov), MLX (Apple MLX team), ONNX Runtime (Microsoft), Core ML (Apple), TensorFlow Lite (Google), ExecuTorch (Meta PyTorch), GGUF, Apple Neural Engine, Qualcomm AI Hub, NVIDIA Jetson.*

From the Systems Engineering research vault · CC BY 4.0 · all notes · raw markdown