Opening article
New in llama.cpp: Model Management
The llama.cpp server now includes a router mode that allows dynamic loading, unloading, and switching between multiple models without restarting the server. This feature, inspired by Ollama, uses a multi‑process architecture so that a crash in one model does not affect others. Models are auto‑discovered from the llama.cpp cache and are loaded on first request, with subsequent requests served instantly. The built‑in web UI supports model switching via a dropdown, and all loaded models use a 8192‑token context with full GPU offload.
Category
Models
Industry DomainThe Core Development
llama.cpp server now ships with router mode , which lets you dynamically load, unload, and switch between multiple models without restarting.
Reminder: llama.cpp server is a lightweight, OpenAI-compatible HTTP server for running LLMs locally.
This feature was a popular request to bring Ollama-style model management to llama.cpp. It uses a multi-process architecture where each model runs in its own process, so if one model crashes, others remain unaffected.
Strategic Perspective & Ecosystem Impact
This auto-discovers models from your llama.cpp cache ( LLAMA_CACHE or ~/.cache/llama.cpp ). If you've previously downloaded models via llama-server -hf user/model , they'll be available automatically.
On the first request, the server automatically loads the model into memory (loading time depends on model size). Subsequent requests to the same model are instant since it's already loaded.
Returns all discovered models with their status ( loaded , loading , or unloaded ).
Why it matters
Dynamic model management simplifies A/B testing and multi‑tenant deployments, reducing downtime and resource overhead in local LLM hosting.