Local-First Routing with Ollama and Cloud Fallback
Serve requests from a local Ollama model, falling back to a cloud model only when local fails or the task needs it.
Ollama exposes an OpenAI-compatible API at http://localhost:11434/v1, so the same client code works for both.
Setup
ollama pull llama3.1:8b
ollama serve # if not already running as a serviceClient
from openai import OpenAI
local = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama") # key is ignored
cloud = OpenAI() # uses OPENAI_API_KEY
def chat(messages, sensitive=False):
try:
r = local.chat.completions.create(model="llama3.1:8b", messages=messages, timeout=60)
return r.choices[0].message.content
except Exception:
if sensitive:
raise # never send sensitive data to the cloud
r = cloud.chat.completions.create(model="gpt-6-luna", messages=messages)
return r.choices[0].message.contentTag requests containing personal or confidential data as sensitive=True so they stay on-device.
Failure modes
- Local model too slow or out of memory on small machines; pick a smaller model.
- Silent cloud fallback can send private data off-device; log and gate it.
- Local and cloud models answer differently; test both paths.
Markdown version: /blueprints/ollama-local-first.md
Want a deeper build of this?
Vote for an extended, fully tested version (more tools, a full workflow file). Votes decide what we build next.
Get one fully built blueprint a week
A weekly email: one blueprint with config, cost, setup time and failure modes, plus a vote on what we build next.
Related blueprints
Complexity-Based Cheap/Strong Model Router
Route easy requests to a cheap model and hard ones to a strong model using a lightweight classifier plus escalation.
LiteLLM Proxy with Fallbacks and Load Balancing
An OpenAI-compatible gateway that load-balances deployments and falls back across providers on errors.
OpenRouter Model Fallback and Provider Ordering
Use OpenRouter's request-level models array and provider preferences for automatic failover.