# Local-First Routing with Ollama and Cloud Fallback

> Serve requests from a local Ollama model, falling back to a cloud model only when local fails or the task needs it.

Category: Multi-Model Routing  
Tags: ollama, local, privacy, fallback  
Setup time (estimate): 30 minutes  
Estimated cost: Free local inference (your hardware and power); cloud fallback billed at provider rates.  
Last verified: 2026-10-14

Ollama exposes an OpenAI-compatible API at `http://localhost:11434/v1`, so the same client code works for both.

## Setup

```bash
ollama pull llama3.1:8b
ollama serve   # if not already running as a service
```

## Client

```python
from openai import OpenAI

local = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")  # key is ignored
cloud = OpenAI()  # uses OPENAI_API_KEY

def chat(messages, sensitive=False):
    try:
        r = local.chat.completions.create(model="llama3.1:8b", messages=messages, timeout=60)
        return r.choices[0].message.content
    except Exception:
        if sensitive:
            raise  # never send sensitive data to the cloud
        r = cloud.chat.completions.create(model="gpt-6-luna", messages=messages)
        return r.choices[0].message.content
```

Tag requests containing personal or confidential data as `sensitive=True` so they stay on-device.

## Failure modes

- Local model too slow or out of memory on small machines; pick a smaller model.
- Silent cloud fallback can send private data off-device; log and gate it.
- Local and cloud models answer differently; test both paths.