# LiteLLM Proxy with Fallbacks and Load Balancing

> An OpenAI-compatible gateway that load-balances deployments and falls back across providers on errors.

Category: Multi-Model Routing  
Tags: litellm, proxy, fallbacks, openai-compatible  
Setup time (estimate): 30-60 minutes  
Estimated cost: LiteLLM is open source; you pay only your providers' token prices plus hosting for the proxy.  
Last verified: 2026-10-14

## config.yaml

```yaml
model_list:
  - model_name: chat-default
    litellm_params:
      model: openai/gpt-6-luna
      api_key: os.environ/OPENAI_API_KEY
  - model_name: chat-default            # same name = load-balanced group
    litellm_params:
      model: azure/my-gpt4o-mini-deployment
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
  - model_name: chat-backup
    litellm_params:
      model: anthropic/claude-haiku-5-5
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2
  timeout: 30
  fallbacks:
    - chat-default: [chat-backup]
```

## Run

```bash
pip install 'litellm[proxy]'
litellm --config config.yaml --port 4000
curl http://localhost:4000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"chat-default","messages":[{"role":"user","content":"hi"}]}'
```

Point any OpenAI SDK at `http://localhost:4000` as the base URL. Check model identifiers against your providers' current model lists.

## Failure modes

- Fallbacks hide outages; log which model actually answered.
- Backup models format structured output differently; test them on the same prompts.
- Retries multiply cost and latency; keep num_retries low and set a timeout.