Skip to main content

Model Fallback

Use CaseLLMAI Gateway

When the deployment behind a model goes down, every application that asks for that model fails, even when another model could have answered.

The Problem

An application that calls one deployment directly has no way out when it can't be reached. Adding a second deployment means adding retry and switch-over logic to every application.

How Zilla Solves It

A route on llm · proxy lists fallback models. When the route's deployment can't be reached, Zilla tries each fallback model in turn, routing it like a new request for that model. The application asks for the same model as always and gets the fallback model's reply.

Architecture

LLM model fallback flow
LLM model fallback flow

Configuration

north_llm_proxy:
  type: llm
  kind: proxy
  routes:
    - when:
        - model:
            - claude-primary
      exit: anthropic_llm_client_primary
      with:
        fallback:
          - claude-haiku-4-5
    - when:
        - model:
            - claude-*
      exit: anthropic_llm_client

Limits

Fallback covers a deployment that can't be reached at all, such as a refused connection. Once a deployment accepts a request, its reply is final: a 429 or 5xx error from the provider is returned to the caller, not retried on a fallback model. See Fallback.

Try It

The LLM Gateway demo routes a claude-primary model to a deployment that can't be reached, and shows it falling back to claude-haiku-4-5.

Try the guide

Walk through Model Fallback to configure a fallback and watch it take over.