Skip to main content

Fallback

A route on llm · proxy can list fallback models. When the deployment the route sends a request to can't be reached, Zilla tries each fallback model in turn until one of them can be reached.

Configuration

north_llm_proxy:
  type: llm
  kind: proxy
  routes:
    - when:
        - model:
            - claude-primary
      exit: anthropic_llm_client_primary
      with:
        fallback:
          - claude-haiku-4-5
    - when:
        - model:
            - claude-*
      exit: anthropic_llm_client

A request for claude-primary goes to anthropic_llm_client_primary. If that deployment can't be reached, Zilla falls back to claude-haiku-4-5, which the second route sends to anthropic_llm_client. The caller receives the reply from claude-haiku-4-5 as if it had asked for it.

with.fallback is a list of exact model names, tried in order. It needs at least one model.

When Fallback Happens

Fallback happens only when the request can't be delivered to the deployment at all, for example when the connection is refused or the host is unreachable. It applies the same way to streaming and non-streaming requests.

Once the deployment accepts the request, its reply is final. An error the provider returns, such as 429 Too Many Requests or a 5xx status, does not trigger fallback, and is returned to the caller.

How Fallback Models Are Routed

Zilla routes each fallback model through the same routes, as a request in the caller's original dialect and with the caller's authorization. A fallback model that matches no route is skipped. When no fallback model can be reached either, the caller receives the error from the last attempt.

Before falling back, Zilla rewrites the model in the request to the fallback model.

Fallback and Guardrails

LLM guardrails are configured per model. After a fallback, the guardrails of the fallback model run in addition to those of the model the caller asked for.

See Fallback in Metrics

A failed attempt still counts as an interaction on the client binding it was routed to. With the llm.interactions metric, the attempts on the unreachable deployment show up next to the successful ones on the fallback deployment.

Next Steps