Skip to main content

Model Fallback

The Model Fallback use case describes keeping applications answering when a deployment is down. This guide routes a claude-primary model to a deployment that can't be reached, with claude-haiku-4-5 as its fallback, and uses the llm.interactions metric to see both attempts.

Prerequisites

docker-compose.yaml

Download docker-compose.yaml

name: zilla-llm-model-fallback
services:
  zilla:
    image: ghcr.io/aklivity/zilla:${ZILLA_VERSION:-2.5.0-rc5}
    restart: unless-stopped
    ports:
      - 7162:7162
      - 7190:7190
    healthcheck:
      interval: 5s
      timeout: 3s
      retries: 15
      start_period: 15s
      start_interval: 250ms
      test: ["CMD", "sh", "-c", "test -f /var/run/zilla/ready"]
    volumes:
      - ./zilla.yaml:/etc/zilla/zilla.yaml:ro
    command: start -v -e
zilla.yaml

Download zilla.yaml

name: llm-model-fallback
telemetry:
  metrics:
    - llm.interactions
  exporters:
    prometheus_exporter:
      type: prometheus
      options:
        endpoints:
          - scheme: http
            port: 7190
            path: /metrics
guards:
  api_key:
    type: inline
bindings:
  north_tcp_server:
    type: tcp
    kind: server
    options:
      host: 0.0.0.0
      port: 7162
    exit: north_http_server
  north_http_server:
    type: http
    kind: server
    exit: north_llm_server
  north_llm_server:
    type: llm
    kind: server
    options:
      dialect: anthropic
      authorization:
        api_key:
          credentials: "{credentials}"
    exit: north_llm_proxy
  #region llm_proxy
  north_llm_proxy:
    type: llm
    kind: proxy
    routes:
      - when:
          - model:
              - claude-primary
        exit: south_llm_client_primary
        with:
          fallback:
            - claude-haiku-4-5
      - when:
          - model:
              - claude-*
        exit: south_llm_client_anthropic
  #endregion llm_proxy
  #region llm_clients
  south_llm_client_primary:
    type: llm
    kind: client
    options:
      dialect: anthropic
      server: http://localhost:4999
      authorization:
        api_key:
          credentials: "{credentials}"
    telemetry:
      metrics:
        - llm.*
    exit: sys:http_client
  south_llm_client_anthropic:
    type: llm
    kind: client
    options:
      dialect: anthropic
      server: https://api.anthropic.com
      authorization:
        api_key:
          credentials: "{credentials}"
    telemetry:
      metrics:
        - llm.*
    exit: sys:http_client
  #endregion llm_clients

The claude-primary route lists claude-haiku-4-5 as its fallback:

north_llm_proxy:
  type: llm
  kind: proxy
  routes:
    - when:
        - model:
            - claude-primary
      exit: south_llm_client_primary
      with:
        fallback:
          - claude-haiku-4-5
    - when:
        - model:
            - claude-*
      exit: south_llm_client_anthropic

south_llm_client_primary stands for a deployment that is down: nothing listens on its server port.

south_llm_client_primary:
  type: llm
  kind: client
  options:
    dialect: anthropic
    server: http://localhost:4999
    authorization:
      api_key:
        credentials: "{credentials}"
  telemetry:
    metrics:
      - llm.*
  exit: sys:http_client
south_llm_client_anthropic:
  type: llm
  kind: client
  options:
    dialect: anthropic
    server: https://api.anthropic.com
    authorization:
      api_key:
        credentials: "{credentials}"
  telemetry:
    metrics:
      - llm.*
  exit: sys:http_client

Start the Stack

docker compose up -d --wait

Ask for the Primary Model

curl -s http://localhost:7162/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{"model":"claude-primary","max_tokens":100,"messages":[{"role":"user","content":"In one sentence, what is an LLM gateway?"}]}'

The reply's model is claude-haiku-4-5: the primary deployment couldn't be reached, so Zilla fell back to it.

See Both Attempts

curl -s http://localhost:7190/metrics | grep '^llm_interactions_total'
llm_interactions_total{namespace="llm-model-fallback",binding="south_llm_client_primary"} 1
llm_interactions_total{namespace="llm-model-fallback",binding="south_llm_client_anthropic"} 1

The failed attempt counts on south_llm_client_primary, and the reply on south_llm_client_anthropic.

What Doesn't Trigger Fallback

Fallback covers only a deployment that can't be reached. An error reply such as 429 Too Many Requests is returned to the caller, not retried on the fallback model. See Fallback.

Stop the Stack

docker compose down

Next Steps