Skip to main content

Model Routing Across Deployments

Use CaseLLMAI Gateway

Models rarely live in one place. Some come from OpenAI, some from Anthropic, a busy model may have a second deployment, and a model reserved for certain teams may run under its own account. Applications shouldn't need to know which is which.

The Problem

When applications call providers directly, each one carries the knowledge of where every model lives: which endpoint, which account, which SDK. Moving a model to another deployment, adding a second deployment, or reserving a model for certain callers means changing every application that uses it.

How Zilla Solves It

Applications send every request to one gateway endpoint and name the model they want. llm · proxy routes each request by its model, and optionally by the API it arrived in, to the llm · client for the provider or deployment that serves it. Moving a model is a route change at the gateway.

Architecture

LLM model routing flow
LLM model routing flow

Routes

Routes match an exact model name or a * pattern, and the first matching route wins, so specific models go before general patterns:

north_llm_proxy:
  type: llm
  kind: proxy
  routes:
    - when:
        - model:
            - gpt-4o
      exit: openai_llm_client_secondary
    - when:
        - model:
            - gpt-*
      exit: openai_llm_client
    - when:
        - model:
            - claude-*
      exit: anthropic_llm_client

A route can also require roles from the guard that authorized the caller, to reserve a model for some callers. See Model Routing.

Limits

Routing is decided per request from its model and API. Zilla doesn't spread one model across several deployments or retry on a provider error. When a deployment can't be reached at all, a route can fall back to another model.

Try It

Try the guide

Walk through Route by Model to add routes and see how their order decides which deployment serves a request.