Skip to main content

Model Routing

llm · proxy sends each request to one llm · client, picked by the request's dialect and the model it asks for. Each llm · client connects to one provider or deployment.

Configuration

north_llm_proxy:
  type: llm
  kind: proxy
  routes:
    - when:
        - model:
            - gpt-*
      exit: openai_llm_client
    - when:
        - model:
            - claude-*
      exit: anthropic_llm_client

A request for gpt-4o-mini goes to openai_llm_client, and a request for claude-haiku-4-5 goes to anthropic_llm_client, whether it arrived in the OpenAI or the Anthropic API. When the client's dialect differs from the request's, the request is translated.

Route Conditions

Each route has a list of when conditions, and matches when any one of them matches. A condition can check two values:

ConditionMatches when
dialectThe request arrived in this API: openai or anthropic
modelThe model named in the request matches one of the listed names or patterns

When a condition sets both, both must match. A condition that leaves one out matches any value for it, and a route with no when matches every request.

Model Patterns

A model entry is an exact model name, such as gpt-4o-mini, or a pattern where * matches any sequence of characters, such as claude-*. Every other character must match exactly.

- when:
    - model:
        - gpt-4o
        - gpt-4o-mini
  exit: openai_llm_client

Route Order

Routes are evaluated top to bottom, and the first matching route wins. Put specific routes before general ones. Here, gpt-4o from an OpenAI client goes to a second OpenAI deployment, and every other gpt-* model goes to the first:

routes:
  - when:
      - dialect: openai
        model:
          - gpt-4o
    exit: openai_llm_client_secondary
  - when:
      - model:
          - gpt-*
    exit: openai_llm_client

Make sure every model your clients ask for matches a route. A last route with no when catches any model the routes above don't.

Routing by Role

A route can also require roles from a guard, to reserve models for callers whose credentials carry those roles:

routes:
  - when:
      - model:
          - gpt-4o
    guarded:
      api_keys:
        - premium
    exit: openai_llm_client

See LLM Credentials for how llm · server authorizes requests.

Fallback

A route can name fallback models to use when the deployment it routes to can't be reached. See Fallback.

Next Steps