Skip to main content

Route by Model

The Model Routing Across Deployments use case describes sending each model to the deployment that serves it. This guide runs a gateway with two OpenAI clients, one for gpt-4o-mini and one for every other gpt-* model, and uses the llm.interactions metric to see which client served each request.

Prerequisites

docker-compose.yaml

Download docker-compose.yaml

name: zilla-llm-route-by-model
services:
  zilla:
    image: ghcr.io/aklivity/zilla:${ZILLA_VERSION:-2.5.0-rc5}
    restart: unless-stopped
    ports:
      - 7161:7161
      - 7190:7190
    healthcheck:
      interval: 5s
      timeout: 3s
      retries: 15
      start_period: 15s
      start_interval: 250ms
      test: ["CMD", "sh", "-c", "test -f /var/run/zilla/ready"]
    volumes:
      - ./zilla.yaml:/etc/zilla/zilla.yaml:ro
    command: start -v -e
zilla.yaml

Download zilla.yaml

name: llm-route-by-model
telemetry:
  metrics:
    - llm.interactions
  exporters:
    prometheus_exporter:
      type: prometheus
      options:
        endpoints:
          - scheme: http
            port: 7190
            path: /metrics
guards:
  api_key:
    type: inline
bindings:
  north_tcp_server:
    type: tcp
    kind: server
    options:
      host: 0.0.0.0
      port: 7161
    exit: north_http_server
  north_http_server:
    type: http
    kind: server
    exit: north_llm_server
  north_llm_server:
    type: llm
    kind: server
    options:
      dialect: openai
      authorization:
        api_key:
          credentials: "Bearer {credentials}"
    exit: north_llm_proxy
  #region llm_proxy
  north_llm_proxy:
    type: llm
    kind: proxy
    routes:
      - when:
          - model:
              - gpt-4o-mini
        exit: south_llm_client_mini
      - when:
          - model:
              - gpt-*
        exit: south_llm_client_openai
  #endregion llm_proxy
  #region llm_clients
  south_llm_client_mini:
    type: llm
    kind: client
    options:
      dialect: openai
      server: https://api.openai.com
      authorization:
        api_key:
          credentials: "Bearer {credentials}"
    telemetry:
      metrics:
        - llm.*
    exit: sys:http_client
  south_llm_client_openai:
    type: llm
    kind: client
    options:
      dialect: openai
      server: https://api.openai.com
      authorization:
        api_key:
          credentials: "Bearer {credentials}"
    telemetry:
      metrics:
        - llm.*
    exit: sys:http_client
  #endregion llm_clients

Two routes, the specific one first:

north_llm_proxy:
  type: llm
  kind: proxy
  routes:
    - when:
        - model:
            - gpt-4o-mini
      exit: south_llm_client_mini
    - when:
        - model:
            - gpt-*
      exit: south_llm_client_openai

Both clients connect to OpenAI; in a real deployment, each would point at its own deployment. Each client's llm.interactions count shows how many requests it served.

Start the Stack

docker compose up -d --wait

Send Two Models

for model in gpt-4o-mini gpt-4o; do
  curl -s -o /dev/null -w "$model -> %{http_code}\n" http://localhost:7161/v1/chat/completions \
    -H "Authorization: Bearer $OPENAI_API_KEY" \
    -H "Content-Type: application/json" \
    -d "{\"model\":\"$model\",\"max_tokens\":20,\"messages\":[{\"role\":\"user\",\"content\":\"Say hello\"}]}"
done

Read the metrics:

curl -s http://localhost:7190/metrics | grep '^llm_interactions_total'
llm_interactions_total{namespace="llm-route-by-model",binding="south_llm_client_mini"} 1
llm_interactions_total{namespace="llm-route-by-model",binding="south_llm_client_openai"} 1

gpt-4o-mini matched the first route and went to south_llm_client_mini. gpt-4o didn't, and fell through to the gpt-* route.

Change the Route Order

Routes are evaluated top to bottom, and the first match wins. In zilla.yaml, move the gpt-* route above the gpt-4o-mini route, then recreate Zilla:

docker compose up -d --force-recreate --wait

Send gpt-4o-mini again:

curl -s -o /dev/null -w "%{http_code}\n" http://localhost:7161/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","max_tokens":20,"messages":[{"role":"user","content":"Say hello"}]}'
curl -s http://localhost:7190/metrics | grep '^llm_interactions_total'
llm_interactions_total{namespace="llm-route-by-model",binding="south_llm_client_mini"} 0
llm_interactions_total{namespace="llm-route-by-model",binding="south_llm_client_openai"} 1

This time gpt-4o-mini matched gpt-* first, and south_llm_client_mini served nothing. Put specific routes before general patterns.

Stop the Stack

docker compose down

Next Steps