Skip to main content

LLM Gateway Setup

The Unified LLM API use case describes applications reaching any provider from the SDK they already use. This guide builds that gateway from scratch: an OpenAI API on port 7161, an Anthropic API on port 7162, and routes that send gpt-* models to OpenAI and claude-* models to Anthropic, forwarding your own API keys. For a ready-made version with a browser UI and dashboards, see Get Started.

Prerequisites

  • Docker Compose
  • An OpenAI API key and an Anthropic API key, exported as OPENAI_API_KEY and ANTHROPIC_API_KEY
docker-compose.yaml

Download docker-compose.yaml

name: zilla-llm-gateway-setup
services:
  zilla:
    image: ghcr.io/aklivity/zilla:${ZILLA_VERSION:-2.5.0-rc5}
    restart: unless-stopped
    ports:
      - 7161:7161
      - 7162:7162
    healthcheck:
      interval: 5s
      timeout: 3s
      retries: 15
      start_period: 15s
      start_interval: 250ms
      test: ["CMD", "sh", "-c", "test -f /var/run/zilla/ready"]
    volumes:
      - ./zilla.yaml:/etc/zilla/zilla.yaml:ro
    command: start -v -e
zilla.yaml

Download zilla.yaml

name: llm-gateway-setup
guards:
  api_key:
    type: inline
bindings:
  north_tcp_server:
    type: tcp
    kind: server
    options:
      host: 0.0.0.0
      port:
        - 7161
        - 7162
    routes:
      - when:
          - port: 7161
        exit: north_http_server_openai
      - when:
          - port: 7162
        exit: north_http_server_anthropic
  #region llm_servers
  north_http_server_openai:
    type: http
    kind: server
    exit: north_llm_server_openai
  north_llm_server_openai:
    type: llm
    kind: server
    options:
      dialect: openai
      authorization:
        api_key:
          credentials: "Bearer {credentials}"
    exit: north_llm_proxy
  north_http_server_anthropic:
    type: http
    kind: server
    exit: north_llm_server_anthropic
  north_llm_server_anthropic:
    type: llm
    kind: server
    options:
      dialect: anthropic
      authorization:
        api_key:
          credentials: "{credentials}"
    exit: north_llm_proxy
  #endregion llm_servers
  #region llm_proxy
  north_llm_proxy:
    type: llm
    kind: proxy
    routes:
      - when:
          - model:
              - gpt-*
        exit: south_llm_client_openai
      - when:
          - model:
              - claude-*
        exit: south_llm_client_anthropic
  #endregion llm_proxy
  #region llm_clients
  south_llm_client_openai:
    type: llm
    kind: client
    options:
      dialect: openai
      server: https://api.openai.com
      authorization:
        api_key:
          credentials: "Bearer {credentials}"
    exit: sys:http_client
  south_llm_client_anthropic:
    type: llm
    kind: client
    options:
      dialect: anthropic
      server: https://api.anthropic.com
      authorization:
        api_key:
          credentials: "{credentials}"
    exit: sys:http_client
  #endregion llm_clients

How the Configuration Fits Together

Two llm · server bindings accept requests, one per API. Each fixes its dialect and reads the caller's API key with the inline guard:

north_http_server_openai:
  type: http
  kind: server
  exit: north_llm_server_openai
north_llm_server_openai:
  type: llm
  kind: server
  options:
    dialect: openai
    authorization:
      api_key:
        credentials: "Bearer {credentials}"
  exit: north_llm_proxy
north_http_server_anthropic:
  type: http
  kind: server
  exit: north_llm_server_anthropic
north_llm_server_anthropic:
  type: llm
  kind: server
  options:
    dialect: anthropic
    authorization:
      api_key:
        credentials: "{credentials}"
  exit: north_llm_proxy

Both send requests to one llm · proxy, which routes them by the model they ask for:

north_llm_proxy:
  type: llm
  kind: proxy
  routes:
    - when:
        - model:
            - gpt-*
      exit: south_llm_client_openai
    - when:
        - model:
            - claude-*
      exit: south_llm_client_anthropic

Each llm · client connects to one provider and sends the caller's key in the header that provider expects:

south_llm_client_openai:
  type: llm
  kind: client
  options:
    dialect: openai
    server: https://api.openai.com
    authorization:
      api_key:
        credentials: "Bearer {credentials}"
  exit: sys:http_client
south_llm_client_anthropic:
  type: llm
  kind: client
  options:
    dialect: anthropic
    server: https://api.anthropic.com
    authorization:
      api_key:
        credentials: "{credentials}"
  exit: sys:http_client

Start the Stack

docker compose up -d --wait

Call OpenAI Through the Gateway

Send an OpenAI Chat Completions request to port 7161:

curl -s http://localhost:7161/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","max_tokens":100,"messages":[{"role":"user","content":"In one sentence, what is an LLM gateway?"}]}'

The reply is a plain OpenAI chat.completion.

Call Claude Through the Gateway

Send an Anthropic Messages request to port 7162:

curl -s http://localhost:7162/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{"model":"claude-haiku-4-5","max_tokens":100,"messages":[{"role":"user","content":"In one sentence, what is an LLM gateway?"}]}'

Reach GPT from the Anthropic API

Keep the Anthropic API on port 7162, but ask for gpt-4o-mini, with your OpenAI key:

curl -s http://localhost:7162/v1/messages \
  -H "x-api-key: $OPENAI_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","max_tokens":100,"messages":[{"role":"user","content":"In one sentence, what is an LLM gateway?"}]}'

Zilla translates the request for OpenAI and the reply back: the response is an Anthropic message, generated by gpt-4o-mini.

Stream a Reply

Add "stream":true to any request:

curl -sN http://localhost:7161/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","stream":true,"max_tokens":100,"messages":[{"role":"user","content":"In one sentence, what is an LLM gateway?"}]}'

The reply arrives as Server-Sent Events, in the event format of the API you called.

Stop the Stack

docker compose down

Next Steps