LLM Gateway

AI GatewayLLM

Zilla's LLM Gateway sits between your applications and the LLM providers they call. Applications keep using the official OpenAI or Anthropic SDK and change only the base URL. Zilla routes each request by model to the right provider, translates between the OpenAI and Anthropic APIs when needed, checks prompts and replies against guardrails, and records token usage.

Features

Unified API

Serve OpenAI and Anthropic clients from one gateway by changing only the base URL.

OpenAI & Anthropic APIsAccept OpenAI Chat Completions and Anthropic Messages requests.StreamingStream replies back as Server-Sent Events when a request asks for a stream.Client SetupPoint the official OpenAI or Anthropic SDK at the gateway by changing its base URL.

Dialect Translation

Let an OpenAI client reach a Claude model, or an Anthropic client reach a GPT model.

Messages & ToolsTranslate messages, tool definitions and tool calls between the OpenAI and Anthropic formats.Replies & ErrorsReturn the provider's reply, and its errors, in the format the client sent the request in.

Model Routing & Fallback

Send each request to the provider or deployment that serves its model, and fall back when that deployment can't be reached.

Route by ModelMatch the requested model by exact name or a * pattern such as claude-*, and the API it arrived on.FallbackTry the next model in a route's fallback list when the deployment it routes to can't be reached.

Guardrails

Reject prompts and replies that contain secrets, personal data, or jailbreak attempts, per model, before they reach the provider or the caller.

LLM GuardrailsCheck each section of a prompt or reply, such as the user text or the assistant text, with a model you configure.ClassifiersDetect secrets by pattern, personal data with Presidio, or jailbreaks by meaning.

Providers

Connect to OpenAI, Anthropic, and AWS Bedrock, one client binding per provider.

OpenAICall the OpenAI Chat Completions API with the caller's own API key.AnthropicCall the Anthropic Messages API with the caller's own API key.AWS Bedrock ★Zilla PlusCall Amazon Bedrock models, signing each request with AWS credentials from the environment.

Secure LLM Access

Check credentials at the gateway and control which keys reach which provider.

Bring Your Own KeyForward the caller's own API key upstream, in the header each provider expects, without storing it in configuration.Inbound AuthorizationPass each request's API key to a guard before it reaches a provider.

Token Usage

See tokens, latency, time to first token, and interactions for every provider, as reported by the provider.

LLM MetricsRecord input, output, cached, and reasoning tokens, duration, time to first token, and interactions with the llm.* metrics.ExportersExport them to Prometheus, OTLP, or any other configured exporter.

Show moreShow less

Architecture

Binding	Role
`llm · server`	Accepts LLM API requests from an `http · server`, resolves which API each request uses, and optionally authorizes its API key with a guard
`llm · proxy`	Routes each request by its API and model, applies guardrails, and falls back to another model when a deployment can't be reached
`llm · client`	Sends the request to one provider in that provider's API, translating the request and reply when the client's API differs

See How It Works for how a request flows through these bindings.

Editions

Component	Zilla	Zilla Plus
`openai` and `anthropic` dialects	✓	✓
`aws-bedrock` dialect: call Amazon Bedrock, and accept the Bedrock Converse API	—	✓
Model routing, fallback, and guardrails	✓	✓
`llm.*` metrics	✓	✓

Get Started

This configuration accepts OpenAI Chat Completions requests on port 7161, routes gpt-* models to OpenAI and claude-* models to Anthropic, and forwards the caller's own API key to whichever provider serves the request.

zilla.yaml

guards:
  api_key:
    type: inline

bindings:
  north_tcp_server:
    type: tcp
    kind: server
    options:
      host: 0.0.0.0
      port: 7161
    exit: north_http_server

  north_http_server:
    type: http
    kind: server
    exit: north_llm_server

  north_llm_server:
    type: llm
    kind: server
    options:
      dialect: openai
      authorization:
        api_key:
          credentials: "Bearer {credentials}"
    telemetry:
      metrics:
        - llm.*
    exit: north_llm_proxy

  north_llm_proxy:
    type: llm
    kind: proxy
    routes:
      - when:
          - model:
              - gpt-*
        exit: openai_llm_client
      - when:
          - model:
              - claude-*
        exit: anthropic_llm_client

  openai_llm_client:
    type: llm
    kind: client
    options:
      dialect: openai
      server: https://api.openai.com
      authorization:
        api_key:
          credentials: "Bearer {credentials}"
    exit: sys:http_client

  anthropic_llm_client:
    type: llm
    kind: client
    options:
      dialect: anthropic
      server: https://api.anthropic.com
      authorization:
        api_key:
          credentials: "{credentials}"
    exit: sys:http_client

telemetry:
  metrics:
    - llm.active.requests
    - llm.duration
    - llm.tokens.input
    - llm.tokens.output
  exporters:
    prometheus_exporter:
      type: prometheus
      options:
        endpoints:
          - scheme: http
            port: 7190
            path: /metrics

Point the OpenAI SDK at the gateway by changing only its base URL:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:7161/v1", api_key="<your-openai-api-key>")
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello"}])
print(resp.choices[0].message.content)

Try the demo

The LLM Gateway demo runs OpenAI and Anthropic behind one gateway with translation, guardrails, fallback, and a Grafana dashboard of token usage.