Skip to main content

How It Works

Zilla's LLM Gateway is built from three llm bindings, layered on top of the http bindings.

Binding Pipeline

LLM Gateway binding pipeline
LLM Gateway binding pipeline
BindingRole
tcp · server, http · serverAccept the application's connection and decode each HTTP request
llm · serverResolves which API the request uses, checks its content type and API key, and validates its body
llm · proxyPicks a client by the request's API and model, applies guardrails, and falls back to another model when needed
llm · clientBuilds the request to one provider, in that provider's API, and translates the reply back
sys:http_clientConnects to the provider over HTTP, or over TLS when the server URL is https://

The llm bindings sit on top of http the same way the mcp bindings do in the MCP Gateway: http handles the wire format, and llm works on the decoded request and reply.

Request Flow

  1. The application sends a request, such as POST /v1/chat/completions, to http · server.
  2. llm · server resolves the request's dialect, the API it is written in: openai or anthropic.
  3. llm · server checks the request's content type and, when options.authorization is set, passes its API key to a guard. A request that fails either check is rejected with 415 or 401, in the error format of its dialect.
  4. llm · server validates the request body and reads the model it asks for.
  5. llm · proxy follows the first route whose dialect and model conditions match. See Model Routing.
  6. llm · proxy applies the request guardrails configured for that model. A rejected request never reaches the provider. See LLM Guardrails.
  7. llm · client builds the request to its provider from its server URL and its own dialect, translating the request when the two dialects differ, and adds the API key to the header its provider expects. See LLM Credentials.
  8. If the provider can't be reached at all, llm · proxy tries the route's fallback models in order. See Fallback.
  9. llm · client translates the reply to the request's dialect, llm · proxy applies the reply guardrails, and the application receives the reply, or the provider's error, in the format it sent the request in.

Each llm binding records the llm.* metrics for the exchange when its telemetry.metrics includes them.

What Each Binding Sees

llm · server is the only binding that reads the request's HTTP headers and path. It passes on the request with the dialect and model it resolved.

llm · proxy routes on those two values. It reads the request and reply content only to apply the guardrails configured for the model.

llm · client builds the upstream request from its own configuration. The upstream path comes from the server URL and the client's dialect, not from the path the application called.

Streaming

A request that asks for a stream, with "stream": true in the body, receives its reply as Server-Sent Events (text/event-stream). A request without it receives a single JSON reply.

Streaming works across translation: each event is translated into the request's dialect as it arrives.

Next Steps