llm server

The llm server binding accepts inbound LLM API requests from an http server, resolving each request's dialect and optionally authorizing its credentials with a guard, before forwarding it to the exit binding.

llm_server:
  type: llm
  kind: server
  options:
    dialect: openai
    authorization:
      my_api_key_guard:
        credentials: "Bearer {credentials}"
  exit: llm_proxy

Configuration (* required)

options

object

The server kind llm specific options.

options:
  dialect: openai
  authorization:
    my_api_key_guard:
      credentials: "Bearer {credentials}"

options.dialect

enum [ openai, anthropic, gemini, aws-bedrock, azure-foundry, gcp-vertex ]

Fixes the dialect assumed for every inbound request, instead of detecting it per request. Mutually exclusive with dialects.

The dialects available to the llm binding are:

Dialect	API	Request path	Credentials header
`openai`	OpenAI Chat Completions	`/v1/chat/completions`	`authorization`
`openai`	OpenAI Responses	`/v1/responses`	`authorization`
`anthropic`	Anthropic Messages	`/v1/messages`	`x-api-key`
`gemini`	Google Gemini generateContent	`/v1beta/models/{model}:generateContent`, `/v1beta/models/{model}:streamGenerateContent`, or the same paths under `/v1`	`x-goog-api-key`
`aws-bedrock` ★Zilla Plus	Amazon Bedrock Converse	`/model/{modelId}/converse`, `/model/{modelId}/converse-stream`	`authorization`
`azure-foundry` ★Zilla Plus	Azure OpenAI Chat Completions	`/openai/v1/chat/completions`, `/openai/deployments/{deployment}/chat/completions`	`api-key`, `authorization`
`gcp-vertex` ★Zilla Plus	Google Vertex AI	`/v1/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:{method}`, `/v1/projects/{project}/locations/{location}/endpoints/openapi/chat/completions`	`authorization`

When dialect is omitted, each request is offered to every available dialect, and must be recognized by exactly one of them:

openai recognizes a POST to /v1/chat/completions or /v1/responses with content-type: application/json, with or without parameters such as charset.
anthropic recognizes a POST to /v1/messages, or any request carrying an anthropic-version header, or an x-api-key header without an authorization header.
gemini recognizes a POST to /v1beta/models/{model}:generateContent or /v1beta/models/{model}:streamGenerateContent, or either path under /v1. The model is taken from the path, not from the request body.
aws-bedrock ★Zilla Plus recognizes a POST to /model/{modelId}/converse or /model/{modelId}/converse-stream.
azure-foundry ★Zilla Plus recognizes a POST with content-type: application/json to /openai/v1/chat/completions, or to /openai/deployments/{deployment}/chat/completions with a non-empty api-version query parameter.
gcp-vertex ★Zilla Plus recognizes a POST to a Vertex AI path: /v1/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:{method}, with generateContent or streamGenerateContent for google, and rawPredict or streamRawPredict for anthropic, or /v1/projects/{project}/locations/{location}/endpoints/openapi/chat/completions. The request resolves to the gemini, anthropic or openai dialect of its API.

A request recognized by no dialect, or by more than one, is rejected. When dialect is set to openai, both OpenAI APIs are served, told apart by the request path, unless apis narrows them. When dialect is set to anthropic, the request path and method are not checked. When dialect is set to gemini, a request that isn't a POST to one of the gemini paths is rejected with 404 Not Found. ★Zilla Plus The same applies to aws-bedrock, for a request that isn't a POST to /model/{modelId}/converse or /model/{modelId}/converse-stream, and to gcp-vertex, for a request that isn't a POST to a Vertex AI path. ★Zilla Plus When dialect is set to azure-foundry, a request to any other path, or to the classic path without api-version, is rejected with 404 Not Found.

Once the dialect is resolved:

a request whose content-type is not application/json is rejected with 415 Unsupported Media Type
an anthropic request must carry the anthropic-version: 2023-06-01 header, or it is rejected with 400 Bad Request
a request body that can't be parsed is rejected with 400 Bad Request

A gemini request to :streamGenerateContent is answered with server-sent events when its query string has alt=sse, and with a JSON array of response chunks otherwise. A request to :generateContent is answered with a single JSON document.

options.apis

array of enum [ chat/completions, responses ]

The OpenAI APIs this server serves. Requires dialect openai; with dialects, set apis on the openai entry instead. The list must have at least one entry, without duplicates. Defaults to both.

options:
  dialect: openai
  apis:
    - chat/completions

A request for an API that is not listed is answered as an unknown dialect, and the stream is rejected. The request path is still checked to tell the APIs apart.

★Zilla Plus For aws-bedrock, apis instead lists the inbound API surfaces the binding serves, named dialect:api, where a bare dialect names all of its APIs: anthropic, openai, openai:chat/completions and openai:responses. It defaults to every API. An InvokeModel request for the model family of an API that is not listed is answered as an unknown operation. Converse requests, and requests in the aws-bedrock dialect, are always served.

options.dialects

array

The dialects this server serves, as an alternative to dialect, with which it is mutually exclusive. The list must have at least one entry, and a dialect may be listed once. Each entry is either a dialect name, openai, anthropic or gemini, which serves every API of that dialect, or an object with a single property named by the dialect, holding the options for that dialect. Only openai has options: apis.

options:
  dialects:
    - openai:
        apis:
          - chat/completions
    - anthropic

The dialects are told apart by the request path. A request for a dialect or API that is not listed is answered as an unknown dialect, and the stream is rejected.

dialects[].openai

object

The options of the openai dialect, for an entry that holds options.

openai.apis

array of enum [ chat/completions, responses ]

The OpenAI APIs this server serves, as for apis. Defaults to both.

dialects[].anthropic

object

The anthropic dialect, as an entry with no options.

dialects[].gemini

object

The gemini dialect, as an entry with no options.

options.operations

array of enum [ converse, converse-stream, invoke, invoke-with-response-stream ]

★Zilla Plus The aws-bedrock operations this server exposes to callers, named by their request path /model/{modelId}/<operation>. Requires dialect aws-bedrock, because a server with no dialect also accepts the other dialects. Defaults to all four.

options:
  dialect: aws-bedrock
  operations:
    - converse
    - converse-stream

A request for an operation that is not listed is rejected with 404 Not Found and the message Unknown operation., the same response as for any path the server does not serve. An llm client with dialect: aws-bedrock has its own operations, which limits what it calls upstream independently of what the server exposes.

options.authorization

object as map of named object properties

Authorization by a named guard, used to validate the credentials of each inbound request. The credentials are read from the header that the resolved dialect expects them in: authorization for openai, x-api-key for anthropic, x-goog-api-key for gemini.

A request whose credentials header is missing, does not match the credentials template, or whose extracted credentials the guard rejects, is rejected with a 401 Unauthorized response, using the dialect's own error body. Once authorized, the guard session is carried to the next binding, where it can satisfy guarded routes and supply outbound credentials to an llm client.

★Zilla Plus For aws-bedrock, the credentials are read from the authorization header, and a rejected request returns 403 Forbidden. For gcp-vertex, the credentials are read from the authorization header, and a rejected request returns 401 Unauthorized.

authorization:
  my_api_key_guard:
    credentials: "Bearer {credentials}"

authorization.credentials

string | Default: Bearer {credentials} Pattern: .*\\{credentials\\}.*

Template matched against the inbound credentials header, extracting the captured value in place of the required {credentials} placeholder and passing it to the named guard to authorize the request.

The default suits the openai authorization: Bearer <key> header. The anthropic x-api-key and gemini x-goog-api-key headers carry the key without a prefix, so set credentials: "{credentials}" for it.

★Zilla Plus For azure-foundry, the credentials are read from the authorization header when the template starts with Bearer and a space, and from the api-key header otherwise. A request that carries only the other header is accepted, with its value matched against the template. A rejected request returns 401 Unauthorized, with code invalid_api_key. When the request is authorized, the header the caller used is relayed to the next binding, so that an azure-foundry llm client can send the credentials in the same header.

The azure-foundry server resolves the request as an openai request. On the classic path the {deployment} is the model, and the model in the request body is ignored. Message content must be a string, so a request with content parts such as images is rejected with 400 Bad Request.

routes

array of object

Routes to the exit binding, selected only by the guarded roles of the inbound request's authorization. The first route whose guarded roles are satisfied is followed. A request that matches no route is rejected.

routes:
  - guarded:
      my_api_key_guard:
        - premium
    exit: llm_proxy_premium

routes[].guarded

object as map of named array of string

Roles required by the named guard.

routes:
  - guarded:
      my_api_key_guard:
        - premium

routes[].exit

string

Next binding when following this route.

routes:
  - exit: llm_proxy_premium

exit

string

Default exit binding when no conditional routes are viable.

exit: echo_server

telemetry

object

Defines the desired telemetry for the binding.

telemetry.metrics

array

Telemetry metrics to track

telemetry:
  metrics:
    - stream.*

telemetry.attributes

object as map of named string properties

Telemetry attributes to add to the binding metrics.

Token usage

A streaming openai request is forwarded with stream_options.include_usage set to true, so that the token usage reported by the upstream LLM server can be recorded by the llm metrics. The final chunk carrying token usage is returned to the caller only when the request itself sets stream_options.include_usage to true.

Errors

Errors are returned in the error body format of the request's dialect, such as {"error":{...}} for openai, {"type":"error","error":{...}} for anthropic or {"error":{"code":...,"message":...,"status":...}} for gemini, where status is the Google API status such as INVALID_ARGUMENT or RESOURCE_EXHAUSTED. ★Zilla Plus For aws-bedrock, the error type is returned in the x-amzn-errortype header, with a {"message":...} body. For gcp-vertex, the error is in the format of the Vertex AI API that the request used.

Status	Cause
`400 Bad Request`	The request body can't be parsed, an `anthropic` request has no supported `anthropic-version` header, a `gemini` request carries an API key that the `authorization` guard rejects (with reason `API_KEY_INVALID`), or a request section was rejected by an `llm` `proxy` guardrail or cannot be translated to the dialect of the `llm` `client`.
`401 Unauthorized`	An `openai` or `anthropic` request's credentials were missing or rejected by the `authorization` guard. ★Zilla Plus The same applies to a `gcp-vertex` request.
`403 Forbidden`	A `gemini` request carries no credentials, with status `PERMISSION_DENIED`. ★Zilla Plus An `aws-bedrock` request's credentials were missing or rejected by the `authorization` guard.
`404 Not Found`	A `gemini` request isn't a `POST` to a `generateContent` or `streamGenerateContent` path. ★Zilla Plus An `aws-bedrock` request isn't a `POST` to a Converse path, or a `gcp-vertex` request isn't a `POST` to a Vertex AI path.
`415 Unsupported Media Type`	The request `content-type` is not `application/json`.
`502 Bad Gateway`	The upstream LLM server could not be reached.
Upstream status	The upstream LLM server rejected the request, for example with `429 Too Many Requests`. The upstream error message is preserved, even when translated from a different dialect.