llm server

The llm server binding accepts inbound LLM API requests from an http server, resolving each request's dialect and optionally authorizing its credentials with a guard, before forwarding it to the exit binding.

llm_server:
  type: llm
  kind: server
  options:
    dialect: openai
    authorization:
      my_api_key_guard:
        credentials: "Bearer {credentials}"
  exit: llm_proxy

Configuration (* required)

options

object

The server kind llm specific options.

options:
  dialect: openai
  authorization:
    my_api_key_guard:
      credentials: "Bearer {credentials}"

options.dialect

enum [ openai, anthropic, aws-bedrock ]

Fixes the dialect assumed for every inbound request, instead of detecting it per request.

The dialects available to the llm binding are:

Dialect	API	Request path	Credentials header
`openai`	OpenAI Chat Completions	`/v1/chat/completions`	`authorization`
`anthropic`	Anthropic Messages	`/v1/messages`	`x-api-key`
`aws-bedrock` ★Zilla Plus	Amazon Bedrock Converse	`/model/{modelId}/converse`, `/model/{modelId}/converse-stream`	`authorization`

When dialect is omitted, each request is offered to every available dialect, and must be recognized by exactly one of them:

openai recognizes a POST to /v1/chat/completions with content-type: application/json, with or without parameters such as charset.
anthropic recognizes a POST to /v1/messages, or any request carrying an anthropic-version header, or an x-api-key header without an authorization header.
aws-bedrock ★Zilla Plus recognizes a POST to /model/{modelId}/converse or /model/{modelId}/converse-stream.

A request recognized by no dialect, or by more than one, is rejected. When dialect is set to openai or anthropic, the request path and method are not checked. ★Zilla Plus When dialect is set to aws-bedrock, a request that isn't a POST to /model/{modelId}/converse or /model/{modelId}/converse-stream is rejected with 404 Not Found.

Once the dialect is resolved:

a request whose content-type is not application/json is rejected with 415 Unsupported Media Type
an anthropic request must carry the anthropic-version: 2023-06-01 header, or it is rejected with 400 Bad Request
a request body that can't be parsed is rejected with 400 Bad Request

options.authorization

object as map of named object properties

Authorization by a named guard, used to validate the credentials of each inbound request. The credentials are read from the header that the resolved dialect expects them in: authorization for openai, x-api-key for anthropic.

A request whose credentials header is missing, does not match the credentials template, or whose extracted credentials the guard rejects, is rejected with a 401 Unauthorized response, using the dialect's own error body. Once authorized, the guard session is carried to the next binding, where it can satisfy guarded routes and supply outbound credentials to an llm client.

★Zilla Plus For aws-bedrock, the credentials are read from the authorization header, and a rejected request returns 403 Forbidden.

authorization:
  my_api_key_guard:
    credentials: "Bearer {credentials}"

authorization.credentials

string | Default: Bearer {credentials} Pattern: .*\\{credentials\\}.*

Template matched against the inbound credentials header, extracting the captured value in place of the required {credentials} placeholder and passing it to the named guard to authorize the request.

The default suits the openai authorization: Bearer <key> header. The anthropic x-api-key header carries the key without a prefix, so set credentials: "{credentials}" for it.

routes

array of object

Routes to the exit binding, selected only by the guarded roles of the inbound request's authorization. The first route whose guarded roles are satisfied is followed. A request that matches no route is rejected.

routes:
  - guarded:
      my_api_key_guard:
        - premium
    exit: llm_proxy_premium

routes[].guarded

object as map of named array of string

Roles required by the named guard.

routes:
  - guarded:
      my_api_key_guard:
        - premium

routes[].exit

string

Next binding when following this route.

routes:
  - exit: llm_proxy_premium

exit

string

Default exit binding when no conditional routes are viable.

exit: echo_server

telemetry

object

Defines the desired telemetry for the binding.

telemetry.metrics

array

Telemetry metrics to track

telemetry:
  metrics:
    - stream.*

telemetry.attributes

object as map of named string properties

Telemetry attributes to add to the binding metrics.

Token usage

A streaming openai request is forwarded with stream_options.include_usage set to true, so that the token usage reported by the upstream LLM server can be recorded by the llm metrics. The final chunk carrying token usage is returned to the caller only when the request itself sets stream_options.include_usage to true.

Errors

Errors are returned in the error body format of the request's dialect, such as {"error":{...}} for openai or {"type":"error","error":{...}} for anthropic. ★Zilla Plus For aws-bedrock, the error type is returned in the x-amzn-errortype header, with a {"message":...} body.

Status	Cause
`400 Bad Request`	The request body can't be parsed, an `anthropic` request has no supported `anthropic-version` header, or a request section was rejected by an `llm` `proxy` guardrail or cannot be translated to the dialect of the `llm` `client`.
`401 Unauthorized`	The request's credentials were missing or rejected by the `authorization` guard.
`403 Forbidden` ★Zilla Plus	An `aws-bedrock` request's credentials were missing or rejected by the `authorization` guard.
`404 Not Found` ★Zilla Plus	An `aws-bedrock` request isn't a `POST` to a Converse path.
`415 Unsupported Media Type`	The request `content-type` is not `application/json`.
`502 Bad Gateway`	The upstream LLM server could not be reached.
Upstream status	The upstream LLM server rejected the request, for example with `429 Too Many Requests`. The upstream error message is preserved, even when translated from a different dialect.