llm client

The llm client binding sends LLM requests to the upstream LLM server configured by its server URL, in its configured dialect, translating from the request's own dialect when they differ.

llm_client_anthropic:
  type: llm
  kind: client
  options:
    dialect: anthropic
    server: https://api.anthropic.com
    authorization:
      my_api_key_guard:
        credentials: "{credentials}"
  exit: http_client

Configuration (* required)

options*

object

The client kind llm specific options.

options:
  dialect: anthropic
  server: https://api.anthropic.com
  authorization:
    my_api_key_guard:
      credentials: "{credentials}"

options.dialect*

enum [ openai, anthropic, aws-bedrock ]

The dialect this client speaks to the upstream LLM server named by server.

The dialects available to the llm binding are:

Dialect	API	Request path	Credentials header
`openai`	OpenAI Chat Completions	`/v1/chat/completions`	`authorization`
`anthropic`	Anthropic Messages	`/v1/messages`	`x-api-key`
`aws-bedrock` ★Zilla Plus	Amazon Bedrock Converse	`/model/{modelId}/converse`, `/model/{modelId}/converse-stream`	`authorization`

When dialect differs from the dialect of the inbound request, resolved by the llm server, the request and response are translated between the two dialects. See Dialect translation.

Base URL of the upstream LLM server, following the base_url convention of the official SDKs: the origin plus a shared path prefix, such as https://api.openai.com/v1. The dialect appends its own operation path to it: /chat/completions for openai and /messages for anthropic. When the URL has no path, or only /, it defaults to /v1, so https://api.openai.com sends requests to https://api.openai.com/v1/chat/completions.

★Zilla Plus For aws-bedrock, only the origin is used. The request path is /model/{modelId}/converse, with {modelId} taken from the model field of each request, so one client binding serves every model. A streaming request from an aws-bedrock caller uses /model/{modelId}/converse-stream instead. An anthropic request for an anthropic.* model, or an openai request for an openai.* model, optionally with a geographic prefix such as us., is sent in its own format to /model/{modelId}/invoke, or /model/{modelId}/invoke-with-response-stream for a streaming request. The AWS region used for signing is taken from the host, such as us-east-1 for https://bedrock-runtime.us-east-1.amazonaws.com.

The server URL only sets the request's :scheme, :authority and :path. The connection itself is made by the next binding, such as an http client followed by tls and tcp client bindings.

options.authorization

object as map of named object properties

Authorization by a named guard, used to add credentials to each outbound request. The credentials are sent in the header that the configured dialect expects them in: authorization for openai, x-api-key for anthropic.

The credentials are resolved by the named guard from the authorization already granted to the inbound request, for example by the same guard on an llm server. This forwards the caller's own credentials upstream. When the inbound request is not authorized by the named guard, no credentials header is sent.

authorization:
  my_api_key_guard:
    credentials: "{credentials}"

authorization.credentials

string | Default: Bearer {credentials} Pattern: .*\\{credentials\\}.*

Template used to build the outbound credentials header, with the required {credentials} placeholder replaced by the value returned by the named guard. The default suits openai. For anthropic, set credentials: "{credentials}".

★Zilla Plus The aws-bedrock dialect signs each request with AWS Signature Version 4 instead, using credentials from the standard AWS default credentials provider chain: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN environment variables, profile files, web identity (IRSA), ECS container credentials, or the EC2 instance profile. options.authorization is not needed. The identity needs the bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream permissions. Requests are rejected with 503 Service Unavailable when credentials cannot be resolved.

routes

array of object

Routes to the exit binding, selected only by the guarded roles of the inbound request's authorization. The first route whose guarded roles are satisfied is followed. A request that matches no route is rejected.

routes:
  - guarded:
      my_api_key_guard:
        - premium
    exit: llm_proxy_premium

routes[].guarded

object as map of named array of string

Roles required by the named guard.

routes:
  - guarded:
      my_api_key_guard:
        - premium

routes[].exit

string

Next binding when following this route.

routes:
  - exit: llm_proxy_premium

exit

string

Default exit binding when no conditional routes are viable.

exit: echo_server

telemetry

object

Defines the desired telemetry for the binding.

telemetry.metrics

array

Telemetry metrics to track

telemetry:
  metrics:
    - stream.*

telemetry.attributes

object as map of named string properties

Telemetry attributes to add to the binding metrics.

Dialect translation

When the client's dialect differs from the dialect of the inbound request, the request is translated to the client's dialect, and the response, streaming or not, is translated back to the request's dialect.

Requests

The request content is translated section by section: system instructions, user text, images, documents and audio, assistant text and refusals, tool definitions, tool calls and tool results. The maximum number of output tokens is also translated, from openai max_completion_tokens or max_tokens, anthropic max_tokens, or ★Zilla Plus aws-bedrock inferenceConfig.maxTokens. Because anthropic requires max_tokens, it is set to 4096 when the inbound request sets no limit.

Other request parameters, such as temperature, top_p, stop, stop_sequences, tool_choice, n, logprobs and response_format, are not translated, and are dropped. The request to the upstream LLM server is not streamed, and a streaming response is produced from its complete response.

Content that the client's dialect cannot represent is dropped or rejected:

Client dialect	Dropped	Rejected
`openai`	reasoning, server tool results, citations	video, tools executed by the LLM provider
`anthropic`		audio, video
★Zilla Plus `aws-bedrock`	server tool results, citations	images, documents or audio referenced by URL or file id, audio not from `openai`, tools executed by the LLM provider

A rejected request returns 400 Bad Request with type invalid_request_error, and a message naming the section and dialects, such as user-video cannot be converted from aws-bedrock to openai.

★Zilla Plus For aws-bedrock, the request is translated to the Bedrock Converse API, and model is moved from the body into the request path:

Content	Bedrock Converse
System instructions	`system[].text`
Text and refusals	`text` content block
Images and documents	`image` and `document` content blocks with `bytes`
Audio, from `openai`	`audio` content block
Reasoning, from `anthropic`	`reasoningContent` content block
Tool definitions	`toolConfig.tools[].toolSpec`
Tool calls	`toolUse` content block
Tool results	`toolResult` content block

Responses

Text, refusals and tool calls are translated, including the anthropic tool_use block and the openai tool_calls entry. Reasoning is translated to an anthropic thinking block, but is not translated to openai.

The finish reason is mapped to its equivalent:

`openai`	`anthropic`	★Zilla Plus `aws-bedrock`
`stop`	`end_turn`, `stop_sequence`, `pause_turn`	`end_turn`, `stop_sequence`
`length`	`max_tokens`	`max_tokens`
`tool_calls`	`tool_use`	`tool_use`
`content_filter`	`refusal`	`content_filtered`, `guardrail_intervened`

Token usage is translated, including cached and reasoning tokens where the request's dialect reports them:

openai reports prompt_tokens, completion_tokens and total_tokens, computing total_tokens from input and output tokens when it is not reported, along with prompt_tokens_details.cached_tokens and completion_tokens_details.reasoning_tokens.
anthropic reports input_tokens, output_tokens, cache_read_input_tokens and cache_creation_input_tokens. In a translated anthropic stream, message_start reports 0 tokens, and the token usage arrives in message_delta.
In a translated openai stream, token usage is sent in a final chunk with empty choices, before data: [DONE], only when the request sets stream_options.include_usage to true.

Some response content has no equivalent in the other dialect, so it is not translated:

Only the first choice of an openai response is translated, so a request with n greater than 1 returns one choice.
openai logprobs are not translated.
Citations and the results of tools executed by the LLM provider are not translated.
★Zilla Plus aws-bedrock metrics are not translated.

Errors

When the upstream LLM server rejects the request, its status, error type and error message are preserved, and re-expressed in the error body format of the inbound request's dialect, so an anthropic 429 rate_limit_error is returned to an openai caller as a 429 with "type": "rate_limit_error". The upstream error code is also preserved for an openai caller, when the upstream LLM server sends one. The same applies to an error that arrives after a streamed response has started.

When the upstream LLM server cannot be reached, 502 Bad Gateway is returned.