llm client

The llm client binding sends LLM requests to the upstream LLM server configured by its server URL, in its configured dialect, translating from the request's own dialect when they differ.

llm_client_anthropic:
  type: llm
  kind: client
  options:
    dialect: anthropic
    server: https://api.anthropic.com
    authorization:
      my_api_key_guard:
        credentials: "{credentials}"
  exit: http_client

Configuration (* required)

options*

object

The client kind llm specific options.

options:
  dialect: anthropic
  server: https://api.anthropic.com
  authorization:
    my_api_key_guard:
      credentials: "{credentials}"

options.dialect*

enum [ openai, anthropic, gemini, aws-bedrock, azure-foundry, gcp-vertex ]

The dialect this client speaks to the upstream LLM server named by server.

The dialects available to the llm binding are:

Dialect	API	Request path	Credentials header
`openai`	OpenAI Chat Completions	`/v1/chat/completions`	`authorization`
`openai`	OpenAI Responses	`/v1/responses`	`authorization`
`anthropic`	Anthropic Messages	`/v1/messages`	`x-api-key`
`gemini`	Google Gemini generateContent	`/models/{model}:generateContent`, `/models/{model}:streamGenerateContent`	`x-goog-api-key`
`aws-bedrock` ★Zilla Plus	Amazon Bedrock Converse	`/model/{modelId}/converse`, `/model/{modelId}/converse-stream`	`authorization`
`azure-foundry` ★Zilla Plus	Azure OpenAI Chat Completions	`/openai/v1/chat/completions`, `/openai/deployments/{deployment}/chat/completions`	`api-key`, `authorization`
`gcp-vertex` ★Zilla Plus	Google Vertex AI	`/v1/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:{method}`, `/v1/projects/{project}/locations/{location}/endpoints/openapi/chat/completions`	`authorization`, obtained by Zilla

When dialect differs from the dialect of the inbound request, resolved by the llm server, the request and response are translated between the two dialects. See Dialect translation.

The openai dialect covers both OpenAI APIs. Use apis to list the ones the upstream LLM server supports.

options.apis

array of enum [ chat/completions, responses ]

The OpenAI APIs the upstream LLM server supports. Requires dialect openai. The list must have at least one entry, without duplicates. Defaults to both.

options:
  dialect: openai
  server: https://api.openai.com
  apis:
    - chat/completions

The client speaks the API of the inbound request when it is listed, and otherwise translates the request to the first listed API, so apis: [ chat/completions ] lets a responses request reach an upstream LLM server that has no /v1/responses. An inbound request from another dialect is sent to the first listed API, which is chat/completions when both are listed or apis is omitted. See Dialect translation.

★Zilla Plus For aws-bedrock, apis instead lists the inbound API surfaces the binding serves, named dialect:api, where a bare dialect names all of its APIs: anthropic, openai, openai:chat/completions and openai:responses. It defaults to every API. The client rejects a request whose dialect is not listed. Converse requests, and requests in the aws-bedrock dialect, are always served.

options.server*

string | Pattern: ^https?://

Base URL of the upstream LLM server, following the base_url convention of the official SDKs: the origin plus a shared path prefix, such as https://api.openai.com/v1. The dialect appends its own operation path to it: /chat/completions or /responses for openai, following the API in use, /messages for anthropic, and /models/{model}:generateContent or /models/{model}:streamGenerateContent for gemini, with {model} taken from the model of each request. When the URL has no path, or only /, it defaults to /v1, so https://api.openai.com sends requests to https://api.openai.com/v1/chat/completions. The default for gemini is /v1beta, so https://generativelanguage.googleapis.com sends requests to https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent.

★Zilla Plus For aws-bedrock, only the origin is used. The request path is /model/{modelId}/converse, with {modelId} taken from the model field of each request, so one client binding serves every model. A streaming request from an aws-bedrock caller uses /model/{modelId}/converse-stream instead. An anthropic request for an anthropic.* model, or an openai request for an openai.* model, optionally with a geographic prefix such as us., is sent in its own format to /model/{modelId}/invoke, or /model/{modelId}/invoke-with-response-stream for a streaming request. The AWS region used for signing is taken from the host, such as us-east-1 for https://bedrock-runtime.us-east-1.amazonaws.com.★Zilla Plus For azure-foundry, server is the Azure OpenAI resource, such as https://my-resource.openai.azure.com/openai/v1. Without classic, the request path is /openai/v1/chat/completions after the path of server, and the model is sent in the request body. With classic, the path of server is ignored and the request path is /openai/deployments/{deployment}/chat/completions?api-version={api-version}, and the model is not sent in the request body. Streaming and non-streaming requests use the same path.★Zilla Plus For gcp-vertex, the URL names the Google Cloud project and location, such as https://us-central1-aiplatform.googleapis.com/v1/projects/my-project/locations/us-central1, and the dialect appends the rest of the path for the surface that hosts the model of each request, so one client binding serves every model. A gemini* model is sent to /publishers/google/models/{model}:generateContent, or :streamGenerateContent, a claude* model to /publishers/anthropic/models/{model}:rawPredict, or :streamRawPredict, and a model whose name contains /, such as meta/llama-3.3-70b-instruct-maas, to /endpoints/openapi/chat/completions.

The server URL only sets the request's :scheme, :authority and :path. The connection itself is made by the next binding, such as an http client followed by tls and tcp client bindings.

options.operations

array of enum [ converse, converse-stream, invoke, invoke-with-response-stream ]

★Zilla Plus The aws-bedrock operations this client may call, for credentials that are granted only some of the Bedrock Runtime permissions. Requires dialect aws-bedrock. Defaults to all four. Each value names an operation by its request path:

Operation	Request path	Permission
`converse`	`/model/{modelId}/converse`	`bedrock:InvokeModel`
`converse-stream`	`/model/{modelId}/converse-stream`	`bedrock:InvokeModelWithResponseStream`
`invoke`	`/model/{modelId}/invoke`	`bedrock:InvokeModel`
`invoke-with-response-stream`	`/model/{modelId}/invoke-with-response-stream`	`bedrock:InvokeModelWithResponseStream`

options:
  dialect: aws-bedrock
  server: https://bedrock-runtime.us-east-1.amazonaws.com
  operations:
    - converse
    - invoke

The client works out the operation a request wants, then calls the nearest allowed operation:

A request that wants streaming, when only the non-streaming operation is allowed, calls the non-streaming operation. The llm server still responds to its caller as a stream.
A request that does not want streaming, when only the streaming operation is allowed, calls the streaming operation. The llm server still responds to its caller with one document.
An anthropic request for an anthropic.* model, or an openai request for an openai.* model, uses Converse when neither invoke operation is allowed.
A request with no allowed operation is rejected with 400 Bad Request and type invalid_request_error, before any request is sent upstream.

options.models

object as map of named object properties

The capabilities of the models this client speaks to, keyed by model identifier or by a * glob pattern, matched against the model of each request. The client uses them to write each request member in a form the model accepts, or to leave it out, before the request is sent upstream. Each dialect has built-in defaults, which apply to a model that has no entry, or to a setting an entry does not set, so models is only needed to describe a model the defaults don't cover or to override them.

options:
  dialect: anthropic
  server: https://api.anthropic.com
  models:
    claude-3-7-sonnet-*:
      thinking: enabled
      budget:
        minimum: 1024
        maximum: 64000

Each entry sets some of these properties:

Property	Type	Dialects	Meaning
`sampling`	`boolean`	all	Whether the model accepts `temperature`, `top_p` and `top_k`. A model that does not is sent none of them.
`thinking`	`enum`	`anthropic`, `gemini`, ★Zilla Plus `aws-bedrock`, `gcp-vertex`	The thinking form the model takes. For `anthropic`, `adaptive` accepts adaptive thinking only, `adaptive-disabled` adds an explicit disabled, `adaptive-enabled` adds the deprecated enabled form, and `enabled` accepts enabled with a budget only, with `adaptive-disabled` for a model with no entry. For `gemini`, `level` or `budget`, with `level` for a model with no entry.
`budget`	`object`, with `minimum` and `maximum`	`anthropic`, `gemini`, ★Zilla Plus `aws-bedrock`, `gcp-vertex`	The range of thinking budget tokens the model accepts. A budget outside the range is clamped into it, and a `minimum` of `0` lets a budget of `0` turn thinking off.
`ladder`	`object`, with `minimal`, `low`, `medium`, `high`, `xhigh` and `max`	all	Thinking budget tokens by reasoning effort, for a model that takes a budget instead of an effort. An effort with no entry takes the nearest entry present, and an empty ladder drops the effort.
`tiers`	`array` of `enum` [ `auto`, `default`, `flex`, `priority` ]	all	The service tiers the model supports. A tier the model doesn't support is dropped.
`unsupported`	`array`	all	Replaces `unsupported` for this model.
`exclusive`	`array`	all	Replaces `exclusive` for this model.
`family`	`enum` [ `claude`, `nova`, `gpt-oss` ]	★Zilla Plus `aws-bedrock`	The model family, which decides where reasoning effort, thinking and `top_k` are written: `outputConfig.effort` and `additionalModelRequestFields.thinking` for `claude`, `additionalModelRequestFields.reasoningConfig` for `nova`, and `additionalModelRequestFields.reasoning_effort` for `gpt-oss`. A model of no known family has its reasoning members dropped.
`efforts`	`array` of `enum` [ `minimal`, `low`, `medium`, `high`, `xhigh`, `max` ]	★Zilla Plus `aws-bedrock`	The reasoning efforts the model offers. Any other effort is clamped to the nearest one offered.
`stop`	`integer`	★Zilla Plus `aws-bedrock`	The most stop sequences the model accepts. A request with more is rejected, and `0` rejects any stop sequence.
`structured`	`boolean`	★Zilla Plus `aws-bedrock`	Whether the model accepts structured output, `outputConfig.textFormat`. A model that does not rejects a response format.
`cache`	`object`, with `ttls` and `tools`	★Zilla Plus `aws-bedrock`	The prompt caching the model supports: `ttls`, the cache point time-to-live values from `5m` and `1h`, and `tools`, whether a cache point may follow a tool definition.
`named-tools`	`boolean`	★Zilla Plus `aws-bedrock`	Whether the model accepts a tool choice that names a tool. A named choice for a model that does not rejects the request.

★Zilla Plus For aws-bedrock, a model is matched whichever inference profile prefix it carries, such as us., eu. or global., and a model with no entry at all is sent every request member unchanged, so the provider's own error is returned. For gcp-vertex, thinking takes the gemini values for a Gemini model and the anthropic values for a Claude model.

options.unsupported

array of enum [ temperature, top_p, top_k, stop, seed, frequency_penalty, presence_penalty, tool_choice, parallel_tool_calls, response_format, reasoning_effort, thinking_budget_tokens, user, service_tier ]

The request members the upstream LLM server does not honor, by their canonical name. The client acts on them before sending the request, instead of relying on the upstream LLM server's answer:

An advisory member listed here is dropped.
A behavioral member, stop, tool_choice, parallel_tool_calls or response_format, rejects the request with 400 Bad Request, naming the member and the dialects, unless its value is harmless, such as null, parallel_tool_calls set to true, tool_choice set to auto or response_format set to text, in which case it is dropped.

The list applies to every model, and a models entry that sets unsupported replaces it for the models that entry matches. An empty list lists nothing. Nothing is listed by default.

options:
  dialect: openai
  server: https://my-openai-compatible-server/v1
  unsupported:
    - seed
    - service_tier

options.exclusive

array of array

Groups of request members the upstream LLM server refuses together, so that the client sends only one member of each group: the first member of the group that is present is kept, and the others are dropped. A group lists reasoning_effort and thinking_budget_tokens. The list applies to every model, and a models entry that sets exclusive replaces it for the models that entry matches. An empty list lists nothing. Nothing is listed by default.

options:
  dialect: gemini
  server: https://generativelanguage.googleapis.com
  exclusive:
    - - reasoning_effort
      - thinking_budget_tokens

options.authorization

object as map of named object properties

Authorization by a named guard, used to add credentials to each outbound request. The credentials are sent in the header that the configured dialect expects them in: authorization for openai, x-api-key for anthropic, x-goog-api-key for gemini.

The credentials are resolved by the named guard from the authorization already granted to the inbound request, for example by the same guard on an llm server. This forwards the caller's own credentials upstream. When the inbound request is not authorized by the named guard, no credentials header is sent.

authorization:
  my_api_key_guard:
    credentials: "{credentials}"

authorization.credentials

string | Default: Bearer {credentials} Pattern: .*\\{credentials\\}.*

Template used to build the outbound credentials header, with the required {credentials} placeholder replaced by the value returned by the named guard. The default suits openai. For anthropic or gemini, set credentials: "{credentials}".

★Zilla Plus The aws-bedrock dialect signs each request with AWS Signature Version 4 instead, using credentials from the standard AWS default credentials provider chain: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN environment variables, profile files, web identity (IRSA), ECS container credentials, or the EC2 instance profile. options.authorization is not needed. The identity needs the bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream permissions, or only one of them when options.operations is set accordingly. Requests are rejected with 503 Service Unavailable when credentials cannot be resolved.★Zilla Plus The azure-foundry dialect sends the credentials in the authorization header as Bearer {credentials} when the template starts with Bearer and a space, and in the api-key header otherwise. When the inbound request was received by an azure-foundry llm server that authorized it with a guard, the header the caller used, api-key or authorization, is used instead, and the template is not applied. When entra is set, authorization is not used.★Zilla Plus The gcp-vertex dialect sends each request with a Google access token as Authorization: Bearer, obtained from the Google application default credentials: the key file named by the GOOGLE_APPLICATION_CREDENTIALS environment variable, the file written by gcloud auth application-default login, or the metadata server of a Google Cloud VM or GKE pod. options.authorization is not needed, and a token sent by the caller is not forwarded. The identity needs the Vertex AI User role, roles/aiplatform.user. Requests are rejected with 503 Service Unavailable when credentials cannot be resolved.

options.classic

object

★Zilla Plus Sends requests to the Azure OpenAI classic surface, where the deployment name is part of the request path. Only valid for the azure-foundry dialect. The v1 surface needs none of it, because the model is already the deployment name.

options:
  dialect: azure-foundry
  server: https://my-resource.openai.azure.com
  classic:
    api-version: 2024-10-21
    deployments:
      gpt-4o: my-gpt-4o-deployment

classic.api-version

string

The api-version query parameter sent on the classic surface. Defaults to 2024-06-01.

classic.deployments*

object as map of named string properties

Maps the model of the request to the Azure OpenAI deployment name placed in the request path. At least one entry is required. A model with no entry uses the model name as the deployment name.

options.entra

object

★Zilla Plus Authenticates with a Microsoft Entra ID access token instead of the api key credentials of authorization. Only valid for the azure-foundry dialect. The token is sent in the authorization header as Bearer {token}.

options:
  dialect: azure-foundry
  server: https://my-resource.openai.azure.com/openai/v1
  entra:
    scope: https://ai.azure.com/.default

The token is requested when the binding attaches and refreshed five minutes before it expires. A request that cannot get a token within two seconds is rejected with 503 Service Unavailable, type server_error and message The Microsoft Entra access token is not available.

Without client-id, the credentials come from the first of these that the environment configures:

workload identity: AZURE_TENANT_ID, AZURE_CLIENT_ID and AZURE_FEDERATED_TOKEN_FILE
client secret: AZURE_TENANT_ID, AZURE_CLIENT_ID and AZURE_CLIENT_SECRET
managed identity, selecting a user-assigned identity with AZURE_CLIENT_ID when it is set

AZURE_AUTHORITY_HOST overrides the default authority https://login.microsoftonline.com.

entra.client-id

string

Selects a user-assigned managed identity.

entra.scope

string

The scope requested for the access token. Defaults to https://ai.azure.com/.default.

options.reasoning

object as map of named object properties

★Zilla Plus The reasoning behavior of the models this client speaks to, keyed by model identifier or by a * glob pattern, matched against the model of each request. Only valid for the azure-foundry dialect. Settings given here override the built-in defaults, which apply to a model that has no entry or to a setting an entry does not set.

Property	Type	Meaning
`default`	`enum` [ `none`, `minimal`, `low`, `medium`, `high`, `xhigh` ]	The reasoning effort the service applies when a request carries none. A model with no default runs without reasoning unless the request asks for it.
`tools`	`boolean`	Whether the model accepts function tools on chat completions while reasoning is on. A request with function tools for a model that does not is rejected, naming the model and the `openai/responses` dialect, unless its effort is `none`. A model with no entry accepts function tools.

options:
  dialect: azure-foundry
  server: https://my-resource.openai.azure.com/openai/v1
  reasoning:
    o4-mini:
      default: medium
      tools: false

routes

array of object

Routes to the exit binding, selected only by the guarded roles of the inbound request's authorization. The first route whose guarded roles are satisfied is followed. A request that matches no route is rejected.

routes:
  - guarded:
      my_api_key_guard:
        - premium
    exit: llm_proxy_premium

routes[].guarded

object as map of named array of string

Roles required by the named guard.

routes:
  - guarded:
      my_api_key_guard:
        - premium

routes[].exit

string

Next binding when following this route.

routes:
  - exit: llm_proxy_premium

exit

string

Default exit binding when no conditional routes are viable.

exit: echo_server

telemetry

object

Defines the desired telemetry for the binding.

telemetry.metrics

array

Telemetry metrics to track

telemetry:
  metrics:
    - stream.*

telemetry.attributes

object as map of named string properties

Telemetry attributes to add to the binding metrics.

Dialect translation

When the client's dialect differs from the dialect of the inbound request, the request is translated to the client's dialect, and the response, streaming or not, is translated back to the request's dialect. An openai client also translates between the two OpenAI APIs.

Requests

The request content is translated section by section: system instructions, user text, images, documents, audio and video, assistant text and refusals, reasoning, tool definitions, tool calls and tool results. The maximum number of output tokens is also translated, from openai max_completion_tokens or max_tokens, anthropic max_tokens, gemini generationConfig.maxOutputTokens, or ★Zilla Plus aws-bedrock inferenceConfig.maxTokens. Because anthropic requires max_tokens, it is set to 4096 when the inbound request sets no limit.

The sampling, tool and format parameters that the dialects share are translated: temperature, top_p, stop, seed, the penalties, tool_choice, parallel_tool_calls, response_format, the reasoning effort, user and the service tier. Each is written in the client dialect's own form, and dropped when that dialect has no equivalent. Some are rejected when the client dialect cannot represent the value, such as stop for an openai Responses upstream. Other parameters are not translated, and are dropped. The request to an openai or anthropic upstream LLM server is not streamed, and a streaming response is produced from its complete response. A request to a gemini upstream LLM server always calls :streamGenerateContent with alt=sse, and the response is translated as it arrives.

Content that the client's dialect cannot represent is dropped or rejected:

Client dialect	Dropped	Rejected
`openai`	reasoning, server tool results, citations	video, tools executed by the LLM provider
`anthropic`		audio, video
`gemini`	reasoning not from `gemini`, server tool results, citations	tools executed by the LLM provider
★Zilla Plus `aws-bedrock`	server tool results, citations	images, documents or audio referenced by URL or file id, audio not from `openai`, tools executed by the LLM provider
★Zilla Plus `azure-foundry`	reasoning, citations	images, documents, audio, video, server tools

★Zilla Plus A gcp-vertex client rejects a url source in a request for a claude* model, because Vertex AI does not support it.

A rejected request returns 400 Bad Request with type invalid_request_error, and a message naming the section and dialects, such as user-video cannot be converted from aws-bedrock to openai.

For gemini, the request is translated to the Gemini generateContent API, and model is moved from the body into the request path:

Content	Gemini
System instructions	`systemInstruction.parts[].text`
Text	`text` part of a `user` or `model` content
Images, documents, audio and video	`inlineData` part, or `fileData` part for a URL or file reference
Reasoning, from `gemini`	`text` part with `thought` set to `true`
Tool definitions	`tools[].functionDeclarations`
Tool calls	`functionCall` part, with the placeholder `thoughtSignature` `skip_thought_signature_validator` on the first call of a turn that has no signature of its own
Tool results	`functionResponse` part, in a `user` content

A fileData part names a Files API handle or a URL, and a file reference from another dialect is passed on as the fileUri unchanged, so only references the Gemini API itself resolves, such as Files API URIs, succeed.

★Zilla Plus For azure-foundry, system instructions, user and assistant text, tool definitions, tool calls and tool results are translated to Azure OpenAI chat completions messages, tools and tool_calls. The maximum number of output tokens is sent as max_completion_tokens. Request parameters of the openai dialect, such as temperature, n, logprobs and parallel_tool_calls, are passed through, as are the azure-foundry parameters data_sources, user_security_context and enhancements. Parameters of other dialects are dropped. A streaming request sets stream_options as the caller gave it, so stream_options.include_usage must be set by the caller to receive token usage.★Zilla Plus For aws-bedrock, the request is translated to the Bedrock Converse API, and model is moved from the body into the request path:

Content	Bedrock Converse
System instructions	`system[].text`
Text and refusals	`text` content block
Images and documents	`image` and `document` content blocks with `bytes`
Audio, from `openai`	`audio` content block
Reasoning, from `anthropic`	`reasoningContent` content block
Tool definitions	`toolConfig.tools[].toolSpec`
Tool calls	`toolUse` content block
Tool results	`toolResult` content block

★Zilla Plus For gcp-vertex, the request is sent to the Vertex AI surface that hosts the model, in the dialect of that surface: gemini for a gemini* model, anthropic for a claude* model, and openai for a model named publisher/model. Content is translated to that dialect as above, and model is moved from the body into the request path, except for openai, which keeps it in the body. A request is sent to the streaming method of the surface, and a response that is not streamed is built from the streamed response, unless the request is already in the dialect of the surface.

OpenAI APIs

The openai dialect covers the Chat Completions API (/v1/chat/completions) and the Responses API (/v1/responses). A request in one OpenAI API is translated to the other when the client's apis does not list the API of the request. Members specific to one OpenAI API are not restored for the other, except store, metadata, prompt_cache_key, user, include_obfuscation and service_tier, which mean the same in both.

A request that carries a member the other API or dialect has no equivalent for is rejected with 400 Bad Request and type invalid_request_error, instead of silently dropping it, when it must be translated to the other API or to another dialect:

Request	Rejected members
Responses	`previous_response_id`, `background`, `conversation`
Chat Completions	`n` greater than `1`, `logprobs` set to `true`, `audio`, `modalities`

The message names the member and the dialects, with openai/ccs for Chat Completions and openai/rs for Responses, such as openai/rs:previous_response_id cannot be converted from openai/rs to openai/ccs. Harmless values, such as null, n set to 1, and logprobs or background set to false, are dropped.

Responses

Text, refusals and tool calls are translated, including the anthropic tool_use block and the openai tool_calls entry. Reasoning is translated to an anthropic thinking block and a gemini text part with thought set to true, but is not translated to openai.

The finish reason is mapped to its equivalent:

`openai`	`anthropic`	`gemini`	★Zilla Plus `aws-bedrock`
`stop`	`end_turn`, `stop_sequence`, `pause_turn`	`STOP`	`end_turn`, `stop_sequence`
`length`	`max_tokens`	`MAX_TOKENS`	`max_tokens`
`tool_calls`	`tool_use`	`STOP`, with a `functionCall` part	`tool_use`
`content_filter`	`refusal`	`SAFETY`, `RECITATION`, `BLOCKLIST`, `PROHIBITED_CONTENT`, `SPII`, `IMAGE_SAFETY`, `LANGUAGE`	`content_filtered`, `guardrail_intervened`

Token usage is translated, including cached and reasoning tokens where the request's dialect reports them:

openai reports prompt_tokens, completion_tokens and total_tokens, computing total_tokens from input and output tokens when it is not reported, along with prompt_tokens_details.cached_tokens and completion_tokens_details.reasoning_tokens.
anthropic reports input_tokens, output_tokens, cache_read_input_tokens and cache_creation_input_tokens. In a translated anthropic stream, message_start reports 0 tokens, and the token usage arrives in message_delta.
gemini reports promptTokenCount, candidatesTokenCount, thoughtsTokenCount, cachedContentTokenCount and totalTokenCount in usageMetadata, repeated and cumulative in every chunk of a stream. The output tokens are candidatesTokenCount plus thoughtsTokenCount, and the last report is the one used.
In a translated openai stream, token usage is sent in a final chunk with empty choices, before data: [DONE], only when the request sets stream_options.include_usage to true.

Some response content has no equivalent in the other dialect, so it is not translated:

Only the first choice of an openai response is translated.
openai logprobs are not translated.
Citations and the results of tools executed by the LLM provider are not translated.
★Zilla Plus aws-bedrock metrics are not translated.

★Zilla Plus For azure-foundry, the finish reasons are those of openai. The Azure content filter results of a response are kept with the content that they describe: content_filter_results, content_filter_offsets and logprobs of a choice, and the prompt_filter_results and prompt_annotations of the response. They are not translated to other dialects, and are dropped when larger than 4096 bytes. Reasoning content, refusals, annotations and audio of a response are dropped.

Errors

When the upstream LLM server rejects the request, its status, error type and error message are preserved, and re-expressed in the error body format of the inbound request's dialect, so an anthropic 429 rate_limit_error is returned to an openai caller as a 429 with "type": "rate_limit_error". The upstream error code is also preserved for an openai caller, when the upstream LLM server sends one. The same applies to an error that arrives after a streamed response has started.

When the upstream LLM server cannot be reached, 502 Bad Gateway is returned.

★Zilla Plus For azure-foundry, a prompt blocked by the Azure content filter is returned to the caller as 400 Bad Request with code content_filter and param prompt, and a completion that the content filter blocks ends with the finish reason content_filter. A request section that the azure-foundry dialect cannot represent returns 400 Bad Request with a message such as user-image cannot be converted from openai to azure-foundry.