llm proxy

The llm proxy binding routes LLM requests to different exit bindings based on the request's dialect and requested model, applying guardrails to the sections of each request and response, and falling back to other models when an exit refuses the request.

llm_proxy:
  type: llm
  kind: proxy
  options:
    guardrails:
      gpt-4o:
        request:
          - section: user-text
            value:
              model: string
              maxLength: 2000
  routes:
    - when:
        - dialect: openai
          model:
            - gpt-4o
      exit: llm_client_openai
      with:
        fallback:
          - claude-3-5-haiku-20241022
    - when:
        - dialect: openai
      exit: llm_client_anthropic

Configuration (* required)

options

object

The proxy kind llm specific options.

options:
  guardrails:
    gpt-4o:
      request:
        - section: user-text
          value:
            model: string
            maxLength: 2000
      response:
        - section: assistant-text
          value:
            model: string
            maxLength: 10000

options.guardrails

object as map of named object properties

Guardrails applied to the sections of each request and response, keyed by the model named in the inbound request. Each key is an exact model name, without * patterns. A request whose model has no guardrails passes through unchanged.

When a request falls back to another model, see with.fallback, the fallback model's guardrails apply after those of the requested model.

guardrails.request

array of object

Stages applied, in order, to the sections of each request.

guardrails.response

array of object

Stages applied, in order, to the sections of each response. A streamed response carries each delta as its own section, so a stage sees one delta at a time.

request[].section*

enum [ system-instruction, user-text, user-image, user-document, user-audio, user-video, assistant-text, assistant-reasoning, assistant-refusal, tool-definition, tool-call, tool-result, server-tool-definition, server-tool-call, server-tool-result, citations ] as string, or array of enum [ system-instruction, user-text, user-image, user-document, user-audio, user-video, assistant-text, assistant-reasoning, assistant-refusal, tool-definition, tool-call, tool-result, server-tool-definition, server-tool-call, server-tool-result, citations ]

The section type, or section types, this request stage applies to. Sections of other types pass through unchanged.

Section	`openai`	`anthropic`
`system-instruction`	`system` or `developer` message	`system`
`user-text`	`user` message text	`user` message text
`user-image`	`image_url` content part	`image` content block
`user-document`	`file` content part	`document` content block
`user-audio`	`input_audio` content part
`user-video`
`assistant-text`	`assistant` message text	`assistant` message text
`assistant-reasoning`		`thinking` and `redacted_thinking` content blocks
`assistant-refusal`	`refusal`
`tool-definition`	`tools` entry	`tools` entry
`tool-call`	`tool_calls` entry	`tool_use` content block
`tool-result`	`tool` message	`tool_result` content block
`server-tool-definition`		`tools` entry for a server tool
`server-tool-call`		`server_tool_use` content block
`server-tool-result`		server tool result content blocks
`citations`		text block `citations`

★Zilla Plus For aws-bedrock, a video content block is a user-video section.

request[].value*

enum [ avro, boolean, bytes, classify, double, float, int32, int64, json, protobuf, string ] as string, or object

The model that the section payload passes through, such as the string model with a maxLength. The model validates the payload, and can also transform it. When more than one stage applies to the same section type, they apply in order.

A model that transforms the payload must do so deterministically, so that the request or response it produces is predictable.

A request section rejected by a stage rejects the request, and a response section rejected by a stage aborts the response. See Errors.

value.model*

enum [ avro, boolean, bytes, classify, double, float, int32, int64, json, protobuf, string ]

A schema or type to validate the section payload. Refer to the individual model docs for type specific implementation.

value.validate

enum [ strict, lenient ] as string, or object | Default: strict

Controls how strictly the model enforces conformance to the schema. Can also be set independently for decode and encode.

validate.decode

enum [ strict, lenient ]

Validation mode applied when decoding the section payload.

validate.encode

enum [ strict, lenient ]

Validation mode applied when encoding the section payload.

response[].section*

enum [ system-instruction, user-text, user-image, user-document, user-audio, user-video, assistant-text, assistant-reasoning, assistant-refusal, tool-definition, tool-call, tool-result, server-tool-definition, server-tool-call, server-tool-result, citations ] as string, or array of enum [ system-instruction, user-text, user-image, user-document, user-audio, user-video, assistant-text, assistant-reasoning, assistant-refusal, tool-definition, tool-call, tool-result, server-tool-definition, server-tool-call, server-tool-result, citations ]

The section type, or section types, this response stage applies to, as for request[].section.

response[].value*

enum [ avro, boolean, bytes, classify, double, float, int32, int64, json, protobuf, string ] as string, or object

The model that the section payload passes through, as for request[].value.

routes

array of object

Conditional llm specific routes. Routes are evaluated in order, and the first route whose conditions match, and whose guarded roles are satisfied, is followed. A request that matches no route is rejected.

routes:
  - when:
      - dialect: openai
        model:
          - gpt-4o
    exit: llm_client_openai
  - when:
      - dialect: openai
    exit: llm_client_anthropic

routes[].guarded

object as map of named array of string

Roles required by the named guard, checked against the authorization granted to the inbound request, for example by options.authorization on the llm server.

routes:
  - guarded:
      my_api_key_guard:
        - premium

routes[].when

array of object

List of conditions (any match) to match this route. A route with no conditions matches every request. Read more: When a route matches

when[].dialect

enum [ openai, anthropic, aws-bedrock ]

Matches when the dialect of the inbound request, resolved by the llm server, equals this value. When omitted, any dialect matches.

when[].model

array of string

Matches when the model named in the request body matches any of these patterns, where * matches any sequence of characters and every other character must match exactly, such as gpt-4o or claude-*. When omitted, any model matches. When present, a request with no model does not match.

When both dialect and model are given in one condition, both must match.

routes[].with

object

Options applied when following this route.

routes:
  - when:
      - model:
          - gpt-4o
    exit: llm_client_openai
    with:
      fallback:
        - claude-3-5-haiku-20241022

with.fallback

array of string

Ordered models to try when this route's exit refuses the request before accepting it, for example when the upstream LLM server cannot be reached. Each fallback model is an exact model name, without * patterns.

Each fallback model is routed again through the routes of this binding, using the dialect and authorization of the inbound request, and the request's model is replaced with the fallback model. A fallback model that matches no route the request is authorized for is skipped.

Once an exit accepts the request, the route is fixed, so an error returned later by the upstream LLM server, such as 429 Too Many Requests or 500 Internal Server Error, is not retried. When every fallback model is refused, the last error, such as 502 Bad Gateway, is returned to the caller.

routes[].exit

string

Next binding when following this route.

routes:
  - exit: llm_client_openai

exit

string

Default exit binding when no conditional routes are viable.

exit: echo_server

telemetry

object

Defines the desired telemetry for the binding.

telemetry.metrics

array

Telemetry metrics to track

telemetry:
  metrics:
    - stream.*

telemetry.attributes

object as map of named string properties

Telemetry attributes to add to the binding metrics.

Errors

Errors are returned in the error body format of the request's dialect, such as {"error":{...}} for openai or {"type":"error","error":{...}} for anthropic.

Status	Cause
`400 Bad Request`	A request section was rejected by a guardrail, with type `invalid_request_error` and message `<section> rejected`, such as `user-text rejected`. For `openai`, `param` names the section and message index, such as `user-text[0]`.
`502 Bad Gateway`	The exit, and every fallback model, refused the request, for example because the upstream LLM server could not be reached.

A response section rejected by a guardrail ends a streaming response with an error event in the response's dialect, and aborts a non-streaming response.