llm Binding

Defines a binding with Large Language Model (LLM) API support, with server, client, or proxy behavior. Each LLM API is a dialect, such as openai or anthropic. Requests can be routed on their dialect and requested model, and translated to a different dialect upstream, so that clients of one LLM API can be served by a model behind another.

A typical pipeline accepts LLM API requests with an llm server, routes them with an llm proxy, and sends them upstream with one llm client per upstream LLM server:

tcp server → http server → llm server → llm proxy → llm client → http client → tls client → tcp client

Both streaming ("stream": true, as Server-Sent Events) and non-streaming responses are supported. Token usage reported by the upstream LLM server can be recorded with the llm metrics.

server

Full config

The server kind llm binding accepts inbound LLM API requests from an http server, resolving each request's dialect and optionally authorizing its credentials with a guard, before forwarding it to the exit binding.

llm_server:
  type: llm
  kind: server
  options:
    dialect: openai
    authorization:
      my_api_key_guard:
        credentials: "Bearer {credentials}"
  exit: llm_proxy

proxy

Full config

The proxy kind llm binding routes LLM requests to different exit bindings based on the request's dialect and requested model, applying guardrails to the sections of each request and response, and falling back to other models when an exit refuses the request.

llm_proxy:
  type: llm
  kind: proxy
  options:
    guardrails:
      gpt-4o:
        request:
          - section: user-text
            value:
              model: string
              maxLength: 2000
  routes:
    - when:
        - dialect: openai
          model:
            - gpt-4o
      exit: llm_client_openai
      with:
        fallback:
          - claude-3-5-haiku-20241022
    - when:
        - dialect: openai
      exit: llm_client_anthropic

client

Full config

The client kind llm binding sends LLM requests to the upstream LLM server configured by its server URL, in its configured dialect, translating from the request's own dialect when they differ.

llm_client_anthropic:
  type: llm
  kind: client
  options:
    dialect: anthropic
    server: https://api.anthropic.com
    authorization:
      my_api_key_guard:
        credentials: "{credentials}"
  exit: http_client