AWS SDK

The AWS SDK's Bedrock Runtime client talks to the LLM Gateway through an llm · server with dialect: aws-bedrock. Set the client's endpoint to the gateway, and keep the rest of the code as it is. The gateway can send the request to Amazon Bedrock, or to OpenAI, Anthropic, or any other provider its routes lead to, and the reply comes back in the Bedrock format. The examples use the AWS SDK for Python, boto3.

Available in Zilla Plus

Configure the Gateway

north_llm_server:
  type: llm
  kind: server
  options:
    dialect: aws-bedrock
  exit: north_llm_proxy

north_llm_proxy:
  type: llm
  kind: proxy
  routes:
    - when:
        - model:
            - gpt-*
      exit: openai_llm_client
    - when:
        - model:
            - amazon.*
            - us.anthropic.*
      exit: bedrock_llm_client

openai_llm_client:
  type: llm
  kind: client
  options:
    dialect: openai
    server: https://api.openai.com
  exit: openai_http_client

openai_http_client:
  type: http
  kind: client
  options:
    overrides:
      authorization: Bearer ${{env.OPENAI_API_KEY}}
  exit: sys:tls_client

The llm · server accepts the Bedrock Runtime API. See Accepting the Bedrock Converse API. Bedrock model IDs go to an AWS Bedrock client, bedrock_llm_client here, which signs each request with the AWS credentials Zilla Plus runs with.

The AWS SDK signs its requests with AWS Signature Version 4 and sends no provider API key, so a route to another provider needs a key that the gateway holds. Here openai_http_client adds the OpenAI key from the OPENAI_API_KEY environment variable. For Anthropic, override x-api-key instead, and for Gemini, x-goog-api-key. See Holding the Provider Key in the Gateway.

Configure the Client

import boto3

client = boto3.client(
    "bedrock-runtime",
    endpoint_url="http://localhost:7164",
    region_name="us-east-1",
    aws_access_key_id="<access-key-id>",
    aws_secret_access_key="<secret-access-key>")

The SDK signs each request with these credentials. Zilla Plus doesn't verify AWS Signature Version 4, and calls Bedrock with its own AWS credentials, so the client's credentials don't need access to Bedrock.

Send a Request

resp = client.converse(
    modelId="amazon.nova-micro-v1:0",
    messages=[{"role": "user", "content": [{"text": "In one sentence, what is an LLM gateway?"}]}],
    inferenceConfig={"maxTokens": 200})
print(resp["output"]["message"]["content"][0]["text"])
print(resp["usage"])

The modelId decides which provider answers, through the routes on llm · proxy, so the same client can ask for gpt-4o-mini, and the reply still comes back as a Converse response, with the token usage.

Stream the Reply

resp = client.converse_stream(
    modelId="amazon.nova-micro-v1:0",
    messages=[{"role": "user", "content": [{"text": "In one sentence, what is an LLM gateway?"}]}])
for event in resp["stream"]:
    if "contentBlockDelta" in event:
        print(event["contentBlockDelta"]["delta"].get("text", ""), end="", flush=True)
    if "metadata" in event:
        print()
        print(event["metadata"]["usage"])

The reply arrives as a Converse event stream, whichever provider serves it.

Call Tools

tool_config = {
    "tools": [{
        "toolSpec": {
            "name": "get_weather",
            "description": "Get the current weather for a city",
            "inputSchema": {"json": {
                "type": "object",
                "properties": {"city": {"type": "string"}},
                "required": ["city"],
            }},
        },
    }],
}

resp = client.converse(
    modelId="gpt-4o-mini",
    messages=[{"role": "user", "content": [{"text": "What is the weather in Paris?"}]}],
    toolConfig=tool_config)
for block in resp["output"]["message"]["content"]:
    if "toolUse" in block:
        print(block["toolUse"]["name"], block["toolUse"]["input"])

Call InvokeModel

The llm · server also accepts InvokeModel for anthropic.* and openai.* models, with the request body in the model's own format:

import json

resp = client.invoke_model(
    modelId="openai.gpt-oss-20b-1:0",
    contentType="application/json",
    body=json.dumps({
        "messages": [{"role": "user", "content": "In one sentence, what is an LLM gateway?"}],
        "max_tokens": 200,
    }))
print(json.loads(resp["body"].read())["choices"][0]["message"]["content"])

Set contentType="application/json", because Zilla Plus rejects a request whose content type isn't JSON. invoke_model_with_response_stream streams the reply in the same way. InvokeModel for any other model is rejected with 400 and the message InvokeModel is supported for anthropic and openai models.

Handle Errors

Zilla returns errors in the Bedrock format, so the SDK raises them as botocore.exceptions.ClientError:

from botocore.exceptions import ClientError

try:
    resp = client.converse(
        modelId="gpt-4o-mini",
        messages=[{"role": "user", "content": [{"text": "My email is jane.doe@example.com"}]}])
except ClientError as error:
    print(error.response["ResponseMetadata"]["HTTPStatusCode"], error.response["Error"]["Message"])

A request rejected by an LLM guardrail, for example, raises a 400 whose message names the rejected section, such as user-text rejected.