Skip to main content

OpenAI SDK on AWS Bedrock

The OpenAI Apps on AWS Bedrock use case describes serving applications written for the OpenAI API from Amazon Bedrock. This guide runs an OpenAI API endpoint that routes Bedrock model IDs to Bedrock, and calls amazon.nova-micro-v1:0 with curl and the OpenAI SDK.

Available in Zilla Plus

Prerequisites

  • Docker Compose
  • A Zilla Plus license key, exported as ZILLA_PLUS_LICENSE_KEY
  • AWS credentials allowed to call bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream on the model in us-east-1, exported as AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and, for temporary credentials, AWS_SESSION_TOKEN
docker-compose.yaml

Download docker-compose.yaml

name: zilla-openai-sdk-on-aws-bedrock
services:
  zilla:
    image: ghcr.io/aklivity/zilla-plus:${ZILLA_VERSION:-2.1.0-rc4}
    restart: unless-stopped
    ports:
      - 7161:7161
    healthcheck:
      interval: 5s
      timeout: 3s
      retries: 15
      start_period: 15s
      start_interval: 250ms
      test: ["CMD", "sh", "-c", "test -f /var/run/zilla/ready"]
    environment:
      ZILLA_PLUS_LICENSE_KEY: ${ZILLA_PLUS_LICENSE_KEY}
      AWS_REGION: ${AWS_REGION:-us-east-1}
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_SESSION_TOKEN: ${AWS_SESSION_TOKEN:-}
    volumes:
      - ./zilla.yaml:/etc/zilla/zilla.yaml:ro
    command: start -v -e
zilla.yaml

Download zilla.yaml

name: openai-sdk-on-aws-bedrock
bindings:
  north_tcp_server:
    type: tcp
    kind: server
    options:
      host: 0.0.0.0
      port: 7161
    exit: north_http_server
  north_http_server:
    type: http
    kind: server
    exit: north_llm_server
  #region llm_server
  north_llm_server:
    type: llm
    kind: server
    options:
      dialect: openai
    exit: north_llm_proxy
  #endregion llm_server
  #region llm_proxy
  north_llm_proxy:
    type: llm
    kind: proxy
    routes:
      - when:
          - model:
              - amazon.*
              - anthropic.*
              - us.amazon.*
              - us.anthropic.*
        exit: south_llm_client_bedrock
  #endregion llm_proxy
  #region llm_client
  south_llm_client_bedrock:
    type: llm
    kind: client
    options:
      dialect: aws-bedrock
      server: https://bedrock-runtime.us-east-1.amazonaws.com
    exit: south_http_client_bedrock
  south_http_client_bedrock:
    type: http
    kind: client
    exit: south_tls_client_bedrock
  south_tls_client_bedrock:
    type: tls
    kind: client
    options:
      sni:
        - bedrock-runtime.us-east-1.amazonaws.com
    exit: south_tcp_client_bedrock
  south_tcp_client_bedrock:
    type: tcp
    kind: client
    options:
      host: bedrock-runtime.us-east-1.amazonaws.com
      port: 443
  #endregion llm_client

llm · server accepts the OpenAI API. It needs no options.authorization, because the caller's API key isn't forwarded to Bedrock:

north_llm_server:
  type: llm
  kind: server
  options:
    dialect: openai
  exit: north_llm_proxy

llm · proxy routes Bedrock model IDs to the Bedrock client. Cross-region inference profile IDs start with a geography, such as us.:

north_llm_proxy:
  type: llm
  kind: proxy
  routes:
    - when:
        - model:
            - amazon.*
            - anthropic.*
            - us.amazon.*
            - us.anthropic.*
      exit: south_llm_client_bedrock

The aws-bedrock client signs each request with the AWS credentials in the Zilla Plus container, and connects through its own http, tls, and tcp clients:

south_llm_client_bedrock:
  type: llm
  kind: client
  options:
    dialect: aws-bedrock
    server: https://bedrock-runtime.us-east-1.amazonaws.com
  exit: south_http_client_bedrock
south_http_client_bedrock:
  type: http
  kind: client
  exit: south_tls_client_bedrock
south_tls_client_bedrock:
  type: tls
  kind: client
  options:
    sni:
      - bedrock-runtime.us-east-1.amazonaws.com
  exit: south_tcp_client_bedrock
south_tcp_client_bedrock:
  type: tcp
  kind: client
  options:
    host: bedrock-runtime.us-east-1.amazonaws.com
    port: 443

No AWS key appears anywhere in zilla.yaml.

Start the Stack

docker compose up -d --wait

Send a Request

curl -s http://localhost:7161/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"amazon.nova-micro-v1:0","max_tokens":60,"messages":[{"role":"user","content":"In one sentence, what is an LLM gateway?"}]}'

The reply is a standard OpenAI chat completion:

{
  "id": "5d2764de-3d2f-4d30-b5cb-1be450212538",
  "object": "chat.completion",
  "model": "amazon.nova-micro-v1:0",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "An LLM gateway is a system or interface that facilitates the interaction between large language models (LLMs) and various applications or users, enabling efficient data exchange and service integration."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 11,
    "completion_tokens": 36,
    "total_tokens": 47
  }
}

Zilla Plus converted the request to the Bedrock Converse API, signed it, and converted the reply back to the OpenAI format.

Use the OpenAI SDK

Point the OpenAI SDK at the gateway. The API key isn't used, but the SDK requires one:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:7161/v1", api_key="unused")

reply = client.chat.completions.create(
    model="amazon.nova-micro-v1:0",
    max_tokens=60,
    messages=[{"role": "user", "content": "In one sentence, what is an LLM gateway?"}],
)
print(reply.choices[0].message.content)

Stream the Reply

Set stream=True, and the reply arrives as OpenAI chunks:

stream = client.chat.completions.create(
    model="amazon.nova-micro-v1:0",
    max_tokens=60,
    stream=True,
    messages=[{"role": "user", "content": "Name three uses of an LLM gateway."}],
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Use Another Region

To use another region, change the host in server, in the tls · client's sni, and in the tcp · client's host.

Stop the Stack

docker compose down

Next Steps

  • AWS Bedrock: what is converted, credentials, and limits.
  • Route by Model: route other models to OpenAI and Anthropic from the same endpoint.