Azure OpenAI SDK

The AzureOpenAI client of the OpenAI SDK talks to the LLM Gateway through an llm · server with dialect: azure-foundry. Set the client's endpoint to the gateway, and keep the rest of the code as it is. The gateway can send the request to Azure OpenAI, or to any other provider its routes lead to.

Available in Zilla Plus

Configure the Gateway

north_llm_server:
  type: llm
  kind: server
  options:
    dialect: azure-foundry
    authorization:
      api_key:
        credentials: "{credentials}"
  exit: north_llm_proxy

The llm · server accepts the Azure OpenAI API, and reads the key from the api-key header the client sends. See Accepting the Azure OpenAI API.

Configure the Client

from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint="http://localhost:7165",
    api_key="<your-provider-api-key>",
    api_version="2024-10-21")

The client sends each request to /openai/deployments/{deployment}/chat/completions, with the key as api-key: <key>. With credential pass-through, send the key for the provider that serves the model you ask for.

Send a Request

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "In one sentence, what is an LLM gateway?"}])
print(resp.choices[0].message.content)

model is the deployment name, and the client puts it in the request path. It decides which provider answers, through the routes on llm · proxy, so the same client can ask for a claude-* model, with an Anthropic key.

Stream the Reply

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "In one sentence, what is an LLM gateway?"}],
    stream=True,
    stream_options={"include_usage": True})
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

stream_options.include_usage adds a final chunk with the token usage.

Call Tools

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "What is the weather in Paris?"}],
    tools=tools)
for call in resp.choices[0].message.tool_calls or []:
    print(call.function.name, call.function.arguments)

Use the v1 API

The llm · server also accepts the Azure OpenAI v1 API, where the model is in the request body. Point the OpenAI client at its /openai/v1 path:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:7165/openai/v1",
    api_key="<your-provider-api-key>")

This client sends the key as Authorization: Bearer <key>, which the llm · server also accepts.

Message Content

Each message's content must be a string. A request with content parts, such as an image, is rejected with 400, which the SDK raises as openai.BadRequestError.

Handle Errors

Zilla returns errors in the OpenAI error format, so the SDK raises its usual exceptions:

import openai

try:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Say hello"}])
except openai.AuthenticationError as error:
    print(error.status_code, error.message)
except openai.APIStatusError as error:
    print(error.status_code, error.message)

A key the provider doesn't accept raises openai.AuthenticationError with status 401, and a request rejected by an LLM guardrail raises openai.BadRequestError with status 400, naming the rejected section.