OpenAI SDK

The OpenAI SDK talks to the LLM Gateway through an llm · server with dialect: openai. Set the SDK's base URL to the gateway, including the /v1 path, and keep the rest of the code as it is.

Configure the Client

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:7161/v1",
    api_key="<your-provider-api-key>")

The SDK sends the key as Authorization: Bearer <key>. With credential pass-through, send the key for the provider that serves the model you ask for.

Send a Request

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    max_tokens=512,
    messages=[{"role": "user", "content": "In one sentence, what is an LLM gateway?"}])
print(resp.choices[0].message.content)

The model decides which provider answers, through the routes on llm · proxy.

Stream the Reply

for chunk in client.chat.completions.create(
        model="gpt-4o-mini",
        max_tokens=512,
        messages=[{"role": "user", "content": "In one sentence, what is an LLM gateway?"}],
        stream=True):
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Check chunk.choices before reading it: a stream can end with a chunk that carries only token usage and no choices.

Call Tools

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    max_tokens=512,
    messages=[{"role": "user", "content": "What is the weather in Paris?"}],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {"city": {"type": "string"}},
                "required": ["city"],
            },
        },
    }])
for call in resp.choices[0].message.tool_calls or []:
    print(call.function.name, call.function.arguments)

Handle Errors

Zilla returns errors in the OpenAI error format, so the SDK raises them as openai.APIStatusError:

import openai

try:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        max_tokens=512,
        messages=[{"role": "user", "content": "My email is jane.doe@example.com"}])
except openai.APIStatusError as error:
    print(error.status_code, error.message)

A request rejected by an LLM guardrail, for example, raises a 400 whose message names the rejected section, such as user-text rejected.