LLM Gateway Benchmark

This benchmark measures what Zilla adds when it routes OpenAI-compatible chat completion requests through the LLM Gateway to an upstream provider at 5,000 req/s.

Full write-up on the Aklivity blog

Summary

Zilla answered every request at 5,000 req/s. All 150,000 measured requests returned 200, with a p50 of 1,502 ms and a p99 of 1,588 ms against a mock provider that takes 1,500 ms.
Zilla added 1.0 ms at p50 and 85 ms at p99 over sending the same load straight to the mock provider.
The median request spent 0.27 ms inside Zilla, across every stage of the request and response paths.
Routing by model took 17 µs for the median request, not counting JSON parsing and encoding, HTTP/2 handling, or network writes.
Streamed responses finished within 2.1 s at p99 at 2,000 req/s over HTTP/1.1, across OpenAI and Anthropic clients and providers.

Setup

A mock provider, not a real model. The mock answers every request after a fixed 1,500 ms, so any latency above 1,500 ms comes from the gateway, the network, or the load generator.
Separate hosts. The gateway, the load generator, and the mock provider each run on their own instance, so they don't compete for CPU.
Open-loop load. The load generator sends at a fixed rate whether or not earlier requests have completed, so a slow gateway shows up as rising latency.
A direct baseline. The same load, sent from the load generator straight to the mock provider over the same protocol and connections, shows the latency the test setup itself adds. The gateway instance stays idle during the baseline.

Setting	Value
Deployment	3 × c7i.xlarge (4 vCPU, 8 GiB) in one AWS availability zone over private IPs: Zilla, the load generator, and the mock provider
Protocol	HTTP/2 cleartext on both sides, with the load generator on four connections
Traffic	5,000 req/s, open loop
Requests	`POST /v1/chat/completions`, model `gpt-4o-mini`, about 0.5 KB
Upstream	Mock provider, about 0.36 KB response after a fixed 1,500 ms delay
Configuration	One OpenAI-compatible provider, routed by model
Duration	10 s warm-up ramping from 500 to 5,000 req/s, then 30 s measurement (150,000 requests)
Client timeout	90 s
Measurement	End-to-end latency from the load generator; time inside Zilla from engine directory traces
Tooling	`benchmark.go` and `mocker` from the Aklivity fork of maximhq/bifrost-benchmarking, which adds the Zilla configuration and HTTP/2 cleartext support

The LLM Gateway is three bindings: one accepts OpenAI requests, one routes them by model, and one sends them to the OpenAI-compatible provider. The full configuration is zilla/zilla.yaml in the benchmarking repository.

north_llm_server:
  type: llm
  kind: server
  options:
    dialect: openai
  exit: north_llm_proxy
north_llm_proxy:
  type: llm
  kind: proxy
  routes:
    - when:
        - model: [ "gpt-*" ]
      exit: south_llm_client_openai
south_llm_client_openai:
  type: llm
  kind: client
  options:
    dialect: openai
    server: ${{env.MOCKER_URL}}
  exit: south_http_client

Routing another model to a different provider adds one route and one client binding.

Results

Metric	Direct to mock provider	Through Zilla	Difference
Warm-up	27,500 of 27,500 returned 200	27,499 of 27,499 returned 200
Requests sent	149,998	150,000
200 responses	149,998 (100%)	150,000 (100%)
Errors	0	0
Throughput	4,762 req/s	4,762 req/s
Average	1,501.0 ms	1,511 ms	+10.0 ms
p50	1,500.9 ms	1,501.9 ms	+1.0 ms
p99	1,502.5 ms	1,588 ms	+85.5 ms
Max	1,526.5 ms	1,719 ms	+192.5 ms

Latencies include the mock provider's 1,500 ms. Without a gateway, the load generator, mock provider, and network added 0.9 ms at p50 and 2.5 ms at p99 above that delay. Through Zilla, the same load measured 1.0 ms at p50 and 85 ms at p99 above the direct baseline. That difference includes an extra network hop and both of Zilla's connections, so it is larger than the time Zilla itself spends on a request. The breakdown below measures the time inside Zilla directly.

Per-Stage Latency

A Zilla configuration is a chain of bindings, each handling one stage of the request, such as HTTP, OpenAI parsing, or routing. Zilla records every message that passes between bindings in its engine directory, in memory-mapped ring buffers, one per engine worker. Tracing those messages shows where each request spends its time.

The median request spent 0.27 ms inside Zilla. The table follows a request close to the median of 19,750 traced at 5,000 req/s through the three LLM bindings. The request path is counted from the request body's arrival, 99 µs after the request stream opens, and the response path from the provider's last byte. All timings are for a 0.36 KB response; larger responses take longer to parse, re-encode, and send.

Operation	Latency	What happens
Parse OpenAI request	12 µs	The OpenAI server reads the chat completion request, including the model name.
Route by model	10 µs	The model router matches `gpt-4o-mini` to the `gpt-*` route and selects the OpenAI provider.
Prepare upstream request	11 µs	The OpenAI client builds the request for the provider.
Parse OpenAI response	13 µs	The OpenAI client reads the provider's chat completion response.
Route response back	7 µs	The model router passes the response back to the client's request.
Format OpenAI response	25 µs	The OpenAI server prepares the response for the client.
LLM stages total	78 µs	33 µs before the provider and 45 µs after.

The total excludes HTTP/2 decoding and encoding and network writes, which any gateway needs. Routing alone, by model and back through the router, took 17 µs.

HTTP/1.1

The main run uses HTTP/2 cleartext. The same 5,000 req/s load also ran over HTTP/1.1 on both sides, with the upstream HTTP client set to versions: [ http/1.1 ]:

Metric	Through Zilla, HTTP/1.1
Requests sent	149,996 of 150,000
200 responses	149,996 (100%)
p50	1,558 ms
p99	2,372 ms
Max	2,838 ms

HTTP/1.1 carries one request at a time on a connection, so this load needs about 22,000 connections. Two settings let Zilla carry them:

Worker capacity. Each worker has a fixed number of connection slots, shared by client and upstream connections. The default comes from host memory, 4,096 per worker on c7i.xlarge, so the run raised it with -P zilla.engine.worker.capacity=16384.
Engine directory on tmpfs.

With both settings, Zilla averaged 263% of the 400% CPU and peaked at 6.3 GB of memory. On HTTP/2, Zilla needed neither setting.

Cross-Dialect Translation

Zilla accepts OpenAI chat completions and Anthropic messages, and translates a request when the provider speaks the other dialect. The provider is chosen by the model, gpt-4o-mini or claude-haiku-4-5. Each pair ran at 2,000 req/s over HTTP/1.1 on both sides, with JSON responses, on three newly launched c7i.xlarge instances:

Client → provider	p99	CPU
OpenAI → OpenAI	1,536 ms	126%
OpenAI → Anthropic	1,536 ms	118%
Anthropic → OpenAI	1,543 ms	132%
Anthropic → Anthropic	1,535 ms	117%

Every run answered every request sent. CPU is the Zilla process, out of 400% on the 4 vCPU host. Translated pairs ran within 7 ms of OpenAI to OpenAI. With Zilla on HTTP/2 cleartext, the JSON p99 was 1,521 to 1,528 ms.

Streaming

The cross-dialect runs were repeated with streamed responses, stream: true. The mock provider sends 10 text deltas spread over 1,500 ms, the first at once, and Zilla relays them to the client as server-sent events in the client's dialect. Each pair ran at 2,000 req/s over HTTP/1.1, with a 10 s warm-up and a 30 s measurement (60,000 requests). A stream that ends without its final event, message_stop for Anthropic or [DONE] for OpenAI, counts as an error.

Client → provider	Stream p99	First event p99	CPU
OpenAI → OpenAI	1,576 ms	81 ms	235%
OpenAI → Anthropic	1,592 ms	103 ms	258%
Anthropic → OpenAI	1,673 ms	226 ms	246%
Anthropic → Anthropic	2,111 ms	676 ms	282%

Every run answered every request sent. Translated streams finished within 100 ms of OpenAI to OpenAI at p99. With Zilla on HTTP/2 cleartext, streaming measured a p99 of 1.6 to 1.7 s, with every stream sharing four client connections.

Caveats

The mock provider answers after a fixed delay with a small response. Real providers vary in latency and stream longer responses, and larger responses take longer to parse, re-encode, and send.
The main run uses HTTP/2 cleartext on both sides. At 5,000 req/s with a 1,500 ms provider, about 7,500 requests are in flight at any moment, and HTTP/1.1 needs one connection for each. HTTP/1.1 results are higher and need the settings in HTTP/1.1.
The cross-dialect and streaming runs use 2,000 req/s over HTTP/1.1, so they aren't directly comparable to the main run.
The configuration has no caller authentication, guardrails, logging, retries, or caching. Each feature you enable adds its own cost.
Results are for one gateway instance on a c7i.xlarge. Measure your own workload before sizing a deployment.

Run the Benchmark

See Run the LLM Gateway Benchmark.