Run the LLM Gateway Benchmark

This page reproduces the LLM Gateway Benchmark. Use three hosts of the same instance type: one for the mock provider, one for Zilla, and one for the load generator.

Clone the Repository

The load generator, mock provider, and Zilla configuration are in the Aklivity benchmarking repository. Clone it on each of the three hosts:

git clone https://github.com/aklivity/bifrost-benchmarking.git
cd bifrost-benchmarking

Start the Mock Provider

On the mock provider host:

cd mocker
go build -o mocker .
./mocker -host 0.0.0.0 -port 8000 -latency 1500 -h2c

Start Zilla

On the gateway host, start Zilla with zilla/zilla.yaml from the repository:

docker run -d --name zilla --network host \
  -e MOCKER_URL=http://<mocker-host>:8000 \
  -v $PWD/zilla/zilla.yaml:/etc/zilla/zilla.yaml:ro \
  ghcr.io/aklivity/zilla:latest \
  start -v -e

Run the Load Generator

On the load generator host:

go build benchmark.go
ZILLA_PORT=7161 ./benchmark -provider zilla -host <zilla-host> -rate 5000 -duration 30 \
  -warmup 10 -h2c -h2c-conns 4 -timeout 90

See the repository's README for every load generator flag.