Track Token Usage with Prometheus

The LLM Token Usage Visibility use case describes measuring token usage at the gateway for every application. This guide records the llm.* metrics for requests to OpenAI, reads them from Zilla's Prometheus endpoint, and queries them in Prometheus.

Prerequisites

• Docker Compose
• An OpenAI API key, exported as OPENAI_API_KEY

docker-compose.yaml

Download docker-compose.yaml

name: zilla-llm-track-token-usage
services:
  zilla:
    image: ghcr.io/aklivity/zilla:${ZILLA_VERSION:-2.5.0-rc5}
    restart: unless-stopped
    ports:
      - 7161:7161
      - 7190:7190
    healthcheck:
      interval: 5s
      timeout: 3s
      retries: 15
      start_period: 15s
      start_interval: 250ms
      test: ["CMD", "sh", "-c", "test -f /var/run/zilla/ready"]
    volumes:
      - ./zilla.yaml:/etc/zilla/zilla.yaml:ro
    command: start -v -e
  prometheus:
    image: prom/prometheus:v3.15.0
    restart: unless-stopped
    ports:
      - 9090:9090
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    depends_on:
      zilla:
        condition: service_healthy

zilla.yaml

Download zilla.yaml

name: llm-track-token-usage
#region telemetry
telemetry:
  metrics:
    - llm.interactions
    - llm.duration
    - llm.time.to.first.token
    - llm.tokens.input
    - llm.tokens.output
    - llm.tokens.total
  exporters:
    prometheus_exporter:
      type: prometheus
      options:
        endpoints:
          - scheme: http
            port: 7190
            path: /metrics
#endregion telemetry
guards:
  api_key:
    type: inline
bindings:
  north_tcp_server:
    type: tcp
    kind: server
    options:
      host: 0.0.0.0
      port: 7161
    exit: north_http_server
  north_http_server:
    type: http
    kind: server
    exit: north_llm_server
  north_llm_server:
    type: llm
    kind: server
    options:
      dialect: openai
      authorization:
        api_key:
          credentials: "Bearer {credentials}"
    exit: north_llm_proxy
  north_llm_proxy:
    type: llm
    kind: proxy
    routes:
      - when:
          - model:
              - gpt-*
        exit: south_llm_client_openai
  #region llm_client
  south_llm_client_openai:
    type: llm
    kind: client
    options:
      dialect: openai
      server: https://api.openai.com
      authorization:
        api_key:
          credentials: "Bearer {credentials}"
    telemetry:
      metrics:
        - llm.*
    exit: sys:http_client
  #endregion llm_client

prometheus.yml

Download prometheus.yml

global:
  scrape_interval: 5s
scrape_configs:
  - job_name: zilla
    fallback_scrape_protocol: PrometheusText0.0.4
    static_configs:
      - targets: ["zilla:7190"]

telemetry lists the llm.* metrics to record, and exposes them in the Prometheus format on port 7190:

telemetry:
  metrics:
    - llm.interactions
    - llm.duration
    - llm.time.to.first.token
    - llm.tokens.input
    - llm.tokens.output
    - llm.tokens.total
  exporters:
    prometheus_exporter:
      type: prometheus
      options:
        endpoints:
          - scheme: http
            port: 7190
            path: /metrics

llm · client records them for the requests it sends to OpenAI:

south_llm_client_openai:
  type: llm
  kind: client
  options:
    dialect: openai
    server: https://api.openai.com
    authorization:
      api_key:
        credentials: "Bearer {credentials}"
  telemetry:
    metrics:
      - llm.*
  exit: sys:http_client

Start the Stack

docker compose up -d --wait

This starts Zilla, and Prometheus scraping Zilla every 5 seconds.

Send Requests

Send the same request twice:

curl -s http://localhost:7161/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","max_tokens":20,"messages":[{"role":"user","content":"Say hello"}]}'

Each reply reports the usage OpenAI counted:

"usage": {
  "prompt_tokens": 9,
  "completion_tokens": 9,
  "total_tokens": 18
}

Read the Metrics from Zilla

curl -s http://localhost:7190/metrics | grep -E '^llm_(interactions_total|tokens_(input|output|total)_(sum|count))'

llm_interactions_total{namespace="llm-track-token-usage",binding="south_llm_client_openai"} 2
llm_tokens_total_sum{namespace="llm-track-token-usage",binding="south_llm_client_openai"} 36
llm_tokens_total_count{namespace="llm-track-token-usage",binding="south_llm_client_openai"} 2
llm_tokens_output_sum{namespace="llm-track-token-usage",binding="south_llm_client_openai"} 18
llm_tokens_output_count{namespace="llm-track-token-usage",binding="south_llm_client_openai"} 2
llm_tokens_input_sum{namespace="llm-track-token-usage",binding="south_llm_client_openai"} 18
llm_tokens_input_count{namespace="llm-track-token-usage",binding="south_llm_client_openai"} 2

Each token metric is a histogram: _sum is the tokens recorded, and _count the exchanges recorded. The binding label names the llm · client, so each provider's usage is a separate series.

Query in Prometheus

Open Prometheus at http://localhost:9090, or query its API.

Total tokens per provider since Zilla started:

curl -s -G http://localhost:9090/api/v1/query \
  --data-urlencode 'query=sum by (binding) (llm_tokens_total_sum)'

Tokens per provider over the last hour:

sum by (binding) (increase(llm_tokens_total_sum[1h]))

Average output tokens per exchange:

sum by (binding) (llm_tokens_output_sum) / sum by (binding) (llm_tokens_output_count)

Stop the Stack

docker compose down

Next Steps

• LLM Metrics: every llm.* metric, and labeling usage per tenant.
• Exporters: send the metrics to OTLP or AWS CloudWatch instead.