Secure LLM Access with JWT

The LLM Credentials page describes the credentials the LLM Gateway handles at each end of a request. This guide authenticates applications with JWT bearer tokens instead of provider keys: a jwt guard on llm · server validates each token, Zilla adds the OpenAI key it holds to each request it sends upstream, and callers with the premium role are routed to their own client.

Prerequisites

• Docker Compose
• An OpenAI API key, exported as OPENAI_API_KEY

docker-compose.yaml

Download docker-compose.yaml

name: zilla-secure-llm-access-jwt
services:
  zilla:
    image: ghcr.io/aklivity/zilla:${ZILLA_VERSION:-2.5.0-rc5}
    restart: unless-stopped
    ports:
      - 7161:7161
      - 7190:7190
    healthcheck:
      interval: 5s
      timeout: 3s
      retries: 15
      start_period: 15s
      start_interval: 250ms
      test: ["CMD", "sh", "-c", "test -f /var/run/zilla/ready"]
    environment:
      OPENAI_API_KEY: ${OPENAI_API_KEY}
    volumes:
      - ./zilla.yaml:/etc/zilla/zilla.yaml:ro
    command: start -v -e

  jwt-cli:
    image: bitnamilegacy/jwt-cli
    profiles:
      - on-demand
    volumes:
      - ./private.pem:/private.pem:ro

zilla.yaml

Download zilla.yaml

name: secure-llm-access-jwt
#region guard
guards:
  llm_jwt:
    type: jwt
    options:
      issuer: https://auth.example.com
      audience: https://llm.example.com
      keys:
        - kty: RSA
          n: tfIpDLlc-tOwAOe-_x1VV1xGohkpkSsxY6LB-3FwXIM6ueRZ--QtRUN4UOcw56Xs6TrQJFcbeF2rmtRZa3FZWC8dentLaO896vvc_w6pTyNQzlO5IHOv2snHFvT9dlG9wxwXv5jd1nNLYLQ9wTnROqJfK9KipUTGTTyOd1B6Ia7xUZT6kTPCSHZAvj4zVfafW37KfWtEdvq0LdDTivavc37yDnGOI7Z27MEAuA8RuAicZlIjF4tqMZ8qw-Y2NZs1OjoPbhGZde5k6hqET5i-hqrumI_ihf7yX7Vs0BLvovwczmcDF6SPHWywRd7Rs6g8A5sClRMGI9w8rJ4kk6KnHQ
          e: AQAB
          alg: RS256
          kid: example
#endregion guard
bindings:
  north_tcp_server:
    type: tcp
    kind: server
    options:
      host: 0.0.0.0
      port: 7161
    exit: north_http_server
  north_http_server:
    type: http
    kind: server
    exit: north_llm_server
  #region llm_server
  north_llm_server:
    type: llm
    kind: server
    options:
      dialect: openai
      authorization:
        llm_jwt:
          credentials: "Bearer {credentials}"
    exit: north_llm_proxy
  #endregion llm_server
  #region llm_proxy
  north_llm_proxy:
    type: llm
    kind: proxy
    routes:
      - when:
          - model:
              - gpt-*
        guarded:
          llm_jwt:
            - premium
        exit: south_llm_client_openai_premium
      - when:
          - model:
              - gpt-*
        exit: south_llm_client_openai
  #endregion llm_proxy
  #region llm_clients
  south_llm_client_openai_premium:
    type: llm
    kind: client
    options:
      dialect: openai
      server: https://api.openai.com
    telemetry:
      metrics:
        - llm.*
    exit: south_http_client_openai
  south_llm_client_openai:
    type: llm
    kind: client
    options:
      dialect: openai
      server: https://api.openai.com
    telemetry:
      metrics:
        - llm.*
    exit: south_http_client_openai
  #endregion llm_clients
  #region http_client
  south_http_client_openai:
    type: http
    kind: client
    options:
      overrides:
        authorization: Bearer ${{env.OPENAI_API_KEY}}
    exit: sys:tls_client
  #endregion http_client
telemetry:
  metrics:
    - llm.interactions
  exporters:
    prometheus_exporter:
      type: prometheus
      options:
        endpoints:
          - scheme: http
            port: 7190
            path: /metrics

Download private.pem

private.pem signs the test tokens in this guide. Use it only for this guide.

The jwt guard validates the token signature, issuer, and audience, and reads the caller's roles from the scope claim:

guards:
  llm_jwt:
    type: jwt
    options:
      issuer: https://auth.example.com
      audience: https://llm.example.com
      keys:
        - kty: RSA
          n: tfIpDLlc-tOwAOe-_x1VV1xGohkpkSsxY6LB-3FwXIM6ueRZ--QtRUN4UOcw56Xs6TrQJFcbeF2rmtRZa3FZWC8dentLaO896vvc_w6pTyNQzlO5IHOv2snHFvT9dlG9wxwXv5jd1nNLYLQ9wTnROqJfK9KipUTGTTyOd1B6Ia7xUZT6kTPCSHZAvj4zVfafW37KfWtEdvq0LdDTivavc37yDnGOI7Z27MEAuA8RuAicZlIjF4tqMZ8qw-Y2NZs1OjoPbhGZde5k6hqET5i-hqrumI_ihf7yX7Vs0BLvovwczmcDF6SPHWywRd7Rs6g8A5sClRMGI9w8rJ4kk6KnHQ
          e: AQAB
          alg: RS256
          kid: example

llm · server reads the token from the Authorization header and passes it to the guard:

north_llm_server:
  type: llm
  kind: server
  options:
    dialect: openai
    authorization:
      llm_jwt:
        credentials: "Bearer {credentials}"
  exit: north_llm_proxy

llm · proxy sends callers with the premium role to their own client, and every other caller to the default client:

north_llm_proxy:
  type: llm
  kind: proxy
  routes:
    - when:
        - model:
            - gpt-*
      guarded:
        llm_jwt:
          - premium
      exit: south_llm_client_openai_premium
    - when:
        - model:
            - gpt-*
      exit: south_llm_client_openai

Neither llm · client forwards the token. Both exit to an http · client that adds the OpenAI key from the OPENAI_API_KEY environment variable to every request:

south_http_client_openai:
  type: http
  kind: client
  options:
    overrides:
      authorization: Bearer ${{env.OPENAI_API_KEY}}
  exit: sys:tls_client

Start the Stack

docker compose up -d --wait

Mint Tokens

Mint a token for an application, with the llm scope:

export JWT_APP=$(docker compose run --rm jwt-cli encode --alg RS256 --kid example \
    --iss https://auth.example.com --aud https://llm.example.com \
    --exp=+1h --no-iat --sub app-1 --payload "scope=llm" \
    --secret @/private.pem | tr -d '\r\n')

And one with the premium role:

export JWT_PREMIUM=$(docker compose run --rm jwt-cli encode --alg RS256 --kid example \
    --iss https://auth.example.com --aud https://llm.example.com \
    --exp=+1h --no-iat --sub app-2 --payload "scope=llm premium" \
    --secret @/private.pem | tr -d '\r\n')

Send a Request with a Token

curl -s http://localhost:7161/v1/chat/completions \
  -H "Authorization: Bearer $JWT_APP" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","max_tokens":20,"messages":[{"role":"user","content":"Say hello"}]}'

OpenAI answers. Zilla validated the token, and sent the request to OpenAI with the key it holds.

Send a Request Without a Valid Token

Leave out the token, or send one Zilla doesn't accept:

curl -s http://localhost:7161/v1/chat/completions \
  -H "Authorization: Bearer not-a-valid-token" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","max_tokens":20,"messages":[{"role":"user","content":"Say hello"}]}'

{"error":{"message":"Incorrect API key provided.","type":"invalid_request_error","param":null,"code":"invalid_api_key"}}

Zilla rejects the request with 401 before it reaches OpenAI. An expired token gets the same 401.

Route by Role

Send the same request with each token, then compare the metrics:

curl -s -o /dev/null http://localhost:7161/v1/chat/completions \
  -H "Authorization: Bearer $JWT_PREMIUM" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","max_tokens":20,"messages":[{"role":"user","content":"Say hello"}]}'

curl -s http://localhost:7190/metrics | grep '^llm_interactions_total'

llm_interactions_total{namespace="secure-llm-access-jwt",binding="south_llm_client_openai_premium"} 1
llm_interactions_total{namespace="secure-llm-access-jwt",binding="south_llm_client_openai"} 1

The premium request went through south_llm_client_openai_premium, and the other through south_llm_client_openai. Point each client at a different deployment, account, or provider to give each role its own.

Stop the Stack

docker compose down

Next Steps

• LLM Credentials: credentials templates, and forwarding each caller's own key instead.
• JWT Bearer: the jwt guard in detail.
• LLM Metrics: label token usage per caller with the token's claims.