Skip to main content

REST over Kafka

Get Started ran the packaged http.kafka.crud example to show a REST API backed by Kafka. This tutorial builds the same kind of API from scratch, with two files you write yourself and a Kafka broker running in the same Docker Compose stack, so you can see exactly how each moving piece fits together and reuse it as a starting template.

The http-kafka binding maps HTTP routes directly onto Kafka produce and fetch operations, no application code in between. Kafka's cleanup.policy=compact feature lets Zilla treat the topic like a keyed record store:

  • GET fetches all items on the topic, or a single item by its key using /items/{id}.
  • POST creates a new item. Sending an Idempotency-Key header sets the Kafka message key; if the header is omitted, Zilla generates a UUID instead.
  • PUT updates an item by producing a new message under the same key using /items/{id}.
  • DELETE removes an item by producing a tombstone (a blank message) for its key using /items/{id}.

Prerequisites

Build the REST API

Create zilla.yaml and docker-compose.yaml in the same directory.

zilla.yaml
name: REST-example
bindings:
  # Proxy service entrypoint
  north_tcp_server:
    type: tcp
    kind: server
    options:
      host: 0.0.0.0
      port: 7114
    exit: north_http_server
  north_http_server:
    type: http
    kind: server
    routes:
      - when:
          - headers:
              :scheme: http
              :authority: localhost:7114
        exit: north_http_kafka_mapping

  # Proxy REST endpoints to a Kafka topic
  north_http_kafka_mapping:
    type: http-kafka
    kind: proxy
    routes:
      #region rest_create
      - when:
          - method: POST
            path: /items
        exit: north_kafka_cache_client
        with:
          capability: produce
          topic: items-snapshots
          key: ${idempotencyKey}
      #endregion rest_create
      #region rest_retrieve_all
      - when:
          - method: GET
            path: /items
        exit: north_kafka_cache_client
        with:
          capability: fetch
          topic: items-snapshots
          merge:
            content-type: application/json
      #endregion rest_retrieve_all
      #region rest_retrieve_id
      - when:
          - method: GET
            path: /items/{id}
        exit: north_kafka_cache_client
        with:
          capability: fetch
          topic: items-snapshots
          filters:
            - key: ${params.id}
      #endregion rest_retrieve_id
      #region rest_update
      - when:
          - method: PUT
            path: /items/{id}
        exit: north_kafka_cache_client
        with:
          capability: produce
          topic: items-snapshots
          key: ${params.id}
      #endregion rest_update
      #region rest_delete
      - when:
          - method: DELETE
            path: /items/{id}
        exit: north_kafka_cache_client
        with:
          capability: produce
          topic: items-snapshots
          key: ${params.id}
      #endregion rest_delete

  # Kafka sync layer
  north_kafka_cache_client:
    type: kafka
    kind: cache_client
    exit: south_kafka_cache_server
  south_kafka_cache_server:
    type: kafka
    kind: cache_server
    options:
      bootstrap:
        - items-snapshots
    exit: south_kafka_client

  # Connect to local Kafka
  south_kafka_client:
    type: kafka
    kind: client
    options:
      servers:
        - ${{env.KAFKA_BOOTSTRAP_SERVER}}
    exit: south_kafka_tcp_client
  south_kafka_tcp_client:
    type: tcp
    kind: client

telemetry:
  exporters:
    stdout_logs_exporter:
      type: stdout

The Compose stack runs Zilla alongside a local Kafka broker and a one-shot kafka-init container that creates the items-snapshots topic before Zilla starts.

Each REST endpoint is a route in zilla.yaml, mapping an HTTP method and path to a Kafka capability (produce or fetch):

Create

Produce a new message keyed by an idempotency key.

- when:
    - method: POST
      path: /items
  exit: north_kafka_cache_client
  with:
    capability: produce
    topic: items-snapshots
    key: ${idempotencyKey}

Run Zilla and Kafka

docker-compose up --detach

Try the CRUD endpoints

Create an item. The Idempotency-Key header becomes the Kafka message key.

curl -X POST http://localhost:7114/items -H 'Content-Type: application/json' -H 'Idempotency-Key: 1234' -d '{"greeting":"Hello, world"}'

List every item on the topic:

curl http://localhost:7114/items
[{"greeting":"Hello, world"}]

Fetch the item back by its key:

curl http://localhost:7114/items/1234

Update the item by producing a new message under the same key:

curl -X PUT http://localhost:7114/items/1234 -H 'Content-Type: application/json' -d '{"greeting":"Hello, again"}'

Delete the item by producing a tombstone for its key:

curl -X DELETE http://localhost:7114/items/1234

Remove the running containers

docker-compose down

Next Steps

  • Compare this from-scratch build with the packaged http.kafka.crud example in Get Started.
  • Walk through the HTTP to Kafka use case for a deeper look at REST-to-Kafka mapping.
  • See the http-kafka binding reference for every route option.
  • Try out more HTTP-to-Kafka examples in the Zilla examples directory: http.kafka.async, http.kafka.cache, http.kafka.oneway, and http.kafka.sync.