Data Governance

Kafka Gateway validates and shapes the data flowing between Kafka and HTTP, MQTT, gRPC, SSE, and WebSocket clients:

• Validate rejects a Kafka message key or value that doesn't conform to a schema or type, on produce or on fetch.
• Transform changes a message's representation, so clients and the topic don't have to agree on the same wire format.

Both build on two resources:

• Catalog defines where schemas live, whether embedded in zilla.yaml, read from the local filesystem, or fetched from a remote schema registry.
• Model defines how a Kafka message key or value is interpreted, from primitive types like string and int32 to schema-backed formats like avro, json, and protobuf.

A kafka binding combines both: the model decides how to parse and validate the message, and for schema-backed models, the catalog supplies the schema to validate against.

north_kafka_cache_client:
  type: kafka
  kind: cache_client
  options:
    topics:
      - name: items-snapshots
        value:
          model: avro
          catalog:
            my_catalog:
              - strategy: topic

Next Steps

• Validate walks through rejecting a malformed message.
• Transform walks through transforming a message between wire and view formats.
• Schema Registry covers each catalog type Kafka Gateway supports and when to use it.
• Schema Enforcement covers each model type Kafka Gateway supports, from primitives to schema-backed formats.