Schema Enforcement

Kafka Gateway parses only the metadata it needs to move a message between protocols. A model adds the type or schema information Zilla needs to deserialize, validate, and transform the rest of the message.

Primitive Models

Primitive models validate a value against a basic type, with a few type-specific properties, and require no catalog.

Boolean

The boolean model validates that a message key or value is a boolean.

model: boolean

Use boolean for simple flag-style keys or values, for example a topic where the value is just a true/false indicator. For structured data with multiple fields, use a schema-backed model like Avro, JSON, or Protobuf instead.

See the boolean model reference for the complete configuration.

Double

The double model validates that a message key or value is a 64-bit floating-point number, accepted in text or binary format.

model: double

Beyond the base type, double supports a format (text or binary), a range constraint, and a multiple constraint that restricts values to a multiple of a given number. For example, to accept only values between -4.5 and 4.5:

model: double
range: (-4.5,4.5)

Use double for numeric fields that need decimal precision and, optionally, bounds checking, such as a price or a measurement. For whole numbers, use Int32 or Int64 instead.

See the double model reference for the full set of options.

Float

The float model validates that a message key or value is a 32-bit floating-point number, accepted in text or binary format.

model: float

Beyond the base type, float supports the same format, range, and multiple constraints as double. For example, to accept only values between -4.5 and 4.5:

model: float
range: (-4.5,4.5)

Use float for numeric fields that need decimal precision but don't require the range of a Double, such as a sensor reading. For whole numbers, use Int32 or Int64 instead.

See the float model reference for the full set of options.

Int32

The int32 model validates that a message key or value is a 32-bit integer, accepted in text or binary format.

model: int32

Beyond the base type, int32 supports a format (text or binary), a range constraint, and a multiple constraint that restricts values to a multiple of a given integer. For example, to accept only multiples of 3:

model: int32
multiple: 3

Use int32 for whole-number fields such as counts or IDs that fit within a 32-bit range. For larger values, use Int64. For decimal values, use Float or Double.

See the int32 model reference for the full set of options.

Int64

The int64 model validates that a message key or value is a 64-bit integer, accepted in text or binary format.

model: int64

Beyond the base type, int64 supports the same format, range, and multiple constraints as int32. For example, to accept only multiples of 3:

model: int64
multiple: 3

Use int64 for whole-number fields that can exceed the 32-bit range, such as timestamps or large counters. For smaller values, Int32 is sufficient. For decimal values, use Float or Double.

See the int64 model reference for the full set of options.

String

The string model validates that a message key or value is textual data, with constraints on encoding, length, and format.

model: string
encoding: utf_8
minLength: 1
maxLength: 100
pattern: ^wait=\d+$

encoding sets the character encoding (utf_8 or utf_16). minLength and maxLength bound the string's size, and pattern restricts it to a regular expression.

Use string for free-form text fields, identifiers, or values that need format validation via pattern, such as a key that must match wait=<number>. For structured, multi-field data, use a schema-backed model like Avro, JSON, or Protobuf instead.

See the string model reference for the full set of options.

Schema Models

Schema-backed models reference a catalog to supply the schema used to validate and transform the message. A schema can be resolved by subject, by the schema attached to a Kafka topic, or by a specific id:

model: avro
catalog:
  my_catalog:
    - subject: my_schema_subject

Avro

The avro model deserializes and validates a Kafka message key or value encoded with an Apache Avro schema.

model: avro
catalog:
  items-catalog:
    - strategy: topic
      version: latest

A view can translate the model to a different format for clients while Avro remains the format stored on the Kafka topic. For example, clients can produce and consume JSON while Zilla stores Avro on the topic:

model: avro
view: json
catalog:
  items-catalog:
    - subject: items-snapshots

Use avro when your Kafka topics carry Avro-encoded data, commonly alongside a schema registry such as Confluent, Karapace, or Apicurio. Combine it with view: json to expose a JSON interface over HTTP, MQTT, or another protocol while keeping Avro on the wire to Kafka.

See the avro model reference for the full set of options, including all catalog lookup strategies.

JSON

The json model validates a Kafka message key or value against a JSON Schema.

model: json
catalog:
  items-catalog:
    - strategy: topic
      version: latest

Use json when a topic's messages are plain JSON and you want Zilla to reject anything that doesn't match a published schema, for example messages entering Kafka from an HTTP API. Unlike avro and protobuf, json has no view option since the wire format and the validated format are the same.

Combine json with an Inline catalog for small, stable schemas kept in zilla.yaml, or a remote registry for schemas that evolve independently.

See the json model reference for the full set of options.

Protobuf

The protobuf model deserializes and validates a Kafka message key or value encoded with Protocol Buffers, and requires a record naming which message type in the schema to use.

model: protobuf
catalog:
  items-catalog:
    - strategy: topic
      version: latest
      record: EchoMessage

A view can translate the model to a different format for clients while Protobuf remains the format stored on the Kafka topic. For example, clients can produce and consume JSON while Zilla stores Protobuf on the topic:

model: protobuf
view: json
catalog:
  items-catalog:
    - subject: items-snapshots
      record: EchoMessage

Use protobuf when your Kafka topics carry Protobuf-encoded data, as in the grpc.kafka.proxy example. Combine it with view: json to expose a JSON interface over HTTP, MQTT, or another protocol while keeping Protobuf on the wire to Kafka.

See the protobuf model reference for the full set of options, including all catalog lookup strategies.

Structured Message Data

The kafka cache_client and cache_server bindings apply models to the messages they proxy.

Validating Message Keys

north_kafka_cache_client:
  type: kafka
  kind: cache_client
  options:
    topics:
      - name: my-kafka-topic
        key:
          model: string

Validating a New Message

The cache_client validates the message value Produced on a topic:

north_kafka_cache_client:
  type: kafka
  kind: cache_client
  options:
    topics:
      - name: my-kafka-topic
        value:
          model: avro
          catalog:
            my_catalog:
              - strategy: topic

Enforcing a Schema on Fetch

The cache_server can reject messages Fetched from a topic that don't match the configured schema, even if they were Produced without validation:

south_kafka_cache_server:
  type: kafka
  kind: cache_server
  options:
    bootstrap:
      - my-kafka-topic
    topics:
      - name: my-kafka-topic
        value:
          model: avro
          catalog:
            my_catalog:
              - strategy: topic

Exposing a Different Model Format

A view translates between the model clients interact with and the model stored on the topic. Here clients produce and consume JSON while Avro is stored on the topic:

north_kafka_cache_client:
  type: kafka
  kind: cache_client
  options:
    topics:
      - name: my-kafka-topic
        value:
          model: avro
          view: json
          catalog:
            my_catalog:
              - strategy: topic
  exit: south_kafka_cache_server
south_kafka_cache_server:
  type: kafka
  kind: cache_server
  options:
    bootstrap:
      - my-kafka-topic
    topics:
      - name: my-kafka-topic
        value:
          model: avro
          view: json
          catalog:
            my_catalog:
              - strategy: topic