Transform

Transform changes a Kafka message's representation as it flows through the gateway, so clients and the topic don't have to agree on the same wire format. It's built on the view option of a schema-backed model.

How It Works

The avro and protobuf models accept a view, which transforms the model from its wire format to the view format when producing, and back from the view format to the wire format when consuming:

north_kafka_cache_client:
  type: kafka
  kind: cache_client
  options:
    topics:
      - name: my-kafka-topic
        value:
          model: avro
          view: json
          catalog:
            my_catalog:
              - strategy: topic
  exit: south_kafka_cache_server
south_kafka_cache_server:
  type: kafka
  kind: cache_server
  options:
    bootstrap:
      - my-kafka-topic
    topics:
      - name: my-kafka-topic
        value:
          model: avro
          view: json
          catalog:
            my_catalog:
              - strategy: topic

With this configuration, clients produce and consume json, while avro is stored on the topic. See the Avro and Protobuf reference for the full set of options.

When to Use It

Reach for Transform when clients speaking HTTP, MQTT, gRPC, SSE, or WebSocket expect one representation of a message (typically JSON) while the topic carries another (Avro or Protobuf), commonly for compatibility with existing Kafka producers and consumers or a schema registry. Rather than writing a translation layer in the client or a stream processor, the view transform does it at the gateway, in configuration.

Next Steps

• Schema Enforcement covers each schema-backed model type that supports a view.
• Validate covers rejecting a message that doesn't conform to a schema in the first place.