Encryption

A Kafka message often carries a field that shouldn't be visible to just anyone: a card number, an SSN, any other PII subject to a compliance requirement. The core protection is keeping that field encrypted at rest on the Kafka topic itself, so it's unreadable even to whatever operates the cluster. Everything past that is about retrieval: not every caller who reaches the field through Kafka Gateway is authorized to see the same thing, so a caller's role decides how much of the protected value comes back: the full decrypted value, or a masked, hashed, redacted, or omitted stand-in instead.

One transforms entry on a schema-backed model configures both halves:

• encryption wraps and unwraps a per-message data encryption key with a configured vault, so the field's real value never sits in the clear on the topic.
• disclosure decides how much of that protected value a given caller gets back on retrieval: full plaintext for an authorized role, or a masked, hashed, redacted, or omitted substitute otherwise.

Available in Zilla Plus

model: json
transforms:
  - select:
      - type: structure
        fields:
          - tagged:
              - x-kind: pii
    guarded:
      my-guard:
        - card:full
    encryption:
      vault:
        my-vault:
          - my-kek
    disclosure:
      action: mask
      pattern: "XX***@***XX"

On the topic itself, the field is never stored in the clear; that's encryption. On retrieval, anyone holding the card:full role gets the real value back; everyone else gets the masked pattern instead; that's disclosure, deciding the degree, not the at-rest protection itself.

How It Works

A transforms entry pairs a select (which fields it applies to), a guarded (which roles get the full plaintext back), an encryption (how the field is protected at rest), and a disclosure (what everyone else's retrieval degrades to instead of the plaintext). With no guarded at all on a rule, nobody ever gets the plaintext back: retrieval always degrades to the disclosure substitute, or to an empty/zero value if no disclosure is configured either.

Selecting Which Fields to Protect

select decides which part of the model a transforms entry applies to:

• A root selector like type: string (no fields) targets the whole message key or value.
• type: structure with fields descends into a schema-backed model and matches individual fields by either:
  • tagged: fields carrying a given custom schema property, for example a JSON Schema or Avro schema annotated with x-kind: pii.
  • named: fields matched by their exact name.

Multiple transforms entries can each match a different tag or field set and bind that selection to its own guarded role, its own vault and KEKs, and its own disclosure action, so different fields, or the same field for different callers, can be protected differently. This works the same regardless of which schema-backed model it's layered onto: json, avro, and protobuf all support both per-field selection and whole-message (root) protection.

Keeping a Field Unreadable at Rest

Encryption follows standard envelope-encryption terms:

• KEK (Key Encryption Key): a key held in a vault, referenced by alias (my-kek above). The KEK itself never leaves the vault.
• DEK (Data Encryption Key): a 256-bit AES key Zilla generates itself, used to actually encrypt a field's plaintext. The vault never sees a DEK's real message data, only the DEK's own bytes when wrapping or unwrapping it.
• EDEK (Encrypted DEK): the DEK after the vault wraps it under a KEK. The EDEK, not the raw DEK, is what travels alongside a field's ciphertext.

On encrypt, Zilla generates a DEK, encrypts the field's plaintext with it (AES-GCM), then asks the named vault to wrap that DEK into an EDEK under the configured KEK. The KEK's alias, the EDEK, and the ciphertext are bundled into one self-describing blob. On decrypt, Zilla reads the KEK alias back out of that same blob, asks the vault to unwrap the EDEK into the DEK, and decrypts the ciphertext with it, never by looking up which key is configured today, only by the alias the blob itself names. A message stays decryptable even after the deployment's encryption config changes, as long as the vault still holds a KEK with that alias, which matters for a long-lived or compacted topic.

Four vault types can hold KEKs:

Vault	Edition	Key material
filesystem	Community	Local PKCS12/JKS keystore
aws-kms	Plus	AWS KMS
hashicorp	Plus	HashiCorp Vault Transit engine
kmip	Plus	Any KMIP-compliant key manager

Note

The encryption/disclosure transforms themselves are a Zilla Plus capability regardless of vault type: filesystem only means the KEK material lives in a local keystore instead of a cloud or on-prem key manager.

Choose the vault type based on where key material is already managed: filesystem for local development, aws-kms if AWS KMS already holds the organization's keys, hashicorp if HashiCorp Vault's Transit engine is already deployed, or kmip for any other KMIP-compliant key manager.

DEK Rotation

Because a vault only ever wraps or unwraps the small DEK, not the message data, Zilla rotates DEKs aggressively without adding a vault round trip per message. A DEK retires on whichever comes first: a TTL (1 hour by default, a lazy swap, where the current DEK keeps serving until its replacement is ready) or a hard operation-count cutoff (about one million operations by default, enforced immediately, no grace period). Bounding how much ciphertext ever depended on one DEK is a safety property, not just good housekeeping, and it happens automatically; no key material to rotate by hand beyond what the vault itself manages for the KEK.

Multiple Keys, One Vault

encryption.vault.<vault-name> accepts a list of KEK aliases instead of a single one, to chain-wrap a field's DEK through more than one key from that vault, applied in the listed order, each layer wraps the complete output of the layer before it:

encryption:
  vault:
    my-vault:
      - kek0
      - kek1

Decrypt reverses the same layers, resolving each by the KEK alias recorded in the blob itself rather than by position in today's config, so a message encrypted under an older chain stays decryptable after the configured chain changes.

Bytes, Strings, and Primitive Types

Where the ciphertext travels depends on the wire shape of the field being protected:

• String and bytes fields have variable-length wire encoding, so the encrypted blob (KEK alias + EDEK + ciphertext) is substituted directly in place of the field's own value: base64-encoded for a string field, since AES-GCM ciphertext isn't valid UTF-8, or raw for a bytes field. The field is independently decryptable from nothing but its own substituted value and vault access.
• Boolean, int, long, float, double, and fixed fields have a fixed wire width that ciphertext can't fit into: AES-GCM output always runs larger than the plaintext it came from. For these, Zilla replaces the field's own wire value with that type's zero value (0, false, and so on) and instead writes the encrypted blob into a zilla:efield Kafka header, one repeated entry per encrypted primitive field, in schema-walk order. Any consumer, decrypting or not, can read the zero-value placeholder without error; only a consumer that resolves zilla:efield and is authorized to decrypt reconstructs the real value.

Disclosure: Degrees of Retrieval

Once a field is protected, guarded alone only gives a binary outcome on retrieval: the full plaintext for an authorized role, or an empty/zero value for everyone else. disclosure fills in that gap with intermediate degrees, from the closest thing to the real value down to nothing at all:

Degree	Result
(full plaintext)	Governed by guarded; no disclosure needed. The caller decrypts the field and gets the real value back.
mask	The field is replaced per a pattern template such as "XX***@***XX": a reveal character (X by default, overridable via reveal) marks positions kept from the original value, a mask character (* by default, overridable via mask) marks positions hidden, and any other character is a literal anchor (like @ above) that must occur in the actual value at that position. If an anchor can't be found in the value, the whole value is masked as a fail-safe rather than risk revealing more than intended.
hash	The field is replaced with a deterministic digest of its real value (SHA-256 by default, or HMAC-SHA256 keyed by a configured secret), rendered as hex. The same input always hashes the same, which supports joins or deduplication downstream without exposing the plaintext.
redact	The field is replaced with a fixed, type-appropriate blank: "" for strings, 0 for numbers, false for booleans.
omit	The field is nulled out entirely (legal only where the schema allows a null there).

transforms:
  - select:
      - type: structure
        fields:
          - tagged:
              - x-kind: pii-hash
    disclosure:
      action: hash
      secret: my-hmac-secret

disclosure needs no encryption configured at all: a field can be stored in Kafka as plain, unencrypted JSON, Avro, or Protobuf, and still have its plaintext withheld from callers who don't hold the right role. But its natural place is downstream of encryption: the two combine on the same rule to say what an unauthorized caller gets back instead of the empty value encryption alone would leave them with.

Authorization and Decrypt

encryption.guarded, nested under encryption itself, overrides which roles gate the decrypt attempt specifically, independent of whatever else the peer-level guarded is used for; omit it and decrypt falls back to that peer-level guard.

When both encryption and disclosure are configured on the same rule, an authorized caller (matching guarded, or encryption.guarded if that's set) decrypts and gets the real value straight off. For anyone else, decrypt is skipped outright: never attempted, never charged against the vault. disclosure then decides what they see in its place. omit/redact substitute their usual fixed blank. hash/mask do the same, rather than running against ciphertext bytes they can't read, so a denied decrypt can never leak a hash or masked fragment derived from real ciphertext.

Editions

Component	Community	Plus
encryption transform	—	✓
disclosure transform	—	✓
filesystem vault	✓	✓
aws-kms, hashicorp, kmip vaults	—	✓

Next Steps

• Schema Enforcement covers the model types encryption and disclosure layer onto.
• Validate covers rejecting a malformed message before it's protected.