Skip to main content

Streaming Data as Tools

Use CaseMCPAI Gateway

Agents often need the same operational data your services already read and write, recent orders, current inventory, live status, and that data frequently lives in an event-streaming system rather than behind a REST API an agent can call directly.

The Problem

Standing up a bespoke MCP server in front of a Kafka topic just to expose two or three operations, "create a record," "list recent records", means writing and operating a new service whose only job is translation. That service has to be kept in sync with the topic's schema and redeployed every time a tool changes, for what is ultimately a thin wrapper.

How Zilla Solves It

mcp-kafka · client connects straight to the broker and exposes produce_message, consume_messages, and a dozen topic, broker, and consumer-group admin operations as fixed MCP tools, no REST API, no MCP server, and no per-tool schema to author:

bindings:
  north_mcp_proxy:
    type: mcp
    kind: proxy
    routes:
      - exit: kafka_mcp_client
        when:
          - toolkit: kafka

  kafka_mcp_client:
    type: mcp-kafka
    kind: client
    options:
      servers:
        - kafka.internal:9092
    routes:
      - when:
          - tool: produce_message
            topics: [ orders ]
      - when:
          - tool: consume_messages
            topics: [ orders ]

See Backend Bindings for the full tool set and route-level topic scoping.

The REST-Hop Alternative

When a Kafka-backed REST API already exists, or the tool shape needs to be more specific than mcp-kafka's fixed set (a custom idempotency-key header, a narrower response shape), chain Kafka Gateway's http-kafka binding with AI Gateway's mcp-http · proxy instead:

  • Kafka Gateway's http-kafka binding exposes a Kafka topic as a REST API with no application code.
  • AI Gateway's mcp-http · proxy binding exposes that REST API as hand-named MCP tools with no MCP server needed upstream.

Architecture

mcp-kafka · client talks to the broker directly, one hop. The REST-hop alternative adds a second Zilla instance (rest-api) fronting Kafka with http-kafka, then mcp-http · proxy fronting that REST API in turn — still the same binding engine and config format at every layer, just one more hop than mcp-kafka needs.

Practical Outcome

Calling produce_message with a topic and value appends a record to the orders topic directly; consume_messages reads it back, both without a line of application code or a custom MCP server. In the REST-hop alternative, an agent instead calls a hand-named tool like create_item, whose idempotency key travels through as an HTTP header on the underlying http-kafka request, useful when a retried call must not produce a duplicate record, a nuance mcp-kafka's fixed tool set doesn't expose.

Try It

Try the example

The MCP Gateway demo routes a kafka toolkit straight to mcp-kafka · client alongside two other toolkit kinds. For the REST-hop alternative, walk through Kafka as MCP Tools for the full rest-api.yaml / mcp-gateway.yaml stack.