Kafka for AI Agents Without Confluent Cloud

An agent is only as good as what it knows when it acts. Kafka gives it two useful forms of memory. A retained topic is history: every event in order, replayable from any offset, which is how you rebuild an agent's view after a bug or test a new prompt against last week's traffic. A compacted topic is current state: Kafka keeps at least the last value for every key, so a consumer reading from the start gets the latest record per customer, order or ticket without a separate database.

Records with the same key go to the same partition, and Kafka guarantees consumers read a partition in the order it was written. That is the property agents need most: if the key is the session or conversation ID, every message in that conversation arrives in sequence at whichever agent instance owns the partition.

Retrieval is only useful if the index matches the source. The common pattern is change data capture: Debezium runs as Kafka Connect source connectors, reads the database transaction log, and writes one change stream per table to Kafka. A processing step turns each change into chunks and embeddings, and a sink writes them to the vector store, whether that is Redis, OpenSearch, PostgreSQL with a vector extension or a dedicated database. Updates re-embed the row; deletes arrive as delete events and tombstones, and the pipeline must remove the matching vectors, or the agent will keep citing records that no longer exist.

Keep the embedding step separate from the connector so you can re-embed from the topic when you change models, which you will. Retain the change topics long enough to make that replay possible.

Two engines do most of this work. Kafka Streams is a library inside your application: it keeps fault-tolerant local state stores, supports exactly-once processing between Kafka topics, and suits enrichment, joins and per-session aggregates that feed an agent's context. Apache Flink is a separate cluster with SQL, the Table API and asynchronous I/O, which is the usual way to call an embedding model or inference endpoint from a stream without blocking it.

The Flink community has also started Apache Flink Agents, a framework for event-driven agents on Flink in Java and Python, with tool calling, short-term memory with TTLs, long-term memory and vector store integrations. Its 0.3 release in June 2026 describes the APIs as experimental and subject to incompatible change, so treat it as something to evaluate, not a foundation for a production system yet.

Confluent publishes an open-source MCP server, mcp-confluent, under the MIT license. It works against Confluent Cloud, Confluent Platform and standalone Apache Kafka through bootstrap servers, exposes more than 50 tools across topics, produce and consume, consumer groups and lag, Flink SQL, connectors, Schema Registry and Tableflow, and runs over stdio, HTTP or SSE. Tools that depend on Confluent services only work where those services exist. It also supports allow-lists and block-lists of tools, which is the feature that matters most. Community Kafka MCP servers exist as well; review them like any third-party code you hand cluster credentials.

On a production cluster, give the MCP server its own principal with ACLs scoped to named topics and groups. Allow describe, read and lag inspection. Block topic deletion, configuration changes, partition reassignment and offset resets outright, and allow produce only to a staging or request topic that a human-reviewed process consumes. Set client quotas so a looping agent cannot saturate a broker, and keep audit logs of every call.

Request and reply for tool calls. The agent writes a request to a tool's topic with a correlation ID and a reply topic in the headers; the tool service consumes it, does the work and writes the result to the reply topic. The agent matches replies by correlation ID and times out the ones that never come. This decouples agents from tools, gives every call a durable record, and lets a slow tool queue rather than fail.

Consumer groups as agent pools. Run agent instances as one consumer group and Kafka spreads partitions across them, but parallelism stops at the partition count. Queues for Kafka (KIP-932), production-ready since Kafka 4.2, adds share groups: consumers can outnumber partitions, records are acknowledged individually and delivery attempts are counted, which fits independent tasks such as document enrichment. Share groups give up per-partition ordering, so keep conversations on ordinary consumer groups.

Ordering per conversation. Key every message by session or conversation ID. One key, one partition, one ordered sequence. If a single conversation can outgrow one consumer, the design problem is the conversation, not the partition count.

Idempotent producers and transactions give exactly-once processing from Kafka topic to Kafka topic, and Kafka Streams uses them to do so. An agent's side effects are rarely Kafka writes: it sends an email, charges a card, files a ticket. Kafka's own design documentation is clear that writing to an external system needs the consumer's position coordinated with the output, either by storing offsets alongside the result or by making the write idempotent. For agents, the practical rule is an idempotency key on every tool call, derived from the conversation and step, checked by the tool before it acts.

Set retention per topic by what the data is for. Conversation and decision logs need long retention for replay and audit. Current agent state belongs in a compacted topic, where a record with a null value is a tombstone that removes the key. Tombstones are themselves cleaned up after delete.retention.ms, 24 hours by default, so a consumer that lags longer than that can miss a deletion; size that setting to your slowest rebuild. For a compacted topic that should also age out, use the combined compact and delete cleanup policy.

Agent messages change shape often, so give them schemas. Confluent Schema Registry is under the Confluent Community License, with its client and serializer modules under Apache 2.0. Apicurio Registry and Karapace are Apache 2.0 registries that implement the Confluent-compatible API, so most existing serializers work against them. Use Avro, Protobuf or JSON Schema, and enforce compatibility rules, so a prompt change that renames a field cannot break every downstream consumer.

Confluent Streaming Agents run on Confluent Cloud for Apache Flink. They let you define agents in SQL, call tools through MCP, invoke models from several providers, and replay agent runs against historical events for testing. That is a real convenience, and it does not exist on open-source Kafka, Amazon MSK or Aiven. Neither does fully managed Flink tied to the same cluster.

The replacements are components, not a product. Run Flink yourself, or on AWS use Amazon Managed Service for Apache Flink, which reads from MSK. Write agents in Kafka Streams or a consumer with your own agent framework, and keep an eye on Apache Flink Agents as it matures. Swap Schema Registry for Apicurio or Karapace if licensing matters. mcp-confluent still works against standalone Kafka for the core topic and consumer tools. What you are really giving up is integration and someone else's pager, which is the question to answer before choosing.

An agent platform on Kafka is a Kafka estate with more consumers, more connectors and stricter audit needs. The work is the same as any production cluster: partition and retention planning, ACLs and quotas, connector operations, upgrades to the release that has the features you need. Decide early whether your team holds that pager or hands it to someone who does.