Kafka
Exactly-once semantics (EOS) with the Spring client
Section titled “Exactly-once semantics (EOS) with the Spring client”In a consume-transform-produce application the goal is that the read → process → write sequence
completes exactly once: the record is processed, the output is published, and the consumer offset
advances — all three, or none of them.
Kafka provides this with two mechanisms working together. The producer writes idempotently, so a
retried send cannot duplicate a record in the log; and the producer transaction covers both the
output records and the consumed input offsets, so a consumer reading with
isolation.level=read_committed sees the output only once the offsets were committed with it.
Spring for Apache Kafka wires this up at the listener container. Configure the container with a
KafkaTransactionManager and it begins a Kafka transaction before invoking the listener. If the
listener processes the record successfully — or the batch of records, when using a
BatchMessageListener — the container adds the consumed offsets to that same transaction before
the transaction manager commits it. If the listener throws, the transaction rolls back and the
AfterRollbackProcessor seeks the consumer back so the failed records are redelivered on the next
poll.
Two details have changed since the 2.x line and are worth checking against the version in the
build: EOSMode.V2 is now the only supported mode, which removes the old requirement to derive a
distinct transactional.id per input partition for consumer-initiated transactions; and
ChainedKafkaTransactionManager is deprecated — to span a second resource, keep the
KafkaTransactionManager on the container and annotate the listener method with @Transactional.
Spring for Apache Kafka is on the 4.1 line, which pairs with Spring Boot 4.1 and the 4.2 Kafka clients. The project publishes a compatibility matrix giving the Kafka client and Spring Boot versions each line was built against; check the build against it before copying configuration from older material.
Topics
Section titled “Topics”Topics should be created to be both available and durable. The usual pairing is a replication
factor of three with min.insync.replicas=2:
kafka-topics --bootstrap-server localhost:9092 --create --topic orders \ --partitions 6 --replication-factor 3 --config min.insync.replicas=2Together these balance availability against durability: three copies of every partition, and an
acknowledged write only once two of them are in sync. Combined with acks=all on the producer, a
write survives the loss of any single broker, while the cluster still accepts writes when one
replica is behind. The replication factor is fixed when the topic is created — the broker-wide
default is default.replication.factor — whereas min.insync.replicas is a topic configuration
that can be changed afterwards.
Running a broker locally
Section titled “Running a broker locally”Apache Kafka 4.0 was the first release to run entirely without ZooKeeper — KRaft is the only
metadata mode — and it raised the floor to Java 17 for brokers, Connect and the command-line tools
(clients and Kafka Streams still run on Java 11). Compose files and runbooks that start a
zookeeper container predate this and will not work against a current broker.
To reach a broker running under Compose:
docker compose exec broker bashkafka-topics --bootstrap-server localhost:9092 --listTo open a ksqlDB session — the product was renamed from KSQL to ksqlDB, and the compose services
are ksqldb-server and ksqldb-cli:
docker compose exec ksqldb-cli ksql http://ksqldb-server:8088