Transactions and Exactly-Once Semantics

A Kafka transaction can commit output records and consumed offsets together. Processing may execute again after a failure, but aborted output is hidden from read-committed consumers. A database mutation or HTTP request needs a separate atomicity mechanism.

Scope: Apache Kafka 4.0.x Java clients. The processing loop is pseudocode that states recovery obligations; it is not compiled or fault-tested application code. Exactly-once here describes committed effects within a defined boundary, not one execution of arbitrary business code.

Four mechanisms with different jobs

An input may be processed once, crash before transaction commit, and be processed again. With the correct Kafka read-process-write transaction, only one committed output effect is retained for the consumed progress. Calling a sink twice remains possible if it is outside that boundary.

The Kafka transaction design explains why including input offsets with output records is essential. Disabling auto-commit and calling commitSync() after ordinary processing provides an at-least-once pattern, not exactly-once effects by itself.

The atomic boundary in a processing loop

Input partition → consumer → transform → output partitions
                   │                       │
                   └── next input offsets ─┤
                                           ▼
                                one Kafka transaction

External DB / HTTP effects are outside this boundary.

Input partition → consumer → transform → output partitions
                   │                       │
                   └── next input offsets ─┤
                                           ▼
                                one Kafka transaction

External DB / HTTP effects are outside this boundary.

For a manual transaction loop, configure the consumer with enable.auto.commit=false and, when reading transactional inputs, isolation.level=read_committed. Configure the producer with a transactional.id whose ownership prevents simultaneously active collisions. Initialize transactions before processing.

There is only one open transaction per producer. A thread-safe producer does not permit unrelated workers to run overlapping independent transactions on the same instance. A replacement using the same transactional identity fences an obsolete owner; treat fencing as loss of authority, not a transient retry opportunity.

Processing pseudocode:

initialize transactional producer and consumer
poll an input batch within the owned assignments
begin a transaction
transform records without irreversible external side effects
send output records
send the next consumed offsets with current consumer group metadata
commit the transaction
only then advance application state that depends on that commit

on a known abortable failure:
    abort the transaction
    restore application state and reset consumer position to committed progress
    retry only while assignment/transaction ownership remains valid
on fencing or another fatal producer failure:
    stop this producer/owner; do not continue using it
on an ambiguous commit outcome:
    follow the API's retry/resolution contract for that same operation
    do not begin an unrelated transaction or assume the previous one aborted

The pseudocode intentionally leaves language-specific exception classification to the KafkaProducer API contract. For example, a commit timeout is not proof of abortion; the API permits retrying the commit operation. Production code needs a state machine that distinguishes this from a failure requiring abort or close.

Abort does not rewind the consumer position or undo application memory. One conservative recovery approach is to close/recreate the clients and restore state from durable committed progress; an optimized loop can explicitly restore offsets and state while preserving valid ownership. A transaction flag alone does neither.

Checkpoint: two executions, one committed result

Given: the group’s committed input position is 42. Record 42 is one order event; processing produces one Kafka result. Output consumers use read_committed. Each transaction includes both that result and input progress, ownership is valid, and recovery uses committed progress without a manual rewind.

Predict what survives a crash after output append but before transaction commit:

1. First attempt: transaction A appends a result and includes next input offset 43. Neither is committed yet.
2. Crash and resolve: suppose recovery resolves A as aborted. Its result remains hidden from read-committed readers, and the group’s committed position remains 42.
3. Second attempt: the replacement reads 42 again, computes the result again, and commits transaction B with next offset 43.
4. Count separately: application executions = 2; committed result records for these attempts = 1. The aborted attempt did not advance the group past its input.

The transaction preserves a pairing: abort leaves neither committed output nor advanced progress; commit records both. Independent commits could leave either half without the other. This is an atomic outcome, not simultaneous delivery across partitions.

Move the crash: if A had durably committed before the process died, recovery from committed position 43 would skip this input. A lost reply alone cannot tell the caller which case occurred; resolve the transaction outcome. If either attempt also charged an external card, that charge would fall outside this proof.

Committed does not mean visible everywhere at once

Within each partition, a read-committed consumer cannot pass the earliest unresolved transaction. Its visible end is constrained by the last stable offset. Later records, including nontransactional ones, may wait behind an earlier open transaction.

Aborted records and transaction control markers occupy log positions but are not returned as ordinary application records by KafkaConsumer. There is no special __transaction_marker user header that reveals them in a debug mode.

Transaction commit is atomic in its outcome, but applications do not receive a cross-partition snapshot in one poll(). Different partitions can be fetched at different times, and consumers may be assigned only part of the transaction’s output. Avoid interpreting “atomic writes” as “every reader sees every output simultaneously.”

Long transactions therefore have an operational cost beyond commit overhead: they can hold back visible progress. Measure open transaction age and last-stable-offset behavior alongside producer acknowledgment latency and consumer freshness. See consumer isolation settings.

Failure outcomes follow the durable decision

Coordinator failover does not indiscriminately abort every incomplete transaction or automatically restart all producer epochs. The 4.0.2 state-manager source resumes pending commit and abort decisions from loaded state.

Database effects: choose the real transaction

For a Kafka-to-database sink, a useful design is:

1. Start a database transaction.
2. Insert a stable event identity into a deduplication table with a uniqueness constraint.
3. If it is a new identity, apply the business mutation in that same transaction.
4. Commit the database transaction, then commit the corresponding completed Kafka prefix.

A duplicate-ID outcome is safe only when it proves the original mutation committed. Ensure transaction rollback, uniqueness/conflict handling and concurrent delivery preserve that fact. The identity retention horizon must cover replay; clearing the dedupe table while old events remain replayable can reintroduce effects.

For a database-to-Kafka publisher, atomically write business data and an outbox row in the database. The relay publishes before marking its row delivered. A crash between those steps can publish the event again; stable IDs and downstream dedupe remain necessary. Concurrent relays also need row claiming/ownership.

These are different directions and different atomicity boundaries. Neither is made globally atomic by placing a database annotation around a Kafka transaction. The database-boundary companion develops the crash-and-replay argument.

An external API may support an idempotency key with a limited lifetime; it may not. If a request succeeds but its response is lost, the application needs reconciliation or an explicit duplicate-risk policy. Kafka cannot infer whether the remote action occurred.

Kafka Streams and transaction cost

Kafka Streams can coordinate input progress, Kafka outputs and managed state changes using processing.guarantee=exactly_once_v2. This simplifies the framework-owned boundary. Arbitrary database writes, emails or HTTP calls from processing code remain external effects.

Larger transactions can amortize coordination overhead, but increase visibility delay, buffered work and the amount that may need reprocessing. Smaller transactions reduce that exposure while increasing commit frequency. There is no fixed “40% slower” rule independent of payload, partitions, brokers, TLS, transaction size and load.

Choose transactions when atomically advancing Kafka-derived output and input progress is the requirement. Choose destination cooperation when the effect lives elsewhere; sometimes both mechanisms are needed. Test lost commit replies, fencing, abort/reset and long-open-transaction visibility before claiming the chosen boundary survives failures.

Continue with event sourcing for application ownership, or stream processing for state and time semantics.

Whiteboard: separate three kinds of progress

Guided variant. An output partition has high watermark 56. Its earliest unresolved transaction begins at offset 50; assume no earlier blocker. A nontransactional record sits at 54. Fill the stable visibility boundary __ and decide whether a read_committed consumer can pass 50 to return 54. If the transaction aborts, does its output become ordinary readable data?

Independent problem. The fulfillment processor consumes order event at input offset 42, writes a database reservation, then starts a Kafka transaction containing its status output and next input offset 43. Draw a table for three cases: transaction aborts; transaction commits but its reply is lost; process crashes after the DB reservation and before starting the transaction. For each, record the DB effect, committed Kafka output, committed input position and safe recovery action. Assume the group previously committed 42 and no manual offset reset occurs.

Pass criterion: keep DB state, committed Kafka output and input progress in separate columns. The next design decision is which system should have authority to accept the original order.