Producer Mechanics - Under the Hood

A producer must do two jobs: use resources efficiently when Kafka is healthy, and give the application a bounded, explicit outcome when it is not. Batching handles the first. Admission control, deadlines and a retry contract handle the second.

Scope: Apache Kafka 4.0.x Java producer. Settings below are configuration fragments and tuning candidates, not a benchmark or a complete application. Other client libraries have different buffering and callback behavior.

From send to acknowledgment

Serialization and partition selection happen on the application path. The producer adds the serialized record to an accumulator for its partition. A background sender builds requests, sends them to partition leaders, receives responses and performs eligible retries.

Application queue → serialize / route → partition accumulator → sender
                                             │                  │
                                        batch / linger     network request
                                                                │
Callback or future completion ← acknowledgment ← leader / replicas

Application queue → serialize / route → partition accumulator → sender
                                             │                  │
                                        batch / linger     network request
                                                                │
Callback or future completion ← acknowledgment ← leader / replicas

send() is asynchronous after admission, but it can block while obtaining metadata or buffer space. It can also throw before returning a future. A recipe that only checks asynchronous callbacks misses synchronous admission/serialization failures.

The Java producer is thread-safe and can be shared. That does not make an application’s mutable serializer buffer safe for simultaneous use. Prefer a stateless serializer or correctly owned buffers before attempting allocation optimizations. Keep callbacks short: blocking the sender with database work delays unrelated sends. The KafkaProducer API defines these lifecycle contracts.

Batches form per producer and partition

A producer maintains separate batches for each partition it writes. One broker request can contain several partition batches. Compression operates over batches, so repeated field names and values can compress together.

batch.size is a batching target/limit for normal accumulation, not the maximum permitted record size. Larger individual records need compatible producer request and broker/topic batch-size limits. In Java 4.0, the default batch size is 16,384 bytes and default linger.ms is 5 ms. The linger default changed from zero in 4.0.

A full batch can be sent before linger expires. A sparse batch can wait for more records. Broker throttling, retries and sender scheduling can extend the actual time in the producer, so linger is not an end-to-end latency ceiling. See producer configurations.

For a simple steady-load approximation, let B be the batch target in bytes and r be serialized bytes/s entering one producer-partition accumulator:

batch fill time ≈ B / r
first record's intentional batching wait ≈ min(linger, B / r)

These estimates ignore record boundaries and scheduling. They explain why total topic throughput alone cannot predict batch size.

Checkpoint: why did adding producers make batches smaller?

You double the producer instances during a rollout. Traffic stays constant, but average batch size falls. Before changing batch.size, predict what happened to the traffic reaching one accumulator.

Given: a hypothetical 20 MB/s of serialized keyed traffic, evenly spread across 10 producers, each writing all 20 partitions; a 65,536-byte batch target and 5 ms linger. Use decimal MB/kB and ignore record granularity for this estimate.

1. Count the independent accumulators: 10 × 20 = 200. Producers do not pool their batches with one another.
2. Divide the traffic: 20,000,000 / 200 = 100,000 bytes/s per accumulator.
3. Compute fill time: 65,536 / 100,000 ≈ 0.655 s, or 655 ms. That is much longer than linger.
4. Compute arrivals during linger: 100,000 × 0.005 = 500 bytes. This is expected additional traffic during the interval, not an exact batch size; the record that opened the batch and record boundaries also matter.
5. Repeat after doubling producers: 400 accumulators receive 50,000 bytes/s each. Fill time becomes 1.31 s, and expected arrivals during linger fall to 250 bytes.

The governing quantity is bytes/s per producer-partition pair, not topic throughput. Increasing the batch target creates room; it does not create arrivals.

Change one assumption: if traffic also doubles to 40 MB/s, each of the 400 accumulators is back at 100,000 bytes/s. The original estimate returns. This reasoning assumes evenly distributed keyed traffic; sticky unkeyed routing need not keep every accumulator active.

Adding producers or partitions can therefore reduce batching efficiency at unchanged aggregate traffic. For keyed traffic, measure the distribution per producer and partition, including hot and sparse keys. For unkeyed traffic, default sticky/adaptive partitioning helps batch formation; selecting a round-robin partitioner changes that behavior.

Tune batching against a latency budget

Start with representative payload sizes, keys, producer counts, TLS and acknowledgment settings. Measure batch size, compression ratio, request rate, CPU, retries, queue time and acknowledgment latency. Then change one variable at a time.

Do not synchronously wait on every send() or flush every record in a throughput-oriented loop. Admit a bounded amount of work, inspect completions, and drain explicitly at shutdown. Compression choices require measurements; there is no universal ratio or fixed throughput multiplier.

Higher linger can sometimes improve loaded p99 latency by reducing CPU/request pressure. Under low load it usually adds waiting. Neither observation predicts source-to-destination latency without measuring the consumer and sink as well.

Three different timeout boundaries

Prefer an explicit delivery budget over a small arbitrary retry count. A request can be retried within that budget; adding application retries outside it can silently extend the business deadline and produce another business event.

A failure response does not always mean “Kafka has no copy.” If the broker stored a record but the reply was lost, the outcome is uncertain from the application’s perspective. Preserve event identity when deciding whether to retry or reconcile.

Reliability settings and their limits

For a durable stream, this Java producer fragment makes the intended constraints explicit:

acks=all
enable.idempotence=true
max.in.flight.requests.per.connection=5

It must be paired with an appropriate topic replication factor and minimum ISR. These three producer settings alone do not establish the failure domain. With idempotence enabled, Kafka preserves ordering with up to five in-flight requests per connection; reducing the value to one is not a universal reliability requirement.

Idempotence requires acks=all, retries greater than zero and at most five in-flight requests. Compatible values are configurable. Explicit idempotence plus conflicting settings causes configuration failure; when idempotence is left implicit, conflicting settings can disable it.

Idempotence deduplicates the producer protocol’s retries. It does not deduplicate a fresh application send() of the same business event or arbitrary operations across restarts. Assign a stable event ID when repeated delivery could duplicate an external effect. Use Kafka transactions when Kafka outputs and input offsets must commit atomically.

Follow the retry identity

Return to the order service’s outbox relay. E17 has a business identity; the producer protocol also carries a producer ID, producer epoch and sequence numbers per partition. They solve different problems.

Assume a healthy idempotent session, one-record batches and valid retained broker producer state. Producer P, epoch 0, sends sequence 10. The broker appends it at offset 42, but the reply is lost. A supported protocol retry of that same batch uses the same identity and sequence; the broker recognizes the duplicate and returns the previous batch’s metadata instead of appending another copy.

Now suppose application code makes a fresh send(E17). Its next sequence is 11. That is new protocol work even though the payload’s business identity matches, so it can occupy offset 43. If the relay crashed before recording publication, an outbox retry can create the same kind of business duplicate.

A newer producer epoch can invalidate older producer work under the relevant protocol; it is not the partition’s leader epoch. Do not implement your own counter-reset recovery from this teaching example. Read producer-state validation alongside duplicate-batch recognition, then follow the client’s error contract.

Bounded admission is the backpressure mechanism

The producer’s buffer is only one queue. An unbounded queue before it can exhaust application memory while the producer itself obeys its configured limit. buffer.memory is also not a hard limit on total producer memory: compression and in-flight requests need additional space.

A useful admission contract declares:

1. Maximum queued bytes and outstanding operations at the application boundary.
2. How long callers may wait for admission and completion.
3. Whether overload causes rejection, durable spooling, explicit dropping or upstream slowdown.
4. How an uncertain delivery is reconciled without losing business identity.

The following is control-flow pseudocode, not a Java implementation:

reserve bounded application capacity before accepting an event
try to send using a stable event ID
    if send throws synchronously:
        release capacity; report the failure
    otherwise, on completion:
        release capacity
        record acknowledged, failed, or uncertain outcome
        notify the caller without blocking the producer's sender
on shutdown:
    stop admission; drain within the shutdown budget
    retain/reconcile any unresolved accepted work under the declared policy

A memory queue is not durable acceptance. If the service tells a caller “accepted” before Kafka acknowledges, it must define what a process crash does to that accepted event. A durable outbox or spool changes that contract; a larger heap does not.

Recipes for different publication paths

A database transaction and producer.commitTransaction() are separate commits. Annotating a method as transactional does not combine them. The external-commit companion gives the failure trace; the downstream consumer recipe defines how repeated delivery becomes a safe effect.

Before adopting a configuration, demonstrate buffer exhaustion, lost acknowledgments and shutdown with pending work. A successful throughput run exercises none of those outcomes by itself.

Whiteboard: scale the relay without losing its contract

Guided variant. The order relay now emits a hypothetical 12 MB/s of serialized keyed records through six producers, each writing ten partitions evenly. Batch target is 32,768 bytes and linger is 10 ms. Fill in: accumulator count 6 × 10 = __; bytes/s per accumulator 12,000,000 / __ = __; fill time 32,768 / __; additional arrivals during linger __ × 0.010. Use decimal units and the checkpoint’s smooth-arrival approximation.

Independent problem. Keep total traffic, batch target and linger unchanged, but use 24 producers writing 20 partitions each. Recalculate the quantities. Then draw the outcomes of a lost broker reply followed by (a) a supported retry of the original batch and (b) a fresh application send of E17. Choose an admission policy if the caller’s deadline expires while publication is uncertain; state who owns later reconciliation.

Pass criterion: derive accumulator traffic and distinguish a caller timeout, a protocol retry and a repeated business command. Next, trace when a consumer may declare E17 finished.