Kafka Architecture - Core Concepts

Kafka stores a partitioned, replicated log. Producers append records; consumers maintain their own positions and can read the same retained history independently. This separation makes Kafka useful when several applications need both a live feed and replay.

The difficult decisions concern boundaries: which records must stay ordered, what an acknowledgment proves, how long recovery may take, and what happens when a destination cannot keep up. This series builds those decisions into one pipeline model.

Version scope: Exact defaults and client behavior refer to Apache Kafka 4.0.x and its Java client. This is a versioned baseline, not a claim that 4.0 is the latest release. Configuration fragments and pseudocode illustrate contracts; they are not a tested application.

Learn it on a whiteboard

Keep one order service on the board throughout this series. Initially, a database accepts orders and commits an outbox row in the same transaction; a worker sends notifications at 200 events/s. Later, search, fulfillment and analytics need independent progress and replay. The final design exercise grows the workload to 100,000 events/s and adds an AZ recovery requirement. These are explicit stages of a hypothetical system.

The database owns command acceptance. Kafka may carry accepted changes; a consumer owns its downstream effect. Draw those three boundaries before adding boxes for products.

Read a worked example, finish the guided variant, then solve the independent problem before opening its answer. Each chapter supplies a pass criterion. An answer may depend on an unknown measurement: identify it and derive the threshold it must meet. The reading-time estimate covers the text; allow additional time to draw and solve.

The log and its readers

A topic names a stream. Each topic has partitions; each partition is an ordered sequence of records identified by offsets. Brokers host partition replicas. A producer discovers the partition leader and sends directly to it. The controller manages metadata and leadership rather than routing each record.

Producer ──append──▶ Partition leader ──replicate──▶ Followers
                        │
                        ├──fetch──▶ Group A: database projection
                        └──fetch──▶ Group B: analytics

Group A and Group B keep separate progress.
Retention applies even when a group has not finished reading.

Producer ──append──▶ Partition leader ──replicate──▶ Followers
                        │
                        ├──fetch──▶ Group A: database projection
                        └──fetch──▶ Group B: analytics

Group A and Group B keep separate progress.
Retention applies even when a group has not finished reading.

Within a conventional consumer group, each assigned partition has one active consumer owner at a time. Different groups can each consume the full topic. More groups therefore add read traffic; they do not divide a single delivery among themselves. An application can dispatch work to additional threads, but must then preserve the ordering and progress guarantees it needs.

Consumption does not delete records. Time, size and compaction policies govern what remains available. A consumer can rewind only as far as retained data permits. Kafka is a buffer with a finite recovery horizon, not an unlimited promise to remember every event.

Follow order-17 through the broker

Suppose the database has committed order-17 and its outbox event E17. Trace publication before drawing a consumer:

1. Admit and batch. The relay serializes E17, chooses a partition and places it in that producer’s accumulator. Buffer admission can wait or fail before a network request exists.
2. Receive the request. A broker network processor receives the produce request; request handling validates it and sends the append to the partition leader’s log. Queueing and handler work are separate from waiting for replication.
3. Append locally. The leader assigns offsets and appends record batches to its active segment, normally through the page cache. A local append alone has not satisfied all acknowledgments.
4. Replicate. Followers fetch from the leader. Their subsequent fetch positions let the leader learn how far they have copied; the acknowledgment condition can remain pending while this progress catches up.
5. Respond. Once the required replication condition is satisfied, the broker can complete the produce request. The reply still has to reach the producer; losing it creates uncertainty for the caller.
6. Fetch and apply. A consumer fetches visible records, decodes them and updates its destination. Only a successful destination operation proves that effect; an earlier producer acknowledgment cannot prove it.

On the board, mark three possible waits: producer admission, broker request handling and replication. Add a fourth at the sink. The same end-to-end delay can arise at different places; raising one timeout does not identify the bottleneck.

For implementation reading, start with Kafka 4.0.2’s produce request handler, then leader append. Ask which state each function changes before following another call. The controller is not a per-record relay on this path.

Decide what must stay ordered

Kafka provides partition log order. It does not provide a total order across partitions or guarantee that concurrent workers finish in that order. Record timestamps can also move backward: event time and append order describe different things.

Use a stable entity key when operations for that entity require one order. Under the Java client’s default keyed routing, the serialized key determines a partition. Increasing the topic’s partition count can change that mapping: new records for a key can land elsewhere while old records remain in the original partition. Plan the migration if readers depend on a single sequence.

Without a key or explicit partition, the Java producer uses sticky/adaptive routing to improve batching and distribution; it is not simple record-by-record round-robin. A hot key remains a bottleneck if the application requires all its work to execute serially. Adding consumers cannot split that dependency.

These routing details follow the producer configuration contract. A partition count is a capacity and future-migration decision, not just a parallelism knob.

What a successful write proves

Each partition has a leader and followers. The in-sync replica set, or ISR, tracks replicas sufficiently caught up under Kafka’s replication protocol. It is dynamic: a slow or unavailable follower can leave it.

acks=all waits for the ISR, not every configured replica and not a fixed majority vote. min.insync.replicas sets the minimum ISR condition for successful all-acknowledged writes. A common starting policy is replication factor 3, minimum ISR 2, all acknowledgments and clean leader election, with replicas placed in distinct failure domains.

Suppose all three replicas are in sync. Losing one can leave two replicas able to accept writes after failover. Losing another can stop successful all-acknowledged writes. This trades write availability for a stronger replication requirement; it is not an unconditional zero-data-loss guarantee. Replica placement, correlated storage loss and the controller quorum still matter.

An append commonly enters the operating system’s page cache. A successful acknowledgment does not mean every replica has forced that record to physical media with fsync. Kafka uses replication as a central durability mechanism. See the replication and persistence design.

Worked trace: two copies are not automatically enough

Assumptions: A leads replicas A/B/C; all remain in a stable ISR; RF=3, minISR=2, acks=all; no transactions, ISR changes or additional joining replicas. ELR is disabled for this exercise. Offsets are exclusive boundaries: log end 11 means the replica has appended through offset 10.

Start with all replicas at log end 10 and high watermark 10. Append one record at offset 10. Each row below is after the leader has learned the stated follower progress and reevaluated its high watermark:

MinISR=2 is a minimum membership condition, not permission to ignore a third member of the ISR. The high watermark also depends on progress known to the leader, not instantaneous knowledge of every disk. If ISR membership changes, recalculate under that new state instead of applying this fixed table mechanically.

The source separates checking acknowledgment eligibility from advancing the high watermark. KRaft’s metadata quorum is another protocol; do not substitute its majority rule for this data-replication trace.

Four positions that must not be confused

1. Log end: where the replica has appended data.
2. Replication high watermark: the boundary of replicated data available under the replication protocol.
3. Consumer position: where the next fetch/poll progress has reached; fetched records may still be unfinished.
4. Committed consumer offset: the restart position the application has recorded for its group.

A transactional reader has another boundary: the last stable offset limits visibility behind open transactions. A record can be replicated and acknowledged while remaining invisible to a read_committed consumer until the transaction resolves. Nontransactional records behind an earlier open transaction can also wait.

Likewise, a consumer may fetch offsets 10–19 but finish only 10. Committing its fetched position can skip unfinished effects after a crash. The consumer chapter explains how to commit completed work; the transaction chapter explains output visibility. Producer acknowledgment, consumer visibility and destination durability are separate events.

KRaft and failure domains

Kafka 4.0 removed ZooKeeper mode. KRaft controllers maintain cluster metadata using a quorum; brokers store and serve partition data. Production role placement must preserve controller quorum as well as enough data replicas. Losing the active controller is not simply an election among arbitrary brokers.

Consider a network split that isolates one of three availability zones. If replica and controller placement preserve the required quorums in the surviving zones, affected partitions can elect eligible leaders and resume service there. Clients stranded in the isolated zone may still be unable to write. A lower acks setting cannot repair missing network reachability or restore a lost controller quorum.

Leader eligibility is version- and feature-dependent. Kafka 4.0 introduces Eligible Leader Replicas, which can preserve safe election choices outside the current ISR. Avoid the blanket rule that every clean election must always choose a member of the current ISR; check the enabled feature and upgrade guidance.

Replication within a region and mirroring between regional clusters solve different problems. A mirrored cluster has its own progress and failure boundary. A successful local write does not imply the remote cluster already has it. Apache’s datacenter guidance discourages stretching one cluster over high-latency links.

Choose Kafka for the required contract

Kafka fits retained event streams, independent subscribers, replayable projections and partitioned processing. Its costs include partition planning, schema evolution, lag/retention management and application-level handling of repeated effects.

A task queue may fit better when individual job acknowledgment, arbitrary retries and routing are the central requirements. A database transaction plus an outbox may fit a service that chiefly needs reliable publication of database changes. There is no message-rate threshold that decides this choice by itself.

Use the remaining chapters as decision tools:

• Producers: batching, deadlines, retries and bounded admission.
• Consumers: ownership, backpressure and durable progress.
• Retention: replay horizon and compacted state.
• Transactions: atomic Kafka effects and external boundaries.
• Event sourcing: command concurrency and projection rebuilds.
• Stream processing: time, joins and state recovery.
• Sizing and deployment: workload, resource budgets and failure capacity.

Before selecting broker sizes, write down the ordering key, accepted loss/duplicate policy, freshness deadline, replay horizon and failure domain. Those requirements determine which configurations and capacity calculations are relevant.

Whiteboard: preserve the acknowledged prefix

Guided variant. Keep the worked trace’s assumptions. All replicas now have log end 11. A appends offset 11; B copies it, but C remains at 11. Complete: A/B/C log ends = 12 / 12 / __; high watermark = __; can the producer receive a successful all-acks reply? Mark the last offset an ordinary consumer can read.

Independent problem. The same three-replica policy has high watermark 21 and log ends A=22, B=22, C=21. Offset 21 was not acknowledged. A fails; the controller elects eligible ISR member C as leader, and B rejoins it. Draw the surviving prefix and the divergent tail. Can B insist that C keep offset 21? What can the publisher infer from its missing reply, and what additional event in the request path would change that inference?

Pass criterion: draw the request path and both offset tables without conflating local append, replicated visibility, producer receipt and destination completion. Carry the uncertain reply into the producer chapter.