Kafka Pipeline Sizing and Deployment Recipes

A Kafka pipeline is limited by its slowest required stage: source admission, broker replication, processing or destination effects. Sizing only retained bytes can produce a cluster that holds the data but cannot meet the freshness or recovery deadline.

This chapter uses one hypothetical workload to connect storage, throughput, latency and deployment. Every hardware and per-partition capacity below is an illustrative assumption, not a measured Kafka benchmark. Replace those assumptions with tests of your records, topology, client versions and failure conditions.

Start with the workload contract

Collect sustained and peak records/s, serialized record-size distribution, key skew, compression, subscriber groups, output expansion, retention and recovery deadlines. Include the largest accepted record/batch, not just an average. Record the latency boundary: broker acknowledgment, visible output, or durable destination effect.

Also define the failure condition under which the contract must hold. Surviving one broker is different from surviving an AZ, restoring full replication while that AZ is unavailable, or failing over to a remote region.

Use decimal MB and TB throughout this example. Mixing MB with MiB or bytes with bits can change the answer before any tuning begins.

Storage and network are different budgets

logical ingress U = λ × s                         [bytes/s]
stored-log rate C = U × c                         [bytes/s]
retained replicated payload D = C × T × R         [bytes]
provisioned disk ≥ (D + extra stored bytes) / u    [bytes]
aggregate log writes W ≈ C × R                    [bytes/s]
inter-broker replication N ≈ C × (R − 1)           [bytes/s]
full-rate consumer egress F ≈ C × g               [bytes/s]
total broker egress ≈ N + F                       [bytes/s]

These equations assume balanced traffic, comparable stored/wire sizes, no replay and no output topics. Add request/TLS overhead, retries, repair, reassignment, remote fetches and each derived topic separately. If the broker recompresses data or changes its format, measure distinct ingress/storage/egress ratios.

The disk floor excludes indexes, segment slack, internal topics, compaction/repair workspace, skew and growth. Retention storage follows time-weighted sustained volume; peak rate times the full retention window is a different, deliberately conservative assumption.

Replication uses both a sender’s egress and a receiver’s ingress. Count each on the relevant NIC direction, rather than pretending they are two independent logical data streams. In this example aggregate broker ingress is approximately 40 MB/s from producers plus 80 MB/s from replication. Egress includes replication plus independent consumers. Network-attached disks also have a storage-network limit separate from client/replication traffic.

AWS’s sizing analysis develops these resource constraints and shows why TLS, burst credits and cold replay change achievable throughput. Its hardware results apply to the stated MSK setup; they are not universal Kafka capacity figures.

Recovery capacity must exceed arrivals

With backlog Q and effective processing capacity μ:

backlog growth = λ − μ                              [records/s]
drain time = Q / (μ − λ), provided μ > λ             [seconds]
capacity for drain deadline D = λ + Q / D            [records/s]

Checkpoint: why isn’t 50% spare capacity enough?

The consumer returns after a 30-minute outage. It can now process 150,000 records/s against arrivals of 100,000/s. That sounds comfortable—but can it catch up in 20 minutes?

Given: constant arrivals, no expired input, balanced work and processing rates measured through the durable destination effect.

1. Count the missed work: 100,000 × (30 × 60) = 180 million records.
2. Reserve capacity for new arrivals: 150,000 − 100,000 = 50,000 records/s is available to reduce backlog. Dividing by total processing rate would incorrectly pretend arrivals stopped.
3. Derive the drain time: 180,000,000 / 50,000 = 3,600 seconds, or 60 minutes. The 20-minute objective fails.
4. Solve backward from the objective: in 1,200 seconds, 120 million new records arrive. Completing those plus 180 million old records requires 300,000,000 / 1,200 = 250,000 records/s.

This is conservation of work: capacity during recovery must cover new arrivals plus the backlog. Algebraically, μD ≥ λD + Q, so μ ≥ λ + Q/D. Spare capacity determines recovery speed.

Change one bottleneck: Kafka can supply 250,000/s, but the database completes only 120,000/s. The effective surplus is 20,000/s; draining takes 180,000,000 / 20,000 = 9,000 seconds, or 150 minutes. Faster fetching alone cannot meet the deadline.

If all three groups must recover simultaneously at 250,000/s each, and wire size remains 400 bytes/record, consumer egress becomes 300 MB/s, not 120 MB/s. Add normal replication and broker egress becomes 380 MB/s, before repair and overhead. Historical reads may also miss page cache and create disk I/O absent from a caught-up benchmark.

Keep peak and recovery scenarios explicit. The 250,000/s requirement assumes arrivals return to 100,000/s. If a 200,000/s peak continues during the same drain interval, the requirement is 350,000/s. Do not quietly combine a peak-ingress broker calculation with a normal-arrival recovery promise.

For variable traffic, model backlog over time rather than assuming constant rates. Retention must still cover the oldest unfinished data throughout the outage and drain, and a byte cap may shorten the available time horizon.

Derive a partition lower bound

Suppose a representative test established these capacities at the required SLO:

• One partition supports 5 MB/s of compressed producer traffic.
• One partition’s ordered processing lane supports 10,000 records/s at the destination.

Those are hypothetical measurements for this calculation. With compressed peak ingress 80 MB/s and normal-arrival recovery requiring 250,000 records/s:

producer partition bound = ceil(80 / 5) = 16
consumer partition bound = ceil(250,000 / 10,000) = 25
initial lower bound = max(16, 25) = 25 partitions

If the recovery deadline must hold during peak arrivals, the consumer bound becomes 35. Also account for the desired number of simultaneously active group members.

This is a balanced-load lower bound, not a guarantee that 25 partitions deliver the result. Test the hottest partition and key. A key requiring serial 15,000 records/s cannot be fixed by spreading unrelated keys across more partitions when its own lane handles 10,000/s. Revisit the key/ordering requirement or processing cost.

More partitions also fragment producer batches, increase metadata/files and potentially increase state-recovery work. Increasing the count changes default key routing. Plan expansion before the ordering contract depends on the old mapping.

Derive a broker floor under AZ loss

Suppose each broker has 8 TB of usable raw disk and the target occupancy is 70%, leaving 5.6 TB per broker for the modeled payload plus later allowances.

Normal payload capacity requires ceil(31.104 / 5.6) = 6 brokers. With six evenly distributed across three AZs, losing one AZ leaves four brokers and 22.4 TB at target occupancy. That is insufficient to restore the full 31.104 TB replicated payload on the survivors.

If the requirement is to restore full RF=3 while one AZ remains unavailable, nine evenly distributed brokers leave six survivors and 33.6 TB at target occupancy. Thus nine is an AZ-aligned payload-only disk lower bound for that particular requirement. Extra data, growth, hot replicas and restore traffic can require more. During the outage, three replicas across two remaining AZs also do not retain the original three-AZ separation.

Keeping the service running temporarily with fewer replicas is a different requirement from restoring full RF while the AZ stays down. State which one the design promises. At RF3/minISR2, losing another relevant replica before repair can stop all-acknowledged writes.

Finally check surviving brokers’ sustained disk-write/read rates, directional NIC capacity, storage-network bandwidth, CPU and controller quorum. For a balanced throughput resource, an initial bound has the form:

surviving broker count ≥ ceil(required resource rate / measured usable rate per broker)

Use the largest resource bound, then test actual leader/replica placement and repair. Do not derive broker count solely from a fixed partitions-per-broker constant or from advertised burst bandwidth.

Latency is a path, not one producer setting

For each record, source-to-durable-effect latency includes source admission, producer buffering/batching, network and broker replication, consumer fetch/queueing, processing and destination commit. Transaction visibility or window finality can add another delay.

Measure that full distribution. Summing independently measured p99 values does not produce an end-to-end p99. Keep broker acknowledgment latency and destination freshness as separate metrics; one can remain healthy while the other degrades.

Little’s law, L = λW, relates average work-in-system, average arrival rate and average time in a stable system with one consistent boundary. It does not predict p99 or provide a steady-state estimate for an indefinitely growing backlog.

Allocate latency to stages as a design budget, then test the actual path at low load, sustainable load, bursts and recovery. Increasing linger can improve loaded efficiency while increasing sparse-traffic wait; more worker concurrency can lower queue time until it overloads the destination.

Six workload recipes

The values below are starting experiments, not production recommendations. Shared durable-publication baseline: explicit idempotence, acks=all, tested replication/minISR and bounded admission. Relax loss guarantees only under an explicit product contract. Java 4.0 settings do not transfer verbatim to another client library.

Fetch minimums are waiting thresholds, not hard memory caps. Batch sizes are producer-partition accumulation settings, not consumer processing batch sizes. Producers and consumers explain these distinctions.

Adapt the recipe to the deployment

MirrorMaker replicates between independent clusters. It is not a cross-region Kafka transaction. DNS switching alone does not establish correct consumer progress or prevent conflicting writers.

Turn estimates into an acceptance test

Worked calibration: which rate may enter the sizing model?

Suppose a test of one candidate end-to-end deployment produces the following synthetic observations for this exercise, with a caught-up p99 freshness objective of 500 ms. Offered rate counts attempted input; completed rate counts durable destination effects. Each row is a separate sustained test window after warm-up, not a Kafka benchmark claim.

The justified result is 250 k/s under the tested warm conditions, not a universal maximum. Even 270 k/s completed in the overloaded row is not evidence of a stable 270 k/s system at the required latency. Locate the transition with more load points, then retest realistic skew, output expansion, cold reads and surviving capacity after failure.

For the order service, timestamp from the declared acceptance boundary through the durable destination operation, including outbox and application queues. Record offered, rejected, published and completed counts separately; otherwise admission rejection can make latency look good while the service drops its workload. A resource estimate becomes defensible when its measured rate and the required operating condition match.

Run the representative workload long enough to expose sustained limits and state growth. Include record-size tails, realistic keys, compression, TLS, all groups, output topics and the actual destination. Test cold replay as well as readers at the log tail.

Record producer queue time/errors/throttling; broker disk/NIC/CPU and replication health; committed consumer progress, oldest unfinished-event age, queue bytes and sink latency; state restore and transaction visibility where relevant. Preserve configuration, versions, duration and warm/cold conditions with the result.

Pass only when the required loss/duplicate contract, p95/p99 freshness, memory bounds, replay horizon and drain deadline hold in the declared failure scenario. Until then, the calculations are useful lower bounds and hypotheses—not evidence that the pipeline has that capacity.

Whiteboard capstone: defend the order pipeline

Guided variant. A group has 120 million records of backlog. New arrivals remain 100,000/s; the deadline is ten minutes. Complete required rate = arrivals + backlog / deadline = __. If three groups recover simultaneously at that rate with 400 wire bytes/record, add 80 MB/s of normal replication to derive broker egress. Keep repair traffic and overhead as separate terms.

Independent architecture problem. The order service now reaches the workload in this chapter’s first table: 100 k/s sustained, 200 k/s peak, 1,000 serialized bytes/record, compressed ratio 0.4, three full-rate groups and 72-hour history. The DB remains the command authority. Require per-order effect ordering, replay-safe external projections and caught-up p99 freshness of 500 ms. After a 30-minute consumer outage, recover within 20 minutes while normal arrivals continue; do not promise 500 ms freshness during that recovery interval.

Place the regional deployment across three AZs. It must tolerate one AZ loss without losing acknowledged Kafka records under the assumed single-AZ failure model, and have room to restore full RF=3 while that AZ remains unavailable. Use the chapter’s illustrative 5 MB/s per-partition ingress, 10 k/s serial processing lane and 8 TB/broker assumptions solely to derive lower bounds. There is no supplied proof that cold recovery or surviving brokers meet their required rates.

Produce these six artifacts before opening the answer:

1. Requirements and unknowns: state accepted loss/duplicate behavior, deadlines, clock boundary and measurements still needed.
2. Invariants: locate command acceptance, event identity, per-order sequencing and safe consumer progress.
3. Data path: draw the outbox relay, topics/keys, independent groups, state/sinks, replicas and controller placement; label queues and owners.
4. Budget: derive storage, steady/recovery network rates, processing capacity and partition/broker floors with units and exclusions.
5. Decision: choose a design, reject one credible alternative against the same requirements, and assign operational ownership and cost drivers.
6. Falsification: specify an experiment and observation that would make you revise the design. Include a migration/rollback path for a key or projection change.

Then change one assumption at a time: the destination can complete only 150 k/s; one business invariant forces 15 k/s through one serial key; peak arrivals persist throughout recovery; the producer’s successful append loses its reply. Revise the affected portion of the board, preserving the other requirements or explicitly renegotiating them.

Pass criterion: explain which invariant or resource budget changes for every new assumption. Then propose the smallest experiment that could disprove your preferred architecture. Solving this on paper establishes a testable design; running those experiments is the next step before production commitment.