Retention and Log Compaction

Kafka retention defines how much history a recovering application can still read. Compaction defines which state changes may disappear while preserving later values for the same key. Neither policy waits for every consumer to finish.

Scope: Apache Kafka 4.0.x. Settings are topic-level unless explicitly labeled as broker defaults. Storage figures are capacity estimates, not guarantees of an exact deletion time.

Retention is a replay contract

A partition log consists of segments. Kafka removes eligible segments rather than deleting each expired record individually. Rolling, retention checks and delayed file deletion make actual retention coarser than the configured duration.

For modern records, time-based eligibility uses the segment’s largest record timestamp. File modification time is a fallback for older data without the relevant timestamp, not the general modern rule. The Kafka 4.0.2 implementation compares current time with segment.largestTimestamp.

Timestamp policy therefore matters. With CreateTime, producer-supplied timestamps affect retention; with LogAppendTime, the broker assigns append timestamps. Validate timestamp bounds and clock behavior when event timestamps can be old or in the future. Do not promise exactly seven segments from seven-day retention and daily segment rolling.

# Topic configuration fragment: bounded event history, no compaction.
cleanup.policy=delete
retention.ms=259200000
segment.ms=3600000
retention.bytes=-1

This illustrative topic keeps roughly a three-day time horizon, subject to timestamp semantics and segment cleanup. It sets no size-based retention limit; provisioned disk must still cover the workload. An hour segment roll helps limit cleanup granularity but does not establish an exact hourly deletion schedule.

Broker defaults use names such as log.retention.ms; topic overrides use retention.ms. Verify the scope before copying a property. The topic configuration reference lists the corresponding broker defaults.

Follow an offset into a segment

An offset is a logical position, not a byte address. Suppose a local partition has segments with base offsets 0, 100 and 200. To read offset 137, first locate the segment whose range contains it. A sparse offset index might identify a file position near offset 130; Kafka then scans record batches forward to find the requested position. It does not require one index entry per record.

Timestamp lookup likewise uses index assistance and a search; timestamps need not rise with every appended record. Compaction can leave holes in logical offsets without making them physical addresses. Kafka 4.0.2’s segment offset translation shows the offset-index lookup followed by a log search.

Now rewind the order service’s analytics consumer by two days. Recent reads may have been served largely from cached pages; older local segments may require storage reads. The replay competes for disk, memory cache and network with live traffic. Remote-tier reads add another path. Efficient file transfer does not make a cache miss, TLS processing or remote fetch free.

If live p99 rises during replay, draw competing resource paths before changing producer linger. Compare replay read volume, storage latency/queueing, broker request time, network usage and live replication progress against their prior values. Flat producer ingress does not imply flat broker work. A rate-limited replay that restores live latency is useful evidence, but does not alone identify which resource was saturated.

Byte retention is per partition

retention.bytes limits a partition’s log size under delete retention. It is not a replicated cluster disk budget. A topic with ten equally sized partitions and a 100 GB per-partition limit has an approximate 1 TB logical retention budget before segment granularity; replication adds physical copies.

If both time and byte retention are enabled, either limit can remove old segments. A burst can therefore shorten the available history below the nominal time horizon. A seven-day setting does not establish seven-day recoverability when the byte cap is reached first.

Provisioned storage and retention configuration answer different questions:

retained replicated payload ≈ stored bytes/s × retention seconds × replicas
provisioned disk ≥ (payload + other stored bytes) / target occupancy

Add indexes, segment slack, internal topics, cleaning workspace, skew and growth. Use measured stored bytes rather than assuming JSON size equals compressed log size. See the pipeline model for a worked example.

Choose the replay horizon from the longest credible detection delay, outage, repair and replay path. A slow consumer can lose data to retention even while its process is healthy. Monitor the oldest unfinished event and the earliest available offset, not only aggregate lag.

Compaction preserves later state, not every transition

Consider one partition containing:

After cleaning, the older balance for account-7 can disappear. The null value is a tombstone: it represents deletion for account-9. The tombstone can eventually disappear too, according to delete-retention behavior.

Compaction does not renumber or reorder surviving records. Reading from an offset whose record was removed proceeds to a later available record. It also does not ensure that only one value per key is visible at every instant: cleaning runs asynchronously, and the active/uncompacted portion can contain several versions.

A consumer rebuilding current state must process the retained updates and tombstones under the documented bootstrap conditions. If it falls behind long enough to miss a tombstone, applying the remaining history to an old local state can resurrect deleted data. A fresh empty-state rebuild and an incremental update to stale state have different prerequisites.

The log-compaction design explains ordering, offsets and tombstone visibility. Compaction is a storage policy, not a point-query API; a consumer or state store must materialize the key-to-current-value view.

Cleaner settings describe eligibility

# Topic configuration fragment: current-state or snapshot changelog.
cleanup.policy=compact
min.compaction.lag.ms=60000
max.compaction.lag.ms=86400000
delete.retention.ms=86400000

These values are illustrative. Minimum compaction lag keeps recent records ineligible for cleaning for a period. Maximum compaction lag bounds how long records can remain ineligible; it does not force the cleaner to finish within a day. Available cleaner threads, I/O budget, dirty data and key distribution determine how quickly eligible work completes.

A compacted topic’s storage depends on live key cardinality and value size, plus dirty history, tombstones and cleaning workspace. A rate-times-retention calculation alone cannot size it. A small set of frequently updated keys can compact well; unique keys for every event preserve nearly all events and gain little from compaction.

Checkpoint: can the last record rebuild the balance?

Given: account-7 starts at zero and receives credits of 100, 25 and 10. All three records use the same key. Assume cleaning has removed the first two records; no snapshot exists elsewhere.

Predict the rebuilt balance for each encoding:

1. Replay deltas before cleaning: 0 + 100 + 25 + 10 = 135.
2. Replay the retained delta after cleaning: only “add 10” remains, so 0 + 10 = 10. The missing 125 cannot be inferred from that record.
3. Encode replacement state instead: publish balance=100, then balance=125, then balance=135.
4. Replay the retained replacement: assigning balance=135 reconstructs the final balance, even when the earlier replacements disappear.

The test is whether replaying the retained representation produces the same required state. Kafka can compare keys; it cannot know that “add 10” depends on two deleted credits.

Change one requirement: an auditor needs the three original credits. Even the correct final balance is insufficient. Keep the domain-event history separately; a current-state representation deliberately answers a narrower question.

Separate history from snapshots

An authoritative event store cannot promise arbitrary historical reconstruction if it deletes the needed events. Snapshots can shorten replay only when their version/progress boundary and retained tail form a complete recovery path. Event sourcing and CQRS covers that contract.

Local and remote storage have different budgets

Tiered storage can move closed segments to remote storage while retaining a smaller local window. Model local disk, remote bytes, upload lag, remote fetch bandwidth and request costs separately. A longer remote history does not imply low-latency cold replay.

In the documented Kafka 4.0 implementation, a remote storage manager must be supplied/configured; compacted topics are not supported by the tiered-storage feature described there. Managed offerings can differ. Check the exact implementation and versioned limitations rather than assuming every compacted topic can be tiered.

Recover from disk pressure without corrupting the log

First determine whether growth comes from ingress, skew, underperforming compaction, internal topics or unexpectedly slow retention. Measure free bytes and growth rate to estimate time until exhaustion. Then control ingress, add capacity, or move replicas using supported Kafka operations while budgeting the resulting network and disk traffic.

Changing retention or deleting a topic can free data only by abandoning some history; confirm the accepted replay/data-loss boundary first. Avoid an automatic retention reduction based solely on how recently a topic was read. Rarely read history may be the disaster-recovery copy.

Do not manually remove broker-managed .log files as a generic cleanup procedure. Kafka must coordinate segment files, indexes, offsets and replication state. A superficially larger free-space reading is not evidence that recovery remains valid.

Test the chosen policy with a consumer outage that approaches the replay horizon, a slow cleaner and a cold bootstrap that needs tombstones. The result to verify is recoverable state within the required time, not merely fewer bytes on disk.

Whiteboard: prove the replay is possible

Guided variant. One order-event partition stores 10 MB/s of compressed data. Time retention is 72 hours, but its byte limit is 300 GB. Ignore segment slack and use decimal units. Complete the size-limited horizon: 300,000 MB / 10 MB/s = __ seconds = __ hours. Which configured limit is reached first at this sustained rate? Does RF=3 triple that logical horizon?

Independent problem. A search snapshot contains all required effects below offset 100, with next replay position 100. The current log begins at 120; required events at 100–119 existed and were deleted. Another topic has a compacted snapshot whose completeness and boundary are unknown. Draw the recovery intervals and decide whether a rebuild can proceed. Then explain why replaying older data could slow live writes even with unchanged producer volume; name two competing hypotheses and observations that distinguish them.

Pass criterion: show a complete snapshot-plus-tail interval and distinguish a retention setting from a recovery guarantee. Bring that same “what survived?” question to transaction failures.