A broker upgrade and a consumer-protocol migration are different changes. Kafka 4.0 made the new consumer rebalance protocol generally available, but its Java consumer still requires group.protocol=consumer to opt in.
A group can therefore run against upgraded brokers while continuing to use the classic protocol. That distinction explains why an upgrade alone may not change the rebalancing behavior you observe.
This note uses the versioned Kafka 4.0 documentation to explain the transition. It is not a claim that 4.0 is the newest Kafka release.
Three things that often get called rebalancing
Classic groups can use eager assignment, which revokes assignments broadly, or cooperative assignment, which moves partitions incrementally. Cooperative assignment is still part of the classic protocol.
The newer consumer protocol, specified in KIP-848, moves assignment coordination to the server and removes the global synchronization barrier through a fully incremental design. Selecting CooperativeStickyAssignor is not the same operation as selecting group.protocol=consumer.
| Concern | Classic protocol | Consumer protocol in Kafka 4.0 |
|---|---|---|
| Client opt-in | group.protocol=classic | group.protocol=consumer |
| Heartbeat interval | Client heartbeat.interval.ms | Server group.consumer.heartbeat.interval.ms |
| Session timeout | Client session.timeout.ms | Server group.consumer.session.timeout.ms |
| Assignment | Client assignment strategy | Server assignors; optional client group.remote.assignor selection |
Under the new protocol, the classic client heartbeat, session-timeout and partition.assignment.strategy settings are no longer usable. Nor are enforceRebalance() and its string overload. Copying the old configuration and changing one field is an incomplete migration.
Start with the group’s dependencies
Inventory the actual client library and version, assignment strategy, rebalance listeners, subscription API, and any custom assignment metadata. Include non-Java clients and framework wrappers: their feature support and exposed settings can differ.
Kafka 4.0’s protocol guide documents an offline path: stop all group members, then restart with the selected protocol. An empty group can be converted in either direction. This is straightforward but includes downtime.
It also documents an online rolling path for classic groups whose assignor does not embed custom metadata. That condition is material. Do not promise a rolling migration merely because both client versions connect to the broker. The 4.0 guide lists additional limitations around client-side assignors and rack-aware assignment; check the documentation for the exact version you plan to run.
Test the handover, not just steady-state throughput
In a staging group, measure consumption pauses, lag recovery, duplicate application effects, and commit failures while adding, removing and restarting members. Include a slow handler and a process killed during a database write. Verify listener assumptions against the new assignment lifecycle.
Incremental reassignment does not make processing instantaneous, remove the need to poll correctly, or make external side effects exactly once. The transaction-boundary note explains why duplicate-safe destination writes remain necessary.
Keep the protocol change separate from a large business-logic change. When metrics move, you want a short list of plausible causes. Record the rollback path and the limitations of the group and broker versions involved before starting the rollout.
The other version boundary: ZooKeeper
Kafka 4.0 removed ZooKeeper mode. Existing ZooKeeper clusters must migrate to KRaft before upgrading to 4.0. The upgrade guide also separates rolling broker replacement from finalizing the feature version and describes downgrade constraints.
The older diagrams in the Kafka architecture chapter now identify their ZooKeeper material as historical. The consumer groups chapter retains its classic-protocol explanation and adds this migration boundary.
The practical deliverable is a configuration ownership map: which settings belong to the client, which belong to the broker, and which assumptions live in application code. That map is more useful than treating “Kafka upgraded successfully” as proof that every consumer changed behavior.