Kafka Consumer Groups & Partition Rebalancing

Consumer Group Fundamentals

A consumer group is a set of consumers that cooperate to consume messages from a set of topics. Each partition is consumed by exactly one consumer within the group.

Topic: payments (6 partitions)
Group: payment-processors (3 consumers)

Consumer-1 → Partition 0, 1
Consumer-2 → Partition 2, 3
Consumer-3 → Partition 4, 5

Partition Assignment Strategies

Kafka provides three built-in assignors:

• RangeAssignor (default): Assigns contiguous ranges per topic
• RoundRobinAssignor: Distributes partitions evenly across consumers
• StickyAssignor: Minimizes partition movement on rebalance

props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
    StickyAssignor.class.getName());

The Rebalance Protocol

A rebalance is triggered when:

1. 1.A consumer joins the group
2. 2.A consumer leaves or crashes
3. 3.New partitions are added
4. 4.Subscription changes

Stop-the-World Rebalance (Classic Protocol)

All consumers stop consuming, rejoin, get new assignments. This causes consumer lag.

Cooperative/Incremental Rebalance (Kafka 2.4+)

Only partitions that need to move are revoked. Other consumers continue consuming.

props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
    CooperativeStickyAssignor.class.getName());

Offset Management

// Auto-commit (risky for exactly-once)
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");

// Manual commit after processing
consumer.commitSync(Collections.singletonMap(
    partition, new OffsetAndMetadata(record.offset() + 1)
));

Interview Questions

1. 1.What is the difference between __consumer_offsets topic and Zookeeper offset storage?
2. 2.How do you prevent duplicate processing during a rebalance?
3. 3.What is session.timeout.ms vs heartbeat.interval.ms?
4. 4.How does the Cooperative Sticky Assignor reduce rebalance impact?
5. 5.What happens to uncommitted offsets when a consumer crashes?