Partition Rebalancing

[chenhengqi]

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.

Sarama	Kafka	Go
v1.23.1	0.10.21	1.11.5

Configuration

What configuration values are you using for Sarama and Kafka?

c := sarama.NewConfig()
c.Consumer.Return.Errors = true
c.Consumer.Offsets.Initial = sarama.OffsetOldest
c.Version = sarama.V0_10_2_1

Logs

When filing an issue please provide logs from Sarama and Kafka if at all
possible. You can set sarama.Logger to a log.Logger to capture Sarama debug
output.

logs: CLICK ME

// producer
[sarama] 2019/09/05 16:57:21 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama] 2019/09/05 16:57:21 producer/broker/10356 starting up
[sarama] 2019/09/05 16:57:21 producer/broker/10356 state change to [open] on partition-test-log/0
[sarama] 2019/09/05 16:57:21 Connected to broker at 9.1.0.28:9094 (registered as #10356)
[sarama] 2019/09/05 17:07:21 client/metadata fetching metadata for all topics from broker 9.1.0.28:9092
[sarama] 2019/09/05 17:07:21 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama] 2019/09/05 17:07:21 producer/broker/10353 starting up
[sarama] 2019/09/05 17:07:21 producer/broker/10353 state change to [open] on partition-test-log/1
[sarama] 2019/09/05 17:07:21 Connected to broker at 9.1.0.28:9093 (registered as #10353)

// consumer
[sarama] 2019/09/05 16:56:38 client/metadata fetching metadata for [partition-test-log] from broker 9.1.0.28:9092
[sarama] 2019/09/05 16:56:38 client/metadata fetching metadata for [partition-test-log] from broker 9.1.0.28:9092
[sarama] 2019/09/05 16:56:38 client/coordinator requesting coordinator for consumergroup partition-test-group from 9.1.0.28:9092
[sarama] 2019/09/05 16:56:38 client/coordinator requesting coordinator for consumergroup partition-test-group from 9.1.0.28:9092
[sarama] 2019/09/05 16:56:38 client/coordinator coordinator for consumergroup partition-test-group is #10356 (9.1.0.28:9094)
[sarama] 2019/09/05 16:56:38 client/coordinator coordinator for consumergroup partition-test-group is #10356 (9.1.0.28:9094)
[sarama] 2019/09/05 16:56:38 Connected to broker at 9.1.0.28:9094 (registered as #10356)
[sarama] 2019/09/05 16:56:38 Connected to broker at 9.1.0.28:9094 (registered as #10356)
[sarama] 2019/09/05 16:56:40 client/coordinator requesting coordinator for consumergroup partition-test-group from 9.1.0.28:9092
[sarama] 2019/09/05 16:56:40 client/coordinator coordinator for consumergroup partition-test-group is #10356 (9.1.0.28:9094)
[sarama] 2019/09/05 16:56:40 consumer/broker/10356 added subscription to partition-test-log/0
[sarama] 2019/09/05 17:06:38 client/metadata fetching metadata for all topics from broker 9.1.0.28:9092
[sarama] 2019/09/05 17:06:38 client/metadata fetching metadata for all topics from broker 9.1.0.28:9092

Problem Description

When I add a new partition to the topic that I am consuming, the consumer do NOT exit ConsumeClaim ASAP.

The messages in new partition stay in Kafka.

Is there a way to consume messages in new partition without restarting the consumer ?

I believe these issues #1237 #1462 #1471 are related.

[chenhengqi]

ping @d1egoaz

[antsbean]

I had commited PR to try fix this problem, please see #1525

[chenhengqi]

Thanks.