Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RePause All Partitions After Rebalance if user paused any and requested #363

Merged
merged 4 commits into from
Oct 19, 2023

Conversation

Gin2022Null
Copy link
Contributor

Resolves #307

In order to achieve graceful shutdown during deployment, We applied the following strategy:

  1. Pause old kafka consumer
  2. Process in-flight messages
  3. Application fully shutdown.
    But when new consumers join the consumer group, after rebalance, old consumers wake up, if the newly assigned partitions are not in the pausedByUser, consumer start poll messages which gracefully shutdown are not fully achieved.

Add pauseAllAfterRebalance configuration
When a user pauses topics/partitions before rebalancing, the behavior depends on the value of pauseAllAfterRebalance.If it is set to False, the paused topics/partitions will remain paused after the rebalance. However, if it is set to True, all assigned topics/partitions will be paused after the rebalance.

Test:
ReceiverOptions.create(props).pauseAllAfterRebalance(Boolean.TRUE);

Consumer-1 assigned partitions: testTopic-2
Consumer-2 assigned partitions: testTopic-0, testTopic3-1

Consumer-2 Pause Topic/Partitions
Kafka Consumer Template status=PAUSE topic=testTopic, partition=0
Kafka Consumer Template status=PAUSE topic=testTopic, partition=1

Consumer-3 join - rebalance
Consumer-2 Revoke previously assigned partitions testTopic-0, testTopic-1
Assignment(partitions=[testTopic-2])}

testTopic-2 paused.

@pivotal-cla
Copy link

@Gin2022Null Please sign the Contributor License Agreement!

Click here to manually synchronize the status of this Pull Request.

See the FAQ for frequently asked questions.

@pivotal-cla
Copy link

@Gin2022Null Thank you for signing the Contributor License Agreement!

@Gin2022Null Gin2022Null marked this pull request as draft October 18, 2023 14:51
@Gin2022Null Gin2022Null marked this pull request as ready for review October 18, 2023 17:07
Copy link
Contributor

@garyrussell garyrussell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution; LGTM, except we should use boolean instead of Boolean.

@garyrussell garyrussell merged commit f535b9c into reactor:main Oct 19, 2023
1 check passed
@violetagg violetagg added the type/enhancement A general enhancement label Nov 14, 2023
@violetagg violetagg added this to the 1.3.22 milestone Nov 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement A general enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Option to Pause All After Rebalance
5 participants