-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Paused Consumer still sends FetchRequest (with empty blocks) #2125
Comments
Sure, I'll try to check it at the weekend. |
hi @dnwe , I've submitted a proposal, please check if this is a valid way. But I fear that skipping the |
Hello, I stumpled upon this issue as well. I tested the associated PR and it solves it. About your concerns of skipping
I don't see anything else critical inside So if there is any further assistance that I could provide here which could potentially help expedite the PR status, please do let me know 🙂 |
ok @dnwe I see that you merged this one, but there is a corner case which I am not sure that it is covered by this issue. So when all topics and the respective partitions are paused then this PR addresses that case and does not send a request, so far so good. But if you have a ConsumerGroup that consumes from two or more topics, and all the partitions of just one topic are paused, the same error resurfaces. My proposed fix to cover that case as well fix would be the following
would you like me to do a separate issue and maybe a PR about that case? |
@pkoutsovasilis yes that sounds interesting, please do raise an issue. Hearing from users who are consuming from multiple topics (potentially of differing partition numbers) from the same consumer group is always interesting as this isn’t as common a scenario and it usually teases out new and unforeseen behaviour |
@raulnegreiros Hi, using the latest v1.37.2 with this PR(#2143) mered, I found that when all the partitions are paused, we would observe high CPU load. (using the example consumergroup and sending SIGUSR1 to the pid would reproduce this problem) Digging into the code, I think it's caused by the following code introuduced by this PR. With Do you think a little
|
I have the same issue sarama v1.38.1 |
Whilst trying out the new "pause consumer" support (#2005) on main, I noticed that Sarama will submit empty FetchRequests if all of the assigned partitions are paused. This is because we only use Pause to skip adding the topicPartition blocks to the FetchRequest and still go ahead and send the Fetch even if it is essentially empty:
https://github.com/Shopify/sarama/blob/f214fdd60afa75e28e1d40308b64029033fc6f9d/consumer.go#L1071-L1077
Ideally we should work the same way as the Java client and not send any FetchRequest if there are no active assigned partitions to consume from
@raulnegreiros might you be able to take a look into what it would take to achieve this?
The text was updated successfully, but these errors were encountered: