Consumer group support for manually comitting offsets

[wclaeys]

Provide manual Commit method and don't create mainloop if AutoCommit is disabled

Fixes #1570

[knoguchi]

I'm just a user but how are you going to return errors?

[wclaeys]

In the same manner as how it's now being done in the 'mainLoop' when AutoCommit.Enable is true.

[BPScott]

Hi @wclaeys, your CLA check is failing because your commits aren't associated with your Github account. You need to connect your email address with your github account per https://help.github.com/en/github/setting-up-and-managing-your-github-user-account/adding-an-email-address-to-your-github-account.

[wclaeys]

@BPScott , thank you for your message. I've updated my profile.

[d1egoaz]

Thanks! 🚀

Could you please add some test for this!

[Gypsying]

Is this OK ?
What time will merge the pr?

[Morriaty-The-Murderer]

Hi guys, how is this pr going now?

[coldfire-x]

fixed yet?

[d1egoaz]

just some small comments.
Thanks

[d1egoaz]

this looks LGTM, but I'd love to have another 👍 from another Sarama maintainer @bai @dnwe @varun06 @mimaison @edoardocomar

[dnwe]

Thanks for working on this PR @wclaeys — the changes overall look good to me, a couple of thoughts below:

[wclaeys]

Can somebody please cancel this new CLA check? Accidentally pushed without a proper e-mail setting (reverted it and did a proper commit). @BPScott maybe? Thank you!

[dnwe]

@wclaeys I think the PR is ready, so why don’t you squash your commits now and force push again with your correct email, that should pass the CLA check

[dnwe]

@wclaeys you need to rebase your branch on master to get the checks to pass

This branch is 1 commit ahead, 24 commits behind Shopify:master.

[dnwe]

@wclaeys ah don't worry, I found that I had permission to push to your branch myself so I performed the rebase. If the checks both pass green then I will go ahead and merge. Thanks for your contribution on this one, really appreciated 👍

[dnwe]

As there's been a lot of interest in this functionality, it would be great if all the people watching this PR and the accompanying issues could test it out with their application code once it's merged, in advance of us cutting any new release version.

Assuming you're using go modules to manage your dependencies, once this has merged, you can update your go.mod to fetch the latest commit from the default branch by doing:

go get github.com/Shopify/sarama@HEAD

[bglmmz]

How about a single consumer? Could a single consumer commit offsets manually?
Maybe we can create a consumer group it only has a single consumer instance.

[aneeskA]

As there's been a lot of interest in this functionality, it would be great if all the people watching this PR and the accompanying issues could test it out with their application code once it's merged, in advance of us cutting any new release version.

Assuming you're using go modules to manage your dependencies, once this has merged, you can update your go.mod to fetch the latest commit from the default branch by doing:

go get github.com/Shopify/sarama@HEAD

@dnwe Is there any timeline for releasing this feature?

[alok87]

Please suggest when are you releasing this, this helpful in cases where you want the commit to happen instantly, systems which care about order and duplicates really need it.

[jameshalsall]

doesn't this break backwards compatibility with the addition of the new interface method?

[d1egoaz]

@jameshalsall yup

I think we could apply this trick if we don't want users to implement the interface

    // A private method to prevent users implementing the
    // interface and so future additions to it will not
    // violate Go 1 compatibility.
    private()

from https://blog.golang.org/module-compatibility

what do you think @dnwe ? ]

this PR was also merged adding a new method to an existing interface #1781

[jameshalsall]

We implement this interface in our tests so we can use mock versions in our handlers, I’m not sure preventing people from implementing it is the right option from my perspective.

…

Sent from my iPhone

On 19 Aug 2020, at 16:36, Diego Alvarez ***@***.***> wrote:  @jameshalsall yup I think we could apply this trick if we don't want users to implement directly the interface // A private method to prevent users implementing the // interface and so future additions to it will not // violate Go 1 compatibility. private() from https://blog.golang.org/module-compatibility what do you think @dnwe ? this PR was also merged adding a new method to an existing interface #1781 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[hipper]

Hey @wclaeys , found a bug that looks like introduced after this change. The problem is when you close the consumer it ignores the AutoCommit flag and commits the offset before it exits. That problem doesn't exists in 1.26.4

Described more here: #1843