Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rack-awareness in Kafka input #560

Closed
nicktelford opened this issue Nov 18, 2020 · 1 comment · Fixed by #878
Closed

Rack-awareness in Kafka input #560

nicktelford opened this issue Nov 18, 2020 · 1 comment · Fixed by #878
Labels
effort: lower A lower effort issue that should be possible to complete (relatively) quickly enhancement inputs Any tasks or issues relating specifically to inputs

Comments

@nicktelford
Copy link
Contributor

To reduce cross rack/DC/AZ bandwidth costs, the kafka input should support the rack-aware consumer API introduced in KIP-392.

Sarama has landed support for this upstream, although it has not yet been released. Once it has been released, it would be good for benthos to support this ASAP.

At the very least, benthos should add support via a new rackId configuration parameter on kafka inputs, which defaults to not being set.
@Jeffail Jeffail added enhancement inputs Any tasks or issues relating specifically to inputs labels Nov 18, 2020
@nicktelford
Copy link
Contributor Author

@Jeffail Rack-awareness landed in Sarama 1.26.2. Would it be possible to look at adding support for it now?

@Jeffail Jeffail added the effort: lower A lower effort issue that should be possible to complete (relatively) quickly label Sep 9, 2021
@mihaitodor mihaitodor mentioned this issue Sep 16, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
effort: lower A lower effort issue that should be possible to complete (relatively) quickly enhancement inputs Any tasks or issues relating specifically to inputs
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add rack_id field for Kafka components mihaitodor/connect
2 participants
@nicktelford @Jeffail