feat(producer): add retry buffer tuning option to prevent OOM

[wanwenli]

This commit adds an optional configuration to Sarama's retry mechanism to limit the size of the retry buffer. The change addresses issues #1358 and #1372 by preventing unbounded memory growth when retries are backlogged or brokers are unresponsive.

Key updates:

• Added Producer.Retry.MaxBufferLength configuration to control the maximum number of messages stored in the retry buffer.
• Implemented logic to handle overflow scenarios, ensuring non-flagged messages are either retried or sent to the errors channel, while flagged messages are re-queued.

This enhancement provides a safeguard against OOM errors in high-throughput or unstable environments while maintaining backward compatibility (unlimited buffer by default).

Additionally, as demonstrated in the graph above, the retry buffer’s length is effectively constrained, confirming that the new feature functions as intended.

[wanwenli]

@dnwe @puellanivis @rtreffer @klauspost I’ve submitted a PR addressing the retry buffer OOM issue (#1358, #1372) by introducing an optional configuration to limit the retry buffer’s size. The changes aim to prevent unbounded memory growth while maintaining backward compatibility (unlimited buffer by default). Your review and feedback would be greatly appreciated!

[puellanivis]

I don’t see any concerns. Though, I don’t know how appropriate a minimum functional retry buffer of 4 KiB would be. (I just knew it should be a power of 2 for being more allocator friendly.)

FYI: I cannot actually “approve” or even “resolve conversation”.

[dnwe]

@wanwenli thanks for submitting your PR and thanks @puellanivis for the review and giving useful early feedback. Looking over the changes I'm happy with the implementation and it provides a useful tuning mechanism for backpressure. Thanks again!