UDP Socket Memory Limit hit / Overflow

[bpereto]

I think there is some issue introduced in updating sarama to 1.30.0 or its deps (#47) as we see UDP Socket Memory limit hit / overflow.

The working commit is: 17a96d9
The Overflow surfaces with: ec08b78

OK: netdata graph with commit 17a96d9

NOK: netdata graph with commit ec08b78

[shyam334]

To isolate the fault, wondering if you have tried the commit/PR just before ec08b78, 7baa828. (where sarama was updated, but not go version)

[bpereto]

yes, I verified the commit 7baa828 with the same result.

[lspgn]

Thank you for reporting this issue, are you able to provide more collected metrics (eg: Prometheus endpoint)?
Does this surface when using a file transport? (assuming no, but would like to know before ruling this out)
Are you able to get the count of messages produced into Kafka with both versions?
Is it NetFlow/IPFIX or sFlow?

[lspgn]

Hello,
Is this still happening?

[bpereto]

Hi, sorry for the delay, I will test again.
The change is the update to sarama 1.30.1 in right? 05b4362

the release 1.30.1 of sarama has a warning, which I don't know if this applies to goflow?

regression for throughput drop:
IBM/sarama#2071

I will test again with the newest commits of goflow.

[bpereto]

The performance issue is resolved with the latest commit of goflow (8d59905) and latest sarama release 1.31.0, probably a regression in sarama.

Thank you for reporting this issue, are you able to provide more collected metrics (eg: Prometheus endpoint)?

I think this is now obsolete.

Does this surface when using a file transport? (assuming no, but would like to know before ruling this out)

No, writing directly to a file (tmpfs) does not show a problem.

Is it NetFlow/IPFIX or sFlow?

It's ipfix (v10).

[lspgn]

Thank you for confirming. Will close the Issue.