Amortize allocations in snappyDecode

[potocnyj]

Profiling the Kafka consumers at VividCortex showed me that a large portion of the allocated memory in our code comes from sarama.snappyDecode. This change reduces the average number of allocated bytes by pre-allocating the destination slice, and by re-using the chunk slice for cases that make multiple calls to snappy.Decode. Doing so should reduce GC pressure by a modest amount.

[eapache]

Makes sense to me. I'd be curious to see the actual before/after benchmark results if you can share those?

[potocnyj]

The numbers I have so far from local testing are fairly noisy, I'd be happy to collect some numbers from our production env and report back later though

[potocnyj]

Right, so here are some numbers on patched vs. un-patched on one of our production consumers:

• snappyDecode went from being 43.9% of total alloc'd_space to 33.9%, with the underlying snappy.Decode call dropping from 6.2% to 0.7%
• On the cpu side, snappyDecode went from 6.6% of cpu to 4.8%. runtime.Growslice and runtime.Makeslice have decreases that appear to make up this difference.
• bgsweep dropped from using 7.8% to 6.7% of cpu, and runtime.sweepone went from 9.7% to 7.8% of total cpu

[eapache]

👍

[wvanbergen]

Awesome improvement!!

On Friday, May 8, 2015, Evan Huus [email protected] wrote:

Merged #446 #446.

—
Reply to this email directly or view it on GitHub
#446 (comment).

[eapache]

Definitely awesome! After rolling this out to our cluster a little while ago, we've seen a distinct ~15% reduction in memory usage across our producers. Thanks!