Slow Producer

[kiro]

Hi,

we observe another peculiar behaviour, if we have a consumer and sync producer for the same topic running in two different goroutines (regardless of the number of partitions), the producer has significantly slower (~10x) produce rate than if it's running by itself.

We're using Sarama version Version 1.8.0 (2016-02-01), Kafka 0.8.2.1 and go1.5.1 linux/amd64

Config is

kafkaConfig.Producer.RequiredAcks = sarama.WaitForAll
kafkaConfig.Producer.Return.Errors = true
kafkaConfig.Producer.Return.Successes = true
kafkaConfig.Producer.Partitioner = sarama.NewManualPartitioner
kafkaConfig.Consumer.Return.Errors = true
kafkaConfig.Consumer.Fetch.Min = 8192
kafkaConfig.Consumer.MaxWaitTime = 250 * time.Millisecond
kafkaConfig.Net.DialTimeout = 5 * time.Second**

This is the profiler graph
https://www.dropbox.com/s/n5s0xssygcojxej/pprof001.svg?dl=0

code is here
https://gist.github.com/kiro/3fb5a80a2196d694cc61

logs look like

TRACE consumer/broker/2 added subscription to test/5
TRACE consumer/broker/1 added subscription to test/6
TRACE producer/broker/2 starting up
TRACE producer/broker/2 state change to [open] on test/1
TRACE consumer/broker/2 added subscription to test/7
TRACE Producer shutting down.
TRACE producer/broker/2 shut down

[eapache]

It's not clear if the producer and consumer are sharing the same client. If they are then that is likely the problem: from a single client, the kafka broker serves requests strictly serially, so while the consumer has requested new messages, the producer cannot proceed. Try using separate clients.

[kiro]

Hi, yes that was the problem, thanks for your help. Maybe the documentation should be updated, currently https://github.com/Shopify/sarama/blob/master/client.go#L12 states "A single client can be safely shared by multiple concurrent Producers and Consumers."