growth fetch session cache slot

[aoslepov]

Versions

github.com/Shopify/sarama v1.26.0

Configuration

c.ClientID = getEnv("sarama")
c.ChannelBufferSize = getEnvInt("256")
c.Version = sarama.V2_4_0_0

c.Net.MaxOpenRequests = 1
c.Net.DialTimeout = time.Duration(getEnvInt("T30")) * time.Second
c.Net.ReadTimeout = time.Duration(getEnvInt("30")) * time.Second
c.Net.WriteTimeout = time.Duration(getEnvInt("30")) * time.Second
c.Net.SASL.Handshake = false

sslCertFile := getEnv("/srv/tracker-consumer/client.crt.pem")
sslKeyFile := getEnv("/srv/tracker-consumer/client.key.pem")
sslCaFile := getEnv("/srv/tracker-consumer/ca.pem")

if (sslCertFile != "") && (sslKeyFile != "") && (sslCaFile != "") {
 if sslConfig, err := NewTLSConfig(sslCertFile, sslKeyFile, sslCaFile); err == nil {
  c.Net.TLS.Enable = true
  c.Net.TLS.Config = sslConfig
 } else {
  panic(err)
 }
}

c.Metadata.Retry.Max = getEnvInt("1")
c.Metadata.Retry.Backoff = time.Duration(getEnvInt("10000")) * time.Millisecond
c.Metadata.RefreshFrequency = time.Duration(getEnvInt("10")) * time.Minute
c.Metadata.Full = true

c.Consumer.Fetch.Min = int32(getEnvInt("122100"))
c.Consumer.Fetch.Max = int32(getEnvInt("2442000"))
c.Consumer.MaxWaitTime = time.Duration(getEnvInt("2")) * time.Second

c.Consumer.Offsets.Initial = sarama.OffsetOldest

Logs

Initializing new client
ClientID is the default of 'sarama', you should consider setting it to something application-specific.
ClientID is the default of 'sarama', you should consider setting it to something application-specific.
client/metadata fetching metadata for all topics from broker 127.0.0.1:9092
Connected to broker at 127.0.0.1:9092 (unregistered)
client/brokers registered new broker #123 at localhost:9092
Successfully initialized new client
client/metadata fetching metadata for [clicks] from broker 127.0.0.1:9092
client/coordinator requesting coordinator for consumergroup tracker_consumer_group from 127.0.0.1:9092
client/coordinator coordinator for consumergroup tracker_consumer_group is #123 (localhost:9092)
ClientID is the default of 'sarama', you should consider setting it to something application-specific.
Connected to broker at localhost:9092 (registered as #123)
client/coordinator requesting coordinator for consumergroup tracker_consumer_group from 127.0.0.1:9092
client/coordinator coordinator for consumergroup tracker_consumer_group is #123 (localhost:9092)
consumer/broker/123 added subscription to clicks/2
consumer/broker/123 added subscription to clicks/0
consumer/broker/123 added subscription to clicks/1
consumer/broker/123 added subscription to clicks/3

Problem Description

continuous growth in slot consumption

# HELP kafka_server_fetchsessioncache_numincrementalfetchsessions Attribute exposed for management (kafka.server<type=F
etchSessionCache, name=NumIncrementalFetchSessions><>Value)
# TYPE kafka_server_fetchsessioncache_numincrementalfetchsessions untyped
kafka_server_fetchsessioncache_numincrementalfetchsessions 421.0
# HELP kafka_server_fetchsessioncache_incrementalfetchsessionevictions_total Attribute exposed for management (kafka.se
rver<type=FetchSessionCache, name=IncrementalFetchSessionEvictionsPerSec><>Count)
# TYPE kafka_server_fetchsessioncache_incrementalfetchsessionevictions_total counter
kafka_server_fetchsessioncache_incrementalfetchsessionevictions_total 0.0
# HELP kafka_server_fetchsessioncache_numincrementalfetchpartitionscached Attribute exposed for management (kafka.serve
r<type=FetchSessionCache, name=NumIncrementalFetchPartitionsCached><>Value)
# TYPE kafka_server_fetchsessioncache_numincrementalfetchpartitionscached untyped
kafka_server_fetchsessioncache_numincrementalfetchpartitionscached 1674.0

Errors in kafka server.log

Node 1 was unable to process the fetch request with (sessionId=1100333569, epoch=591834): FETCH_SESSION_ID_NOT_FOUND. (org.apache.kafka.clients.FetchSessionHandler)

[dnwe]

@aoslepov yes this was a known regression in 1.26.0 and is mentioned in the releases tab:

Fetch requests will cause the Kafka broker to continously allocate new Fetch sessions in its cache on every request. Fixed via #1644

Recommend upgrading to the test fix release v1.26.4