TCP packet capture found that the Partitions.Offset returned by Fetch V1 Request increased, but MessageSet was empty

[memory-overflow]

Description

A topic consumption is suddenly stuck.

Using the packet capture tool, it is found that for normal topics, the return packet of Fetch V1 Request will have an increased offset and will return MessageSet.

As shown below, the normal request and response packets:

Abnormal request and response packets:

Once an exception occurs, the consumer will keep retrying indefinitely, causing the topic consumption to be stuck.

Versions

Sarama	Kafka	Go
v1.29.1	v3.5.1	1.18

Configuration

func setupKafka() (c *cluster.Consumer, err error) {
	topics := []string{}
	for {
		topics, err = dao.TIMatrixTemplateDao{}.GetAllTopics(context.Background())
		if err != nil {
			log.Error("get all topics error: ", err)
		}

		if len(topics) > 0 {
			break
		}
		log.Error("topic_id not update yet")
		time.Sleep(2 * time.Second)
	}

	clusterConfig := cluster.NewConfig()
	clusterConfig.Consumer.Return.Errors = true
	clusterConfig.Group.Return.Notifications = true
	clusterConfig.Consumer.Offsets.Initial = sarama.OffsetNewest
	clusterConfig.Consumer.Offsets.CommitInterval = time.Second
	clusterConfig.Consumer.Fetch.Default = int32(config.GetKafkaConsumer().KafkaMaxMessageBytes)
	if len(config.Get().Kafka.User) > 0 {
		clusterConfig.Net.SASL.Enable = true
		clusterConfig.Net.SASL.User = config.Get().Kafka.User
		clusterConfig.Net.SASL.Password = config.Get().Kafka.PassWord
	}
	consumer, err := cluster.NewConsumer(strings.Split(config.Get().Kafka.Host, ","),
		config.GetKafkaConsumer().KafkaGroupName, topics, clusterConfig)
	go dynamicConsumerTopic(consumer, topics)
	log.Infof("create kakfa consumer success, topics: %v", topics)
	return consumer, nil
}

additional

We have some complete packet capture data, but it involves sensitive information and cannot be provided directly. If you need it, you can contact me for assistance at [email protected]

[dnwe]

@memory-overflow did you resolve this issue?

[memory-overflow]

@memory-overflow did you resolve this issue?

Yes, we found that the v1 version of the fetch protocol will force the max_bytes check. If a packet larger than max_bytes appears, the client will be stuck and can neither get the message nor skip it.