Searching messages is currently ridiculously slow

[jonasvoelcker]

When searching in a topic containing much data, the search will never finish due to a combination of a poll-timeout of 1 second combined with max.poll.records of 50.

[jonasvoelcker]

If the max.poll.records are increased there is also a problem with the oldest listing as it returns a list, which is way too large:

#1840

[AlexisSouquiere]

I tried your 2 fixes (config update + limit on oldest consume) but for me there is still the default 50 records limit applied consumer max.poll.records because of this line:

akhq/src/main/java/org/akhq/repositories/RecordRepository.java

Line 656 in ff3acaf

properties.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, options.getSize());

size will always be 50 if you keep the default value. But increasing it will increase the number of records that will be displayed

akhq/src/main/java/org/akhq/repositories/RecordRepository.java

Line 1269 in ff3acaf

this.size = environment.getProperty("akhq.topic-data.size", Integer.class, 50);

Don't you have the same behaviour ?

[AlexisSouquiere]

I tried to remove the property from the consumer in the search function but the 50 records limit to be displayed in the UI is not applied anymore I was showing hundreds of matching records. We probably have to apply it later, probably here

akhq/src/main/java/org/akhq/repositories/RecordRepository.java

Lines 728 to 731 in b566b8c

	// More records than expected, send the records and then stop
	else if (matchesCount.get() >= options.getSize()) {
	currentEvent.emptyPoll = 666;
	emitter.onNext(currentEvent.progress(options));

[jonasvoelcker]

Hi @AlexisSouquiere,

thanks a lot for the detailed checks 🙂

I've worked on that and came to this solution: #1843

Feel free to doublecheck it as well 😉

Best Regards
Jonas