[improve][broker] Optimize PersistentTopic.getLastDispatchablePosition

[dao-jun]

Motivation

PersistentTopic#getLastDispatchablePosition is using by
Reader#hasMessageAvailable , ConsumerImpl#hasMessageAvailable, Consumer#getLastMessageIdAsync.

The current implementation is read entries from Bookkeeper(or sth else), which leads to low throughput, high latency and heavy load, this PR is for the purpose of optimization.

Modifications

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[shibd]

Nice job! LGTM

[dao-jun]

@coderzc PTAL

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 73.17073% with 11 lines in your changes missing coverage. Please review.

Project coverage is 73.29%. Comparing base (bbc6224) to head (780000b).
Report is 366 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #22707      +/-   ##
============================================
- Coverage     73.57%   73.29%   -0.29%     
- Complexity    32624    33015     +391     
============================================
  Files          1877     1891      +14     
  Lines        139502   141969    +2467     
  Branches      15299    15571     +272     
============================================
+ Hits         102638   104051    +1413     
- Misses        28908    29893     +985     
- Partials       7956     8025      +69     

Flag	Coverage Δ
inttests	27.21% <56.09%> (+2.62%)	⬆️
systests	24.70% <39.02%> (+0.37%)	⬆️
unittests	72.30% <73.17%> (-0.55%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...ransaction/buffer/impl/TopicTransactionBuffer.java	88.21% <100.00%> (+0.46%)	⬆️
...sar/broker/service/persistent/PersistentTopic.java	79.12% <90.90%> (+0.66%)	⬆️
...nsaction/buffer/impl/TransactionBufferDisable.java	57.57% <71.42%> (+1.05%)	⬆️
...ransaction/buffer/impl/InMemTransactionBuffer.java	54.85% <0.00%> (-2.72%)	⬇️

... and 407 files with indirect coverage changes

[liangyepianzhou]

Oh, Sorry, I overlooked this. If you do not set it to null here, but set a flag that requires reading BK, you can reduce the response time of the next request. I mentioned here before, but maybe not in detail.

1. end txn.
2. set a flag 'readFromBK' as true
3. update normal message
4. set a flag 'readFromBK' as false
5. when call getLastDispatchablePosition, if the flag == true, the lastDispatchablePosition is valid to return. And read from BK after return.
6. Call getLastDispatchablePosition again, it can get the last getLastDispatchablePosition.

How do you think about this?