[fix][broker] Fix broker OOM when upload a large package.

[shibd]

Motivation

Using package API to upload a large package may cause broker OOM.

Because in the current implementation, will copy all data to the heap.

pulsar/pulsar-package-management/bookkeeper-storage/src/main/java/org/apache/pulsar/packages/management/storage/bookkeeper/DLOutputStream.java

Lines 60 to 65 in 2c6fcc7

	while ((read = inputStream.read(readBuffer)) != -1) {
	log.info("write something into the ledgers offset: {}, length: {}", offset, read);
	ByteBuf writeBuf = Unpooled.copiedBuffer(readBuffer, 0, read);
	offset += writeBuf.readableBytes();
	LogRecord record = new LogRecord(offset, writeBuf);
	records.add(record);

Modifications

Each time, only copy the size of one record and then upload it directly.

There is no need to worry about batching, as the distributedlog client will handle the batching.

Verifying this change

• All related to package management test will pass.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 85.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 73.43%. Comparing base (bbc6224) to head (f5b8da1).
Report is 427 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #22989      +/-   ##
============================================
- Coverage     73.57%   73.43%   -0.15%     
- Complexity    32624    33266     +642     
============================================
  Files          1877     1907      +30     
  Lines        139502   142951    +3449     
  Branches      15299    15578     +279     
============================================
+ Hits         102638   104972    +2334     
- Misses        28908    29946    +1038     
- Partials       7956     8033      +77     

Flag	Coverage Δ
inttests	27.52% <75.00%> (+2.93%)	⬆️
systests	24.73% <0.00%> (+0.40%)	⬆️
unittests	72.45% <85.00%> (-0.39%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
.../management/storage/bookkeeper/DLOutputStream.java	90.62% <85.00%> (-0.29%)	⬇️

... and 480 files with indirect coverage changes

[liangyepianzhou]

Good Catch!