Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve][log] Print ZK path if write to ZK fails due to data being too large to persist #23652

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

[improve][log] Print ZK path if write to ZK fails due to data being too large to persist #23652

wants to merge 5 commits into from

Conversation

poorbarcode
Copy link
Contributor

@poorbarcode poorbarcode commented Nov 28, 2024 •
edited
Loading

Motivation & Modifications

Improvement 1

java.io.IOException: Len error. A message from /127.0.0.6:43889 with advertised length of 17890775 is either a malformed message or too large to process (length is greater than jute.maxbuffer=10485760)
     at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:567) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2]
     at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:352) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2]
     at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:508) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2]
     at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:153) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2]
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
     at java.lang.Thread.run(Thread.java:840) ~[?:?]

We can hardly find the ZK node when encountering the above error.

Improvement 2

There is a mechanism that persists cursor ack info into ZK if it matches the two conditions:

  • can not create a cursor ledger successfully.
  • all messages were acknowledged.

But it will print Error while using MetaStore, try to persist the position... even if it will not write ZK because there are still messages to consume.

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: x

@poorbarcode poorbarcode added this to the 4.1.0 milestone Nov 28, 2024
@poorbarcode poorbarcode self-assigned this Nov 28, 2024
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Nov 28, 2024
@lhotari lhotari changed the title [improve] [log] Print zk path if failed write ZK due to data is too large to persist [improve][log] Print zk path if failed write ZK due to data is too large to persist Nov 28, 2024
@lhotari
Copy link
Member

lhotari commented Nov 28, 2024

@poorbarcode One small detail about the PR titles. Please don't add a space between the 2 prefixes. Instead of [improve] [log], it should be [improve][log].

@lhotari lhotari changed the title [improve][log] Print zk path if failed write ZK due to data is too large to persist [improve][log] Print ZK path if write to ZK fails due to data being too large to persis Nov 28, 2024
@lhotari lhotari changed the title [improve][log] Print ZK path if write to ZK fails due to data being too large to persis [improve][log] Print ZK path if write to ZK fails due to data being too large to persist Nov 28, 2024
@poorbarcode poorbarcode requested a review from lhotari November 28, 2024 14:41
lhotari
lhotari reviewed Nov 28, 2024
View reviewed changes
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/ZKMetadataStore.java Outdated
Comment on lines 205 to 206
List<Pair> opsForLog = ops.stream().map(op -> Pair.of(op.getPath(), op.size()))
.collect(Collectors.toList());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this list is only for logging, I think it would be more useful to build a String directly in the desired format. It might be good to limit how many entries are logged and split to multiple log messages in case there could be a large amount of ops. Is the size of the String going to be a problem?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

java.util.StringJoiner (or Collectors.joining) could be used instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a filter op -> op.size > 256K

@poorbarcode poorbarcode requested a review from lhotari December 2, 2024 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zjxxzjwang zjxxzjwang zjxxzjwang approved these changes

@congbobo184 congbobo184 congbobo184 approved these changes

@lhotari lhotari Awaiting requested review from lhotari

Requested changes must be addressed to merge this pull request.

Assignees

@poorbarcode poorbarcode

Labels
doc-not-needed Your PR changes do not impact docs release/3.0.9 release/4.0.2
Projects
None yet
Milestone
4.1.0
Development

Successfully merging this pull request may close these issues.
4 participants
@poorbarcode @lhotari @congbobo184 @zjxxzjwang