Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Pulsar logs are truncated when logging to files and Pulsar runs out of direct memory when -Dpulsar.allocator.exit_on_oom=true #17587

Closed
2 tasks done
lhotari opened this issue Sep 12, 2022 · 1 comment · Fixed by #17589
Assignees
Labels
type/bug The PR fixed a bug or issue reported a bug

Comments

@lhotari
Copy link
Member

lhotari commented Sep 12, 2022

Search before asking

  • I searched in the issues and found nothing similar.

Version

2.10.x

Minimal reproduce step

It's possible to reproduce the issue when running Pulsar in a configuration where Pulsar logs to log files. (a VM based deployment). To reproduce, configure the direct memory to a low value and stress the broker with sufficient load so that the direct memory runs out.
This problem doesn't appear in Kubernetes environments since logging uses System.out / stdout which flushes after each line by default.

What did you expect to see?

the Pulsar logs should be writted to disk before Pulsar exits the Java process

What did you see instead?

Pulsar log files are truncated and don't contain the last logged lines

Anything else?

No response

Are you willing to submit a PR?

  • I'm willing to submit a PR!
@lhotari lhotari added the type/bug The PR fixed a bug or issue reported a bug label Sep 12, 2022
@lhotari lhotari changed the title [Bug] Pulsar logs are truncated when logging to files and Pulsar runs out of direct memory [Bug] Pulsar logs are truncated when logging to files and Pulsar runs out of direct memory when -Dpulsar.allocator.exit_on_oom=true Sep 12, 2022
@lhotari
Copy link
Member Author

lhotari commented Sep 12, 2022

It seems that LogManager.shutdown(); should be called before Runtime.getRuntime().halt is called in the Pulsar code base.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
1 participant