-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BugFix] close hudi fsview to avoid resource leak #52738
Conversation
Signed-off-by: yanz <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for catching this!
6d7a8c5
Quality Gate passedIssues Measures |
[Java-Extensions Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[FE Incremental Coverage Report]✅ pass : 29 / 30 (96.67%) file detail
|
[BE Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
@Mergifyio backport branch-3.3 |
@Mergifyio backport branch-3.2 |
✅ Backports have been created
|
✅ Backports have been created
|
Signed-off-by: yanz <[email protected]> (cherry picked from commit ec8eae4) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/connector/RemoteFileScanContext.java # fe/fe-core/src/main/java/com/starrocks/connector/RemotePathKey.java # fe/fe-core/src/main/java/com/starrocks/connector/hudi/HudiRemoteFileIO.java
Signed-off-by: yanz <[email protected]> (cherry picked from commit ec8eae4) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/connector/RemoteFileScanContext.java # fe/fe-core/src/main/java/com/starrocks/connector/RemotePathKey.java # fe/fe-core/src/main/java/com/starrocks/connector/hudi/HudiRemoteFileIO.java
…52777) Signed-off-by: yanz <[email protected]> Co-authored-by: yanz <[email protected]>
…52778) Signed-off-by: yanz <[email protected]> Co-authored-by: yanz <[email protected]>
Why I'm doing:
Hudi bitcask temp files descriptors are leaked.
What I'm doing:
Close hudi filesystem view when we don't need it.
Fixes #52737
This fix has a potential drawback but we can avoid that.
Before this PR, we create fileststem view when the request comes, and we only create the fsview for once. However, we don't know when to close this fsview. And fsview is left in opened state, and resource leak happens.
To fix this problem, we have to close this fsview when no requests need this fsview. We can use reference count to solve it. However we don't know how many total requests will be issued. Because some request will be cached without using this fsview, we don't know how many requests will use fsview in advance.
However there is a potential drawback. If some requests are not overlapped, fsview will be created and closed for multiple times. In practice it seldom happens.
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: