-
Notifications
You must be signed in to change notification settings - Fork 485
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ORC-1813: [C++] fix has_null forward compatibility #2082
base: main
Are you sure you want to change the base?
Conversation
Thank you for making a PR, @suxiaogang223 . |
I have fix some orc c++ uts because the orc file |
Thanks for fixing this! Could you create a JIRA issue? We use JIRA instead of GitHub issue for tracking purpose. |
How to create jira? it seems that your jira system cannot be used :( |
You may request a JIRA account via: https://selfserve.apache.org/jira-account.html It was open for registration. Unfortunately it has been disabled due to spam :( |
Thank you for fling JIRA and update the PR title, @suxiaogang223 . BTW, does it affect all ORC versions like 1.9.x and 1.8.x? The JIRA issue seems to be reported on Apache ORC 2.0.x only. |
For now, I set the milestone, 2.0.4, based on |
What changes were proposed in this pull request?
close: #2079
relate pr: #2055
Introduce fallback logic in the C++ reader to set hasNull to true when the field is missing, similar to the Java implementation.
The Java implementation includes the following logic:
In contrast, the C++ implementation directly uses the has_null value without any fallback logic:
Why are the changes needed?
We encountered an issue with the C++ implementation of the ORC reader when handling ORC files written with version 0.12. Specifically, files written in this version do not include the hasNull field in the column statistics metadata. While the Java implementation of the ORC reader handles this gracefully by defaulting hasNull to true when the field is absent, the C++ implementation does not handle this scenario correctly.
This issue prevents predicates like IS NULL from being pushed down to the ORC reader!!! As a result, all rows in the file are filtered out, leading to incorrect query results :(
How was this patch tested?
I have tested this using Doris external pipeline:
apache/doris#45104
apache/doris-thirdparty#259
Was this patch authored or co-authored using generative AI tooling?
No