Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ORC-1813: [C++] fix has_null forward compatibility #2082

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

suxiaogang223
Copy link

@suxiaogang223 suxiaogang223 commented Dec 10, 2024

What changes were proposed in this pull request?

close: #2079
relate pr: #2055
Introduce fallback logic in the C++ reader to set hasNull to true when the field is missing, similar to the Java implementation.
The Java implementation includes the following logic:

if (stats.hasHasNull()) {
    hasNull = stats.getHasNull();
} else {
    hasNull = true;
}

In contrast, the C++ implementation directly uses the has_null value without any fallback logic:

ColumnStatisticsImpl::ColumnStatisticsImpl(const proto::ColumnStatistics& pb) {
    stats_.setNumberOfValues(pb.number_of_values());
    stats_.setHasNull(pb.has_null());
}

Why are the changes needed?

We encountered an issue with the C++ implementation of the ORC reader when handling ORC files written with version 0.12. Specifically, files written in this version do not include the hasNull field in the column statistics metadata. While the Java implementation of the ORC reader handles this gracefully by defaulting hasNull to true when the field is absent, the C++ implementation does not handle this scenario correctly.
This issue prevents predicates like IS NULL from being pushed down to the ORC reader!!! As a result, all rows in the file are filtered out, leading to incorrect query results :(

How was this patch tested?

I have tested this using Doris external pipeline:
apache/doris#45104
apache/doris-thirdparty#259

Was this patch authored or co-authored using generative AI tooling?

No

@github-actions github-actions bot added the CPP label Dec 10, 2024
@dongjoon-hyun
Copy link
Member

Thank you for making a PR, @suxiaogang223 .

@suxiaogang223
Copy link
Author

suxiaogang223 commented Dec 11, 2024

I have fix some orc c++ uts because the orc file orc_index_int_string.orc is written by old orc version which don't write the has_null statistics.

c++/src/sargs/PredicateLeaf.cc Outdated Show resolved Hide resolved
c++/src/sargs/PredicateLeaf.cc Outdated Show resolved Hide resolved
c++/src/sargs/PredicateLeaf.cc Outdated Show resolved Hide resolved
c++/src/sargs/PredicateLeaf.cc Outdated Show resolved Hide resolved
c++/test/TestPredicateLeaf.cc Outdated Show resolved Hide resolved
@wgtmac
Copy link
Member

wgtmac commented Dec 11, 2024

Thanks for fixing this! Could you create a JIRA issue? We use JIRA instead of GitHub issue for tracking purpose.

@suxiaogang223
Copy link
Author

Thanks for fixing this! Could you create a JIRA issue? We use JIRA instead of GitHub issue for tracking purpose.

How to create jira? it seems that your jira system cannot be used :(

@wgtmac
Copy link
Member

wgtmac commented Dec 11, 2024

You may request a JIRA account via: https://selfserve.apache.org/jira-account.html

It was open for registration. Unfortunately it has been disabled due to spam :(

@suxiaogang223 suxiaogang223 changed the title fix has_null forward compatibility ORC-1813: [C++] fix has_null forward compatibility Dec 11, 2024
@dongjoon-hyun
Copy link
Member

Thank you for fling JIRA and update the PR title, @suxiaogang223 .

BTW, does it affect all ORC versions like 1.9.x and 1.8.x? The JIRA issue seems to be reported on Apache ORC 2.0.x only.

@dongjoon-hyun dongjoon-hyun added this to the 2.0.4 milestone Dec 11, 2024
@dongjoon-hyun
Copy link
Member

For now, I set the milestone, 2.0.4, based on Affected Versions of ORC-1813.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Incompatibility with ORC files written in version 0.12 due to missing hasNull field in C++ Reader
4 participants