You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
From Hive 1.1.0 onwards, the column statistics will also record if there are any null values within the row group by setting the hasNull flag. The hasNull flag is used by ORC’s predicate pushdown to better answer ‘IS NULL’ queries.
We encountered an issue with the C++ implementation of the ORC reader when handling ORC files written with version 0.12. Specifically, files written in this version do not include the hasNull field in the column statistics metadata. While the Java implementation of the ORC reader handles this gracefully by defaulting hasNull to true when the field is absent, the C++ implementation does not handle this scenario correctly. This issue prevents predicates like IS NULL from being pushed down to the ORC reader!!! As a result, all rows in the file are filtered out, leading to incorrect query results :(
Steps to Reproduce:
Use an ORC file written with ORC version 0.12 (without hasNull in its column statistics).
Attempt to read the file using the C++ ORC reader.
Expected Behavior:
The C++ ORC reader should default the hasNull field to true when it is absent, ensuring compatibility with older file versions.
Observed Behavior:
The C++ ORC reader default the hasNull field to false, resulting in incorrect metadata interpretation.
Comparison with Java Implementation:
The Java implementation includes the following logic:
HI @suxiaogang223 , there is a patch #2055 to disable PPD when pb.has_hasnull() == false, but ColumnStatistics::hasNull still return false when the field in pb is missing. It Seems your patch would make it return a right value, would you like to make a PR? CC @wgtmac
HI @suxiaogang223 , there is a patch #2055 to disable PPD when pb.has_hasnull() == false, but ColumnStatistics::hasNull still return false when the field in pb is missing. It Seems your patch would make it return a right value, would you like to make a PR? CC @wgtmac
Issue Description:
We encountered an issue with the C++ implementation of the ORC reader when handling ORC files written with version 0.12. Specifically, files written in this version do not include the hasNull field in the column statistics metadata. While the Java implementation of the ORC reader handles this gracefully by defaulting hasNull to true when the field is absent, the C++ implementation does not handle this scenario correctly.
This issue prevents predicates like IS NULL from being pushed down to the ORC reader!!! As a result, all rows in the file are filtered out, leading to incorrect query results :(
Steps to Reproduce:
Expected Behavior:
The C++ ORC reader should default the hasNull field to true when it is absent, ensuring compatibility with older file versions.
Observed Behavior:
The C++ ORC reader default the hasNull field to false, resulting in incorrect metadata interpretation.
Comparison with Java Implementation:
The Java implementation includes the following logic:
In contrast, the C++ implementation directly uses the has_null value without any fallback logic:
Suggested Fix:
Introduce fallback logic in the C++ reader to set hasNull to true when the field is missing, similar to the Java implementation.
The text was updated successfully, but these errors were encountered: