-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Minor: Add tests showing aggregate behavior for NaNs #10634
Conversation
query RRRRI | ||
select min(col_f64_nan), max(col_f64_nan), avg(col_f64_nan), sum(col_f64_nan), count(col_f64_nan) from float_table; | ||
---- | ||
-128.2 32768.3 10889.133333333333 32667.4 3 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it is surprising to me that the aggregates on f32 seem to handle Nan consistently with postgres, but the aggregates on f64 do not (I would expect max(col_f64_nan), avg(col_f64_nan), sum(col_f64_nan)
to all return NaN
query RRRRI | ||
select min(col_f32), max(col_f32), avg(col_f32), sum(col_f32), count(col_f32) from float_table; | ||
---- | ||
-128.2 32768.3 10889.13359451294 32667.40078353882 3 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What concerns me is below:
DataFusion CLI v38.0.0
> CREATE TABLE float_table (
col_f32 FLOAT,
col_f32_nan FLOAT,
col_f64 DOUBLE,
col_f64_nan DOUBLE
) as VALUES
( -128.2, -128.2, -128.2, -128.2 ),
( 32768.3, arrow_cast('NAN','Float32'), 32768.3, 32768.3 ),
( 27.3, 27.3, 27.3, arrow_cast('NAN','Float64') );
0 row(s) fetched.
Elapsed 0.028 seconds.
> select min(col_f32), max(col_f32), avg(col_f32), sum(col_f32), count(col_f32) from float_table;
+--------------------------+--------------------------+--------------------------+--------------------------+----------------------------+
| MIN(float_table.col_f32) | MAX(float_table.col_f32) | AVG(float_table.col_f32) | SUM(float_table.col_f32) | COUNT(float_table.col_f32) |
+--------------------------+--------------------------+--------------------------+--------------------------+----------------------------+
| -128.2 | 32768.3 | 10889.13359451294 | 32667.40078353882 | 3 |
+--------------------------+--------------------------+--------------------------+--------------------------+----------------------------+
1 row(s) fetched.
Elapsed 0.011 seconds.
but in both duckdb and postgres
>
> select -128.2 col_f32, -128.2 col_f32_nan, -128.2 col_f64, -128.2 col_f64_nan union all
> select 32768.3, 'NaN'::DOUBLE PRECISION, 32768.3, 32768.3 union all
> select 27.3, 27.3, 27.3, 'NaN'::DOUBLE PRECISION) x
> ;
┌──────────────┬──────────────┬────────────────────┬───────────────┬────────────────┐
│ min(col_f32) │ max(col_f32) │ avg(col_f32) │ sum(col_f32) │ count(col_f32) │
│ decimal(6,1) │ decimal(6,1) │ double │ decimal(38,1) │ int64 │
├──────────────┼──────────────┼────────────────────┼───────────────┼────────────────┤
│ -128.2 │ 32768.3 │ 10889.133333333333 │ 32667.4 │ 3 │
└──────────────┴──────────────┴────────────────────┴───────────────┴────────────────┘
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AVG is incorrect precision
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AVG is incorrect precision
One reason the AVG result may different is that your example used single precision (f32
). When I run the same query with double precision (f64
) DataFusion gets the same 10889.133333333333
as duckdb
> select min(col_f64), max(col_f64), avg(col_f64), sum(col_f64), count(col_f64) from float_table;
+--------------------------+--------------------------+--------------------------+--------------------------+----------------------------+
| MIN(float_table.col_f64) | MAX(float_table.col_f64) | AVG(float_table.col_f64) | SUM(float_table.col_f64) | COUNT(float_table.col_f64) |
+--------------------------+--------------------------+--------------------------+--------------------------+----------------------------+
| -128.2 | 32768.3 | 10889.133333333333 | 32667.4 | 3 |
+--------------------------+--------------------------+--------------------------+--------------------------+----------------------------+
1 row(s) fetched.
Elapsed 0.001 seconds.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm thanks @alamb
however I think there is some issue with UNION ALL and type derivation
> select avg(a) from (select -128.2::float a union all select 32768.3 union all select 27.3);
+--------------------+
| AVG(a) |
+--------------------+
| 10889.133334350587 |
+--------------------+
1 row(s) fetched.
Elapsed 0.011 seconds.
> select avg(a) from (select -128.2::float a union all select 32768.3::float union all select 27.3::float);
+-------------------+
| AVG(a) |
+-------------------+
| 10889.13359451294 |
+-------------------+
1 row(s) fetched.
Elapsed 0.011 seconds.
I expect those numbers to be the same. And that what duck db does
Filed #10688 |
* Minor: Add tests showing aggregate behavior for NaNs * Fix NaN setup
Which issue does this PR close?
Related to #10627 and #8031
Rationale for this change
While looking at #10627 from @westonpace I couldn't find any existing tests for how Nans worked in aggregates.
What changes are included in this PR?
Add a test that documents how Nans work today in the basic aggregates
Are these changes tested?
Only tests
Are there any user-facing changes?
No