Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minor: Add tests showing aggregate behavior for NaNs #10634

Merged
merged 3 commits into from
May 27, 2024

Conversation

alamb
Copy link
Contributor

@alamb alamb commented May 23, 2024

Which issue does this PR close?

Related to #10627 and #8031

Rationale for this change

While looking at #10627 from @westonpace I couldn't find any existing tests for how Nans worked in aggregates.

What changes are included in this PR?

Add a test that documents how Nans work today in the basic aggregates

Are these changes tested?

Only tests

Are there any user-facing changes?

No

@alamb alamb marked this pull request as ready for review May 23, 2024 10:50
@github-actions github-actions bot added the sqllogictest SQL Logic Tests (.slt) label May 23, 2024
query RRRRI
select min(col_f64_nan), max(col_f64_nan), avg(col_f64_nan), sum(col_f64_nan), count(col_f64_nan) from float_table;
----
-128.2 32768.3 10889.133333333333 32667.4 3
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is surprising to me that the aggregates on f32 seem to handle Nan consistently with postgres, but the aggregates on f64 do not (I would expect max(col_f64_nan), avg(col_f64_nan), sum(col_f64_nan) to all return NaN

query RRRRI
select min(col_f32), max(col_f32), avg(col_f32), sum(col_f32), count(col_f32) from float_table;
----
-128.2 32768.3 10889.13359451294 32667.40078353882 3
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What concerns me is below:

DataFusion CLI v38.0.0
> CREATE TABLE float_table (
     col_f32 FLOAT,
     col_f32_nan FLOAT,
     col_f64 DOUBLE,
     col_f64_nan DOUBLE
 ) as VALUES
 ( -128.2,  -128.2,                      -128.2,  -128.2 ),
 ( 32768.3, arrow_cast('NAN','Float32'), 32768.3, 32768.3 ),
 ( 27.3,    27.3,                        27.3,    arrow_cast('NAN','Float64') );
0 row(s) fetched. 
Elapsed 0.028 seconds.

> select min(col_f32), max(col_f32), avg(col_f32), sum(col_f32), count(col_f32) from float_table;
+--------------------------+--------------------------+--------------------------+--------------------------+----------------------------+
| MIN(float_table.col_f32) | MAX(float_table.col_f32) | AVG(float_table.col_f32) | SUM(float_table.col_f32) | COUNT(float_table.col_f32) |
+--------------------------+--------------------------+--------------------------+--------------------------+----------------------------+
| -128.2                   | 32768.3                  | 10889.13359451294        | 32667.40078353882        | 3                          |
+--------------------------+--------------------------+--------------------------+--------------------------+----------------------------+
1 row(s) fetched. 
Elapsed 0.011 seconds.


but in both duckdb and postgres

>
>  select -128.2 col_f32,  -128.2 col_f32_nan,                      -128.2 col_f64,  -128.2 col_f64_nan union all
>  select 32768.3, 'NaN'::DOUBLE PRECISION, 32768.3, 32768.3 union all
> select 27.3,    27.3,                        27.3,    'NaN'::DOUBLE PRECISION) x
> ;
┌──────────────┬──────────────┬────────────────────┬───────────────┬────────────────┐
│ min(col_f32) │ max(col_f32) │    avg(col_f32)    │ sum(col_f32)  │ count(col_f32) │
│ decimal(6,1) │ decimal(6,1) │       double       │ decimal(38,1) │     int64      │
├──────────────┼──────────────┼────────────────────┼───────────────┼────────────────┤
│       -128.2 │      32768.3 │ 10889.133333333333 │       32667.4 │              3 │
└──────────────┴──────────────┴────────────────────┴───────────────┴────────────────┘

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AVG is incorrect precision

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AVG is incorrect precision

One reason the AVG result may different is that your example used single precision (f32). When I run the same query with double precision (f64) DataFusion gets the same 10889.133333333333 as duckdb

> select min(col_f64), max(col_f64), avg(col_f64), sum(col_f64), count(col_f64) from float_table;
+--------------------------+--------------------------+--------------------------+--------------------------+----------------------------+
| MIN(float_table.col_f64) | MAX(float_table.col_f64) | AVG(float_table.col_f64) | SUM(float_table.col_f64) | COUNT(float_table.col_f64) |
+--------------------------+--------------------------+--------------------------+--------------------------+----------------------------+
| -128.2                   | 32768.3                  | 10889.133333333333       | 32667.4                  | 3                          |
+--------------------------+--------------------------+--------------------------+--------------------------+----------------------------+
1 row(s) fetched.
Elapsed 0.001 seconds.

Copy link
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm thanks @alamb
however I think there is some issue with UNION ALL and type derivation

> select avg(a) from (select -128.2::float a union all select 32768.3 union all select 27.3);
+--------------------+
| AVG(a)             |
+--------------------+
| 10889.133334350587 |
+--------------------+
1 row(s) fetched. 
Elapsed 0.011 seconds.

> select avg(a) from (select -128.2::float a union all select 32768.3::float union all select 27.3::float);
+-------------------+
| AVG(a)            |
+-------------------+
| 10889.13359451294 |
+-------------------+
1 row(s) fetched. 
Elapsed 0.011 seconds.

I expect those numbers to be the same. And that what duck db does

@comphead
Copy link
Contributor

Filed #10688

@comphead comphead merged commit 5afa801 into apache:main May 27, 2024
23 checks passed
@alamb alamb deleted the alamb/min_max_nan branch May 28, 2024 13:06
findepi pushed a commit to findepi/datafusion that referenced this pull request Jul 16, 2024
* Minor: Add tests showing aggregate behavior for NaNs

* Fix NaN setup
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sqllogictest SQL Logic Tests (.slt)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants