Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

External aggregation reserves more memory than actual usage #13089

Closed
2010YOUY01 opened this issue Oct 24, 2024 · 0 comments · Fixed by #13377
Closed

External aggregation reserves more memory than actual usage #13089

2010YOUY01 opened this issue Oct 24, 2024 · 0 comments · Fixed by #13377
Labels
bug Something isn't working

Comments

@2010YOUY01
Copy link
Contributor

Describe the bug

The below query requires 65M memory to run, if we set memory limit to 50M, it can not run successfully
Run in datafusion-cli:

cargo run -- --mem-pool-type fair -m 50M -c "
select t1.v1,  sum(t2.v1)
from
unnest(generate_series(1,1000)) as t1(v1)
, unnest(generate_series(1,1000)) as t2(v1)
group by t1.v1, t2.v1"

Error: External error: Resources exhausted: Failed to allocate additional 47616 bytes for GroupedHashAggregateStream[0] with 3995896 bytes already allocated for this reservation - 4031073 bytes remain available for the total pool

The issue is when doing sort-merge memory usage is over-estimated

self.reservation.try_grow(batch.get_array_memory_size())?;

For example, a RecordBatch with 3 arrays, arrays are sharing the same buffers, record_batch.get_array_memory_size() will estimate 3X actual memory consumption.
(The original RecordBatches passing through datafusion operators don't share Buffer between different columns, but in spilling queries, RecordBatches are first written to disk and read back, then it will reuse Buffers among different column arrays)

The root cause is already reported in arrow-rs apache/arrow-rs#6363
Once it's fixed in the arrow we should check if this aggregation query can run successfully, and also add tests.

To Reproduce

No response

Expected behavior

No response

Additional context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant