Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

perf(rust): improve json_extract #8858

Merged
merged 7 commits into from
May 16, 2023
Merged

perf(rust): improve json_extract #8858

merged 7 commits into from
May 16, 2023

Conversation

ritchie46
Copy link
Member

@ritchie46 ritchie46 commented May 15, 2023 •
edited
Loading

Vendor json serialization we use and start branching of our specific needs. This will save compile times in the future, save memory and improve performance by using simd-json.

Memory usage reduced by 50% and performance increased by ~25%.

Can also follow up parallelizing this. Currently it is completely single threaded.

main

peak memory: 11992.79 MiB, increment: 11548.17 MiB
CPU times: user 3.52 s, sys: 2.56 s, total: 6.08 s
Wall time: 6.19 s

PR

peak memory: 5375.58 MiB, increment: 4929.63 MiB
CPU times: user 2.08 s, sys: 2.02 s, total: 4.1 s
Wall time: 4.21 s

Will follow up with rewriting the generic json parsing

@ritchie46 ritchie46 marked this pull request as draft May 15, 2023 18:24
@github-actions github-actions bot added enhancement New feature or an improvement of an existing feature rust Related to Rust Polars labels May 15, 2023
@ritchie46 ritchie46 marked this pull request as ready for review May 16, 2023 12:06
@ritchie46 ritchie46 changed the title feat(rust): vendor json perf(rust): improve json_extract May 16, 2023
@github-actions github-actions bot added the performance Performance issues or improvements label May 16, 2023
@ritchie46 ritchie46 merged commit af354fa into main May 16, 2023
@ritchie46 ritchie46 deleted the json branch May 16, 2023 13:26
@cmdlineluser cmdlineluser mentioned this pull request May 16, 2023
Closed
2 tasks
ritchie46 added a commit that referenced this pull request May 20, 2023
@ritchie46
perf(rust): improve json_extract (#8858)
06f215b
@josh josh mentioned this pull request May 30, 2023
Merged
c-peters pushed a commit to c-peters/polars that referenced this pull request Jul 14, 2023
@ritchie46 @c-peters
perf(rust): improve json_extract (pola-rs#8858)
1b462ef
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
enhancement New feature or an improvement of an existing feature performance Performance issues or improvements rust Related to Rust Polars
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@ritchie46