This repository has been archived by the owner on Feb 18, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 224
Improve arity_assign
: ~2x
improvement if we share data.
#1076
Merged
jorgecarleitao
merged 8 commits into
jorgecarleitao:main
from
ritchie46:improve_arity_assign
Jun 16, 2022
Merged
Improve arity_assign
: ~2x
improvement if we share data.
#1076
jorgecarleitao
merged 8 commits into
jorgecarleitao:main
from
ritchie46:improve_arity_assign
Jun 16, 2022
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: Jorge Leitao <[email protected]>
Codecov Report
@@ Coverage Diff @@
## main #1076 +/- ##
==========================================
- Coverage 81.01% 80.95% -0.06%
==========================================
Files 366 366
Lines 35056 35134 +78
==========================================
+ Hits 28399 28442 +43
- Misses 6657 6692 +35
Continue to review full report at Codecov.
|
jorgecarleitao
approved these changes
Jun 15, 2022
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Love it! This is great!
Co-authored-by: Jorge Leitao <[email protected]>
Co-authored-by: Jorge Leitao <[email protected]>
@jorgecarleitao in light of #1078 and the possible I added the allocation branch for the validity as well. |
ritchie46
commented
Jun 16, 2022
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This continues upon #1073 (you may close that one if you prefer in one PR, otherwise I rebase).
I did a local benchmark to see what the cost of the new
arity_assign
would be in the case we share data. In polars expressions that is every first operation:This showed, that in case of shared data, we first do a
memcpy
and this is~2x
slower than the kernels where we allocate a new memory buffer.Luckily we can have best of both worlds. :)
This PR branches depending on the shared state:
I also added a distinction between
set_validity
andwith_validity
as we now have mutable data, it is not always logical return a newPrimitiveArray
.