Caching #59
Closed
Caching #59
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Running into some Polars issues, namely the fact that it can't handle Pega's OOTB scientific notation format when reading from json. I've filed an issue there - as reading with polars is quite a bit faster than the previous pandas-based method. So I think i'll wait for that to be resolved before continuing with this PR, especially since these are not essential changes. |
|
Underlying issue seems to be fixed in the backend json parser lib, so once that's propagated to Polars I can continue this work. Will probably be a few days still. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds caching of tables, currently implemented in the ADMDatamart class. Simply call
save_data(path)which will save the model data and predictor data as .feather files to the path supplied in the function. The default scanning table names in ADMDatamart now also recognise these cached files.Also, I changed the default importing behaviour by using Polars rather than PyArrow, and added logging during that importing function in
cdh_utils.