Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: reduce object sizes up to 88% #687

Merged
merged 3 commits into from
Apr 14, 2023
Merged

feat: reduce object sizes up to 88% #687

merged 3 commits into from
Apr 14, 2023

Conversation

laresbernardo
Copy link
Collaborator

Based on demo data, 1 trial and 1000 iterations, reduced OutputModels from 40Mb to 5.3Mb (87% size reduction) by not storing the xDecompVec data.frame which contained all transformed values for all models, including carryover and immediate effects. Now we re-calculate when exporting the objects (OutputCollect) with little to no performance tradeoff, and reducing it's size substantially too.

@laresbernardo laresbernardo added the enhancement New feature or request label Apr 13, 2023
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 13, 2023
@laresbernardo
Copy link
Collaborator Author

laresbernardo commented Apr 13, 2023 •
edited
Loading

Ran one more test in my local computer, with 5 trials and 2000 iterations, and the size for OutputModels was:

  • main: 391.9Mb
  • PR: 48.8Mb

IMPROVEMENT: OutputModels is now 87.55% lighter!!

Note: OutputCollect object also reduced size but hard to compare given a randomness factor. Nevertheless, for the mentioned OutputModels we can generate a 125Mb-ish OutputCollect R object for 10K models (which is manageable-scalable). Also, note than OutputModels is also stored within OutputCollect.

@laresbernardo laresbernardo self-assigned this Apr 13, 2023
gufengzhou
gufengzhou approved these changes Apr 14, 2023
View reviewed changes
Copy link
Contributor

@gufengzhou gufengzhou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gufengzhou
Copy link
Contributor

Can confirm the %size reduction of OutputModels is reproducible. Good job!
For OutputCollect it'd probably need further reduction because its size is more related to the stored plotting objects.

@laresbernardo laresbernardo merged commit 8a05070 into main Apr 14, 2023
@laresbernardo laresbernardo deleted the size_reduction branch April 14, 2023 14:06
@laresbernardo laresbernardo changed the title feat: reduce object sizes up to 87% feat: reduce object sizes up to 88% Apr 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gufengzhou gufengzhou gufengzhou approved these changes

Assignees

@laresbernardo laresbernardo

Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@laresbernardo @gufengzhou @facebook-github-bot