Mock factory memory improvements #330
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 5 commits
February 2, 2016 11:05
…nly the columns requested by the model. Should significantly improve memory efficiency of mock generation
aphearin
added a commit
that referenced
this pull request
Feb 2, 2016
Mock factory memory improvements
|
When we run halotools via the nbodykit wrapper we see the memory usage increase every time mock.populate is rerun. I think it is related to this. I am working on a reproducible test case, but any insights? |
|
@rainwoodman - does the memory usage taper off after a few repetitions, or grow without bound? I think @johannesulf has reported a related issue before? |
|
It doesn't taper off. See #917 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, when populating mocks, the entire halo catalog was bound to the mock object. This memory usage was needless. Now only the rows passing the appropriate cuts are stored, and also only the columns required by the model. This should reduce the dominant source of memory usage of the mock factories factory by a few hundred percent.
CC @mjvakili - this is a pre-requisite to implementing the sample variance feature we have been discussing, but it should immediately improve the memory leak issue you may have been encountering.