Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce memory requirements of running tool #163

Closed
saraadams opened this issue Dec 20, 2023 · 1 comment
Closed

Reduce memory requirements of running tool #163

saraadams opened this issue Dec 20, 2023 · 1 comment
Assignees
@saraadams
Labels
type/feat Suggests new features.

Comments

@saraadams
Copy link
Collaborator

saraadams commented Dec 20, 2023 •
edited
Loading

Problem

Especially when analyzing larger Bazel profiles, the tool requires a lot of memory. This may lead to OOMs.

Preferred solution

Investigate what consumes the most memory and look into implementing it more performantly.

Additional information

When analyzing a Bazel profile that is gzipped 34 MB and unzipped 35 MB, running the analyzer consumed about 4 GB.
@saraadams saraadams added the type/feat Suggests new features. label Dec 20, 2023
@saraadams saraadams mentioned this issue Dec 20, 2023
Merged
@saraadams
Copy link
Collaborator Author

The main culprit seems to be how we read the Bazel profile. We read the whole profile as a JsonObject. Instead, we should consider reading it in a stream.

saraadams added a commit that referenced this issue Dec 20, 2023
@saraadams
[Performance] Use interner to de-duplicate strings (#164)
8019030 
When reading the profile, many strings are encountered multiple times.
Use `String.interner()` to reduce the memory requirements of running the
analyzer.

Benchmark with a Bazel profile of unzipped size 357MB Using YourKit as a
profiler and checking the shallow size just before generating the
suggestions:

Before: 692 MB
After:  349 MB

Progress on #163

Signed-off-by: Sara Adams <[email protected]>
saraadams added a commit that referenced this issue Dec 21, 2023
@saraadams
[Performance] Read Bazel profile as a stream
a47724e 
Instead of reading the Bazel profile all at once, use a stream
to reduce memory consumption.

Benchmark with a Bazel profile of unzipped size 357MB
Using YourKit as a profiler and checking the shallow size at the end
of the `BazelProfile` constructor:

Before: 3.6 GB
After: 349 MB

Contributes to #163

Signed-off-by: Sara Adams <[email protected]>
@saraadams saraadams mentioned this issue Dec 21, 2023
Merged
@saraadams saraadams self-assigned this Dec 21, 2023
saraadams added a commit that referenced this issue Dec 21, 2023
@saraadams
[Performance] Read Bazel profile as a stream (#165)
ef5c4d5 
Instead of reading the Bazel profile all at once, use a stream to reduce
memory consumption.

Benchmark with a Bazel profile of unzipped size 357MB
Using YourKit as a profiler and checking the shallow size at the end of
the `BazelProfile` constructor:

Before: 3.6 GB
After: 349 MB

Contributes to #163

---------

Signed-off-by: Sara Adams <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@saraadams saraadams

Labels
type/feat Suggests new features.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@saraadams