Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Repo size - Submodules / Parallel Building Memory Usage #75

Closed
danieldjewell opened this issue Aug 8, 2021 · 2 comments · Fixed by #80 · May be fixed by #79
Closed

Repo size - Submodules / Parallel Building Memory Usage #75

danieldjewell opened this issue Aug 8, 2021 · 2 comments · Fixed by #80 · May be fixed by #79

Comments

@danieldjewell
Copy link

danieldjewell commented Aug 8, 2021 •
edited
Loading

Two issues:

  1. With all the submodules, the git pack files alone are 3.7GiB and with everything checked out it's in the neighborhood of 6 GiB... It would be nice to try and slim that down if possible. (Can git submodules be set to do a shallow fetch?)

  2. Building in parallel (make -j) on a VM with 32 cores exposed (the physical hardware is 2x Xeon 8-core with hyperthreading, so 16 threads/CPU) results in ~64 parallel jobs running - at least in the llvm/bolt phase. The trouble with this is:

  • Massive memory usage (something like 20GB)
  • Probably slower build times (depending on what I'm building, I find that 24-30 jobs is actually fastest ... because the hyperthreading isn't actually another core.)
  • The normal make options like -j16 aren't respected - when I try this, bolt and llvm appear to build in parallel but using one thread each.
  • I haven't tested the I/O impact specifically, but using all the cores can definitely run into interesting situations where the build is I/O limited and not processor limited. (I'm not using NVMe drives in this case...)
@kmod
Copy link
Contributor

kmod commented Aug 9, 2021

  1. It looks like there are some flags that allow you to do shallow submodules
  2. You're right, the issue here is that we're using ninja to build llvm, and ninja uses its own parallelism. I think the solution is just to switch our cmake invocation to use the makefile generator instead of ninja.

I'm on vacation and away from my work computer, but I could review a PR if you're interested in tackling these!

kmod added a commit to kmod/pyston that referenced this issue Aug 30, 2021
@kmod
Configure our submodules to be shallow
c29f3f7 
Fixes pyston#75 issue 1
@kmod kmod mentioned this issue Aug 30, 2021
Draft
kmod added a commit to kmod/pyston that referenced this issue Aug 30, 2021
@kmod
Switch our cmake projects from ninja to make
8ac30da 
Ninja is nicer but make has the key feature of integrating with the jobserver
of the top-level Makefile

Fixes pyston#75 part 2
kmod added a commit to kmod/pyston that referenced this issue Aug 30, 2021
@kmod
Switch our cmake projects from ninja to make
a50c08d 
Ninja is nicer but make has the key feature of integrating with the jobserver
of the top-level Makefile

Fixes pyston#75 part 2
@kmod kmod mentioned this issue Aug 30, 2021
Merged
@kmod kmod closed this as completed in #80 Aug 31, 2021
@kmod
Copy link
Contributor

kmod commented Aug 31, 2021

Ok we implemented the second part, and the first part is now being discussed in #79

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@kmod @danieldjewell