Poetry install gets killed OOM

[rednaks]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: ubuntu 20.04.1
• memory: 512M, free: 253M
• Poetry version: 1.1.5
  logs:

Installing dependencies from lock file

Package operations: 60 installs, 0 updates, 0 removals

  • Installing asgiref (3.3.1): Installing...
  • Installing numpy (1.20.1): Installing...
  • Installing pycparser (2.20): Installing...
  • Installing pytz (2021.1): Installing...
  • Installing sqlparse (0.4.1): Installing...
  • Installing vine (5.0.0): Installing...
Killed

Issue

Noticed that today my server deployments fails during the poetry install task. after investigating, figured out that poetry is getting killed by the OS, Out Of Memory.

I have a machine with very limited resources, it was working fine until I updated poetry.

After doing some research I figured out that installer.parallel was added true by default in version 1.1.4. when I disabled the option the deployments are not failing anymore.

[abn]

Out of curiosity, is this machine a development environment or a target environment?

I suspect that the issue here is that the memory consumption is real since it is in parallel downloading, unpacking and installing (which might involve build/compile for packages like numpy). OS might be killing off the parent process. If this is not a leak, then the workaround that you have listed might be the actual solution for this scenario.

Maybe generating some profiles with something like https://pythonspeed.com/products/filmemoryprofiler/ might help triage the issue.

[rednaks]

well, it's a dev machine, I don't have enough ram for everything so I minimized the VMs locally, I'll try the profiler but I'm wondering, how many tasks poetry executes in parallel ? I see here he's trying to install 6 packages simultaneously I would be really concerned if the same thing happens on production and creates 100 process for installing all the dependencies at once.

My machine have 2 cores, poetry could use the number of CPUs (or 2 * CPUs) to execute parallel tasks and of course an option would be welcome.

[abn]

I believe you are looking for

poetry/poetry/installation/executor.py

Lines 66 to 76 in 129ed0e

	if parallel:
	# This should be directly handled by ThreadPoolExecutor
	# however, on some systems the number of CPUs cannot be determined
	# (it raises a NotImplementedError), so, in this case, we assume
	# that the system only has one CPU.
	try:
	self._max_workers = os.cpu_count() + 4
	except NotImplementedError:
	self._max_workers = 5
	else:
	self._max_workers = 1

.

I suspect we will need a new configuration and update the defaults maybe.

[rednaks]

Hello, any news on this issue ? the only solution I'm using right now is to disable parallel.
I think it's really dangerous especially when you download modules that compile things in C. if for example make is also using parallel jobs -jX it becomes a systematic issue ..

[dimbleby]

I expect it's compiling numpy that is burning through RAM anyway - there's nothing much poetry can do about that.

If disabling parallel works for you, great, do that. Or allocate more RAM to your VMs, 512M really isn't very much at all these days.

Pretty sure that there'll be no poetry fix here.

[neersighted]

Correct, this is wontfix. We can tune the default parallelism, but at the end of the day if the default is too aggressive for your system resources, that's why we have the knob.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.