Skip to content
This repository has been archived by the owner on Jun 18, 2023. It is now read-only.

YATSM >100% CPU utilization with conda #51

Closed
ceholden opened this issue Oct 24, 2015 · 3 comments
Closed

YATSM >100% CPU utilization with conda #51

ceholden opened this issue Oct 24, 2015 · 3 comments

Comments

@ceholden
Copy link
Owner

Looks like the NumPy included in conda can do multithreaded computation for linear algebra routines by linking to OpenBLAS. You can disable this by setting OPENBLAS_NUM_THREADS=1 as an environment variable, but having all my jobs killed by our cluster's process reaper gave me quite the fright before I found the solution to turning it off!

If we linked against the Intel MKL, we'd also have MKL_NUM_THREADS to set.

Proposed solutions:

  1. Document it and hope people find it
  2. By default, have yatsm set the environment variables for multiprocessing to just use one thread. Use some --num_threads optional argument in the YATSM cli to turn multiprocessing on.

I'm actually in favor of option 2, but it seems pretty oppressive. Any thoughts @valpasq or @bullocke ?

@parevalo
Copy link
Contributor

I think option two makes a lot more sense. Better to go with the safe option and give an option to override it if necessary, no?

ceholden added a commit that referenced this issue Oct 26, 2015
@ceholden
Copy link
Owner Author

Ignore the previous reference to a commit. I had the wrong issue number

@ceholden ceholden added this to the v0.5.4 milestone Oct 26, 2015
@ceholden
Copy link
Owner Author

Commit 6bf7371 doesn't solve the problem. The idea is right, but it looks like we need to set the environment variables BEFORE numpy is ever imported.

This means that we need to import numpy even after we load all of the CLI program entry points (e.g., map, line, train, etc.). This could be a whole lot of work that makes things very messy...

ceholden added a commit that referenced this issue Oct 28, 2015
- Check sys.argv for num_threads option and set before click CLI imports
  anything that imports YATSM
- Don't import anything in yatsm/__init__.py for now. Maybe use lazy
  importing of numpy in CLI functions later on
ceholden added a commit that referenced this issue Oct 28, 2015
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants