PyCall doesn't work with Enthought Canopy / EPD

[celestrist]

When I type @pyimport math, I get this error message:

ERROR: could not load module /Library/Frameworks/Python.framework/Versions/Current/bin/python: dlopen(/Library/Frameworks/Python.framework/Versions/Current/bin/python.dylib, 9): image not found
 in pyinitialize at /Users/Celestrist/.julia/PyCall/src/PyCall.jl:365
 in pyimport at /Users/Celestrist/.julia/PyCall/src/PyCall.jl:105

Any idea what happens?

[stevengj]

Which version of Python are you using? The one that comes with OSX, Anaconda, or Enthought?

[celestrist]

I am using Enthought Canopy python 2.7

Sent from my iPad

On 16 Sep, 2013, at 11:56 PM, "Steven G. Johnson" [email protected] wrote:

Which version of Python are you using? The one that comes with OSX, Anaconda, or Enthought?

—
Reply to this email directly or view it on GitHub.

[malmaud]

I too was having continual problems with pycall and Canopy. I just ended up switching to anaconda, which worked without incident.

[stevengj]

See also JuliaLang/IJulia.jl#86

[pluskid]

Same here:

ERROR: could not load module python: dlopen(python.dylib, 9): image not found
 in pyinitialize at /Users/chiyuan/.julia/PyCall/src/PyCall.jl:372
 in pyimport at /Users/chiyuan/.julia/PyCall/src/PyCall.jl:105

Everything looks fine when I switch to the system built-in Python.
Is there any workaround by explicitly specifying some PATH?

[pluskid]

I just found that calling

pyinitialize("python2.7")

before using any PyCall functions looks good. I guess making a symlink from libpython2.7.dylib to libpython.dylib in Canopy's distribution will also work.

[stevengj]

Are you sure that python2.7 is Canopy's Python executable? Type which python2.7 at the shell prompt....it might be the /usr/bin/python2.7 that ships with MacOS.

(The basic problem here seemsto be that Canopy's python does not seem to set up its sysutils.distconfig variables correctly.)

[pluskid]

@stevengj You are right. I superficially thought that the error was caused by libpython2.7.dylib as the error message was not being able to find python.dylib. It seems it turns out to be what you said -- I switched to the system python when explicitly initialize with python2.7. :(

[thoth291]

I have tried loading pythonxy version of python (PYTHON were set properly). Had the same issues during initialization under Windows 7 x64 (Julia Version 0.3.1 x86_64-w64-mingw32). Would be nice to get fixed as well.

in pyinitialize at C:\Users....julia\v0.3\PyCall\src\PyCall.jl:457
in pyinitialize at C:\Users....julia\v0.3\PyCall\src\PyCall.jl:451

[stevengj]

@dpinte, any thoughts?

[dpinte]

@stevengj I started looking into the initialization code just after EuroScipy.

On Windows, the issue seems to be with the initialization code in PyCall. From what I remember, the initialization will fail with any venv (on Python 3) or virtualenv setup or the venv backport that Canopy uses on Python 2.7. I'll try to dig a bit more into the issue and share my notes to make it work on Windows. The fix was not trivial though.

On my OSX install, it worked like a charm without any changes (which is surprizing looking at what I've seen for Windows). I am using Canopy 1.4.1 and have the Canopy Python registered on my .profile.

[stevengj]

Thank, I appreciate any tips. The overriding goal here is to initialize Python correctly given only the name of the python executable (or the full path if the executable is not in the PATH), since that is the only thing that an end user can reasonably be expected to know. It seems insane how hard this turns out to be to accomplish in a portable way.

[thoth291]

Thank you, guys!
Just to follow: Now I'm using
Version 0.3.2+2 (2014-10-22 01:21 UTC)
release-0.3/6babc84* (fork: 180 commits, 77 days)
x86_64-w64-mingw32
and having exactly the same problems.
I would be more than happy to test any possible solutions.
Thanks,
Thoth

[mishmash]

I'm using OS X 10.10 with Enthought Canopy Python 2.7.3. Basic imports seem to work fine:

julia> using PyCall
julia> @pyimport math
julia> math.pi
3.141592653589793

But things involving MKL fail:

julia> @pyimport scipy.optimize as so
ERROR: PyError (PyImport_ImportModule) <type 'exceptions.ImportError'>
ImportError('dlopen(/Users/mishmash/Library/Enthought/Canopy_64bit/System/lib/python2.7/site-packages/scipy/optimize/_lbfgsb.so, 2): Library not loaded: @rpath/libmkl_intel_lp64.dylib\n  Referenced from: /Users/mishmash/Library/Enthought/Canopy_64bit/System/lib/python2.7/site-packages/scipy/optimize/_lbfgsb.so\n  Reason: image not found',)
  File "/Users/mishmash/Library/Enthought/Canopy_64bit/System/lib/python2.7/site-packages/scipy/optimize/__init__.py", line 147, in <module>
    from ._minimize import *
  File "/Users/mishmash/Library/Enthought/Canopy_64bit/System/lib/python2.7/site-packages/scipy/optimize/_minimize.py", line 29, in <module>
    from .lbfgsb import _minimize_lbfgsb
  File "/Users/mishmash/Library/Enthought/Canopy_64bit/System/lib/python2.7/site-packages/scipy/optimize/lbfgsb.py", line 40, in <module>
    from . import _lbfgsb

 in pyerr_check at /Users/mishmash/.julia/v0.3/PyCall/src/exception.jl:58
 in pyimport at /Users/mishmash/.julia/v0.3/PyCall/src/PyCall.jl:85

Not sure if this is related to the above. Everything is fine using Anaconda.

[dpinte]

@mishmash yes, it is more than likely related to the same issue. I have zero time at the moment to propose a solution but plan to do it ...

[stevengj]

Whoops, didn't mean to close.

[stevengj]

@dpinte, a possibly related issue: Canopy Python fails to find site.py if I set the PYTHONHOME environment variable to match Python's sys.prefix and sys.exec_prefix, even though the Python documentation implies that PYTHONHOME is equivalent to these. (On my system, sys.prefix is in /Users/foo/Library/Enthought/.... while site.py is in /Applications/Canopy.app/...)

As a result, I can't really call Py_SetPythonHome sensibly to set up the paths.

In general, it seems like the difficulty is that Canopy is doing something unusual in how it sets up the Python paths, which is hard to replicate when loading libpython (as opposed to using python) in an embedded-Python situation like ours. Any light you could shed on this would be helpful.

[stevengj]

(Note that if I just omit the Py_SetPythonHome call for EPD, which is what I've done in the past, then some basic modules work but various scipy imports fail, e.g. pyimport("scipy.optimize") because it can't find the scimath module.)