Problems when installing for python3

[vevenom]

We are facing problems while setting up spore with python3. We follow the guidelines
from Pattern Matching Experiment readme with following changes:

export PYTHONPATH=$PYTHONPATH:$TARGET_DIR/lib/python3.4/site-packages
export PYTHONPATH=$PYTHONPATH:$TARGET_DIR/lib64/python3.4/site-packages

We use pip3 instead of pip like instructed.

For MUSIC we change the following line:

PYTHON=/usr/bin/python3 ./configure --prefix=$TARGET_DIR --disable-isend

For NEST we change:
cmake -DCMAKE_INSTALL_PREFIX:PATH=$TARGET_DIR -Dwith-music=ON -Dwith-mpi=ON -Dwith-python=3 ..

For SPORE:
cmake -Dwith-python=3 ..

We tested with both master and py3 branches of the spore repository. Also with NEST revision 58fd190f5e4 and latest master. When performing make test we fail two tests:

9/10 Test  #9: test_reward_in_proxy .............***Failed    0.23 sec
      Start 10: test_music_integration
10/10 Test #10: test_music_integration ...........***Failed    0.31 sec

We encounter no problems for nest.Install("sporemodule") but again for pattern mattching experiment script it fails with:

Traceback (most recent call last):
  File "python/zmq_proxy_node.py", line 6, in <module>
    import snn_utils.comm.music
  File "/home/sinisa/Uni/Project_CI/spore-nest-module/examples/pattern_matching_showcase/python/snn_utils/comm/music/__init__.py", line 4, in <module>
    import music
  File "/home/sinisa/opt/lib/python3.4/site-packages/music/__init__.py", line 25, in <module>
    from .pymusic import *
  File "pybuffer.pxd", line 49, in init pymusic (pymusic.cpp:11666)
ImportError: /home/sinisa/opt/lib/python3.4/site-packages/music/pybuffer.so: undefined symbol: PyString_Type
Traceback (most recent call last):
  File "python/control_node.py", line 6, in <module>
    import music
  File "/home/sinisa/opt/lib/python3.4/site-packages/music/__init__.py", line 25, in <module>
    from .pymusic import *
  File "pybuffer.pxd", line 49, in init pymusic (pymusic.cpp:11666)
ImportError: /home/sinisa/opt/lib/python3.4/site-packages/music/pybuffer.so: undefined symbol: PyString_Type
--------------------------------------------------------------------------
mpirun has exited due to process rank 1 with PID 30396 on
node sinisa-Lenovo-V110-15ISK exiting improperly. There are two reasons this could occur:

1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"

This may have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------

[mhoff]

Does the following work in your shell? python3 -c 'import music'

[vevenom]

Nope. We get following:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/sinisa/opt/lib/python3.4/site-packages/music/__init__.py", line 25, in <module>
    from .pymusic import *
  File "pybuffer.pxd", line 49, in init pymusic (pymusic.cpp:11666)
ImportError: /home/sinisa/opt/lib/python3.4/site-packages/music/pybuffer.so: undefined symbol: PyString_Type

[mhoff]

http://stackoverflow.com/questions/38869506/cython-python-symbols-not-found-pystring-type

My guess would be a mismatch between Python 2 (where PyString_Type exists) and Python3 (where PyBytes_Type) exists. Are you calling setup.py with the same version of Python as the headers for, and the same version of Python as the link libraries are for?

I had python3 and python3.4 installed. The former I used for compiling, the latter was used by ipython/jupyter which I used to import the module. That caused the linking error.

I suppose you should use this for configuring MUSIC:
PYTHON=/usr/bin/python3.4 ./configure --prefix=$TARGET_DIR --disable-isend

@sinisastekovic Does this solve your problem?

[mhoff]

@anandtrex https://github.com/IGITUGraz/spore-nest-module/blame/master/examples/pattern_matching_showcase/README.md#L95 seems to be incorrect (or not precise enough) for multiple installed versions of python3.

[vevenom]

I have just tried the same thing. It does not help. Anyhow when I run the command
ls -l /usr/bin/python3 i get the following output which seems fine to me:

lrwxrwxrwx 1 root root 9 Dez 21 12:45 /usr/bin/python3 -> python3.4
We are running installation tests on two computers. @dsalaj passed test_reward_in_proxy
after recompiling MUSIC with PYTHON=/usr/bin/python3.5 ./configure --prefix=$TARGET_DIR --disable-isend(he is using python3.5) but still failing the last test test_music_integration.

I have recompiled MUSIC with PYTHON=/usr/bin/python3.4 ./configure --prefix=$TARGET_DIR --disable-isend and am still failing both tests.

[mhoff]

Just to confirm: The error message remains the same?

Did you delete all the remainders from your last build and installation? If so, that is clearly a MUSIC bug in combination with python3. I see no connection to SPORE or NEST.

@anandtrex Ideas?

[vevenom]

For me the message when importing music stays the same. @dsalaj can import
music with no problems(but the experiment still fails with same error message).

I will recheck if I have missed anything but I did delete everything from before.

[vevenom]

Ok, after deleting the complete MUSIC repository and recompiling everything with python3.4 again I have
managed to pass all tests. Thank you four your assistance @mhoff .

[mhoff]

Great! I will close this issue then.