Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error compiling Python 3.6 on SL6 #1653

Open
marcovanleeuwen opened this issue May 24, 2019 · 12 comments
Open

Error compiling Python 3.6 on SL6 #1653

marcovanleeuwen opened this issue May 24, 2019 · 12 comments

Comments

@marcovanleeuwen
Copy link
Contributor

Dear all,

I get an error message when aliBuild it trying to compile Python 3.6 on an SL6 machine. We use a Python 2.7 module to run aliBuild; I can also get Python 3.5 via modules, but not 3.6.

The error message in the log file is:

Fatal Python error: Py_Initialize: Unable to get the locale encoding
File "/cm/shared/apps/python2.7/lib/python2.7/encodings/init.py", line 123
raise CodecRegistryError,
^
SyntaxError: invalid syntax

Current thread 0x00002aaaac6b8ce0 (most recent call first):
/data1/software/alisoft/SPECS/slc6_x86-64/Python/v3.6.8-1/Python.sh: line 62: 5294 Aborted env PATH="$INSTALLROOT/bin:$PATH" LD_LIBRARY_PATH="$INSTALLROOT/lib:$LD_LIBRARY_PATH" PYTHONHOME="$INSTALLROOT" pip3 install 'certifi==2019.3.9'


Are there any suggestions for how to address/fix this? Can I ask aliBuild not to compile Python 3.6? The package that I am trying to build at the moment is pythia.sh (PYTHIA8).

Marco.

@marcovanleeuwen
Copy link
Contributor Author

After some googling, I found that this error can be avoided by doing 'unset PYTHONPATH' before invoking alibuild. I am not sure what this does, but in the post that I found it was suggested that that if PYTHONPATH is set (incorrectly?), python may be picking up files from a different python installation.

Is there a way to deal with this in the build recipe? E.g. by unsetting PYTHONPATH or by setting it to point to the new Python install?

@marcovanleeuwen
Copy link
Contributor Author

Dear all,

Someone else also ran into this problem and reported it on the mailing list. I don't know enough about Python to judge whether this simple solution is the proper one. Any comments from the Python afficionados?

Marco.

@ktf
Copy link
Member

ktf commented Jun 5, 2019

What is not clear to me is why do you need python2.7 at all. Can't you simply run aliBuild with python 3?

@marcovanleeuwen
Copy link
Contributor Author

marcovanleeuwen commented Jun 5, 2019

Hi Guilio,

The reported problem was on a Linux system, which I think had python 2.7 as the system version. I don't know who sets the PYTHONPATH there; it may have been a module that was loaded.

On my laptop, a Macbook, I just realised that some of this was due to a virtualenv that I was using on recommendation of Dario, to avoid another issue (which I don't remember).

Python 2.7 seems to be the system version, though:
mvl@macowiec:various $ python -v
# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.py
import site # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.pyc

however, it also looks like the vanilla environment does not have $PYTHONPATH set.

Best regards,

Marco.

@ktf
Copy link
Member

ktf commented Jun 5, 2019

Running in a virtualenv with python2.7 and then trying to use python3 is indeed up to troubles. I myself had to redo my whole laptop setup since we moved to python3 exactly because of this issue. The only solution I've for the moment is making sure that you have aliBuild both in your python2 and python3 environments, to avoid "cross-overs".

@ktf
Copy link
Member

ktf commented Jan 30, 2020

Can you check if #2034 simplifies your setup / works for you?

@marcovanleeuwen
Copy link
Contributor Author

marcovanleeuwen commented Jan 31, 2020

Hi Giuilio,

I tried this, but it does quite work. However, it does work when I unset PYTHONPATH (as in the past); that may be a specific issue on this particular system, where python 2.7 is loaded as a module, and:

echo $PYTHONPATH
/cm/shared/apps/python2.7/lib/python2.7

the error message that I get without unsetting pythonpath is:

/data1/software/alisoft/BUILD/7d9fd197b2480f46ef88122c7bbc6a651322bb2f/Python

env PATH=/data1/software/alisoft/INSTALLROOT/7d9fd197b2480f46ef88122c7bbc6a651322bb2f/slc6_x86-64/Python/v3.6.10-1/bin:/data1/software/alisoft/slc6_x86-64/libffi/v3.2.1-2/bin:/data1/software/alisoft/slc6_x86-64/AliEn-Runtime/v2-19-le-6/bin:/data1/software/alisoft/slc6_x86-64/UUID/v2.27.1-4/bin:/data1/software/alisoft/slc6_x86-64/ApMon-CPP/v2.2.8-alice2-4/bin:/data1/software/alisoft/slc6_x86-64/libtirpc/libtirpc-1-1-4-2/bin:/data1/software/alisoft/slc6_x86-64/AliEn-CAs/v1-4/bin:/data1/software/alisoft/slc6_x86-64/OpenSSL/v1.0.2o-4/bin:/data1/software/alisoft/slc6_x86-64/GCC-Toolchain/v7.3.0-alice1-4/bin:/data1/software/alisoft/slc6_x86-64/autotools/v1.5.0-1/bin:/data1/software/alisoft/slc6_x86-64/defaults-release/v1-2/bin:/nethome/leeuw179/bin:/cm/local/apps/environment-modules/3.2.10/Modules/3.2.10/bin/:/cm/shared/apps/python2.7/bin:/cm/shared/apps/gcc/4.8.1/bin:/cm/shared/apps/sge/2011.11p1/bin/linux-x64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/sbin:/usr/sbin:/nethome/leeuw179/bin LD_LIBRARY_PATH=/data1/software/alisoft/INSTALLROOT/7d9fd197b2480f46ef88122c7bbc6a651322bb2f/slc6_x86-64/Python/v3.6.10-1/lib:/data1/software/alisoft/slc6_x86-64/libffi/v3.2.1-2/lib:/data1/software/alisoft/slc6_x86-64/libffi/v3.2.1-2/lib64:/data1/software/alisoft/slc6_x86-64/AliEn-Runtime/v2-19-le-6/lib:/data1/software/alisoft/slc6_x86-64/UUID/v2.27.1-4/lib:/data1/software/alisoft/slc6_x86-64/ApMon-CPP/v2.2.8-alice2-4/lib:/data1/software/alisoft/slc6_x86-64/libtirpc/libtirpc-1-1-4-2/lib:/data1/software/alisoft/slc6_x86-64/AliEn-CAs/v1-4/lib:/data1/software/alisoft/slc6_x86-64/OpenSSL/v1.0.2o-4/lib:/data1/software/alisoft/slc6_x86-64/GCC-Toolchain/v7.3.0-alice1-4/lib:/data1/software/alisoft/slc6_x86-64/GCC-Toolchain/v7.3.0-alice1-4/lib64:/data1/software/alisoft/slc6_x86-64/autotools/v1.5.0-1/lib:/data1/software/alisoft/slc6_x86-64/defaults-release/v1-2/lib:/cm/shared/apps/gcc/4.8.1/lib:/cm/shared/apps/gcc/4.8.1/lib64:/cm/shared/apps/sge/2011.11p1/lib/linux-x64 PYTHONHOME=/data1/software/alisoft/INSTALLROOT/7d9fd197b2480f46ef88122c7bbc6a651322bb2f/slc6_x86-64/Python/v3.6.10-1 pip3 install certifi==2019.3.9
Fatal Python error: Py_Initialize: Unable to get the locale encoding
File "/cm/shared/apps/python2.7/lib/python2.7/encodings/init.py", line 123
raise CodecRegistryError,
^
SyntaxError: invalid syntax
Current thread 0x00002aaaac6b9ce0 (most recent call first):
/data1/software/alisoft/SPECS/slc6_x86-64/Python/v3.6.10-1/Python.sh: line 66: 10514 Aborted env PATH="$INSTALLROOT/bin:$PATH" LD_LIBRARY_PATH="$INSTALLROOT/lib:$LD_LIBRARY_PATH" PYTHONHOME="$INSTALLROOT" pip3 install 'certifi==2019.3.9'

I will now try with 'unset PYTHONPATH', which has been useful in the past.

On the Mac, I am not using virtualenv from the shell anymore; I can try/play a bit more with in the next days.

Marco.

@marcovanleeuwen
Copy link
Contributor Author

marcovanleeuwen commented Feb 2, 2020

Hi Guilio,

Compilation works now (even without 'unset PYTHONPATH')

but when using alienv enter, I get:

[leeuw179@quark ~]$ alienv enter AliRoot/latest-v5-09-53a-next-root6
Python/v3.6.10-1(33):ERROR:102: Tcl command execution failed: setenv SSL_CERT_FILE [exec python3 -c "import certifi; print(certifi.where())"]

Python/v3.6.10-1(33):ERROR:102: Tcl command execution failed: setenv SSL_CERT_FILE [exec python3 -c "import certifi; print(certifi.where())"]

Marco.

@ktf
Copy link
Member

ktf commented Feb 2, 2020

Can you try again with the latest fix?

@marcovanleeuwen
Copy link
Contributor Author

Hi Giuiio,

This seems to be a step back, now I get this error while compiling:

Fatal Python error: Py_Initialize: Unable to get the locale encoding
File "/cm/shared/apps/python2.7/lib/python2.7/encodings/init.py", line 123
raise CodecRegistryError,
^
SyntaxError: invalid syntax

Current thread 0x00002aaaac6b9ce0 (most recent call first):
/data1/software/alisoft/SPECS/slc6_x86-64/Python/v3.6.10-2/Python.sh: line 66: 13537 Aborted env PATH="$INSTALLROOT/bin:$PATH" LD_LIBRARY_PATH="$INSTALLROOT/lib:$LD_LIBRARY_PATH" PYTHONHOME="$INSTALLROOT" pip3 install 'certifi==2019.3.9'
~

i.e. it seems to be loading some init.py script from the system (or rather from a previously module, which we used to get/install aliBuild itself). I think this means that the PYTHONPATH is not set correctly (maybe because I load the python module in my .bashrc ?)...

Marco.

@marcovanleeuwen
Copy link
Contributor Author

I could compile with 'unset PYTHONPATH', but still get the same error message.

What I noticed is that there is a python3 executable in python-modules. Is that expected?

which python3
/data1/software/alisoft/slc6_x86-64/Python-modules/1.0-9/share/python-modules/bin/python3

@drsh2306
Copy link

SL6 eol 30.11.2020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants