IndexError occurs when installing qiskit 0.7.2 with pip 19.0.2

[t-imamichi]

Informations

• Qiskit version: 0.7.2
• Python version: 3.6.7
• Operating system: Ubuntu 18.10, macOS High Sierra
• pip version: 19.0.2

What is the current behavior?

IndexError occurs when installing qiskit 0.7.2 with pip 19.0.2. pip <= 19.0.1 does not have this issue.

Steps to reproduce the problem

Run the following commands.

$ python -m venv test
$ source test/bin/activate
$ pip install -U pip wheel setuptools
$ pip install qiskit

Get the following result.

Building wheels for collected packages: qiskit
  Building wheel for qiskit (setup.py) ... done
Exception:
Traceback (most recent call last):
  File "/home/ima/envs/test/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 179, in main
    status = self.run(options, args)
  File "/home/ima/envs/test/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 355, in run
    session=session, autobuilding=True
  File "/home/ima/envs/test/lib/python3.6/site-packages/pip/_internal/wheel.py", line 980, in build
    python_tag=python_tag,
  File "/home/ima/envs/test/lib/python3.6/site-packages/pip/_internal/wheel.py", line 813, in _build_one
    python_tag=python_tag)
  File "/home/ima/envs/test/lib/python3.6/site-packages/pip/_internal/wheel.py", line 821, in _build_one_inside_env
    wheel_path = builder(req, temp_dir.path, python_tag=python_tag)
  File "/home/ima/envs/test/lib/python3.6/site-packages/pip/_internal/wheel.py", line 898, in _build_one_legacy
    return os.path.join(tempd, sorted(os.listdir(tempd))[0])
IndexError: list index out of range

Update(Feb 13): I confirmed that the same errors occurs on mac as well.

Update(Feb 14): I confirmed that pip==19.0.2 causes the error; pip<=19.0.1 has no problem.
pypa/pip@61fd802 seems cause it.

What is the expected behavior?

Install without error.

Suggested solutions

Use pip==19.0.1 for a while. I recommend the following workaround to create a new environment for a while.

$ python -m venv test
$ source test/bin/activate
$ pip install -U pip==19.0.1 wheel setuptools
$ pip install qiskit

[t-imamichi]

Note that I can install qiskit 0.7.1 without error.

$ pip install qiskit==0.7.1
...
Building wheels for collected packages: qiskit
  Building wheel for qiskit (setup.py) ... done
  Stored in directory: /home/ima/.cache/pip/wheels/66/fc/63/aae1a59c6487fa6ca5455725f01fe97211cf95c545dd5bb6fe
Successfully built qiskit
Installing collected packages: qiskit
Successfully installed qiskit-0.7.1

[jaygambetta]

@atilag this is because of aer. Could you look into it

[t-imamichi]

@delapuente I think that it is related to Qiskit/qiskit#45.
If I comment out setup.py#L61, pip install works.
pip install qiskit==0.7.2 does not create tempd directory in wheel.py#L891 because of the override of bdist_wheel. So, os.listdir(tempd) is an empty list in the following backtrace.

  File "/home/ima/envs/test/lib/python3.6/site-packages/pip/_internal/wheel.py", line 898, in _build_one_legacy
    return os.path.join(tempd, sorted(os.listdir(tempd))[0])
IndexError: list index out of range

[t-imamichi]

I found that the cause is the latest version of pip==19.0.2. I can install qiskit==0.7.2 with pip==19.0.1.

[t-imamichi]

I recommend the following workaround to create a new environment for a while. Installing pip==19.0.1 is essential.

$ python -m venv test
$ source test/bin/activate
$ pip install -U pip==19.0.1 wheel setuptools
$ pip install qiskit

Another workaround.

$ python -m venv test
$ source test/bin/activate
$ pip install -U pip wheel setuptools
$ pip install qiskit==0.7.1

[t-imamichi]

I found that this particular commit of pip causes this issue.
pypa/pip@61fd802

[t-imamichi]

If it raises an exception at setup.py#L58, it succeeds the installation at last. But, the traceback looks ugly as follows.

Building wheels for collected packages: qiskit
  Building wheel for qiskit (setup.py) ... error
  Complete output from command /Users/ima/envs/test/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/00/n_n054yn7nb_v2j6vyvgymt40000gn/T/pip-req-build-3jc_42f8/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /private/var/folders/00/n_n054yn7nb_v2j6vyvgymt40000gn/T/pip-wheel-ot6ijw1s --python-tag cp36:
  running bdist_wheel
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/private/var/folders/00/n_n054yn7nb_v2j6vyvgymt40000gn/T/pip-req-build-3jc_42f8/setup.py", line 108, in <module>
      cmdclass=_COMMANDS
    File "/Users/ima/envs/test/lib/python3.6/site-packages/setuptools/__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/private/var/folders/00/n_n054yn7nb_v2j6vyvgymt40000gn/T/pip-req-build-3jc_42f8/setup.py", line 58, in run
      raise Exception('Intentional exception. Do not care')
  Exception: Intentional exception. Do not care
  
  ----------------------------------------
  Failed building wheel for qiskit
  Running setup.py clean for qiskit
Failed to build qiskit
Installing collected packages: qiskit
  Found existing installation: qiskit 0.7.2
    Uninstalling qiskit-0.7.2:
      Successfully uninstalled qiskit-0.7.2
  Running setup.py install for qiskit ... done
Successfully installed qiskit-0.7.2

[ghost]

I confirm this solution is feasible!

[delapuente]

Hi @t-imamichi, thank you for digging into this. I came with the same conclusion you notice at Qiskit/qiskit#78 (comment). I believe there is some way for suppressing the exception but also produccing an error in the pip code allowing the creation of the wheels to be aborted. Let me check and thank you agains for all the thoroughful work.

[t-imamichi]

@delapuente FYI. pip 19.0.3 is also released to fix the issue. I confirmed that pip 19.0.3 can install qiskit 0.7.2 without error as follows.

Building wheels for collected packages: qiskit
  Building wheel for qiskit (setup.py) ... done
  Legacy build of wheel for 'qiskit' created no files.
  Command arguments: ['/Users/ima/envs/test/bin/python', '-u', '-c', "import setuptools, tokenize;__file__='/private/var/folders/00/n_n054yn7nb_v2j6vyvgymt40000gn/T/pip-install-ss7dikz3/qiskit/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\\r\\n', '\\n');f.close();exec(compile(code, __file__, 'exec'))", 'bdist_wheel', '-d', '/private/var/folders/00/n_n054yn7nb_v2j6vyvgymt40000gn/T/pip-wheel-riejgp5q', '--python-tag', 'cp36']
  Command output: [use --verbose to show]
  Running setup.py clean for qiskit
Failed to build qiskit
Installing collected packages: numpy, scipy, marshmallow, psutil, certifi, urllib3, idna, chardet, requests, decorator, networkx, pillow, ply, marshmallow-polyfield, mpmath, sympy, six, pycparser, cffi, asn1crypto, cryptography, ntlm-auth, requests-ntlm, jsonschema, qiskit-terra, qiskit-aer, qiskit
  Running setup.py install for qiskit ... done
Successfully installed asn1crypto-0.24.0 certifi-2018.11.29 cffi-1.12.1 chardet-3.0.4 cryptography-2.5 decorator-4.3.2 idna-2.8 jsonschema-2.6.0 marshmallow-2.18.1 marshmallow-polyfield-3.2 mpmath-1.1.0 networkx-2.2 ntlm-auth-1.2.0 numpy-1.16.1 pillow-5.4.1 ply-3.11 psutil-5.5.1 pycparser-2.19 qiskit-0.7.2 qiskit-aer-0.1.1 qiskit-terra-0.7.0 requests-2.21.0 requests-ntlm-1.1.0 scipy-1.2.1 six-1.12.0 sympy-1.3 urllib3-1.24.1

[mtreinish]

@t-imamichi oh, that's a good point I'd like to revert #119 then if the issue is fixed in pip 19.0.3. The workaround in #119 causes an error during the install and it can be confusing.

[t-imamichi]

I think 0.7.2 still causes an error intentionally as the relase note says https://github.com/Qiskit/qiskit/releases/tag/0.7.2. It is introduced to deal with Qiskit/qiskit#27

[mtreinish]

@t-imamichi right, the workaround in #27 will always be necessary as long as we support migrating from qiskit-terra<0.7.0 to qiskit>=0.7. I'm specifically talking about the wheel building error being printed by pip with #119. #119 was added to workaround the breaking change in behavior that was reverted in pip 19.0.3 so it's no longer needed. #119 introduces a non-fatal error to the build process, for example on my system:

$ test/bin/pip install -U qiskit
Collecting qiskit
  Downloading https://files.pythonhosted.org/packages/37/a9/1268fcc048c29f5c779d3c56873ca449b317e3e35bacdcaca5c8c33a3e20/qiskit-0.7.3.tar.gz

...

Building wheels for collected packages: qiskit
  Building wheel for qiskit (setup.py) ... error
  Complete output from command /tmp/test/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-aal4gb_3/qiskit/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-dh_c10u4 --python-tag cp37:
  running bdist_wheel
  
  ----------------------------------------
  Failed building wheel for qiskit
  Running setup.py clean for qiskit
Failed to build qiskit
Installing collected packages: pycparser, cffi, six, asn1crypto, cryptography, ntlm-auth, certifi, chardet, idna, urllib3, requests, requests-ntlm, mpmath, sympy, ply, marshmallow, marshmallow-polyfield, psutil, pillow, numpy, scipy, decorator, networkx, jsonschema, qiskit-terra, qiskit-aer, qiskit
  Running setup.py install for qiskit ... done
Successfully installed asn1crypto-0.24.0 certifi-2018.11.29 cffi-1.12.1 chardet-3.0.4 cryptography-2.5 decorator-4.3.2 idna-2.8 jsonschema-2.6.0 marshmallow-2.18.1 marshmallow-polyfield-3.2 mpmath-1.1.0 networkx-2.2 ntlm-auth-1.2.0 numpy-1.16.1 pillow-5.4.1 ply-3.11 psutil-5.5.1 pycparser-2.19 qiskit-0.7.3 qiskit-aer-0.1.1 qiskit-terra-0.7.0 requests-2.21.0 requests-ntlm-1.1.0 scipy-1.2.1 six-1.12.0 sympy-1.3 urllib3-1.24.1

This is confusing to users, and I've had several people ask me about it already (Especially because on my system the error is printed in red). Without #119 applied we just get a warning about the wheel build failing, like in your example above or on my local system:

$ test/bin/pip install 'qiskit==0.7.2'
Collecting qiskit
 Using cached https://files.pythonhosted.org/packages/7a/4a/bc00cc835794bb5247146d895586284759fe8e726648cfbb2f977e484bec/qiskit-0.7.2.tar.gz

...

Building wheels for collected packages: qiskit
  Building wheel for qiskit (setup.py) ... done
  Legacy build of wheel for 'qiskit' created no files.
  Command arguments: ['/tmp/test/bin/python', '-u', '-c', "import setuptools, tokenize;__file__='/tmp/pip-install-4aepbgx8/qiskit/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\\r\\n', '\\n');f.close();exec(compile(code, __file__, 'exec'))", 'bdist_wheel', '-d', '/tmp/pip-wheel-1vsytyfg', '--python-tag', 'cp37']
  Command output: [use --verbose to show]
  Running setup.py clean for qiskit
Failed to build qiskit
Installing collected packages: qiskit
  Running setup.py install for qiskit ... done
Successfully installed qiskit-0.7.2

The warning also is yellow on my system, not red, which isn't as scary to users not familiar with the pip internals or why we need to have all these workarounds in place.

[mtreinish]

@t-imamichi Qiskit/qiskit#155 is another example of the user confusion having #119 in place causes.

[t-imamichi]

I agree. The warning message scares users. But, I have no idea to solve the upgrading issue without the warning message.