pip install fails on fresh ubuntu 16.04 instance

[dankegel]

While pip is not the main install method preferred by the scancode authors,
it's still an important mainstream way to distribute python code, so we should
keep it working.

Trying it on a fresh ubuntu 16.04 instance fails at the moment, e.g.

$ lxc launch ubuntu:16.04 ubu1604-demo
$ lxc exec ubu1604-demo bash
# apt update
# apt install python-pip
# su ubuntu
$ cd
$ pip install --user scancode-toolkit
...
Collecting pyicu>=1.9.3 (from normality>=0.4.0->fingerprints==0.5.4->scancode-toolkit)
...
    RuntimeError:
    Please set the ICU_VERSION environment variable to the version of
    ICU you have installed.

i.e. there's a missing native package that can't be installed by pip. That's easy to work around:

$ sudo apt install libicu-dev
$ pip install --user scancode-toolkit
...
Collecting more-itertools (from jaraco.functools>=1.20->tempora->scancode-toolkit)
  Using cached https://files.pythonhosted.org/packages/dd/f4/cdfbb6f07f767e0cd8a11b16adfecc735b6eb87dbebda026ce8898e1cc22/more-itertools-6.0.0.tar.gz
...

That succeeds, but note:

$ pip list | grep itertools
more-itertools (6.0.0)

That's not going to end well, since more-itertools dropped support for python 2.7,
and indeed:

$ .local/bin/scancode --json-pp - --license --package .
Setup plugins...
ERROR: failed to setup plugin: scan:licenses:
Traceback (most recent call last):
...
  File "/home/ubuntu/.local/lib/python2.7/site-packages/jaraco/functools.py", line 27, in <module>
    import more_itertools.recipes
  File "/home/ubuntu/.local/lib/python2.7/site-packages/more_itertools/__init__.py", line 1, in <module>
    from more_itertools.more import *  # noqa
  File "/home/ubuntu/.local/lib/python2.7/site-packages/more_itertools/more.py", line 329
    def _collate(*iterables, key=lambda a: a, reverse=False):
                               ^
SyntaxError: invalid syntax

Filed jaraco/jaraco.functools#10 and
jaraco/jaraco.functools#11 for that.

That's also not too hard to work around:

$ pip uninstall more-itertools
$ pip install 'more-itertools<6.0.0'
$ .local/bin/scancode --json-pp - --license --package .

And that goes swimmingly, using up 1.3 gigabytes of RAM as expected :-)

So, the proper action is probably to wait a while for jaraco.functools to push a fixed version to pypi. If they don't, we could paper over the problem by adding a dependency spec in scancode's setup.{cfg,py}.

Oh, and maybe we could have a line of code to hint to users they need to install native libicu-dev, written generally enough to apply regardless of distro.

[pombredanne]

@dankegel thank you ++ for this detailed report and sorry for that. I guess this is why I like bundled deps ... that said this needs to be fixed alright. FWIW the icu-dev issue is fairly new and there is a bug reported there. pudo/normality#6 (comment)
If we cannot get @pudo to allow some option for this to be optional, I guess we can maintain a patched version alright. I am not looking forward to get icu as a dependency

[dankegel]

jarco replied, and pointed out that ubuntu 16.04's pip ignores metadata that would avoid the problem.
Ubuntu 18.04 and newer don't run into this problem... and he doesn't have much sympathy for people using the system pip. So potentially it's worth adding the requirement in to scancode's setup.py.

[Abhishek-Dev09-zz]

@dankegel :on ubuntu,I tried first time on fresh system on py3.6, it working fine .But after (second time) that it says segmentation fault(core dumped).But after installing pipev.It works fine.Last line is omitted that "scancode-tookit installed sucessfully."
http://dpaste.com/1VFKS5W
See similar problem: https://askubuntu.com/questions/467901/segmentation-fault-core-dumped-in-nam-ubuntu-14-04

[dankegel]

The core dump is because the system pip in ubuntu 18.04 and earlier interacts poorly with simplejson, see #1464

[dankegel]

I forgot to mention one other failure symptom on Ubuntu 18.04:

ImportError: cannot import name os

This is caused by having an old configparser installed, see
PiDelport/backports.os#11
and jaraco/configparser#17

If this happens to you, and you're on Ubuntu 18.04, you can get a fixed configparser from https://launchpad.net/~dank/+archive/ubuntu/python-fixes/

[pombredanne]

@dankegel GitHub autoclosed this because of your comments in #1485 but this is likely inappropriate. I am reopening

[dankegel]

yeah, this bug is pretty broad. I should have written a couple tighter bugs with just one failure mode each.

[pombredanne]

@dankegel that's OK. Eventually that's the issue that need to be ultimately solved!

[pombredanne]

The supported OS/Python versions have changed quite a bit and I think this is no longer relevant.
Install runs fine on a fresh Ubuntu 16, 18 and 20 and these are all part of the test suites now.
Closing at last!