more-itertools dependency broken on python 2.7 #10

dankegel · 2019-03-21T18:43:05Z

On Ubuntu 16.04, using python 2.7.

Installing something via pip that depends on jaraco.functools succeeds, but:

Collecting more-itertools (from jaraco.functools>=1.20->tempora->scancode-toolkit)
  Using cached https://files.pythonhosted.org/packages/dd/f4/cdfbb6f07f767e0cd8a11b16adfecc735b6eb87dbebda026ce8898e1cc22/more-itertools-6.0.0.tar.gz

The resulting installed program fails at runtime, since more-itertools-6 dropped support for python 2.6; here's the symptom:

Traceback (most recent call last):
  File "/home/buildbot/.local/lib/python2.7/site-packages/scancode/cli.py", line 592, in scancode
    plugin.setup(**kwargs)
  File "/home/buildbot/.local/lib/python2.7/site-packages/licensedcode/plugin_license.py", line 118, in setup
    from licensedcode.cache import get_index
  File "/home/buildbot/.local/lib/python2.7/site-packages/licensedcode/cache.py", line 36, in <module>
    import yg.lockfile  # NOQA
  File "/home/buildbot/.local/lib/python2.7/site-packages/yg/lockfile/__init__.py", line 15, in <module>
    from tempora import timing
  File "/home/buildbot/.local/lib/python2.7/site-packages/tempora/timing.py", line 12, in <module>
    import jaraco.functools
  File "/home/buildbot/.local/lib/python2.7/site-packages/jaraco/functools.py", line 27, in <module>
    import more_itertools.recipes
  File "/home/buildbot/.local/lib/python2.7/site-packages/more_itertools/__init__.py", line 1, in <module>
    from more_itertools.more import *  # noqa
  File "/home/buildbot/.local/lib/python2.7/site-packages/more_itertools/more.py", line 329
    def _collate(*iterables, key=lambda a: a, reverse=False):
                               ^
SyntaxError: invalid syntax

The fix is probably to change setup.cfg to be more careful about the more_itertools requirement.

The text was updated successfully, but these errors were encountered:

… 6.0.0; fixes jaraco#10

jaraco · 2019-03-21T19:48:16Z

I suspect the issue you're having is that your pip is old and installing inappropriate versions. When I use pip (via pip-run) to install more_itertools on Python 2.7, I get 5.x:

configparser master $ python2.7 -m pip-run more-itertools                                                                                                                                                
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting more-itertools
  Using cached https://files.pythonhosted.org/packages/2f/9d/dcfe59e213093695f108508af1214cf9cd95cc5489e46877ec5cb56369e5/more_itertools-5.0.0-py2-none-any.whl
Collecting six<2.0.0,>=1.0.0 (from more-itertools)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: six, more-itertools
Successfully installed more-itertools-5.0.0 six-1.12.0
Python 2.7.16 (default, Mar 14 2019, 14:07:40) 
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

I think you need pip 9.x or later to honor the python-requires metadata when discovering versions.

dankegel · 2019-03-21T23:31:43Z

I'm using pip 9.0.1, the one that is bundled with ubuntu 18.04. That's supposed to honor the metadata, as you noted, and usually seems to.

jaraco · 2019-03-21T23:46:49Z

Indeed, it works for me under pip 9.0.1.

~ $ python2.7 -m pip-run pip==9.0.1 -- -m pip-run more-itertools                                                                                                                                         
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting pip==9.0.1
  Downloading https://files.pythonhosted.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 7.4MB/s 
Installing collected packages: pip
Successfully installed pip-9.0.1
Collecting more-itertools
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/2f/9d/dcfe59e213093695f108508af1214cf9cd95cc5489e46877ec5cb56369e5/more_itertools-5.0.0-py2-none-any.whl (52kB)
    100% |████████████████████████████████| 61kB 1.6MB/s 
Collecting six<2.0.0,>=1.0.0 (from more-itertools)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: six, more-itertools
Successfully installed more-itertools-5.0.0 six-1.12.0
You are using pip version 9.0.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Python 2.7.16 (default, Mar 14 2019, 14:07:40) 
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

I wonder what it is about your environment that prevents the metadata from being honored.

jaraco · 2019-03-22T00:04:09Z

I was able to replicate the issue on stock Xenial.

draft $ cat Dockerfile                                                                                                                                                                                    
FROM ubuntu:xenial

RUN apt update -y > /dev/null && apt install -y python-pip > /dev/null
RUN pip install more_itertools
draft $ docker build .                                                                                                                                                                                    
Sending build context to Docker daemon  54.79MB
Step 1/3 : FROM ubuntu:xenial
 ---> 9361ce633ff1
Step 2/3 : RUN apt update -y > /dev/null && apt install -y python-pip > /dev/null
 ---> Running in 19593636bcae

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.


WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

debconf: delaying package configuration, since apt-utils is not installed
Removing intermediate container 19593636bcae
 ---> 935ff8756790
Step 3/3 : RUN pip install more_itertools
 ---> Running in 96d5073d2efc
Collecting more_itertools
  Downloading https://files.pythonhosted.org/packages/dd/f4/cdfbb6f07f767e0cd8a11b16adfecc735b6eb87dbebda026ce8898e1cc22/more-itertools-6.0.0.tar.gz (68kB)
Building wheels for collected packages: more-itertools
  Running setup.py bdist_wheel for more-itertools: started
  Running setup.py bdist_wheel for more-itertools: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/d7/8a/37/026dec21baebd47a0c08276bf3a4be178889e0856bc59fa17f
Successfully built more-itertools
Installing collected packages: more-itertools
Successfully installed more-itertools-6.0.0
You are using pip version 8.1.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Removing intermediate container 96d5073d2efc
 ---> 1c16d5fb1102
Successfully built 1c16d5fb1102

dankegel · 2019-03-22T00:04:58Z

I wuz wrong! This only affects ubuntu 16.04, which has an older pip (8.1.1). Apologies, I'm fighting three or so python problems, and got sloppy.

Thank you for fixing the continuous build, and explaining what's going on.

While Ubuntu 16.04 is still actively used by lots of people, and workarounds for these users are difficult -- they can't just replace the system pip easily -- I agree that babying a weak installer may be seen as somewhat unreasonable.

And I don't even know if the change I proposed would fix the problem on Ubuntu 16.04, if its pip isn't paying attention to metadata...

I'll probably try making the change in scancode-toolkit; it's the wrong place, but at least it'll let me verify the fix without asking you to commit it.

jaraco · 2019-03-22T00:06:55Z

Confirmed that upgrading pip is sufficient to bypass the issue.

draft $ cat Dockerfile                                                                                                                                                                                    
FROM ubuntu:xenial

RUN apt update -y > /dev/null && apt install -y python-pip > /dev/null
RUN pip install -U 'pip < 10'
RUN pip install more_itertools
draft $ docker build .                                                                                                                                                                                    
Sending build context to Docker daemon  54.79MB
Step 1/4 : FROM ubuntu:xenial
 ---> 9361ce633ff1
Step 2/4 : RUN apt update -y > /dev/null && apt install -y python-pip > /dev/null
 ---> Using cache
 ---> 935ff8756790
Step 3/4 : RUN pip install -U 'pip < 10'
 ---> Running in abd51895551c
Collecting pip<10
  Downloading https://files.pythonhosted.org/packages/ac/95/a05b56bb975efa78d3557efa36acaf9cf5d2fd0ee0062060493687432e03/pip-9.0.3-py2.py3-none-any.whl (1.4MB)
Installing collected packages: pip
  Found existing installation: pip 8.1.1
    Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed pip-9.0.3
You are using pip version 9.0.3, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Removing intermediate container abd51895551c
 ---> bbfe6f7cc807
Step 4/4 : RUN pip install more_itertools
 ---> Running in 889c98c3160d
Collecting more_itertools
  Downloading https://files.pythonhosted.org/packages/2f/9d/dcfe59e213093695f108508af1214cf9cd95cc5489e46877ec5cb56369e5/more_itertools-5.0.0-py2-none-any.whl (52kB)
Collecting six<2.0.0,>=1.0.0 (from more_itertools)
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: six, more-itertools
Successfully installed more-itertools-5.0.0 six-1.12.0
You are using pip version 9.0.3, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Removing intermediate container 889c98c3160d
 ---> 9199c0aeb6ec
Successfully built 9199c0aeb6ec

dankegel · 2019-03-22T00:15:11Z

As I said above, upgrading system pip is scary. I tried it, and it broke the system. And if the system pip is installed, you can't run a newer pip, in my recent experience. So whatever fix or workaround I use, it'll be something low-tech that tiptoes around those problems (assuming something else like simplejson/simplejson#245 doesn't force my hand).

jaraco · 2019-03-22T00:42:45Z

I'll probably try making the change in scancode-toolkit;

That does feel worse to me.

And since we're talking about Xenial, maybe it makes sense to have a workaround.

We can pin more_itertools here, but that's only going to work for this package... and implies that the workaround would be to do the same for other packages that depend on more_itertools... and the same would have to happen for every package whose dependency drops support for a Python version.

I really think the only option is to ensure that a later version of pip is available or use another installer technique (install relevant dependencies through a separate requirements.txt).

If permanently installing a suitable pip version isn't suitable, another idea is you could use something like pip-run to temporarily make the necessary version of pip available long enough to get the packages installed (without touching the system pip):

draft $ cat Dockerfile                                                                                                                                                                                    
FROM ubuntu:xenial

RUN apt update -y > /dev/null && apt install -y python-pip > /dev/null
RUN pip install pip-run
RUN pip-run 'pip>9' -- -m pip install more_itertools
draft $ docker build .                                                                                                                                                                                    
Sending build context to Docker daemon  54.79MB
Step 1/4 : FROM ubuntu:xenial
 ---> 9361ce633ff1
Step 2/4 : RUN apt update -y > /dev/null && apt install -y python-pip > /dev/null
 ---> Using cache
 ---> 935ff8756790
Step 3/4 : RUN pip install pip-run
 ---> Running in b8176d10852b
Collecting pip-run
  Downloading https://files.pythonhosted.org/packages/41/1f/1644445eba7b80abebc5b12e0d41cd2ed5120f4d26b8d0ef00bdad17e36b/pip_run-5.3-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): pip in /usr/lib/python2.7/dist-packages (from pip-run)
Installing collected packages: pip-run
Successfully installed pip-run-5.3
You are using pip version 8.1.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Removing intermediate container b8176d10852b
 ---> b8fc3084292c
Step 4/4 : RUN pip-run 'pip>9' -- -m pip install more_itertools
 ---> Running in 11a701d65f12
Collecting pip>9
  Downloading https://files.pythonhosted.org/packages/d8/f3/413bab4ff08e1fc4828dfc59996d721917df8e8583ea85385d51125dceff/pip-19.0.3-py2.py3-none-any.whl (1.4MB)
Installing collected packages: pip
Successfully installed pip-8.1.1
You are using pip version 8.1.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting more_itertools
  Downloading https://files.pythonhosted.org/packages/2f/9d/dcfe59e213093695f108508af1214cf9cd95cc5489e46877ec5cb56369e5/more_itertools-5.0.0-py2-none-any.whl (52kB)
Collecting six<2.0.0,>=1.0.0 (from more_itertools)
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: six, more-itertools
Successfully installed more-itertools-5.0.0 six-1.12.0
Removing intermediate container 11a701d65f12
 ---> 2015f0bebff9
Successfully built 2015f0bebff9

(weird that pip detects itself as 8.1.1 even though clearly it's managed to use 19.0.3)

dankegel · 2019-03-22T00:56:17Z

Oh, what a cesspool python is... I'm looking forward to being able to use pipenv for everything, but I'm not in that world yet, and somehow I suspect pipenv won't solve all the problems.

It's good that python 2.7 finally has an end date, at least.

I'm going to muddle through with bad workarounds, since I'm unwilling to abandon ubuntu 16.04 and 18.04 system python.

Abhishek-Dev09-zz · 2019-03-22T01:56:46Z

@dankegel :on ubuntu,I tried first time on fresh system on py3.6, it working fine .But after (second time) that it says segmentation fault(core dumped).But after installing pipev.It works fine.Last line is omitted that "scancode-tookit installed sucessfully."
http://dpaste.com/1VFKS5W
See similar problem: https://askubuntu.com/questions/467901/segmentation-fault-core-dumped-in-nam-ubuntu-14-04

pombredanne · 2019-03-22T17:47:46Z

@dankegel

Oh, what a cesspool python is...

That's not a fair qualifier IMHO ;) ....

I'm going to muddle through with bad workarounds, since I'm unwilling to abandon ubuntu 16.04 and 18.04 system python.

... as somehow these (the Ubuntu-bundled "system" pip) are the culprit and source of some of your problems, not Python per-se or its libraries. And scancode-toolkit has been designed to work with modern pip (atm it bundles 18.1)

dankegel · 2019-03-22T18:23:37Z

OK, I revise my statement: the current obsession with a rolling update of everything is a cesspool :-)

The impedence mismatch between that and stable, curated collections of software causes lots of tension and hair-pulling. It's frustrating when devotees of everything-up-to-date-all-the-time are unwilling to spare a thought for those in the last-long-term-support-release-must-keep-working world.

But that's life.

jaraco · 2019-08-07T18:31:36Z

I don't think there's anything more to be done here, but feel free to revive the conversation if there is.

dankegel changed the title ~~more-itertools dependency too weak, broken by 6.0.0~~ more-itertools dependency broken on python 2.7 Mar 21, 2019

dankegel added a commit to dankegel/jaraco.functools that referenced this issue Mar 21, 2019

setup.cfg: more_itertools dropped support for python 2.7 with release…

14e415e

… 6.0.0; fixes jaraco#10

This was referenced Mar 21, 2019

setup.cfg: more_itertools dropped support for python 2.7 with release 6.0 #11

Closed

pip install fails on fresh ubuntu 16.04 instance aboutcode-org/scancode-toolkit#1463

Closed

jaraco closed this as completed Aug 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

more-itertools dependency broken on python 2.7 #10

more-itertools dependency broken on python 2.7 #10

dankegel commented Mar 21, 2019

jaraco commented Mar 21, 2019

dankegel commented Mar 21, 2019

jaraco commented Mar 21, 2019

jaraco commented Mar 22, 2019 •

edited

Loading

dankegel commented Mar 22, 2019

jaraco commented Mar 22, 2019

dankegel commented Mar 22, 2019 •

edited

Loading

jaraco commented Mar 22, 2019

dankegel commented Mar 22, 2019

Abhishek-Dev09-zz commented Mar 22, 2019 •

edited

Loading

pombredanne commented Mar 22, 2019

dankegel commented Mar 22, 2019

jaraco commented Aug 7, 2019

more-itertools dependency broken on python 2.7 #10

more-itertools dependency broken on python 2.7 #10

Comments

dankegel commented Mar 21, 2019

jaraco commented Mar 21, 2019

dankegel commented Mar 21, 2019

jaraco commented Mar 21, 2019

jaraco commented Mar 22, 2019 • edited Loading

dankegel commented Mar 22, 2019

jaraco commented Mar 22, 2019

dankegel commented Mar 22, 2019 • edited Loading

jaraco commented Mar 22, 2019

dankegel commented Mar 22, 2019

Abhishek-Dev09-zz commented Mar 22, 2019 • edited Loading

pombredanne commented Mar 22, 2019

dankegel commented Mar 22, 2019

jaraco commented Aug 7, 2019

jaraco commented Mar 22, 2019 •

edited

Loading

dankegel commented Mar 22, 2019 •

edited

Loading

Abhishek-Dev09-zz commented Mar 22, 2019 •

edited

Loading