Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python 2 installs broken with next pyparsing release #313

Closed
AlanCoding opened this issue Jun 8, 2020 · 8 comments
Closed

python 2 installs broken with next pyparsing release #313

AlanCoding opened this issue Jun 8, 2020 · 8 comments

Comments

@AlanCoding
Copy link

On April 5, 2020 your dependency pyparsing released version 3.0.0a1 to PyPI

https://pypi.org/project/pyparsing/#history

You might have guessed this from the version number, but it dropped python2 support

Version 2.4.5 - November, 2019
------------------------------
- NOTE: final release compatible with Python 2.x.

The way I hit this is by trying to do a manual python setup.py install of Ansible on a fresh RHEL7 machine. This results in the error:

[2020-05-30T02:29:51.352Z]     Installed /usr/lib/python2.7/site-packages/ansible_base-2.10.0.dev0-py2.7.egg
[2020-05-30T02:29:51.352Z]     Processing dependencies for ansible-base==2.10.0.dev0
[2020-05-30T02:29:51.352Z]     Searching for packaging
[2020-05-30T02:29:51.352Z]     Reading https://pypi.python.org/simple/packaging/
[2020-05-30T02:29:51.352Z]     Best match: packaging 20.4
[2020-05-30T02:29:51.352Z]     Downloading https://files.pythonhosted.org/packages/55/fd/fc1aca9cf51ed2f2c11748fa797370027babd82f87829c7a8e6dbe720145/packaging-20.4.tar.gz#sha256=4357f74f47b9c12db93624a82154e9b120fa8293699949152b22065d556079f8
[2020-05-30T02:29:51.352Z]     Processing packaging-20.4.tar.gz
[2020-05-30T02:29:51.352Z]     Writing /tmp/easy_install-R9Lrlo/packaging-20.4/setup.cfg
[2020-05-30T02:29:51.352Z]     Running packaging-20.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-R9Lrlo/packaging-20.4/egg-dist-tmp-oofGOu
[2020-05-30T02:29:51.352Z]     Adding packaging 20.4 to easy-install.pth file
[2020-05-30T02:29:51.352Z]   
[2020-05-30T02:29:51.352Z]     Installed /usr/lib/python2.7/site-packages/packaging-20.4-py2.7.egg
[2020-05-30T02:29:51.352Z]     Searching for pyparsing>=2.0.2
[2020-05-30T02:29:51.352Z]     Reading https://pypi.python.org/simple/pyparsing/
[2020-05-30T02:29:51.352Z]     Best match: pyparsing 3.0.0a1
[2020-05-30T02:29:51.352Z]     Downloading https://files.pythonhosted.org/packages/4f/49/f5ef814360ce13ab6eab4e05762ce555ed5bbc169ac941229a3355e9197a/pyparsing-3.0.0a1.tar.gz#sha256=700d17888d441604b0bd51535908dcb297561b040819cccde647a92439db5a2a
[2020-05-30T02:29:51.352Z]     Processing pyparsing-3.0.0a1.tar.gz
[2020-05-30T02:29:51.352Z]     Writing /tmp/easy_install-XgLaMc/pyparsing-3.0.0a1/setup.cfg
[2020-05-30T02:29:51.352Z]     Running pyparsing-3.0.0a1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-XgLaMc/pyparsing-3.0.0a1/egg-dist-tmp-kQzHCn

in stderr

[2020-05-30T02:29:51.053Z]     zip_safe flag not set; analyzing archive contents...
[2020-05-30T02:29:51.053Z]     Traceback (most recent call last):
[2020-05-30T02:29:51.053Z]       File "setup.py", line 338, in <module>
[2020-05-30T02:29:51.053Z]         main()
[2020-05-30T02:29:51.053Z]       File "setup.py", line 333, in main
[2020-05-30T02:29:51.053Z]         setup(**setup_params)
[2020-05-30T02:29:51.053Z]       File "/usr/lib64/python2.7/distutils/core.py", line 152, in setup
[2020-05-30T02:29:51.053Z]         dist.run_commands()
[2020-05-30T02:29:51.053Z]       File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands
[2020-05-30T02:29:51.053Z]         self.run_command(cmd)
[2020-05-30T02:29:51.053Z]       File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
[2020-05-30T02:29:51.053Z]         cmd_obj.run()
[2020-05-30T02:29:51.053Z]       File "/usr/lib/python2.7/site-packages/setuptools/command/install.py", line 73, in run
[2020-05-30T02:29:51.053Z]         self.do_egg_install()
[2020-05-30T02:29:51.053Z]       File "/usr/lib/python2.7/site-packages/setuptools/command/install.py", line 101, in do_egg_install
[2020-05-30T02:29:51.053Z]         cmd.run()
[2020-05-30T02:29:51.053Z]       File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 380, in run
[2020-05-30T02:29:51.053Z]         self.easy_install(spec, not self.no_deps)
[2020-05-30T02:29:51.053Z]       File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 604, in easy_install
[2020-05-30T02:29:51.053Z]         return self.install_item(None, spec, tmpdir, deps, True)
[2020-05-30T02:29:51.053Z]       File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 655, in install_item
[2020-05-30T02:29:51.053Z]         self.process_distribution(spec, dist, deps)
[2020-05-30T02:29:51.053Z]       File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 706, in process_distribution
[2020-05-30T02:29:51.053Z]         [requirement], self.local_index, self.easy_install
[2020-05-30T02:29:51.053Z]       File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 618, in resolve
[2020-05-30T02:29:51.053Z]         dist = best[req.key] = env.best_match(req, self, installer)
[2020-05-30T02:29:51.053Z]       File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 862, in best_match
[2020-05-30T02:29:51.053Z]         return self.obtain(req, installer) # try and download/install
[2020-05-30T02:29:51.053Z]       File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 874, in obtain
[2020-05-30T02:29:51.053Z]         return installer(requirement)
[2020-05-30T02:29:51.053Z]       File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 623, in easy_install
[2020-05-30T02:29:51.053Z]         return self.install_item(spec, dist.location, tmpdir, deps)
[2020-05-30T02:29:51.053Z]       File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 653, in install_item
[2020-05-30T02:29:51.053Z]         dists = self.install_eggs(spec, download, tmpdir)
[2020-05-30T02:29:51.053Z]       File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 849, in install_eggs
[2020-05-30T02:29:51.053Z]         return self.build_and_install(setup_script, setup_base)
[2020-05-30T02:29:51.053Z]       File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1130, in build_and_install
[2020-05-30T02:29:51.053Z]         self.run_setup(setup_script, setup_base, args)
[2020-05-30T02:29:51.053Z]       File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1115, in run_setup
[2020-05-30T02:29:51.053Z]         run_setup(setup_script, args)
[2020-05-30T02:29:51.053Z]       File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 69, in run_setup
[2020-05-30T02:29:51.053Z]         lambda: execfile(
[2020-05-30T02:29:51.053Z]       File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 120, in run
[2020-05-30T02:29:51.053Z]         return func()
[2020-05-30T02:29:51.053Z]       File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 71, in <lambda>
[2020-05-30T02:29:51.053Z]         {'__file__':setup_script, '__name__':'__main__'}
[2020-05-30T02:29:51.053Z]       File "setup.py", line 10, in <module>
[2020-05-30T02:29:51.053Z]   
[2020-05-30T02:29:51.053Z]       File "/tmp/easy_install-XgLaMc/pyparsing-3.0.0a1/pyparsing/__init__.py", line 104, in <module>
[2020-05-30T02:29:51.053Z]       File "/tmp/easy_install-XgLaMc/pyparsing-3.0.0a1/pyparsing/core.py", line 166
[2020-05-30T02:29:51.053Z]         nonlocal found_arity, limit
[2020-05-30T02:29:51.053Z]                            ^
[2020-05-30T02:29:51.053Z]     SyntaxError: invalid syntax

The requirements for the ansible package are:

https://github.com/ansible/ansible/blob/devel/requirements.txt

# Note: this requirements.txt file is used to specify what dependencies are
# needed to make the package run rather than for deployment of a tested set of
# packages.  Thus, this should be the loosest set possible (only required
# packages, not optional ones, and with the widest range of versions that could
# be suitable)
jinja2
PyYAML
cryptography
packaging

This is passed into its setup.py file install_requires option, mostly as-is (it might remove the cryptography dependency but nothing that would matter).

It's totally fine if you want to remove python 2 support #305, but it would be nice to have a final python 2 version that will keep working, and the lack of maximum version pins of dependencies is going to create a big problem for anything trying to keep an environment on life support still working.

Exactly how to do the pin is tricky, the minimum pin is:

install_requires=["pyparsing>=2.0.2", "six"], # Needed to avoid issue #91

You could add an entry with pyparsing<3.0; python_version < '3.0'. That won't work with all versions of pip and setuptools, or you could add code to handle more cases. These decisions are why I am filing an issue and not creating a PR.

@di
Copy link
Member

di commented Jun 8, 2020

Packaging needs to remain compatible with Python 2.7 until pip drops support (pypa/pip#6148), so we should update the dependency to be pyparsing>=2.0.2,<3.0 and add a note in #305 to remove it once that issue is resolved.

@pradyunsg
Copy link
Member

pradyunsg commented Jun 8, 2020

I don't think we need the <3.0 constraint.

pyparsing has correctly declared Requires-Python in their metadata, so any reasonably modern versions of pip will respect this metadata and function properly. For the 3.0.0a0 release, they've set it to >= 3.5 which will exclude Python 2.

$ cat requirements.txt 
jinja2
PyYAML
cryptography
packaging
$ pip install -r requirements.txt --pre                      
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting jinja2
  Using cached Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
Collecting PyYAML
  Using cached PyYAML-5.3.1.tar.gz (269 kB)
Collecting cryptography
  Downloading cryptography-2.9.2-cp27-cp27m-macosx_10_9_x86_64.whl (1.8 MB)
     |████████████████████████████████| 1.8 MB 1.5 MB/s 
Collecting packaging
  Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting MarkupSafe>=0.23
  Using cached MarkupSafe-1.1.1-cp27-cp27m-macosx_10_6_intel.whl (17 kB)
Collecting ipaddress; python_version < "3"
  Using cached ipaddress-1.0.23-py2.py3-none-any.whl (18 kB)
Collecting enum34; python_version < "3"
  Using cached enum34-1.1.10-py2-none-any.whl (11 kB)
Collecting six>=1.4.1
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting cffi!=1.11.3,>=1.8
  Using cached cffi-1.14.0-cp27-cp27m-macosx_10_9_x86_64.whl (173 kB)
Collecting pyparsing>=2.0.2
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pycparser
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: PyYAML
  Building wheel for PyYAML (setup.py) ... done
  Created wheel for PyYAML: filename=PyYAML-5.3.1-cp27-cp27m-macosx_10_14_x86_64.whl size=45648 sha256=8ae54e677c93103d55f4c8d1db4217d9cadda45e5da4218b2749fe75a596501f
  Stored in directory: /Users/pradyunsg/Library/Caches/pip/wheels/d1/d5/a0/3c27cdc8b0209c5fc1385afeee936cf8a71e13d885388b4be2
Successfully built PyYAML
Installing collected packages: MarkupSafe, jinja2, PyYAML, ipaddress, enum34, six, pycparser, cffi, cryptography, pyparsing, packaging
Successfully installed MarkupSafe-1.1.1 PyYAML-5.3.1 cffi-1.14.0 cryptography-2.9.2 enum34-1.1.10 ipaddress-1.0.23 jinja2-2.11.2 packaging-20.4 pycparser-2.20 pyparsing-2.4.7 six-1.15.0

OP is using python setup.py install, which will attempt to install the dependencies using easy_install, which isn't going to respect the Requires-Python metadata. It's also trying to install a pre-release despite that being against the ordering behavior described by PEP 440 (the version standard).

easy_install and setup.py install are both on-their-way-to-deprecation in favor of pip (pypa/setuptools#917, pypa/setuptools#2088 (comment)). I don't think that this is an issue that needs fixing in packaging's dependency declarations.

@AlanCoding
Copy link
Author

yeah, that's a fair point. Their github didn't tag the pre-release the same as the other ones, but PyPI release page reflects that python 2 isn't allowed. I see the argument for wanting the dependency resolver to handle this instead.

@pradyunsg
Copy link
Member

pradyunsg commented Jun 8, 2020

pip's current resolver should handle this just fine TBH (as long as it doesn't see a pyparsing >= 3.0 constraint/requirement basically).

@pradyunsg pradyunsg removed the bug label Jun 8, 2020
@di
Copy link
Member

di commented Jun 8, 2020

Ah, good catch @pradyunsg. Seems like there's nothing we need to do here then.

@pradyunsg
Copy link
Member

Thanks for filing this issue @AlanCoding! ^.^

@ptmcg
Copy link

ptmcg commented Jun 14, 2020

I am about to push out pyparsing 3.0.0a2 as another pre-release. Let me know if there are any additional steps I need to take. (The most recent Py2-compatible version is 2.4.7, released last December in April 2020.)

mattclay added a commit to mattclay/ansible that referenced this issue Jun 15, 2020
The upcoming pyparsing 3 release will require Python 3.5 or later, see:

pypa/packaging#313

Unfortunately pip 8.x and earlier versions do not support python version requirements, which is why this constraint is needed.
mattclay added a commit to ansible/ansible that referenced this issue Jun 16, 2020
The upcoming pyparsing 3 release will require Python 3.5 or later, see:

pypa/packaging#313

Unfortunately pip 8.x and earlier versions do not support python version requirements, which is why this constraint is needed.
@pradyunsg
Copy link
Member

pradyunsg commented Jun 22, 2020

@ptmcg Nothing needed -- I think w.r.t. this issue, pyparsing doesn't really need to change any of the declared metadata. All's good on pyparsing's side. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants