TestDistutilsScheme.test_distutils_config_file_read broken in python 2.7 with tox using virtualenv 20.0.6

[xavfernandez]

With previous tox using virtualenv 20.0.5 (during py27 setup)

$ tox -e py27 -- -k test_distutils_config_file_read
GLOB sdist-make: /home/xafer/other_dev/pip/setup.py
py27 inst-nodeps: /home/xafer/other_dev/pip/.tox/.tmp/package/1/pip-20.1.dev0.zip
py27 installed: DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support,apipkg==1.5,atomicwrites==1.3.0,attrs==19.3.0,cffi==1.14.0,configparser==4.0.2,contextlib2==0.6.0.post1,coverage==5.0.3,cryptography==2.8,csv23==0.1.6,enum34==1.1.9,execnet==1.7.1,freezegun==0.3.15,funcsigs==1.0.2,importlib-metadata==1.5.0,ipaddress==1.0.23,mock==3.0.5,more-itertools==5.0.0,pathlib2==2.3.5,pip==20.1.dev0,pluggy==0.13.1,pretend==1.0.9,py==1.8.1,pycparser==2.19,pytest==3.8.2,pytest-cov==2.8.1,pytest-forked==1.1.3,pytest-rerunfailures==6.0,pytest-timeout==1.3.4,pytest-xdist==1.27.0,python-dateutil==2.8.1,PyYAML==5.3,scandir==1.10.0,scripttest==1.3,setuptools==44.0.0,six==1.14.0,virtualenv==16.7.10,Werkzeug==0.16.0,wheel==0.34.2,zipp==1.2.0
py27 run-test-pre: PYTHONHASHSEED='3646811570'
py27 run-test-pre: commands[0] | python -c 'import shutil, sys; shutil.rmtree(sys.argv[1], ignore_errors=True)' /home/xafer/other_dev/pip/tests/data/common_wheels
py27 run-test-pre: commands[1] | python /home/xafer/other_dev/pip/tools/tox_pip.py wheel -w /home/xafer/other_dev/pip/tests/data/common_wheels -r /home/xafer/other_dev/pip/tools/requirements/tests-common_wheels.txt
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Looking in indexes: https://xfernandez:****@pypi.polydev.blue/root/ci/+simple/
Collecting setuptools>=40.8.0
  Downloading https://pypi.polydev.blue/root/pypi/%2Bf/180/081a244d0888b/setuptools-44.0.0-py2.py3-none-any.whl (583 kB)
     |████████████████████████████████| 583 kB 523 kB/s
  Saved ./tests/data/common_wheels/setuptools-44.0.0-py2.py3-none-any.whl
Collecting wheel
  Using cached https://pypi.polydev.blue/root/pypi/%2Bf/df2/77cb51e61359a/wheel-0.34.2-py2.py3-none-any.whl (26 kB)
  Saved ./tests/data/common_wheels/wheel-0.34.2-py2.py3-none-any.whl
Skipping setuptools, due to already being wheel.
Skipping wheel, due to already being wheel.
py27 run-test: commands[0] | pytest --timeout 300 -k test_distutils_config_file_read
============================================================================================================ test session starts =============================================================================================================
platform linux2 -- Python 2.7.17, pytest-3.8.2, py-1.8.1, pluggy-0.13.1
rootdir: /home/xafer/other_dev/pip, inifile: setup.cfg
plugins: timeout-1.3.4, cov-2.8.1, rerunfailures-6.0, xdist-1.27.0, forked-1.1.3
timeout: 300.0s
timeout method: signal
timeout func_only: False
collected 1838 items / 1837 deselected

tests/unit/test_locations.py .                                                                                                                                                                                                         [100%]

================================================================================================= 1 passed, 1837 deselected in 1.63 seconds ==================================================================================================
__________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________
  py27: commands succeeded
  congratulations :)

while with a freshly installed tox (with virtualenv 20.0.6 at py27 setup):

$ tox -e py27 -- -k test_distutils_config_file_read                                              
GLOB sdist-make: /home/xafer/other_dev/pip/setup.py
py27 create: /home/xafer/other_dev/pip/.tox/py27
py27 installdeps: -r/home/xafer/other_dev/pip/tools/requirements/tests.txt
py27 inst: /home/xafer/other_dev/pip/.tox/.tmp/package/1/pip-20.1.dev0.zip
py27 installed: DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Pytho
n 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support,apipkg==1.5,atomicwrites==1.3.0,attrs==19.3.0,cffi==1.14.0,configparser==4.0.2,contextlib2==0.6.0.post1,coverage==5.0.3,crypto
graphy==2.8,csv23==0.1.6,enum34==1.1.9,execnet==1.7.1,freezegun==0.3.15,funcsigs==1.0.2,importlib-metadata==1.5.0,ipaddress==1.0.23,mock==3.0.5,more-itertools==5.0.0,pathlib2==2.3.5,pip==20.1.dev0,pluggy==0.13.1,pretend==1.0.9,py==1.8.1,p
ycparser==2.19,pytest==3.8.2,pytest-cov==2.8.1,pytest-forked==1.1.3,pytest-rerunfailures==6.0,pytest-timeout==1.3.4,pytest-xdist==1.27.0,python-dateutil==2.8.1,PyYAML==5.3,scandir==1.10.0,scripttest==1.3,setuptools==44.0.0,six==1.14.0,vir
tualenv==16.7.10,Werkzeug==0.16.0,wheel==0.34.2,zipp==1.2.0 
py27 run-test-pre: PYTHONHASHSEED='1984603763'
py27 run-test-pre: commands[0] | python -c 'import shutil, sys; shutil.rmtree(sys.argv[1], ignore_errors=True)' /home/xafer/other_dev/pip/tests/data/common_wheels
py27 run-test-pre: commands[1] | python /home/xafer/other_dev/pip/tools/tox_pip.py wheel -w /home/xafer/other_dev/pip/tests/data/common_wheels -r /home/xafer/other_dev/pip/tools/requirements/tests-common_wheels.txt                        
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in p
ip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support                                                                                                                                               
Looking in indexes: https://xfernandez:****@pypi.polydev.blue/root/ci/+simple/                                                                                                                                                                
Collecting setuptools>=40.8.0                                                                                                                                                                                                                 
  Downloading https://pypi.polydev.blue/root/pypi/%2Bf/180/081a244d0888b/setuptools-44.0.0-py2.py3-none-any.whl (583 kB)
     |████████████████████████████████| 583 kB 638 kB/s 
  Saved ./tests/data/common_wheels/setuptools-44.0.0-py2.py3-none-any.whl
Collecting wheel         
  Using cached https://pypi.polydev.blue/root/pypi/%2Bf/df2/77cb51e61359a/wheel-0.34.2-py2.py3-none-any.whl (26 kB)
  Saved ./tests/data/common_wheels/wheel-0.34.2-py2.py3-none-any.whl
Skipping setuptools, due to already being wheel.
Skipping wheel, due to already being wheel.
py27 run-test: commands[0] | pytest --timeout 300 -k test_distutils_config_file_read
============================================================================================================ test session starts =============================================================================================================
platform linux2 -- Python 2.7.17, pytest-3.8.2, py-1.8.1, pluggy-0.13.1
rootdir: /home/xafer/other_dev/pip, inifile: setup.cfg
plugins: timeout-1.3.4, cov-2.8.1, rerunfailures-6.0, xdist-1.27.0, forked-1.1.3
timeout: 300.0s
timeout method: signal
timeout func_only: False
collected 1838 items / 1837 deselected                                                                                                                                                                                                       

tests/unit/test_locations.py F                                                                                                                                                                                                         [100%]

================================================================================================================== FAILURES ==================================================================================================================
____________________________________________________________________________________________ TestDistutilsScheme.test_distutils_config_file_read _____________________________________________________________________________________________

self = <tests.unit.test_locations.TestDistutilsScheme instance at 0x7f420583cc80>, tmpdir = Path('/tmp/pytest-of-xafer/pytest-0/test_distutils_config_file_rea0'), monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f4204f97190>

    @pytest.mark.incompatible_with_venv
    def test_distutils_config_file_read(self, tmpdir, monkeypatch):
        # This deals with nt/posix path differences
        install_scripts = os.path.normcase(os.path.abspath(
            os.path.join(os.path.sep, 'somewhere', 'else')))
        f = tmpdir / "config" / "setup.cfg"
        f.parent.mkdir()
        f.write_text("[install]\ninstall-scripts=" + install_scripts)
        from distutils.dist import Distribution
        # patch the function that returns what config files are present
        monkeypatch.setattr(
            Distribution,
            'find_config_files',
            lambda self: [f],
        )
        scheme = distutils_scheme('example')
>       assert scheme['scripts'] == install_scripts
E       AssertionError: assert '/home/xafer/....tox/py27/bin' == '/somewhere/else'
E         - /home/xafer/other_dev/pip/.tox/py27/bin
E         + /somewhere/else

tests/unit/test_locations.py:110: AssertionError
========================================================================================================== short test summary info ===========================================================================================================
FAIL tests/unit/test_locations.py::TestDistutilsScheme::()::test_distutils_config_file_read
================================================================================================= 1 failed, 1837 deselected in 1.72 seconds ==================================================================================================
ERROR: InvocationError for command /home/xafer/other_dev/pip/.tox/py27/bin/pytest --timeout 300 -k test_distutils_config_file_read (exited with code 1)
__________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________
ERROR:   py27: commands failed

This likely comes from pypa/virtualenv@9201422

[xavfernandez]

I'm tempted to drop the test in Python 2 also (in Python 3 they are currently skipped because Incompatible with venv).

Ideally this test test_distutils_config_file_read should be moved to some kind of Docker based test not using a venv/virtualenv as suggested in #6712.

[xavfernandez]

Any opinion @pypa/pip-committers ?
An other solution would be to patch parse_config_files instead of find_config_files in

pip/tests/unit/test_locations.py

Lines 94 to 110 in 6d86a04

	@pytest.mark.incompatible_with_venv
	def test_distutils_config_file_read(self, tmpdir, monkeypatch):
	# This deals with nt/posix path differences
	install_scripts = os.path.normcase(os.path.abspath(
	os.path.join(os.path.sep, 'somewhere', 'else')))
	f = tmpdir / "config" / "setup.cfg"
	f.parent.mkdir()
	f.write_text("[install]\ninstall-scripts=" + install_scripts)
	from distutils.dist import Distribution
	# patch the function that returns what config files are present
	monkeypatch.setattr(
	Distribution,
	'find_config_files',
	lambda self: [f],
	)
	scheme = distutils_scheme('example')
	assert scheme['scripts'] == install_scripts

like virtualenv is doing.

[uranusjr]

The new virtualenv are really more similar to venv than virtualenv<20 in many ways, contributing a lot of roadblocks to #7718. We probably need to revisit all incompatible_with_venv and incompatible_with_test_venv tests (including to remove or entirely reimplement them).

[pfmoore]

Agreed. Any test that's marked as incompatible_with_[test_]venv is basically saying "relies on details of how virtualenv works". Those details have all changed with the virtualenv rewrite, and we can expect issues in any or all of those tests.

Pinning to the old virtualenv (as #7798 does) is probably the best immediate solution. But longer term we need to find a way of re-implementing the intent of the marked tests, or accept that we can't test the behaviour on a case by case basis.