4.0.2: test suite is failing in two units

[kloczek]

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix>
• run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-twine-4.0.2-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-twine-4.0.2-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ xvfb-run -a /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/twine-4.0.2, configfile: pytest.ini
plugins: hypothesis-6.61.0, services-2.2.1, socket-0.5.1
collected 190 items

tests/test_auth.py ...................                                                                                                                               [ 10%]
tests/test_check.py .............                                                                                                                                    [ 16%]
tests/test_cli.py ..                                                                                                                                                 [ 17%]
tests/test_commands.py .....                                                                                                                                         [ 20%]
tests/test_integration.py EE........                                                                                                                                 [ 25%]
tests/test_main.py ...                                                                                                                                               [ 27%]
tests/test_package.py .........................                                                                                                                      [ 40%]
tests/test_register.py ....                                                                                                                                          [ 42%]
tests/test_repository.py .........................                                                                                                                   [ 55%]
tests/test_settings.py ..............                                                                                                                                [ 63%]
tests/test_upload.py .............................                                                                                                                   [ 78%]
tests/test_utils.py ...............................                                                                                                                  [ 94%]
tests/test_wheel.py ..........                                                                                                                                       [100%]

================================================================================== ERRORS ==================================================================================
____________________________________________________________________ ERROR at setup of test_pypi_upload ____________________________________________________________________

tmp_path_factory = TempPathFactory(_given_basetemp=None, _trace=<pluggy._tracing.TagTracerSub object at 0x7fb2914a8340>, _basetemp=PosixPath('/tmp/pytest-of-tkloczko/pytest-729'))

    @pytest.fixture(scope="session")
    def sampleproject_dist(tmp_path_factory):
        checkout = tmp_path_factory.mktemp("sampleproject", numbered=False)
        subprocess.run(
            ["git", "clone", "https://github.com/pypa/sampleproject", str(checkout)]
        )
>       with (checkout / "setup.py").open("r+") as setup:

tests/test_integration.py:31:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib64/python3.8/pathlib.py:1222: in open
    return io.open(self, mode, buffering, encoding, errors, newline,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = PosixPath('/tmp/pytest-of-tkloczko/pytest-729/sampleproject/setup.py'), name = '/tmp/pytest-of-tkloczko/pytest-729/sampleproject/setup.py', flags = 524290
mode = 438

    def _opener(self, name, flags, mode=0o666):
        # A stub for the opener argument to built-in open()
>       return self._accessor.open(self, flags, mode)
E       FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-tkloczko/pytest-729/sampleproject/setup.py'

/usr/lib64/python3.8/pathlib.py:1078: FileNotFoundError
-------------------------------------------------------------------------- Captured stderr setup ---------------------------------------------------------------------------
Cloning into '/tmp/pytest-of-tkloczko/pytest-729/sampleproject'...
____________________________________________________________________ ERROR at setup of test_pypi_error _____________________________________________________________________

tmp_path_factory = TempPathFactory(_given_basetemp=None, _trace=<pluggy._tracing.TagTracerSub object at 0x7fb2914a8340>, _basetemp=PosixPath('/tmp/pytest-of-tkloczko/pytest-729'))

    @pytest.fixture(scope="session")
    def sampleproject_dist(tmp_path_factory):
        checkout = tmp_path_factory.mktemp("sampleproject", numbered=False)
        subprocess.run(
            ["git", "clone", "https://github.com/pypa/sampleproject", str(checkout)]
        )
>       with (checkout / "setup.py").open("r+") as setup:

tests/test_integration.py:31:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib64/python3.8/pathlib.py:1222: in open
    return io.open(self, mode, buffering, encoding, errors, newline,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = PosixPath('/tmp/pytest-of-tkloczko/pytest-729/sampleproject/setup.py'), name = '/tmp/pytest-of-tkloczko/pytest-729/sampleproject/setup.py', flags = 524290
mode = 438

    def _opener(self, name, flags, mode=0o666):
        # A stub for the opener argument to built-in open()
>       return self._accessor.open(self, flags, mode)
E       FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-tkloczko/pytest-729/sampleproject/setup.py'

/usr/lib64/python3.8/pathlib.py:1078: FileNotFoundError
============================================================================= warnings summary =============================================================================
tests/test_integration.py:21
  /home/tkloczko/rpmbuild/BUILD/twine-4.0.2/tests/test_integration.py:21: PytestUnknownMarkWarning: Unknown pytest.mark.flaky - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    pytest.mark.flaky(reruns=3, reruns_delay=1),

tests/test_check.py::TestWarningStream::test_write_match
  /usr/lib/python3.8/site-packages/_pytest/fixtures.py:900: PytestRemovedIn8Warning: Support for nose tests is deprecated and will be removed in a future release.
  tests/test_check.py::TestWarningStream::test_write_match is using nose-specific method: `setup(self)`
  To remove this warning, rename it to `setup_method(self)`
  See docs: https://docs.pytest.org/en/stable/deprecations.html#support-for-tests-written-for-nose
    fixture_result = next(generator)

tests/test_check.py::TestWarningStream::test_write_nomatch
  /usr/lib/python3.8/site-packages/_pytest/fixtures.py:900: PytestRemovedIn8Warning: Support for nose tests is deprecated and will be removed in a future release.
  tests/test_check.py::TestWarningStream::test_write_nomatch is using nose-specific method: `setup(self)`
  To remove this warning, rename it to `setup_method(self)`
  See docs: https://docs.pytest.org/en/stable/deprecations.html#support-for-tests-written-for-nose
    fixture_result = next(generator)

tests/test_check.py::TestWarningStream::test_str_representation
  /usr/lib/python3.8/site-packages/_pytest/fixtures.py:900: PytestRemovedIn8Warning: Support for nose tests is deprecated and will be removed in a future release.
  tests/test_check.py::TestWarningStream::test_str_representation is using nose-specific method: `setup(self)`
  To remove this warning, rename it to `setup_method(self)`
  See docs: https://docs.pytest.org/en/stable/deprecations.html#support-for-tests-written-for-nose
    fixture_result = next(generator)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================================= short test summary info ==========================================================================
ERROR tests/test_integration.py::test_pypi_upload - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-tkloczko/pytest-729/sampleproject/setup.py'
ERROR tests/test_integration.py::test_pypi_error - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-tkloczko/pytest-729/sampleproject/setup.py'
=========================================================== 188 passed, 4 warnings, 2 errors in 92.95s (0:01:32) ===========================================================

Here is list of installed modules in build env

Package                       Version
----------------------------- -----------------
alabaster                     0.7.12
appdirs                       1.4.4
asn1crypto                    1.5.1
asttokens                     2.2.0
attrs                         22.1.0
Babel                         2.11.0
backcall                      0.2.0
bcrypt                        3.2.2
beautifulsoup4                4.11.1
bleach                        5.0.0
Brlapi                        0.8.3
build                         0.9.0
cffi                          1.15.1
charset-normalizer            3.0.1
colorama                      0.4.6
commonmark                    0.9.1
contourpy                     1.0.6
cryptography                  38.0.4
cssselect                     1.1.0
cycler                        0.11.0
decorator                     5.1.1
defusedxml                    0.7.1
distlib                       0.3.6
distro                        1.8.0
dnspython                     2.2.1
docutils                      0.19
entrypoints                   0.4
exceptiongroup                1.0.0
executing                     1.2.0
extras                        1.0.0
fastjsonschema                2.16.1
filelock                      3.8.2
fixtures                      4.0.0
fonttools                     4.38.0
gpg                           1.18.0-unknown
html5lib                      1.1
idna                          3.4
imagesize                     1.4.1
importlib-metadata            5.1.0
importlib-resources           5.10.1
iniconfig                     1.1.1
jaraco.classes                3.2.3
jaraco.envs                   2.4.0
jaraco.functools              3.5.2
jedi                          0.18.2
jeepney                       0.8.0
Jinja2                        3.1.2
jsonschema                    4.17.3
jupyter_client                7.4.8
jupyter_core                  5.1.0
jupyterlab-pygments           0.1.2
keyring                       23.11.0
kiwisolver                    1.4.4
libcomps                      0.1.19
louis                         3.24.0
lxml                          4.9.1
MarkupSafe                    2.1.1
matplotlib                    3.6.2
matplotlib-inline             0.1.6
mistune                       2.0.4
more-itertools                9.0.0
munch                         2.5.0
mypy-extensions               0.4.3
nbclient                      0.7.2
nbformat                      5.7.0
nest-asyncio                  1.5.6
numpy                         1.23.1
olefile                       0.46
packaging                     21.3
pandocfilters                 1.5.0
parso                         0.8.3
path                          16.6.0
pbr                           5.9.0
pep517                        0.13.0
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        9.3.0
pip                           22.3.1
pkginfo                       1.9.2
pkgutil_resolve_name          1.3.10
platformdirs                  2.6.0
pluggy                        1.0.0
ply                           3.11
portend                       3.1.0
pretend                       1.0.9
prompt-toolkit                3.0.29
psutil                        5.9.2
ptyprocess                    0.7.0
pure-eval                     0.2.2
py                            1.11.0
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pycparser                     2.21
Pygments                      2.13.0
PyGObject                     3.42.2
pyparsing                     3.0.9
pyrsistent                    0.19.2
pytest                        7.2.0
pytest-services               2.2.1
pytest-socket                 0.5.1
python-dateutil               2.8.2
pytz                          2022.4
PyYAML                        6.0
pyzmq                         24.0.0
readme-renderer               37.3
requests                      2.28.1
requests-toolbelt             0.10.1
rfc3986                       2.0.0
rich                          12.6.0
rpm                           4.17.0
scour                         0.38.2
SecretStorage                 3.3.2
setuptools                    65.6.3
setuptools-scm                7.0.5
six                           1.16.0
snowballstemmer               2.2.0
sortedcontainers              2.4.0
soupsieve                     2.3.2.post1
Sphinx                        5.3.0
sphinxcontrib-applehelp       1.0.2.dev20221204
sphinxcontrib-devhelp         1.0.2.dev20221204
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1.dev20221204
sphinxcontrib-programoutput   0.17
sphinxcontrib-qthelp          1.0.3.dev20221204
sphinxcontrib-serializinghtml 1.1.5
stack-data                    0.6.2
tempora                       5.1.0
testtools                     2.5.0
tinycss2                      1.2.1
tomli                         2.0.1
tornado                       6.2
tox                           3.26.0
tpm2-pkcs11-tools             1.33.7
tpm2-pytss                    1.1.0
tqdm                          4.64.1
traitlets                     5.7.1
typing_extensions             4.4.0
typing-inspect                0.7.1
urllib3                       1.26.12
virtualenv                    20.17.1
wcwidth                       0.2.5
webencodings                  0.5.1
wheel                         0.38.4
zc.lockfile                   2.0
zipp                          3.11.0

[bhrutledge]

This is a duplicate of #911, which went into depth on how the tests are meant to be run, and why it's okay to skip the integration tests.

That said, I think the particular error that you're getting re: setup.py has been resolved in main via #947. I haven't made a release yet because it's only test code.

[kloczek]

Indeed after apply patches from master pytest is now OK

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-twine-4.0.2-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-twine-4.0.2-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ xvfb-run -a /usr/bin/pytest -ra -m 'not network'
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/twine-4.0.2, configfile: pytest.ini
plugins: services-2.2.1, socket-0.5.1
collected 181 items

tests/test_auth.py ...................                                                                                                                                                [ 10%]
tests/test_check.py .............                                                                                                                                                     [ 17%]
tests/test_cli.py ..                                                                                                                                                                  [ 18%]
tests/test_commands.py .....                                                                                                                                                          [ 21%]
tests/test_main.py ...                                                                                                                                                                [ 23%]
tests/test_package.py .........................                                                                                                                                       [ 37%]
tests/test_register.py ....                                                                                                                                                           [ 39%]
tests/test_repository.py .........................                                                                                                                                    [ 53%]
tests/test_settings.py ..............                                                                                                                                                 [ 60%]
tests/test_upload.py ..............................                                                                                                                                   [ 77%]
tests/test_utils.py ...............................                                                                                                                                   [ 94%]
tests/test_wheel.py ..........                                                                                                                                                        [100%]

==================================================================================== 181 passed in 4.88s ====================================================================================
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-05f8ce49-87c4-4fe9-9a53-bb64483c1cc4/test_rmtree_errorhandler_rerai0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_rmtree_errorhandler_rerai0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-05f8ce49-87c4-4fe9-9a53-bb64483c1cc4/test_rmtree_errorhandler_reado0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_rmtree_errorhandler_reado0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-05f8ce49-87c4-4fe9-9a53-bb64483c1cc4/test_safe_delete_no_perms0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_safe_delete_no_perms0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-05f8ce49-87c4-4fe9-9a53-bb64483c1cc4/test_safe_set_no_perms0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_safe_set_no_perms0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-05f8ce49-87c4-4fe9-9a53-bb64483c1cc4/test_safe_get_no_perms0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_safe_get_no_perms0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-05f8ce49-87c4-4fe9-9a53-bb64483c1cc4
<class 'OSError'>: [Errno 39] Directory not empty: '/tmp/pytest-of-tkloczko/garbage-05f8ce49-87c4-4fe9-9a53-bb64483c1cc4'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-062af784-9d84-4d25-9b85-1e4838790f31/test_rmtree_errorhandler_rerai0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_rmtree_errorhandler_rerai0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-062af784-9d84-4d25-9b85-1e4838790f31/test_rmtree_errorhandler_reado0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_rmtree_errorhandler_reado0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-062af784-9d84-4d25-9b85-1e4838790f31/test_safe_delete_no_perms0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_safe_delete_no_perms0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-062af784-9d84-4d25-9b85-1e4838790f31/test_safe_set_no_perms0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_safe_set_no_perms0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-062af784-9d84-4d25-9b85-1e4838790f31/test_safe_get_no_perms0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_safe_get_no_perms0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-062af784-9d84-4d25-9b85-1e4838790f31
<class 'OSError'>: [Errno 39] Directory not empty: '/tmp/pytest-of-tkloczko/garbage-062af784-9d84-4d25-9b85-1e4838790f31'
  warnings.warn(

Thank you 👍

[kloczek]

BTW .. do you know why those errors about not empty directories are reported? 🤔

[bhrutledge]

BTW .. do you know why those errors about not empty directories are reported? 🤔

No; I don't see those warnings when running locally. That said, it looks like they're referring to tests that aren't from this project, e.g. test_safe_set and test_rmtree_errorhandler. A GitHub search indicates that they're part of pip's set suite:

pip/test_network_cache.py at ba11a20b78f9194ef35f3625f9daa77e08b9214c · pypa/pip
pip/test_utils.py at ba11a20b78f9194ef35f3625f9daa77e08b9214c · pypa/pip