Document that with pytest.warns() captures all warnings, not just those it tests were emitted

[Kojoley]

I have a test that ensures a specific warning type is produced by a function, but since I switched to pytest.warns I no longer see Python 3.10 compatibility warnings in Pytest warnings summary the code was emitting before switching to pytest.warns. Is it expected behavior (in which case it contradicts https://docs.pytest.org/en/6.2.x/warnings.html#deprecationwarning-and-pendingdeprecationwarning) and how I can opt-out of hiding DeprecationWarnings by pytest.warns (there is nothing about that in the documentation https://docs.pytest.org/en/6.2.x/reference.html#pytest-warns)?

MVCE:

import pytest
import warnings

def test_warning():
    with pytest.warns(DeprecationWarning):
        with pytest.warns(UserWarning):  # catches UserWarning, but also every other type too
            warnings.warn("my warning", UserWarning)
            warnings.warn("some deprecation warning", DeprecationWarning)

================================== FAILURES ===================================
________________________________ test_warning _________________________________

    def test_warning():
>       with pytest.warns(DeprecationWarning):
E       Failed: DID NOT WARN. No warnings of type (<class 'DeprecationWarning'>,) was emitted. The list of emitted warnings is: [].

[nicoddemus]

Hi @Kojoley,

This is the expected behavior, pytest.warns acts as a net which captures all warnings inside the block, ensuring the given warning is raised, or failing otherwise.

We might add a note to https://docs.pytest.org/en/6.2.x/warnings.html#deprecationwarning-and-pendingdeprecationwarning to mention that behavior does not apply for code inside pytest.warns, what do you think?

[Kojoley]

The pytest.warns documentation does not say it will filter everything (actually it does not say it will filter anything at all), so it should be fixed too. It preferably should say how to make pytest.warns to not capture everything or an alternative solution, but I still myself have no idea how to achieve that with tools Pytest provides, could you please suggest anything?

[nicoddemus]

The pytest.warns documentation does not say it will filter everything (actually it does not say it will filter anything at all), so it should be fixed too.

True, thanks!

I still myself have no idea how to achieve that with tools Pytest provides, could you please suggest anything?

Hmm I can't think of anything right now, pytest.warns uses warnings.catch_warnings under the covers, which by design captures all warnings.

[nicoddemus]

Created #9291 to track the docs change.

[Kojoley]

If I understand correctly pytest.deprecated_call hides all warnings by design too?

Also, https://docs.pytest.org/en/6.2.x/warnings.html#warns says You can check that code raises a particular warning using func:pytest.warns, which works in a similar manner to raises: but raises does not hide all exceptions, example:

def test_raises():
    with pytest.raises(RuntimeError):              # passes,
        with pytest.raises(NotImplementedError):   # because this will not catch and hide the exception
            raise RuntimeError("some exception")

[nicoddemus]

If I understand correctly pytest.deprecated_call hides all warnings by design too?

Yes. 👍

but raises does not hide all exceptions, example

Good point, we need to mention that difference in the docs too.

[Zac-HD]

Alternatively we could make pytest.warns() re-emit all captured warnings on exit, which might better match users' intuitions - including mine, from before I went and looked up warnings.catch_warnings(record=True).

[QuLogic]

I think it would make sense to re-raise all unmatched warnings. For example, say you are testing your own deprecated code so you add pytest.warns(DeprecationWarning, match='something specific to your project'). Then one of your dependencies (or maybe some other part of your code) adds a deprecation that is only triggered by that original code's tests. Then you will be blind-sided if the second deprecation is removed first because you never saw it.

[zanieb]

Note that this also means that if you pass an incorrect value to warnings.warn it will hide the error. See #10865

[Zac-HD]

As of #10937, expected in Pytest 8.0, we re-emit all non-matching warnings.