Confusion: Missing subtest message in 4 failed subtest but 1 test passed

[buhtz]

I have two problems here and not sure if I misunderstand the purpose of pytest-subtests or if there really is a bug.

I have a unittest like this

expect_folder = pathlib.Path.cwd() / 'Beverly'
self.assertTrue(expect_folder.exists())

expect = [
    expect_folder / '_Elke.pickle',
    expect_folder / '_Foo.pickle',
    expect_folder / 'Bar.pickle',
    expect_folder / 'Wurst.pickle',
]
for fp in expect:
    with self.subTest(str(fp)):
        self.assertTrue(fp.exists())

You see assertTrue() is called five times.

The pytest output

===================================================================================== short test summary info ==
BFAIL tests/test_bandas_datacontainer.py::FolderModeFS::test_build_container - AssertionError: False is not true
SUBFAIL tests/test_bandas_datacontainer.py::FolderModeFS::test_build_container - AssertionError: False is not true
SUBFAIL tests/test_bandas_datacontainer.py::FolderModeFS::test_build_container - AssertionError: False is not true
SUBFAIL tests/test_bandas_datacontainer.py::FolderModeFS::test_build_container - AssertionError: False is not true
=================================================================================== 4 failed, 1 passed in 1.97s ==

Problem 1 : passed but not passed

The message 4 failed, 1 passed might technically be correct but it is wrong from the users perspective. There is one test defined by the def test_...() method. It doesn't matter how often self.assert...() is called in there. One fail means the whole test fails. Saying 1 passed is a bug.

Problem 2 : sub test message missing

I do use subtest (self.subTest(str(fp))) to better see which part of the test failed. But the output just tells me four times AssertionError: False is not true without more details about which subtest is involved. I would expect something like AssertionError: False is not true (/Beverly/Foo.pickle) as an output.

[nicoddemus]

Hi @buhtz,

The current output tries to mimic how the standard unittest outputs subtests, but there has been discussions that we should just follow what we think makes more sense for pytest. I'm sure there are open issues discussing this, please take a look (I can't search for them right now I'm afraid).

[Dominik1123]

@nicoddemus The current output - regarding Problem 1 - does not mimic the behavior of the builtin unittest subtests. Consider the following example:

import unittest

class Test(unittest.TestCase):
    def test(self):
        for i in [0, 2]:
            with self.subTest(i=i):
                self.assertEqual(0, i%2)

if __name__ == '__main__':
    unittest.main()

Running this with python test.py gives the following output:

.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

or, with -v flag:

test (__main__.Test) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

Hence, the standard unittest subtests usage reports 1 passed test.

Let's compare this to pytest test.py:

=========================== test session starts ===========================
platform linux -- Python 3.9.0, pytest-7.1.3, pluggy-1.0.0
rootdir: /tmp
plugins: subtests-0.12.0, hydra-core-1.1.1
collected 1 item                                                          

test.py .                                                           [100%]

============================ 1 passed in 0.03s ============================

So far, so good.

But now, what if one of the subtests fails? Let's change the above code example from for i in [0, 2] to for i in [1, 2], i.e., make the first subtest fail.
Running python test.py produces:

======================================================================
FAIL: test (__main__.Test) (i=1)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/test.py", line 7, in test
    self.assertEqual(0, i%2)
AssertionError: 0 != 1

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (failures=1)

Specifically, at the top of the output there is no . to indicate success of a test; running with -v produces test (__main__.Test) ..., i.e., there is no ok.

However, running this with pytest test.py produces:

=========================== test session starts ===========================
platform linux -- Python 3.9.0, pytest-7.1.3, pluggy-1.0.0
rootdir: /tmp
plugins: subtests-0.12.0, hydra-core-1.1.1
collected 1 item                                                          

test.py .                                                          [100%]

================================ FAILURES =================================
_____________________________ Test.test (i=1) _____________________________

self = <test.Test testMethod=test>

    def test(self):
        for i in [1, 2]:
            with self.subTest(i=i):
>               self.assertEqual(0, i%2)
E               AssertionError: 0 != 1

test.py:7: AssertionError
========================= short test summary info =========================
(i=1) SUBFAIL test.py::Test::test - AssertionError: 0 != 1
======================= 1 failed, 1 passed in 0.30s =======================

While pytest reports 1 failure, similar to unittest, it also reports 1 passed test. This behavior is different from unittest's where no passed test is reported. That behavior of pytest (adding a passed test) is even present when all subtests fail. Let's change for i in [1, 2] to for i in [1, 3] and run pytest test.py:

=========================== test session starts ===========================
platform linux -- Python 3.9.0, pytest-7.1.3, pluggy-1.0.0
rootdir: /tmp
plugins: subtests-0.12.0, hydra-core-1.1.1
collected 1 item                                                          

test.py .                                                         [100%]

================================ FAILURES =================================
_____________________________ Test.test (i=1) _____________________________

self = <test.Test testMethod=test>

    def test(self):
        for i in [1, 3]:
            with self.subTest(i=i):
>               self.assertEqual(0, i%2)
E               AssertionError: 0 != 1

test.py:7: AssertionError
_____________________________ Test.test (i=3) _____________________________

self = <test.Test testMethod=test>

    def test(self):
        for i in [1, 3]:
            with self.subTest(i=i):
>               self.assertEqual(0, i%2)
E               AssertionError: 0 != 1

test.py:7: AssertionError
========================= short test summary info =========================
(i=1) SUBFAIL test.py::Test::test - AssertionError: 0 != 1
(i=3) SUBFAIL test.py::Test::test - AssertionError: 0 != 1
======================= 2 failed, 1 passed in 0.12s =======================

2 failures are reported correctly, but again, 1 passed is wrong.

This behavior is different from the builtin unittest subtest behavior and should thus be considered a bug. Also, it's very counterintuitive. Instead, the success of a test with respect to its subtests should be computed as test.has_failed = any(sub_test.has_failed for sub_test in test.sub_tests) (pseudo-code).