Change in data source passed with pytest.mark.parametrize does not cause test rerun

[skyegecko]

Our testsuite has a number of tests which are run with parametrized data. The data source in turn is a variable (usually a list of some kind) in another module. When the data is altered, testmon fails to rerun the affected test.

Simple dummy project example:

File test_list.py:

import pytest
import data

@pytest.mark.parametrize("testdata", data.listdata)
def test_list(testdata):
    assert testdata

File data.py:

listdata = [
    True,
    True,
    True,
]

In this example, appending an item to listdata will not result in test_list being selected in the next test run.

Python version: 3.4.5
Pytest version: 3.8.1 (source: pip)
pytest-testmon version: 0.9.13 (source: pip)

[tarpas]

Thanks for the clear bug report. At a first sight I think this is conceptual weakness of testmon and it might take longer to fix. I'll write more when I have more time.

[tarpas]

@dutchgecko unfortunately won't fix (in the near future). It would be possible to react to the change but that would eliminate most of the potential time savings that testmon is designed to achieve. The (data) library is imported (and the data is initialized) before any test is run. Usually that's true for all tests and all files under test. We ignore what is executed during the import(collection) phase and assume dependency on any file in your project which executes at least on line from the setup, call or teardown phase of a test.

There is probably a workaround where you could make pytest execute a function during the setup phase to get the parametrization data:
https://raphael.codes/blog/pytest-parametrize-with-indirect/

It seems quite verbose and I don't have experience with it.

[tarpas]

@dutchgecko or in a primitive, heavy handed but clearer way:

• to data.py add:

def ping():
    return 0

• and then call it from inside the test function: data.ping() The rest stays the same as in your original version

[tarpas]

This is a duplicate of #191 (which described the more general case)