--testmon-readonly does not work

[boxed]

The --testmon-readonly flag doesn't seem to work. I would expect this:

> rm .testmondata
> python -m pytest --testmon --testmon-readonly

to give me an error message saying there is no .testmondata file.

I would also expect that if I change a function so that just one test is selected, that if I run python -m pytest --testmon --testmon-readonly multiple times it should keep running that test every time and not update the database.

Since it does update the database, I have to assume it's running with coverage analysis on too with this flag, which makes tests much slower. I haven't confirmed this but it seems the only reasonable conclusion.

We have some issues with using testmon with xdist which we believe might be due to multiple processes trying to update the database at the same time. This is speculation on our part though.

I have attached a simple example project that reproduces this issue. Just modify one of the methods in foo.py and run like above.

testmon-readonly-broken.zip

[Matlino]

"I would also expect that if I change a function so that just one test is selected, that if I run python -m pytest --testmon --testmon-readonly multiple times it should keep running that test every time and not update the database."

I tested this on actual version (v1.0.2) and it is working as you would expect. Coverage is not running and database is not updated. You can test it out by running:
pytest --testmon-nocollect
Let us now, if the problem persist on your side.

[Matlino]

I was thinking about the error message you proposed. The testmon header message changed since you submit the issue. For your use-case, the message looks like this:
testmon: collection deactivated, new DB, environment: default

So you know you are not collecting anything. And there is also info about new DB, so there was no data before this run. Therefore there will be nothing to select and all tests will run.

Although, I think this "new DB" message was there before. Do you think the meaning is not clear enough for what to expect when you use nocollect option? Or is it that you did not notice it in the header?

[boxed]

I don't think that's very clear no. But worse, if you are running this in CI and you're misssing the DB you want this to be a hard error and not silently running all tests.

Maybe --testmon-readonly-strict for this use case?

[boxed]

I tried the new version now:

> la .testmondata
-rw-r--r--  1 andersh  staff   4.6M Feb 23 17:35 .testmondata

[changed code so it breaks]

> python -m pytest -x --testmon-nocollect

> la .testmondata
-rw-r--r--  1 andersh  staff   4.6M Feb 23 17:39 .testmondata

I would have expected the test database to not be updated at all. Clearly it's touched, but it's hard to understand if this is ok or not from the outside.

[boxed]

That being said, I just tried mutmut on iommi with testmon and holy crap, the speedup is crazy. The previous version was just a wash, this version is a huge speedup:

without testmon: 7.5m for 100 mutants
with testmon: 5.9m for 171 mutants

[boxed]

Hmm.. on the other hand, without testmon 55 killed mutants (out of 100), with testmon 17 out of 171. That seems very bad. Might be mutmut that handles testmon badly.. I'll have to investigate.

[tarpas]

@boxed So it seems it is a bug, right? Do you think some of the mutants might have the same file timestamp? (could happen if you create the mutants very quickly and if the timestamp resolution of your filesystem is too coarse).

[boxed]

Well, mutmut doesn't just execute pytest when testmon is present, it has code to handle it specially. That code can be wrong with the new major version for example.

It copies the testmon databse right now which seems like it's redundant. I'll have to look at it at some point. For now I actually created a new system for mutmut where you can hook in your own python code to do special logic. With iommi I've made it so it has a list of which test files are relevant for which modules and it executes just those. It has the added benefit of bypassing some slow pytest collect and you don't get incidental testing like you would get with testmon (this might be good or bad depending on perspective).