cargo test does not always replay corpus

[vasumv]

Hello!

I wanted to replay corpus using cargo test (documented here) but was noticing that it will not fully replay a corpus, and always terminates after 1 second.

In the case that the corpus is small, the corpus will be fully replayed and then random inputs will be generated for the remainder (you can verify this with any of the tests in examples or the fibonacci tutorial). In the case that the corpus is large, only a subset of the inputs will be run until the test terminates after 1 second.

Is there a specific flag or profile I need to ensure that the corpus and only the corpus is replayed when running cargo test? Sorry if I'm missing something simple here.

[camshaft]

Yeah this is due to the change made in https://github.com/camshaft/bolero/pull/206/files#diff-e3f9b342c59a9db62c2f297cc5bfd9823e8f90b2833b234edce7c46b865dc333R31, where it currently defaults to running tests for at most a second. If you want to increase the default you can specify a time in the harness using .with_test_time. Additionally, specifying the number of iterations also disables the time limit.

That being said, we should probably document this better.

[camshaft]

Additionally, it might be better to always replay the corpus have the time limit only apply to the random iterations. Not sure.

[vasumv]

I see, thanks for the quick response!

So if I wanted to just do this in cmdline without changing the test harness, I'd get the exact size of the corpus, and use BOLERO_RANDOM_ITERATIONS=<size_of_corpus> cargo test (which seems to have worked).

[camshaft]

Yeah the environment variable should work! One thing to note, though, is the BOLERO_RANDOM_ITERATIONS value is on top of the corpus replay. So if you have 5 corpus files and then specify BOLERO_RANDOM_ITERATIONS=6, you'll end up with 11 runs.

[vasumv]

Oh gotcha, thanks for the clarification. So really what I want is BOLERO_RANDOM_ITERATIONS=0.