Tests timeout is too short, especially on slow architectures

[rwmjones]

On s390x in particular:

+ dune runtest
  test_stack alias examples/runtest
...
Ran: 3 tests in: 0.11 seconds.
OK
        test alias test/lwt/runtest
.
Ran: 1 tests in: 0.12 seconds.
OK
testRunnerProcesses alias test/lwt/testRunnerProcesses/runtest
..................................................
Ran: 50 tests in: 0.14 seconds.
OK
   test_list alias examples/runtest
..
Ran: 2 tests in: 0.12 seconds.
OK
        test alias test/runtest (exit 1)
(cd _build/default/test && ./test.exe -fakeHTML fakeHTML/fakeHTML.exe -fakeRunner fakeRunner/fakeRunner.exe -fakeShared fakeShared/fakeShared.exe -fakeBadFinaliser fakeBadFinaliser/fakeBadFinaliser.exe)
........................................~~............
==============================================================================
Error: OUnit:7:Runner:5:Processes+SIGSEGV.
File "/builddir/build/BUILD/ounit-v2.2.2/_build/default/test/oUnit-OUnit-buildvm-s390x-13.s390.fedoraproject.org#00.log", line 1137, characters 1-1:
Error: OUnit:7:Runner:5:Processes+SIGSEGV (in the log).
Timeout after 60.0s
------------------------------------------------------------------------------
==============================================================================
Error: OUnit:7:Runner:2:ProcessesWithBadFinaliser.
File "/builddir/build/BUILD/ounit-v2.2.2/_build/default/test/oUnit-OUnit-buildvm-s390x-13.s390.fedoraproject.org#00.log", line 1131, characters 1-1:
Error: OUnit:7:Runner:2:ProcessesWithBadFinaliser (in the log).
Timeout after 60.0s
------------------------------------------------------------------------------
Ran: 54 tests in: 73.42 seconds.
FAILED: Cases: 54 Tried: 54 Errors: 0 Failures: 0 Skip:  0 Todo: 0 Timeouts: 2.

TBH 60s is simply too short on these architectures. Best to make it 10 minutes, unless the 60s rule is there for a good reason (like the test must never take longer than 60s).

[gildor478]

60s is a standard that I have copied from other places. I have no problem going up if needed for non amd64 architecture.

What about using 60s for i386/amd64 and going up for anything else.

As an alternative, we can have a OUNIT_MAX_TIMEOUT_MINUTES=10 so that you can set it yourself depending on the architecture.

[rwmjones]

I think timeouts are a necessary evil for some tests, but I try to avoid them if possible. Some questions:

• Are these tests required to run in < 60s? eg: If they take longer then some vital real time deadline will be missed. (The nuclear reactor will blow up unless we insert the control rods in under a minute!)

• Or was 60s chosen arbitrarily because the test sometimes hangs are you want to have it fail instead of waiting forever?

• Or has the test never hung, and it only either succeeds or fails?

If we're in the second case then I would chose some long timeout on all architectures, like 10 or 20 minutes. If we're in the third case then we shouldn't have a timeout at all. If it's the first case then of course we need to keep the timeout and investigate why the test fails on s390x.

[gildor478]

This was an arbitrary choice, nothing will fail if we change the delay. We are in the second case.

The real thing that I wanted to avoid is to have tests that never ends and an overall testing step that takes hours in a CI.