manage long and special tests effectively

[ilgooz]

No description provided.

[grepsuzette]

Confirmed (cd tests; go test).
Hangs after outputing (just copying the end):

stack 1: file{ package main; func init.0() { (const (println func(xs ...interface{})()))((const ("here" string))) }; func main() { init<VPUverse(0)>() } }
stack 0: package(main)
------------------------
OUTPUT:
 
in
inCall
Hello &{foo}
foo
code: foo
2
inCall
Hello {foo}
foo
inCall
a: {}
A
1
1
A
<nil>
1
1
inCall
Hello {foo}
foo
test 2
test 2
test
2 test
{test}
<nil>
inCall
Hello {foo}
foo
0
<nil>
{}
test
Myint: 3 3
Myint: 4 4
hi
foo
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115]
AAAAAA
0 1 2
0 1 2 3
0 1 2
0 1 2 3
test
12
map[b:one round:12]
foo
map[] true
[0 0 0 0]
[hellobar hellofoo worldmachin worldtruc]
[hellobar hellofoo worldmachin worldtruc]
[hellobar worldmachin]
&{ map[Accept-Encoding:[gzip]]}
true false
v: 
 false
not exists
0
1 test
&{}
&{}
{test 1s}
 false
bidule true
[hellobar hellofoo worldmachin worldtruc]
[hellobar hellofoo worldmachin worldtruc]
-1
[187 141 73 89 101 229 33 106 226 63 117 234 117 149 230 21]

Maybe run a git bisect to identify when it started to block.

[thehowl]

This is actually not blocking, it's just because it's running a very long test. It is also not reproduced on the CI as the CI runs the commands from make test, and never runs a simple go test on the tests folder.

The problem is that there are in the files/ directory some tests which have the _long suffix, and are actually not executed using the makefile (as the -test.short flag is passed to go test). These are very very long and more likely meant for those delving deep into optimisation- and internals-land.

I think this issue still raises two good questions: shouldn't we at least sporadically run some of these on the CI, and should we make it clearer that the "proper" way to run files in tests is through make test?

For the first one, I think a good strategy could be to run the long file tests with continue-on-error set to true (I think if this is set to true, github may mark the workflow as passing even though this one is still running?). If they still take >30m, we could subdivide them further.

(While looking at this, I also found that we probably don't test go files in examples/, nor all the stdlibs, because make test only specifies a subset of packages from stdlibs. @moul should I make an issue for this, or is this intended)?

[moul]

This test is too lengthy and prone to timeouts for most users, so it should be disabled and reserved for advanced users who are conducting fuzzing tests. We must find a balance between never running the test and having a slow CI.

To ensure that tests are completely disabled when someone runs cd tests; go test ., we should split the Makefile into two parts. The main Makefile should be shorter and easier to use for most people, while the second Makefile should contain advanced scripts and rules that are available for autonomous advanced users only. Additionally, we should document how to run these advanced tests.

@thehowl shared an intriguing idea for the CI that is reminiscent of fuzzing. This involves randomly disabling tests and running them iteratively until a specified timeout is reached, which would be considered a "success" in our case.

[moul]

You beat me to it, @thehowl!

I believe we can continue discussing this issue. But up to you.

I just remembered that we have one more type of ignored tests: the func TestFlappy ones. By default, these tests are skipped when running go test, but they can be executed from the Makefile and are run multiple times by the CI, looking for at least one successful. Although this isn't ideal, it's still better than completely skipping the test.