ci: consider adding back the gnovm parallel testing #2826
Comments
|
cc @ajnavarro. Can we make the gnovm test shorter without having to split it up again? |
|
Most of the 17 minutes running the tests are spent on testing StdLibs: My proposal: fix all race conditions we have in our code to be able to parallelize our tests. Stdlibs can be tested in much less than 6:30 mins. |
|
Sounds reasonable. Does anyone have an estimation of the feasibility and difficulty of removing all the race conditions? -> #2097 |
|
I did a bit of a research and adding t.Parallel() to stdlib tests we improved a bit the time needed to run them: #2864 They take 3 minutes less to run in total. |
mvertes
pushed a commit
that referenced
this issue
Oct 18, 2024
…2864) Check #2826 for more context. --------- Signed-off-by: Antonio Navarro <[email protected]>
|
PR #2864 makes The failure prints: |
|
of course it's uverse. (#2067) |
r3v4s
pushed a commit
to gnoswap-labs/gno
that referenced
this issue
Dec 10, 2024
This PR tackles a large amount of improvements in our current internal testing systems for the gnovm, as well as those for `gno test`. The primary target is the execution of filetests; however many improvements have been implemented to remove dead or duplicate code, and bring over some of the improvements that have been implemented in tests to filetests, when they come at little added "cost". The biggest headline concerns the execution of filetests. I wrote the specific improvements undertaken in a [blog post on "diary of a gnome"](https://gno.howl.moe/faster-tests/), but here's a side-by-side comparison of the execution in this PR (left) and the execution on master (right).  - Fixes gnolang#1084 - Fixes gnolang#2826 (by addressing root cause of slowness) - Fixes gnolang#588, only running `_long` filetests on master and generally speeding up test execution. ## Impact - Test context (tests and filetests) - Tests and filetests now share the same `test.Store` when running. Coupled with `test.LoadImports`, it allows us to "eagerly load" imports ahead of the test execution, and save it in the store. This is the primary performance improvement of this PR, as all pure packages can be run and preprocessed only once, whereas before it was once per package, and once per filetest. (More info in the blog post.) - One of the consequences of this is that package initialization happens outside of the test; so a filetest can no longer check the output of a `println` used in package initialization. - The default user no longer has 200 gnot by default. There are two mechanisms that make this unnecessary: `// SEND:`, and `std.TestIssueCoin`. Some test balances had to be updated. - Filetests - Running filetests in `gno test -v` now also prints their output. - Realm tests are no longer executed in `main.gno`, but in a filename following the name of the filetest; this changed some `// Realm:` directives. - The sytem to read directives and update them in the filetests has been re-written, and should now be more resilient and with fewer "exceptions". This means that leading and trailing whitespace in output now is correctly considered as output, leading to the addition of many empty `//` in the tests. - `// Error:` directives are now treated the same as everything else; and are updated if the `-update-golden-tests` flag is passed. - Removed the `imports` metric from the runtime metrics, as it's now no longer straightforward to calculate (given that imports are loaded "eagerly"). - Removed support for different "modes" of importing stdlibs; further removing support for gonative (gnolang#1361). The remaining gonative libraries are `os`, `fmt`, `encoding/json`, `internal/os_test` and `math/big`. This removes the `-with-native-fallback` flag from `gno test`. - Consequently, filetests ending with `_native.gno` have largely been removed, and those ending with `_stdlibs.go` have had their suffix removed. - Some files testing `gonative` types and functions which were only used in a small subset of these tests, have been removed. - Tests - `gno test`, for testing packages, created a `main_test.gno` file that is then called directly from the machine on each test. This creates two identifiers, `tests` and `runtest`, which could come into conflict with those defined by the tests themselves. We now call `testing.RunTest` directly, without an intermediary. - Exports in the normal tests (ie. defined in `pkg`) are now visible in the tests of `pkg_test`. This is most useful for some standard library tests, where there often is an `export_test.go` with the sole scope of exporting some internal functions and methods for testing in the "real" testing file. - `gno lint`, and other occasions where we use `issueFromError` (like when parsing errors in `gno test`), will now also print the column of the error if given. ## Summary of internal changes - pkg/gnolang - `TestFiles` is the new function running the filetests. - `eval_test.go` has been removed, moving some of its improvements over to `TestFiles`, and reducing the scope of the corresponding tests in `debugger.go`, to what it's actually supposed to test for. - As a consequence of removing all of type mappings in `imports.go`, I removed `SetStrictGo2GnoMapping` in favour of always being "strict", and not allowing defining native types of defined types any more. - The tests in `gonative_test` where primarily related to this, so I removed them. Similarly for `preprocess_test`. - `TestFunc` / `TestMemPackage` have been removed as redundant, including some of their features in `cmd/go/test.go` (like supporting exporting symbols in `_test.gno` files) - The `Store` no longer has `ClearCache` and `SetStrictGo2GnoMapping`; `ClearCache` now has a better substitute in `BeginTransaction`. - the `testing` stdlib no longer caches `matchPat` and `matchString` as pure packages should not be able to modify their values during execution, and as a result of other changes this was failing. - The tests/ directory has been removed / moved to `gnovm/pkg/test`. This directory should eventually contain the internal code for `gno test` as well; but for now I wanted to clean `tests` to ensure it is a directory just for integration tests, rather than code and testing systems. - `TestMachine` and `TestStore` have been renamed to Machine and Store, to avoid redundancy with the `test` package name. - Removed plenty instructions in `gnovm/Makefile` as now most tests are just in `pkg/gnolang`. - I removed `MsgContext.Msg` as unused. It can be re-added later if necessary. - In the CI, tests are now run with `-short`, at least until we figure out how to make the VM efficient enough to run these tests. Aside from some filetests, this now excludes some stdlibs: `bytes`, `strconv`, `regexp/syntax`. I accept other proposals that could make us have fast tests on PR's, while still testing (mostly) everything. --- [](https://lu.ma/open-source-saturday-torino) --------- Co-authored-by: Marc Vertes <[email protected]>
albttx
pushed a commit
that referenced
this issue
Jan 10, 2025
](https://lu.ma/open-source-saturday-torino){kind=link}
This PR tackles a large amount of improvements in our current internal testing systems for the gnovm, as well as those for `gno test`. The primary target is the execution of filetests; however many improvements have been implemented to remove dead or duplicate code, and bring over some of the improvements that have been implemented in tests to filetests, when they come at little added "cost". The biggest headline concerns the execution of filetests. I wrote the specific improvements undertaken in a [blog post on "diary of a gnome"](https://gno.howl.moe/faster-tests/), but here's a side-by-side comparison of the execution in this PR (left) and the execution on master (right).  - Fixes #1084 - Fixes #2826 (by addressing root cause of slowness) - Fixes #588, only running `_long` filetests on master and generally speeding up test execution. ## Impact - Test context (tests and filetests) - Tests and filetests now share the same `test.Store` when running. Coupled with `test.LoadImports`, it allows us to "eagerly load" imports ahead of the test execution, and save it in the store. This is the primary performance improvement of this PR, as all pure packages can be run and preprocessed only once, whereas before it was once per package, and once per filetest. (More info in the blog post.) - One of the consequences of this is that package initialization happens outside of the test; so a filetest can no longer check the output of a `println` used in package initialization. - The default user no longer has 200 gnot by default. There are two mechanisms that make this unnecessary: `// SEND:`, and `std.TestIssueCoin`. Some test balances had to be updated. - Filetests - Running filetests in `gno test -v` now also prints their output. - Realm tests are no longer executed in `main.gno`, but in a filename following the name of the filetest; this changed some `// Realm:` directives. - The sytem to read directives and update them in the filetests has been re-written, and should now be more resilient and with fewer "exceptions". This means that leading and trailing whitespace in output now is correctly considered as output, leading to the addition of many empty `//` in the tests. - `// Error:` directives are now treated the same as everything else; and are updated if the `-update-golden-tests` flag is passed. - Removed the `imports` metric from the runtime metrics, as it's now no longer straightforward to calculate (given that imports are loaded "eagerly"). - Removed support for different "modes" of importing stdlibs; further removing support for gonative (#1361). The remaining gonative libraries are `os`, `fmt`, `encoding/json`, `internal/os_test` and `math/big`. This removes the `-with-native-fallback` flag from `gno test`. - Consequently, filetests ending with `_native.gno` have largely been removed, and those ending with `_stdlibs.go` have had their suffix removed. - Some files testing `gonative` types and functions which were only used in a small subset of these tests, have been removed. - Tests - `gno test`, for testing packages, created a `main_test.gno` file that is then called directly from the machine on each test. This creates two identifiers, `tests` and `runtest`, which could come into conflict with those defined by the tests themselves. We now call `testing.RunTest` directly, without an intermediary. - Exports in the normal tests (ie. defined in `pkg`) are now visible in the tests of `pkg_test`. This is most useful for some standard library tests, where there often is an `export_test.go` with the sole scope of exporting some internal functions and methods for testing in the "real" testing file. - `gno lint`, and other occasions where we use `issueFromError` (like when parsing errors in `gno test`), will now also print the column of the error if given. ## Summary of internal changes - pkg/gnolang - `TestFiles` is the new function running the filetests. - `eval_test.go` has been removed, moving some of its improvements over to `TestFiles`, and reducing the scope of the corresponding tests in `debugger.go`, to what it's actually supposed to test for. - As a consequence of removing all of type mappings in `imports.go`, I removed `SetStrictGo2GnoMapping` in favour of always being "strict", and not allowing defining native types of defined types any more. - The tests in `gonative_test` where primarily related to this, so I removed them. Similarly for `preprocess_test`. - `TestFunc` / `TestMemPackage` have been removed as redundant, including some of their features in `cmd/go/test.go` (like supporting exporting symbols in `_test.gno` files) - The `Store` no longer has `ClearCache` and `SetStrictGo2GnoMapping`; `ClearCache` now has a better substitute in `BeginTransaction`. - the `testing` stdlib no longer caches `matchPat` and `matchString` as pure packages should not be able to modify their values during execution, and as a result of other changes this was failing. - The tests/ directory has been removed / moved to `gnovm/pkg/test`. This directory should eventually contain the internal code for `gno test` as well; but for now I wanted to clean `tests` to ensure it is a directory just for integration tests, rather than code and testing systems. - `TestMachine` and `TestStore` have been renamed to Machine and Store, to avoid redundancy with the `test` package name. - Removed plenty instructions in `gnovm/Makefile` as now most tests are just in `pkg/gnolang`. - I removed `MsgContext.Msg` as unused. It can be re-added later if necessary. - In the CI, tests are now run with `-short`, at least until we figure out how to make the VM efficient enough to run these tests. Aside from some filetests, this now excludes some stdlibs: `bytes`, `strconv`, `regexp/syntax`. I accept other proposals that could make us have fast tests on PR's, while still testing (mostly) everything. --- [](https://lu.ma/open-source-saturday-torino) --------- Co-authored-by: Marc Vertes <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
With the overhaul of GitHub Actions in PR (#2040), we switched from nine parallel CI jobs for gnovm to a single job. This change has made the CI check longer, and it slows down our (my?) ability to identify the root cause of a problem when we need to sift through a large log for errors.
What do you think about setting up a parallel gnovm test as we did before?
See https://github.com/gnolang/gno/pull/2040/files#diff-c4113e9299b96c08f4bd91c5bf070f0f0feb6ae61f3f19d845a6501d419c9658L62-L71
The text was updated successfully, but these errors were encountered: