feat: gno test support /... pattern

[harry-hov]

BREAKING CHANGE: Altered behavior of the gno test command

Adds support for /... pattern in gno test command. Now args can have /... pattern in the directory path.

Using gno test ./path/to/pkg would trigger the execution of test files solely within the specified package directory, excluding any subdirectories like ./path/to/pkg/subpkg.

To execute test files within subdirectories as well, use gno test ./path/to/pkg/...

It supports all variations of /... such as ./path/.../pkg, ./.../pkg, ,./.../path/... and more

Contributors' checklist...

• Added new tests, or not needed, or not feasible
• Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory
• Updated the official documentation or not needed
• No breaking changes were made, or a BREAKING CHANGE: xxx message was included in the description
• Added references to related issues and PRs
• Provided any useful hints for running manual tests
• Added new benchmarks to generated graphs, if any. More info here.

[moul]

LGTM, but take comments into accounts, thank you

[zivkovicmilos]

Can you please add some unit tests for this functionality you've introduced?

It seems like there are a lot of edge-cases, and I'm not sure how you've verified the functionality, apart from manually running it over a data set or comparing outputs for a few cases.

Not to mention that the PR decreases the overall coverage we have:
https://app.codecov.io/gh/gnolang/gno/pull/1078

@moul I propose we start adding a CI rule that stops a PR from being merged if it does not increase or maintain coverage at a certain margin

[moul]

What if we configure codecov to comment on PRs instead of auto-blocking them?

This way, we can monitor coverage without hindering the PR process.

[zivkovicmilos]

What if we configure codecov to comment on PRs instead of auto-blocking them?

This way, we can monitor coverage without hindering the PR process.

@moul
I agree that we need to increase CodeCov visibility, it should be generating reports on each PR and how it affects coverage in relation to the master branch.

I also believe that we should have the responsibility to not merge uncovered code upstream, because it opens the door to a bad practice of "fixing it later" (or much more likely, never).

It's part of a bigger issue we have in the repo, which is a non-standardized testing suite. I am not a huge fan of testing functionality by a command output as we're doing in this PR, and in other PRs (that have since been merged).

Overall, I want to push for greater accountability and responsibility of upstream changes, not just because we are approaching a launch (and many launches after that), but also because I want us to maintain a codebase that is maintainable by us and the community, and a codebase that won't come back at us like a boomerang down the road

I would love to hear other opinions as well on this issue, @gfanton @ajnavarro @thehowl

[ajnavarro]

For me, the code itself is the least important part of a PR. Code is a mere tool to accomplish a better state of the program.

To be able to verify that the state of the program/application will be better with the PR merged, we need ways to check it, and we need to make that change discoverable for the users.

To check it, we need reproducible tests. Depending on the case: golden tests/regression tests/integration tests/unit tests...

To discover the change: we need an entry on our changelog and/or go doc and user docs.

Those are the minimum parts IMHO of a PR in priority order:

• tests
• documentation
• changelog
• the code itself

If we want to verify further, we have to add benchmarks checking our new code performance too, but maybe we are too far away from that yet.

[moul]

Alright, let's experiment with enabling codecov by default for a while. I'm wary it might lead to valuable work being delayed. While I'm a proponent of atomic PRs with integrated tests, I also value dedicated sessions for refining tests (frameworks and units alike). So, while I'm open to temporarily blocking PRs based on codecov, I suspect we might eventually revert to just enabling comments. (easy to change).

Tests, as you said and in my opinion, are possibly even more critical than the code they assess.

Regarding documentation: Let's also raise the bar, but I believe in focusing on inline code documentation or making the code self-explanatory. Let's prioritize embedded documentation over separate standalone documentation within the context of PRs, keeping the latter as an independent area of improvement. An ideal PR, for me, is one that impacts only a specific folder without modifying root files. By the way, check out #1059.

On the changelog topic: I concur with its significance. However, ensuring clear and comprehensive PR titles and descriptions should suffice. While maintaining side changelog files might have value, I'd lean toward not imposing too many requirements on individual PRs. It could make sense to make releases parties where we update changelog files.

Ultimately, my advocacy lies in:

• Clean and clear code with relevant comments.
• Comprehensive tests.
• Well-described PR titles and descriptions.

I see code generation as a significant tool to aid in these goals, allowing our primary focus on the aforementioned aspects.

Edit: #1120

[gfanton]

To me, one of the most crucial aspects is enforcing rules within the CI. Code coverage, if not mandated, is often overlooked due to complacency. While exceptions exist, they should be made explicit, similar to using a // nolint tag. The more rules we enforce and automate, the more we can focus on the core of a PR. This benefits both the contributor and the reviewer by reducing back-and-forth exchanges.

Regarding non-standard testing suites, specifically golang's txtar tests, I'm relatively unfamiliar with this approach. However, I believe they can complement standard tests. We should always prioritize standard tests, but from what I understand, txtar tests could be useful for specific scenarios, such as testing edge cases or situations involving reading from a directory tree or multiple files. This can make tests more readable and maintainable. Nonetheless, standard tests should never be relegated to a secondary option. As example, for this specific PR, using txtar doesn't seem appropriate since pattern matching can be tested with standard tests, as shown in the original test at https://tip.golang.org/src/cmd/internal/pkgpattern/pat_test.go.
I'm not sure if txtar should contribute to test coverage, as it might lead to neglecting standard tests.

[moul]

Txtar aims to create integration tests that closely mirror real-world usage, employing the exact tools that end-users will utilize.

Consider a scenario where a company boasts 100% code coverage. Everything appears perfect, but the sign-up flow - registering and then logging in - was overlooked. Developers were already logged in, so while CI tests passed, no new user could register.

The primary objectives of txtar are:

1. Real-World Simulation: Txtar prioritizes tests that minimize mock features. The assurance is that if CI tests pass, the instructions provided in our README files remain functional. In an ideal scenario, all README files would be auto-tested with txtar. I'm excited to hear @gfanton is spearheading this initiative!

2. Workflow Readability: Txtar enhances the clarity of intricate workflows, especially those that involve setting up new directories, initializing files, and executing command chains.

Currently, txtar leverages the compiled binary, which enriches the mock-free testing experience. I propose we rename gnokey and gno in txtar to gnokeybin and gnobin. This change will enable some tests to utilize compiled binaries. However, for a majority of tests, instead of using generated binaries, we should directly import the CLI into the unit test. This strategy not only streamlines testing but also ensures that code coverage metrics are retained for such test scenarios.

[codecov]

Codecov Report

Attention: 11 lines in your changes are missing coverage. Please review.

Comparison is base (dce345f) 47.79% compared to head (9bf971e) 47.86%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1078      +/-   ##
==========================================
+ Coverage   47.79%   47.86%   +0.06%     
==========================================
  Files         369      369              
  Lines       62710    62764      +54     
==========================================
+ Hits        29975    30040      +65     
+ Misses      30312    30299      -13     
- Partials     2423     2425       +2     

Files	Coverage Δ
gnovm/cmd/gno/test.go	24.72% <33.33%> (ø)
gnovm/cmd/gno/util.go	45.29% <83.33%> (+11.41%)	⬆️

... and 3 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[zivkovicmilos]

@harry-hov

Code introduced in this PR has still not been properly covered, please see the CodeCov report associated with this PR.

Please ping me when the issue has been resolved 🙏

[moul]

Related with #1215.

[zivkovicmilos]

Thank you for the fixes and changes 🙏

Clear to go 🚀