Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: fix gno test for _test.gno and _filetest.gno files #945

Merged
merged 20 commits into from
Aug 10, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion gno.land/cmd/gnoland/start.go
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ func makeGenesisDoc(

for _, pkg := range nonDraftPkgs {
// open files in directory as MemPackage.
memPkg := gno.ReadMemPackage(pkg.Path(), pkg.Name())
memPkg := gno.ReadMemPackage(pkg.Dir, pkg.Name)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this usage of Dir is OK since you need to file location to load.

var tx std.Tx
tx.Msgs = []std.Msg{
vmm.MsgAddPackage{
Expand Down
51 changes: 26 additions & 25 deletions gnovm/cmd/gno/test.go
Original file line number Diff line number Diff line change
Expand Up @@ -136,9 +136,13 @@ func execTest(cfg *testCfg, args []string, io *commands.IO) error {
cfg.rootDir = guessRootDir()
}

pkgPaths, err := gnoPackagesFromArgs(args)
paths, err := gnoPackagesFromArgs(args)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better to leave it as pkgPaths so as to differentiate from file paths.

Copy link
Contributor Author

@harry-hov harry-hov Jul 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but they can be file paths too.

e.g. paths can contain /path/to/pkg, /path/to/pkg/file_test.gno, or /path/to/pkg/z[N]_filetest.gno


Later I'm also planning to support patterns in args. e.g. gno test ./..., gno test /examples/.../pkg etc (just like go test)

paths can have pkgpath. filepath or both depends on the args

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's do the renaming in a new PR so we can focus on names.


Suggestion:

Suggested change
paths, err := gnoPackagesFromArgs(args)
targets, err := targetsFromPatterns(args)
// targetsFromPatterns returns a list of local dirs or single files
// intended to be used by gno commands such as `gno test`.
func targetsFromPatterns(args []string) []target {
	// TODO: support './...'
}

if err != nil {
return fmt.Errorf("list packages from args: %w", err)
return fmt.Errorf("list package paths from args: %w", err)
}
if len(paths) == 0 {
io.ErrPrintln("no packages to test")
return nil
}

if cfg.timeout > 0 {
Expand All @@ -148,75 +152,72 @@ func execTest(cfg *testCfg, args []string, io *commands.IO) error {
}()
}

subPkgs, err := gnomod.SubPkgsFromPaths(paths)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we CAN have a function called "SubPkgsFromDir" (singular), but not "SubPkgsFromDirs" or "SubPkgsFromPath" or "SubPkgsFromPaths".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on how go list ./crypto/... ./foo ./crypto/dsa works (https://github.com/golang/go/blob/499a12009938fe2ffff90775832b9d67ca3e46b2/src/cmd/go/internal/list/list.go#L569C24-L569C35), they also use a single helper that takes a list of targets.

The main difference is that they are only supporting dirs and not single file, while we want our CLI to basically support this at the end: gno test ./crypto/... ./foo ./bar_test.gno with files too.

I think we should consider that we have:

  • PkgPath when onchain or when identified by gno.mod or other methods
  • Dir in the case a PkgPath is located in the file system (but mabe we won't need it anymore), and
  • Target, in my opinion target should be the only way we identify a os.Args that can be a pattern, a dir or a file.

Edit: @harry-hov found this which is better: https://github.com/golang/go/blob/master/src/cmd/go/internal/load/pkg.go#L2816; it consists of looking for packages, but creating virtual packages of a single file when a file is passed instead of a directory or pattern.

if err != nil {
return fmt.Errorf("list sub packages: %w", err)
}

buildErrCount := 0
testErrCount := 0
for _, pkgPath := range pkgPaths {
for _, pkg := range subPkgs {
if cfg.precompile {
if verbose {
io.ErrPrintfln("=== PREC %s", pkgPath)
io.ErrPrintfln("=== PREC %s", pkg.Dir)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my main point is that here, we should use pkg.Path, which is not the same as any Dir. because "PkgPath/pkgPath/pkgpath/pkg.Path" are all the same "package path" identifier, whereas anything that ends with "Dir" is a filesystem location. They can diverge. Also a package path starts with a domain unless it is special, like a stdlibs package or something special like "main". Dir is a different beast based on the filesystem. Either absolute or relative, not something that starts with "gno.land" nor "main". It might be "." or "./something" though, or "/something". We should ideally make sure all Dirs start with a dot or slash.

Copy link
Contributor Author

@harry-hov harry-hov Aug 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree we should print pkg.Path instead of Dir. original var pkgPath also contains Dir. So this didn't changed anything.

Will fix this. But in another PR. (Out of scope for this PR)

}
precompileOpts := newPrecompileOptions(&precompileCfg{
output: tempdirRoot,
})
err := precompilePkg(importPath(pkgPath), precompileOpts)
err := precompilePkg(importPath(pkg.Dir), precompileOpts)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto. importPath should take pkg.Path.
we should use the type system to help here.
notice that in values.go, type PackageValue has Path as a Name type.
a package path is a Name (it is a name in its function).
a package path is a string (like all file paths and dir paths are strings).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok I see that importPath is declared as a type importPath string.
this is confusing to me.
something is either a package path, in which case we can just use Name.
we could say type packagePath Name but I don't think it's necessary.

there can be some function that converts a package path to a file path,
but that function would need more context, more than just 1 argument.
it could be a method attached to a structure that holds more context.

Other than such a function, we shouldn't be converting package path Names to file paths or dir paths with type conversions, because nobody should be doing such a thing without calling the official function for such conversions (above).

An "import path" I don't see why we need to create such a name, because it is nothing more than package paths. So ImportPkgPaths is less confusing. Or just Imports of Names.

Any path that is not a package path, should be clearly named as such, by calling it a "file path" or a "dir path" that is a string, never a Name type. Anything like package path, pkgPath, pkg.Path, are all package paths. If you want to associate a file path or a dir path it will have to be called pkg.FilePath or pkg.DirPath.

Copy link
Contributor Author

@harry-hov harry-hov Aug 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

original var pkgPath also contains Dir

Yeah, importPath is confusing. need to modify precompile related code for that. Will fix in another PR (Out of scope for this PR)

if err != nil {
io.ErrPrintln(err)
io.ErrPrintln("FAIL")
io.ErrPrintfln("FAIL %s", pkgPath)
io.ErrPrintfln("FAIL %s", pkg.Dir)
io.ErrPrintln("FAIL")

buildErrCount++
continue
}

if verbose {
io.ErrPrintfln("=== BUILD %s", pkgPath)
io.ErrPrintfln("=== BUILD %s", pkg.Dir)
}
tempDir, err := ResolvePath(tempdirRoot, importPath(pkgPath))
tempDir, err := ResolvePath(tempdirRoot, importPath(pkg.Dir))
if err != nil {
return errors.New("cannot resolve build dir")
}
err = goBuildFileOrPkg(tempDir, defaultBuildOptions)
if err != nil {
io.ErrPrintln(err)
io.ErrPrintln("FAIL")
io.ErrPrintfln("FAIL %s", pkgPath)
io.ErrPrintfln("FAIL %s", pkg.Dir)
io.ErrPrintln("FAIL")

buildErrCount++
continue
}
}

unittestFiles, err := filepath.Glob(filepath.Join(pkgPath, "*_test.gno"))
if err != nil {
log.Fatal(err)
}
filetestFiles, err := filepath.Glob(filepath.Join(pkgPath, "*_filetest.gno"))
if err != nil {
log.Fatal(err)
}
if len(unittestFiles) == 0 && len(filetestFiles) == 0 {
io.ErrPrintfln("? %s \t[no test files]", pkgPath)
if len(pkg.TestGnoFiles) == 0 && len(pkg.FiletestGnoFiles) == 0 {
io.ErrPrintfln("? %s \t[no test files]", pkg.Dir)
continue
}

sort.Strings(unittestFiles)
sort.Strings(filetestFiles)
sort.Strings(pkg.TestGnoFiles)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GnoTests string
GnoFiletests string
?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or TestFiles, FiletestFiles?

sort.Strings(pkg.FiletestGnoFiles)

startedAt := time.Now()
err = gnoTestPkg(pkgPath, unittestFiles, filetestFiles, cfg, io)
err = gnoTestPkg(pkg.Dir, pkg.TestGnoFiles, pkg.FiletestGnoFiles, cfg, io)
duration := time.Since(startedAt)
dstr := fmtDuration(duration)

if err != nil {
io.ErrPrintfln("%s: test pkg: %v", pkgPath, err)
io.ErrPrintfln("%s: test pkg: %v", pkg.Dir, err)
io.ErrPrintfln("FAIL")
io.ErrPrintfln("FAIL %s \t%s", pkgPath, dstr)
io.ErrPrintfln("FAIL %s \t%s", pkg.Dir, dstr)
io.ErrPrintfln("FAIL")
testErrCount++
} else {
io.ErrPrintfln("ok %s \t%s", pkgPath, dstr)
io.ErrPrintfln("ok %s \t%s", pkg.Dir, dstr)
}
}
if testErrCount > 0 || buildErrCount > 0 {
Expand Down
8 changes: 7 additions & 1 deletion gnovm/cmd/gno/test_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@ func TestTest(t *testing.T) {
errShouldContain: "no such file or directory",
},
{
args: []string{"test", "../../tests/integ/empty-dir"},
args: []string{"test", "../../tests/integ/empty-dir"},
stderrShouldContain: "no packages to test",
},
{
// FIXME: should have an output
Expand Down Expand Up @@ -98,6 +99,11 @@ func TestTest(t *testing.T) {
stdoutShouldContain: "RUN TestSprintf",
stderrShouldContain: "ok ./../../../examples/gno.land/p/demo/ufmt",
},
{
args: []string{"test", "../../../examples/gno.land/p/demo/ufmt/ufmt_test.gno"},
stdoutShouldContain: "RUN TestSprintf",
stderrShouldContain: "ok ../../../examples/gno.land/p/demo/ufmt",
},
{
args: []string{"test", "--verbose", "../../../examples/gno.land/p/demo/ufmt"},
stdoutShouldContain: "RUN TestSprintf",
Expand Down
11 changes: 8 additions & 3 deletions gnovm/cmd/gno/util.go
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,14 @@ func gnoPackagesFromArgs(args []string) ([]string, error) {
}
visited[parentDir] = true

// cannot use path.Join or filepath.Join, because we need
// to ensure that ./ is the prefix to pass to go build.
pkg := "./" + parentDir
pkg := parentDir
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be better also to rename this pkgDir
since pkg is ambiguous, pkgpath or dirpath

if !filepath.IsAbs(parentDir) {
// cannot use path.Join or filepath.Join, because we need
// to ensure that ./ is the prefix to pass to go build.
// if not absolute.
pkg = "./" + parentDir
}
harry-hov marked this conversation as resolved.
Show resolved Hide resolved

paths = append(paths, pkg)
return nil
})
Expand Down
165 changes: 124 additions & 41 deletions gnovm/pkg/gnomod/pkg.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,40 +5,32 @@ import (
"io/fs"
"os"
"path/filepath"
"strings"
)

type Pkg struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think type Module would be better.
They are all modules, by virtue of having .mod files right?
This differentiates from PackageValue, and amino Package too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I initially had the same thought.

But we don't seem to use word module anywhere.
We only use the word pkg (and pkg with state = realm)

name string
path string
draft bool
requires []string
Dir string // absolute path to package dir
harry-hov marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

@jaekwon jaekwon Jul 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dir here is fine but there should also be a PkgPath.
Module.PkgPath Name.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have Name string field for pkg name/path.

Maybe rename it to Path? (s/Name/Path)

Name string // package name
Requires []string // dependencies
Draft bool // whether the package is a draft
zivkovicmilos marked this conversation as resolved.
Show resolved Hide resolved
}

type SubPkg struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't have anything called Sub*.
From the perspective of Go/Gno, "sub" packages are just independent packages.
We can have maybe one utility function that returns all "sub" Modules inside a dir, but that should be about it. (And btw the filetests folder once we support that, is not itself a module. For safety we should also disallow a module in a folder with that name too).

Dir string // absolute path to package dir
ImportPath string // import path of package
Root string // Root dir containing this package, i.e dir containing gno.mod file
Imports []string // imports used by this package
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ImportPaths []Name // I think ImportPkgPath is just too long.
or
ImportDirs []string // but I don't think this is what we want.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have ImportPath string // import path of package

Don't you think ImportPaths []Name will make things more confusing?

ImportDirs []string? No. since they are not dirs


GnoFiles []string // .gno source files (excluding TestGnoFiles, FiletestGnoFiles)
TestGnoFiles []string // _test.gno source files
FiletestGnoFiles []string // _filetest.gno source files
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe
FilePaths []string
TestFilePaths []string
FiletestFilePaths []string

BTW we might support other langauges that compile to .gno, like .gnoffee

Copy link
Contributor Author

@harry-hov harry-hov Jul 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The original thought process behind this is to have pkg in a structured/sorted/filtered way.

If we want to support another language we can easily add another field here.
e.g. PyFiles []string GnoffeeFiles []string...

This way we can apply specific operations on them accordingly.
e.g.
GnoffeeFiles []string - needs to be compiled to .gno
GnoFiles []string - vm understands it natively
TestGnoFiles []string - if you wants to run tests, these are your files

}

type (
PkgList []Pkg
SortedPkgList []Pkg
)

// Name returns the name of the package.
func (p Pkg) Name() string {
return p.name
}

// Path returns the path of the package.
func (p Pkg) Path() string {
return p.path
}

// Draft returns whether the package is a draft.
func (p Pkg) Draft() bool {
return p.draft
}

// Requires returns the required packages of the package.
func (p Pkg) Requires() []string {
return p.requires
}

// sortPkgs sorts the given packages by their dependencies.
func (pl PkgList) Sort() (SortedPkgList, error) {
visited := make(map[string]bool)
Expand All @@ -57,21 +49,21 @@ func (pl PkgList) Sort() (SortedPkgList, error) {

// visitNode visits a package's and its dependencies dependencies and adds them to the sorted list.
func visitPackage(pkg Pkg, pkgs []Pkg, visited, onStack map[string]bool, sortedPkgs *[]Pkg) error {
if onStack[pkg.name] {
return fmt.Errorf("cycle detected: %s", pkg.name)
if onStack[pkg.Name] {
return fmt.Errorf("cycle detected: %s", pkg.Name)
}
if visited[pkg.name] {
if visited[pkg.Name] {
return nil
}

visited[pkg.name] = true
onStack[pkg.name] = true
visited[pkg.Name] = true
onStack[pkg.Name] = true

// Visit package's dependencies
for _, req := range pkg.requires {
for _, req := range pkg.Requires {
found := false
for _, p := range pkgs {
if p.name != req {
if p.Name != req {
continue
}
if err := visitPackage(p, pkgs, visited, onStack, sortedPkgs); err != nil {
Expand All @@ -81,11 +73,11 @@ func visitPackage(pkg Pkg, pkgs []Pkg, visited, onStack map[string]bool, sortedP
break
}
if !found {
return fmt.Errorf("missing dependency '%s' for package '%s'", req, pkg.name)
return fmt.Errorf("missing dependency '%s' for package '%s'", req, pkg.Name)
}
}

onStack[pkg.name] = false
onStack[pkg.Name] = false
*sortedPkgs = append(*sortedPkgs, pkg)
return nil
}
Expand Down Expand Up @@ -120,10 +112,10 @@ func ListPkgs(root string) (PkgList, error) {
}

pkgs = append(pkgs, Pkg{
name: gnoMod.Module.Mod.Path,
path: path,
draft: gnoMod.Draft,
requires: func() []string {
Dir: path,
Name: gnoMod.Module.Mod.Path,
Draft: gnoMod.Draft,
Requires: func() []string {
var reqs []string
for _, req := range gnoMod.Require {
reqs = append(reqs, req.Mod.Path)
Expand All @@ -147,15 +139,15 @@ func (sp SortedPkgList) GetNonDraftPkgs() SortedPkgList {
draft := make(map[string]bool)

for _, pkg := range sp {
if pkg.draft {
draft[pkg.name] = true
if pkg.Draft {
draft[pkg.Name] = true
continue
}
dependsOnDraft := false
for _, req := range pkg.requires {
for _, req := range pkg.Requires {
if draft[req] {
dependsOnDraft = true
draft[pkg.name] = true
draft[pkg.Name] = true
break
}
}
Expand All @@ -165,3 +157,94 @@ func (sp SortedPkgList) GetNonDraftPkgs() SortedPkgList {
}
return res
}

// SubPkgsFromPaths returns a list of subpackages from the given paths.
func SubPkgsFromPaths(paths []string) ([]*SubPkg, error) {
ajnavarro marked this conversation as resolved.
Show resolved Hide resolved
for _, path := range paths {
fi, err := os.Stat(path)
if err != nil {
return nil, err
}
if fi.IsDir() {
continue
}
if filepath.Ext(path) != ".gno" {
return nil, fmt.Errorf("files must be .gno files: %s", path)
}

subPkg, err := GnoFileSubPkg(paths)
if err != nil {
return nil, err
}
return []*SubPkg{subPkg}, nil
}

zivkovicmilos marked this conversation as resolved.
Show resolved Hide resolved
subPkgs := make([]*SubPkg, 0, len(paths))
for _, path := range paths {
subPkg := SubPkg{}

matches, err := filepath.Glob(filepath.Join(path, "*.gno"))
if err != nil {
return nil, fmt.Errorf("failed to match pattern: %w", err)
}

subPkg.Dir = path
for _, match := range matches {
if strings.HasSuffix(match, "_test.gno") {
subPkg.TestGnoFiles = append(subPkg.TestGnoFiles, match)
continue
}

if strings.HasSuffix(match, "_filetest.gno") {
subPkg.FiletestGnoFiles = append(subPkg.FiletestGnoFiles, match)
continue
}
subPkg.GnoFiles = append(subPkg.GnoFiles, match)
}

subPkgs = append(subPkgs, &subPkg)
}

return subPkgs, nil
}

// GnoFileSubPkg returns a subpackage from the given .gno files.
func GnoFileSubPkg(files []string) (*SubPkg, error) {
ajnavarro marked this conversation as resolved.
Show resolved Hide resolved
subPkg := SubPkg{}
firstDir := ""
for _, file := range files {
if filepath.Ext(file) != ".gno" {
return nil, fmt.Errorf("files must be .gno files: %s", file)
}

fi, err := os.Stat(file)
if err != nil {
return nil, err
}
if fi.IsDir() {
return nil, fmt.Errorf("%s is a directory, should be a Gno file", file)
}

dir := filepath.Dir(file)
if firstDir == "" {
firstDir = dir
}
if dir != firstDir {
return nil, fmt.Errorf("all files must be in one directory; have %s and %s", firstDir, dir)
}

if strings.HasSuffix(file, "_test.gno") {
subPkg.TestGnoFiles = append(subPkg.TestGnoFiles, file)
continue
}

if strings.HasSuffix(file, "_filetest.gno") {
subPkg.FiletestGnoFiles = append(subPkg.FiletestGnoFiles, file)
continue
}
subPkg.GnoFiles = append(subPkg.GnoFiles, file)
}
subPkg.Dir = firstDir

return &subPkg, nil
}
Loading