GRACE_MODULE_PATH does not behave as expected

[IsaacOscar]

If I have a file main.grace

import "util" as U
U.printhelp

And a file util.grace (in the same folder)

method printhelp { print("help") }

It prints:

Usage: /usr/bin/grace [Mode] [Option]... FILE
Compile, process, or run a Grace source file.
...

Indicating it is using the util.grace file from /usr/lib/grace/modules, and not the local directory.
This behaviour has caused me quite some confusion! at the very least I would like a warning so I could tell what was going on. A better option is probably to have some kind of standard prefix, e.g. platform/util, like kernan has.

[apblack]

From the command line, what minigrace ought to do is find files on your GRACE_MODULE_PATH in the order of the directories on that path. minigrace --verbose 60 (or anything ≥ 60) will show the progress of the search. I don't believe that minigrace does anything special for util, but it might. (util is treated specially in the browser.)

If minigrace is not following GRACE_MODULE_PATH correctly, then you should rename this issue and perhaps take a look at fixing it. That will mean that I have tried and failed, so fresh eyes may be valuable.

Providing path names (relative to the importing file) does work at compile time, but can be tricky at runtime. Will the object code correspond to the source code? Other compiled language have the same problem. I honestly can't remember the current status of importing files at runtime when the importing module uses a path. What I know will not work is having two modules with the same file name.

[IsaacOscar]

Nope, chaning GRACE_MODULE_PATH dosn't fix it:

# env GRACE_MODULE_PATH=$PWD:/usr/lib/grace/modules mgc --verbose 60 main.grace
minigrace: main: 0.1 (+0.1): starting compilation
minigrace: main: 0.1 (+0): lexing.
minigrace: main: 0.12 (+0.02): parsing.
minigrace: main: 0.14 (+0.02): checking dialect standardGrace used by module main
minigrace: main: 0.14 (+0): checking module "standardGrace"
minigrace: main: 0.14 (+0): found module "standardGrace" in /usr/lib/grace/modules/./standardGrace.js
minigrace: main: 0.18 (+0.04): checking module "collectionsPrelude"
minigrace: main: 0.18 (+0): found module "collectionsPrelude" in /usr/lib/grace/modules/./collectionsPrelude.js
minigrace: main: 0.22 (+0.04): no need to load dialect "standardGrace": it does not define `thisDialect`
minigrace: main: 0.22 (+0): rewriting tree.
minigrace: main: 0.24 (+0.02): checking module "util"
minigrace: main: 0.24 (+0): found module "util" in /usr/lib/grace/modules/./util.js
minigrace: main: 0.26 (+0.02): checking module "fastDict"
minigrace: main: 0.26 (+0): found module "fastDict" in /usr/lib/grace/modules/./fastDict.js
minigrace: main: 0.27 (+0.01): checking module "unixFilePath"
minigrace: main: 0.27 (+0): found module "unixFilePath" in /usr/lib/grace/modules/./unixFilePath.js
minigrace: main: 0.29 (+0.02): symbol tables built.
minigrace: main: 0.31 (+0.02): generating JavaScript code.
AST nodes compiled:
    identifier  2
    import      1
    member      1
    method      1
    dialect     1
minigrace: main: 0.35 (+0.04): done.

(Using GRACE_MODULE_PATH=.:/usr/lib/grace/modules also has the same behaviour).

However if I run mgc util.js first, running env GRACE_MODULE_PATH=$PWD:/usr/lib/grace/modules mgc main.grace works as expected.

[apblack]

@IsaacOscar, can you please create a test for this issue, and ascertain whether or not it is fixed by Pull Request 296 ?

[apblack]

Here is a proposal for the semantics of import, when running minigrace at the command line. It tries to comply with the language specification, which says "The distribution medium for Grace programs, objects, and libraries is Grace source code." (When running in the browser-based IDE, there are no paths, and no search, because the IDE's "file system" ensures that module names are unique.)

1. The import statement import "moduleName" will cause minigrace to search for a source file with name moduleName.grace. If the file name is absolute, we look there. Otherwise, we look for a file moduleName.grace, starting in the directory containing the file with the import statement, and then proceeding to each directory on GRACE_MODULE_PATH, in order.

2. If no such file is found, an EnvironmentException is raised.

3. If a file is found, its SHA256 is calculated.

4. The compiler will then search for a moduleName.js file that has been generated from source code with the same SHA256, and by the current version of the compiler. To enable this, the source SHA256 and the version of the compiler are written at the head of every output file that the compiler creates. The search will look in

   • the directory in which the source file was found,
   • the compiler's output directory,
   • the current directory, and
   • the directories on GRACE_MODULE_PATH.
     The order of the search matters only for performance, not for correctness, because outdated files (whether created by an old compiler or from different source code) are simply ignored.

5. If no suitable moduleName.js is found, minigrace will compile a new one. This will be written to the directory in which the source file was found, unless a different directory was specified to minigrace with the --dir flag, in which case it will be written there. (This is the rationale for looking in these places first when searching for an existing compiled module.)

What about the case of import "path/to/moduleName". One possibility is to ban this altogether, and rely on GRACE_MODULE_PATH. But I hates environment variables. My precious. Curse them, we hates them! So, instead I propose:

   6. If path/to/moduleName starts with a /, we use the absolute file name as given as the location of the source file. If it is found, we proceed from step 3 above. If it is not found, from step 2 above: there is no search.

   7. If path/to/moduleName does not start with a /, we append the whole path path/to/moduleName to the places listed in (1) above.

   8. The compiler will then search for a moduleName.js file that has been generated from source code with the same
   SHA256, and by the current version of the compiler. The search will look in
     * the directory in which the source file was found,
     * the compiler's output directory,
     * the current directory, .
     * in the directory, ./path/to
     * the directories on GRACE_MODULE_PATH.

    Should it also look, for each p in GRACE_MODULE_PATH, in directory p/path/to? I find it hard to answer these questions, because I don't have a use case. That's why I'm tempted to simply prohibit paths in import statements.

[KimBruce]

Suppose I have a folder f that contains all my libraries. Hence I will place f in my GRACE_MODULE_PATH. Now for organization reasons, I have two sub-directories, one for my intro class libraries “intro” and the other for my data structures related libraries “ds”. How do I get to the libraries in “intro”? import “f/intro/mylib” If that doesn’t work, is there any other way I can get to it without included the complete path (/…)? This kind of access could be useful if students are working both in lab (on our computers) and their own home machines. They will not have to change the import statements as they move back and forth. Admittedly I could put all of the needed directories in the path, but this likely would be a useful scenario. Not critical, just useful.

[apblack]

If f is on GRACE_MODULE_PATH, then you would presumably like to write

import "intro/mylib"

in the importing module, and have minigrace find the module in f/intro/mylib.grace. So you are arguing for a "yes" answer to the final question above (Should it also look, for each p in GRACE_MODULE_PATH, in directory p/path/to?).

Thanks for the use case! I can see that this might be handy.

[apblack]

This proposal, including allowing absolute paths and paths with slashes, has been implemented in 0cf0402.