truffle compile becomes quite unpredictable if you move your artifacts from another location on disk

[wbt]

Summary

Truffle is compiling contracts from a directory way outside the project where the command is entered, to include the \contracts\ directory of a different copy of the project elsewhere on the computer.

Issue

I have a Truffle project on a Windows 10 machine, stored in a path like Documents/repositories/myprojectname. In an update some months back, OneDrive decided to move the Documents folder and all subfolders under OneDrive, for automatic backup of files even when not committed to a version control system.

When having some difficulties with OneDrive, I unlinked the computer and made a complete copy of the directory to a location outside of OneDrive syncing, prior to a re-initialization. On the command line, I navigated to that copy and ran truffle migrate --reset which first runs truffle compile. This output a list of files it was compiling, which included all the .sol files in the .\contracts\ directory (as I expected), as well as all the files in the OneDrive folder's \contracts\ directory, showing a file path going a few levels up .\..\..\... and back down into that folder.

This was highly unexpected. Why is that?

My Truffle config file only specifies a compiler version and a couple networks, with no directories mentioned.

Steps to Reproduce

Unfortunately, I don't have a set of steps to reliably reproduce this issue handy at the moment. The migration failed, on the grounds that a called function had the wrong number of parameters ("Got 1 expected 2!") which appears to be a separate issue I observe on occasion, that also appears to be a serious but intermittent compiler bug.

Usually the fix is to just run truffle compile again (sometimes first deleting the \build\ directory manually), so I tried that. This time, it reported compiling only the contract files I expected. In both cases, the "Artifacts written to" directory was the \build\contracts directory of the project I was currently in, as expected.

However, the first run remains concerning, because Truffle should not be compiling files outside the current project.

Expected Behavior

Only the contracts under the current folder/project's .\contracts\ directory get compiled.

Actual Results

The contracts in the current folder/project were compiled, as well as contents of the \contracts\ directory of another (much older) copy of the project elsewhere on the same hard drive. Other solidity files within the project (either version) but not in its projectName\contracts\ directory were not compiled, nor were contracts from other projects under \repositories\ or anywhere else on the machine. Per expectation, copies of contracts in an unusually named directory's \contracts\ subfolder (within the same project) were also not compiled.

Environment

• Operating System: Windows 10 Pro
• Ethereum client: Doesn't matter
• Truffle version (truffle version): v5.0.14
• node version (node --version): v9.3.0
• npm version (npm --version): 6.9.0

I am not aware of any shortcuts, junctions, hard links, mappings, etc. between the directory structure where I was editing and the other one where things were compiled from.

[eggplantzzz]

Thanks for reporting this @wbt, we'll check it out. Very strange.

[haltman-at]

Huh, I'd noticed this problem as well. I hadn't thought of it as a real problem, but I guess maybe it should be considered one.

[gnidan]

This might be related to #2599 ("Resolver.resolve doesn't make any sense")

[gnidan]

This will be a challenging one to sort out. We've discussed possibly modifying the sourcePath inside the artifact so as to use relative paths, but of course, that solution has trickiness.

Might be worth noting @eggplantzzz raises the point that if you specify truffle compile --all, this problem won't occur.

@haltman-at confirms that this problem still exists (but with slightly different symptoms today).

Unclear how to proceed in short-term vs. long-term. @cds-amal suggests maybe we can start addressing this with an "error better" style fix.

[wbt]

Why is Truffle using absolute paths at all, and where is it getting those paths from?
truffle compile should be pulling from smart contract sources within the project, and writing to the build/contracts directory. It should not be reading from the build/contracts directory except possibly to copy the deployed addresses of a contract that compiles to the same [deployed]bytecode hash as a prior version.

We've discussed possibly modifying the sourcePath inside the artifact so as to use relative paths, but of course, that solution has trickiness.

Until this post, I did not know the build/contracts directory (artifacts) even had any absolute paths in them. In our workflow, we are regularly moving those files around even between computers with very different file systems (and even different operating systems) for their ABI + deployed address information; some are even publicly served for in-browser use and the folder structure information revealed by absolute file paths unnecessarily reduces security. From a user perspective, there's no communication or understanding of the reason for including that, and its inclusion is quite unexpected behavior in itself (plus it leads to even stranger behavior like what's documented at the start of this thread above).

Perhaps using an IPFS CID (even if the file is not stored in IPFS) or other content-based file identification system would be better?

[eggplantzzz]

Currently Truffle uses that path to optimize compilation. It checks when the artifacts' were last updated and compares that to when the source files were updated to see which files require compilation and which ones don't. So it uses it to find source material for a given compiled artifact. Maybe there is a different way to do this without requiring the use of absolute paths?

[wbt]

A couple strategies come to mind:

1. Use content addressing: compute the CID of each file in the contracts source directory, store those in the artifacts instead, and otherwise follow a procedure pretty similar to the status quo. This is slightly slower but much more robust and solves the security + unanticipated behavior problems of the status quo. It also allows for some interesting potential added functionality, e.g. if a dapp/wallet with access to the artifact (for ABI + deployed location information) is able to check IPFS for the source it can do its own verification and present more usable UI for transaction confirmation without having to rely on another centralized service like Sourcify. Much of that can in theory be done today since the artifact contains the source, but optional IPFS pinning of particular versions allow for connections to endorsements, audit reports, trust-level certifications, easier Remix integrations, and other exciting potential expansions of functionality outside Truffle.

2. Use relative paths in the artifact files.

3. Emit an additional file, perhaps a sibling to the truffle-config file, which lists relative paths and last-modified times, and remove in-artifact references to the source path.

4. Drop the optimization and always recompile all smart contract source files. Of these four, it's the slowest, but it is still strongly preferred over the unexpected behavior of the status quo. This is probably also the only option that doesn't require a major version bump.

In our workflow, smart contracts are actually written by scripts as part of the build process, so their last modified dates will always be extremely recent and the truffle compile stage will always recompile them on that basis, so the first option is strongly preferred. Use of absolute paths is definitely to be avoided, especially for collaborative development and review (incl. auditing) processes.

[eggplantzzz]

@gnidan thoughts?

[gnidan]

I concur that we should get rid of absolute paths inside the artifacts. Now that we use that project: prefix for paths, we could probably switch to that. (Is that a breaking change? Maybe not, if we continue to support the existing absolute paths, but new artifacts get output with project:/ paths instead.)

Note that this would require @truffle/resolver to be updated so that resolver.resolve() accepts these project:/ style paths, which it currently does not. I'll make a new issue for that.