Find reliable method for verifying contracts on Etherscan

[mrice32]

Why

We've repeatedly had issues deploying via truffle and using truffle flattener to verify the contracts on etherscan. We should find a new pipeline that reliably works.

How

I think either a new flattening or a new deployment method will be required.

Resolution

Contracts are verified either programmatically or manually, but with an obvious, always-working process.

Difficulty Score [1-10]

6: ~1 day or 7: ~3 days (rough estimate, lots of uncertainty)

[adrianmcli]

Turns out @kendricktan was working on this exact issue at Omisego (he was their devops lead after all) and seems to have an understanding of why verifying doesn't work sometimes. So maybe @kendricktan you can share your analysis of the actual problem to get the ball rolling?

[adrianmcli]

Tagging @chrismaree here as well.

[kendricktan]

@mrice32 @chrismaree

Verifying contracts on etherscan has been more of a horror story for me.

I've made an example repository showcasing how slight changes to the contracts location/headers can lead to different generated bytecode. The example repository uses the solc binary to compile the contracts, and not via a build-tool, so the effects should be seen by every build-tool that uses solc.

Findings

1. Contract directory location affects generated bytecode

This is likely because of the contract metadata inside solc. For example, the bytecode generated from compiling identical contracts ContractA from ./contracts/ContractA and from ./contracts/inner-directory/ContractA will be different.

This in theory should not be too much of a issue because swarm hashes should not affect (in most cases) etherscan contract verification

2. pragma experimental ABIEncoderV2 affects generated bytecode

This might explain why some of the flattened contract source code differs from the deployed bytecode, as some sections of the contract might not be using pragma experimental ABIEncoderV2.

When you flatten them, you are forcing the ABIEncoderV2 feature onto them.

3. pragma experimental ABIEncoderV2 inserted location affects generation bytecode

This could be related to the contract metadata, might be something we need to keep in mind.

Next Steps

Given the amount of things that can affect the generated bytecode, perhaps the most consistent way for verifying would be to flatten it as you've mentioned. Steps could roughly be:

1. Flatten the to-be-deployed contract along with its imports into a single file
2. Compile the flatten contract and then deploy respective contract.

What do you guys think?

[nicholaspai]

@kendricktan Even after fixing the ABIEncoderV2 issues, I've still run into mysterious errors. The one process that has worked in the past is to the compile the flattened contract (after fixing the pragma errors as you list above) in remix, deploy via Remix, and use that ABI/bytecode to verify the contracts.

If there's a way to see the Remix source code or dynamically leverage it, this might be the best strategy

[adrianmcli]

@nicholaspai this is the code Remix uses for compiling: https://github.com/ethereum/remix/blob/master/remix-solidity/src/compiler/compiler.ts

They're just using solc under the hood: https://github.com/ethereum/remix/blob/master/remix-solidity/src/compiler/compiler-worker.ts#L3

[mrice32]

@kendricktan I think this is really solid analysis -- +1!

Verifying contracts on etherscan has been more of a horror story for me.

Same. The sensitivity of the compilation system to minute and non-logic-related details has always been quite worrisome to me.

When you flatten them, you are forcing the ABIEncoderV2 feature onto them.

Does this only apply to code that sits below the ABIEncoderV2 declaration or does it affect all code in the file? If it's the former, it might work to just move all of the contracts that had ABIEncoderV2 declared in them below the declaration and move those that didn't above the declaration when trying to verify. Although, that may be just as hard as pre-flattening the files before compilation 🤷.

1. Flatten the to-be-deployed contract along with its imports into a single file
2. Compile the flatten contract and then deploy respective contract.

Yeah, I think this makes sense. Mimicking the exact source that Etherscan sees is probably prudent if we want our compilation system to match consistently, even if it is a little strange to flatten in the local compilation process. This could be why we've seen Remix work.

I do have one final piece of confusion, however. Etherscan has a multi-file upload verification, where you can upload many files that import one another and it will compile them that way. This should in theory almost exactly match what we're doing locally. However, I've found that even that fails to match sometimes (although it has also fixed the problem on occasion too). The difference here may be caused by the metadata differences you mentioned in 1 since the directory structure changes to one single flattened directory. Note: when using this method, you do have to rework the imports so they work in a flat dir rather than our directory structure (we wrote a quick-and-dirty script to do this).

That context aside, I think a good testing ground to ensure the flattening method you have proposed will work is compiling a pre-flattenedVoting.sol, ExpiringMultiPartyCreator.sol, and ExpiringMultiPartyLib.sol and seeing if they all verify successfully. Then, as a final test, we should ensure that an ExpiringMultiParty.sol contract deployed using this ExpiringMultiPartyCreator method can be verified (core/scripts/local/DeployEMP.js will probably be helpful here).

Thoughts?

[kendricktan]

@nicholaspai @mrice32

If there's a way to see the Remix source code or dynamically leverage it, this might be the best strategy

Remix uses solcjs under the hood, which is javascript bindings for solc 😃

Does this only apply to code that sits below the ABIEncoderV2 declaration or does it affect all code in the file? If it's the former, it might work to just move all of the contracts that had ABIEncoderV2 declared in them below the declaration and move those that didn't above the declaration when trying to verify. Although, that may be just as hard as pre-flattening the files before compilation

It seems like it'll affect all code in the file. In the example repo, the generated bytecode from contracts A, and D differs quite significantly in structure (more than just a swarm hash difference). For context, Contract D is basically Contract A, but with pragma experimental ABIEncoderV2 appended at the bottom.

I do have one final piece of confusion, however. Etherscan has a multi-file upload verification, where you can upload many files that import one another and it will compile them that way. This should in theory almost exactly match what we're doing locally. However, I've found that even that fails to match sometimes (although it has also fixed the problem on occasion too). The difference here may be caused by the metadata differences you mentioned in 1 since the directory structure changes to one single flattened directory. Note: when using this method, you do have to rework the imports so they work in a flat dir rather than our directory structure (we wrote a quick-and-dirty script to do this).

Sounds like a horror story 😛

That context aside, I think a good testing ground to ensure the flattening method you have proposed will work is compiling a pre-flattenedVoting.sol, ExpiringMultiPartyCreator.sol, and ExpiringMultiPartyLib.sol and seeing if they all verify successfully. Then, as a final test, we should ensure that an ExpiringMultiParty.sol contract deployed using this ExpiringMultiPartyCreator method can be verified (core/scripts/local/DeployEMP.js will probably be helpful here).

Sounds good!

[kendricktan]

A bit of an update. Without any additional scripts, I've managed to verify the contracts ExpiringMultiPartyCreator and Voting on etherscan.

Steps

Without any additional config, in the core directory

1. npx buidler compile
2. There should be a new file in core/cache which contains the standardized input method for solidity, solc-input.json
3. Edit solc-input.json and make sure its using whatever runs truffle is (199 at this time of writing), and add in the following to the settings key:

"settings": {
    ...
    "libraries": {
      "contracts/financial-templates/expiring-multiparty/ExpiringMultiPartyLib.sol": {
        "ExpiringMultiPartyLib": "0xB6693A4cB6CFd70680145e733388C7FE8b778841" // ExpiringMultiPartyLib deployed address
      }
    },
    ...
  }

4. Verify contracts using etherscan's Standard-Json-Input
   

[nicholaspai]

@kendricktan Is there an easy way to determine the ABI-encoded constructor arguments passed into contracts? Or do we just use an external tool

[kendricktan]

@nicholaspai iirc etherscan automatically finds the ABI-encoded constructor agrs for you.