TIP #1: Migrations as Modular Scripts

[tcoulter]

  TIP: 1
  Title: Contract Migrations as Modular Scripts
  Status: Draft
  Type: Awesome
  Author: Tim Coulter
  Created: 2016-04-18

Introduction

Truffle's current deploy process only allows for a very strict type of contract: One that doesn't take constructor parameters, and one that's meant to exist as a singleton. For many people these restrictions are limiting and confusing, as deployment for different styles of contracts is needed. Furthermore, no matter the contract, Truffle's deploy process fails to address the needs of future deployments. This proposal aims to provide a solution that allows for deploying complex contracts in a way that doesn't introduce that complexity back into Truffle itself; as well, this proposal will outline a method for managing multiple deployments across the lifetime of a project.

Currently, the only recourse for developers who want to perform complex deployments is to run after_deploy scripts. These are scripts that can be run with truffle exec, and will be run directly after deployment and whenever a deployment is needed. The benefit of after_deploy scripts is they can run arbitrary code during deployment, allowing for as complex of a deployment as the user needs. The negatives, on the other hand, is that after_deploy scripts are hard to use and that the addresses of deployed contracts within the scripts are not saved.

after_deploy scripts, as well, don't address future deployments, as the initial scripts written would assume contracts related to the dapp don't already exist on the desired network. When a future deployment is needed, when a project's contracts do exist -- which need to be modified or updated as a course of the deployment -- the user would be required to manage those scripts depending on the environment in a way that's tedious and error prone.

Solution: Migrations

To address all of the issues mentioned above, let me introduce the concept of Migrations. This idea is heavily influenced by Rail's ActiveRecord migrations, but is much lighter weight.

1. Migrations, in general, are a set of scripts identified by their file name and prefixed by a number, that are responsible for initializing and maintaining the state of a set of contracts on a given network. The numbered prefix represents the date and time of the migration, and the rest of the file name is meant to communicate the purpose of the migration. Example: 1461005828324_initial_deploy.js. Here, the numbered prefix was produced by new Date().getTime(), but it can theoretically be any number.
2. Migrations assume that regardless of intentions, over the lifetime of a dapp contract state will need to be modified and contract code will either need to be updated, modified, or destroyed in order to support new features and bug fixes.
3. Migrations are simple Javascripts modules that export the code they want to run and are expected to call a done() callback when finished, outlined below. This structure allows users to execute complex deployment steps without running into the same issues with after_deploy scripts mentioned above.
4. Migrations will be run in the order of their filename, lowest first. Migration state -- i.e., a record of which migrations have been run on a given network -- will be saved in a special Migrator contract deployed to the network, so on future deployments Truffle can "intuit" which migration to run next. This contract will be added automatically to the user's project upon creation (truffle init). If the Migrator contract doesn't exist or is deleted by the user, Truffle will instead run all migrations in order, and will not save the migiration state upon completion.
5. A new top-level directory will be created for migrations, called migrations. Truffle will determine which migrations need to be run based on the network's saved migration state as well as the files within the migrations directory.
6. Truffle's current deployment architecture will be completely removed except for automatic library linking. This includes removing the deploy configuration from the project's truffle.js file. The truffle deploy command will now run the migrations instead of deploying contracts specified within the deploy configuration.

Discussion

One cause of contention is saving the migration state on the blockchain. Theoretically this state could be saved in a file within the target environment. The benefits of having it on the blockchain is that in my experience, it's easier to manage and the migrator contract could be edited to act differently if the user desired. On the other hand, the largest negative is that deploying and interacting with the Migrator contract would cost real Ether on the public network. I'm open to other's opinions on this (as well as anything else) and would appreciate your thoughts.

Another cause for contention is that as of now, migrations aren't atomic. If a migration fails, there's no way to automatically revert the transactions within the migration that succeeded, as you can with Rails. This likely isn't a feature that can be provided due to the nature of the Ethereum network, but ideas welcome.

Lastly, there is currently no notion of migrating both "up" and "down" as provided in Rails -- currently they only happen in one direction. I would love to hear if this feature would be useful.

Code & Examples

Here are some example migrations as well as proposed solidity code for the migrator contract:

Example Migration: Initial deploy

This will deploy a new contract to the network and save that contract's address.

File name: 1461005828324_initial_deploy.js

module.exports = function(accounts, done) {
  // Add a new contract to the network
  MyContract.new(function(instance) {
    // Tell Truffle that this new contract represents
    // the deployed version of this contract
    MyContract.address = instance.address;
  }).then(done).catch(done); 
};

Example Migration: Upgrading a Contract

This will expect a currently deployed contract and update a value (Hub & Spoke model).

File name: 1461005828324_update_spoke.js

module.exports = function(accounts, done) {
  // Expect an already deployed Hub.
  var hub = Hub.deployed();

  // Deploy a new spoke. 
  Spoke.new(function(spoke) {
    // Tell the hub about the new spoke.
    return hub.setSpoke(spoke.address);
  })).then(done).catch(done);
};

Proposed Migrator Contract

Effectively pseudo-code. Subject to change.

contract Migrator {
  uint[] public completed_migrations;

  function Migrator() {}

  function migrate(uint migration) {
    completed_migrations[completed_migrations.length++] = migration;
  }
}

Feedback

All feedback is welcome. This TIP will remain active until a sufficient consensys is reached. Please leave all comments on github. Thanks!

[rfikki]

Any possibily for a suicide/selfdestruct feature for removing unwanted contracts.

[tcoulter]

selfdestruct is a function of your contract and can be called from within a migration like any other transaction. However, you would need to add the self-destruct command do your contract youreself: http://solidity.readthedocs.org/en/latest/units-and-global-variables.html?highlight=destroy#contract-related

[danfinlay]

First, a couple clarifications:

• In the 1461005828324_initial_deploy.js, MyContract is the Hub contract?
• When updating the Spoke contract, we're actually deploying a new Spoke, and updating a pointer within the Hub?

I'm really excited about this realm of thought. Truffle as it stands is a fantastic way to build simple Dapps, but for sustainable contracts, this is important stuff to deal with.

I think the migration structure is clever, and I like how it allows multi-phase architectures to be constructed.

Of course, since this is such a big change, I'd just like to make sure we get as many concerns on the table as possible before committing to the work.

First off, if we're going to adopt migrations, it seems like we might as well have the ability to roll them back. This could be achieved by storing an array of the previous migration addresses, or maybe just the ability for a future migration to point back to an older one. Whatever the method, I think we'd regret if we could only migrate to new contracts.

Secondly, as a member of the Ethereum community who loves its distributed nature, leaving one contract creator with the keys to change the contract at all times seems like a re-centralization of power. A user, even on reviewing the contract, has no guarantee that the contract won't be spontaneously updated to different terms.

I guess this doesn't need to be an issue, as long as we keep the possibility for different types of migrations. Some migrations may require a vote of an approving board, which is essentially the premise of Boardroom by @SilentCicero.

I guess I'm just making sure that the migration structure is open-ended enough that people could write their own migration contracts, which might require an asynchronous block step of waiting for a third party to sign off on a change.

[tcoulter]

Thanks @FlySwatter. Responses inline:

In the 1461005828324_initial_deploy.js, MyContract is the Hub contract?

The example migrations provided were meant to be completely separate codebases, existing at different stages of a project's lifecycle. However, they could be the same for example purposes.

In the first example, no contract exists on the network yet -- this is the project's initial deploy. Here, most everyone would need to deploy new contracts like shown in the first example. The second example is a hypothetical future migration on a different project, where a hub and spoke architecture exists and some changes have been made to the spoke, thus necessitating a new deploy of the spoke and an update of the Hub.

When updating the Spoke contract, we're actually deploying a new Spoke, and updating a pointer within the Hub?

Correct. Again, this is a hypothetical situation but one that is likely to happen in real life (I've coded dapps like this before).

First off, if we're going to adopt migrations, it seems like we might as well have the ability to roll them back. This could be achieved by storing an array of the previous migration addresses, or maybe just the ability for a future migration to point back to an older one. Whatever the method, I think we'd regret if we could only migrate to new contracts.

It's important to note that the examples given are arbitrary code. A migration doesn't necessarily create new contracts -- just as with Rails migrations, a migration could move data around, update a value, remove an item from the list, etc. Given this, it's unclear how to create an automatic way to rollback a migration. At best we can do what Rails does, and have an up and a down target, that specifies direction, where we require the developer to write the down, but in my experience most people don't test their downs, and they are rarely used (in practice, it's mostly for emergencies). I could be persuaded to adopt this model as it aids in development, especially when testing migrations themselves, but it adds significant complication to the architecture including, as you mentioned, saving old addresses. Moreover, even if we were to implement this architecture, since migrations are non-atomic, depending on where a migration fails there's no guarantee the down will run correctly either, leaving the network and your contracts in a worse state. Given this, it's likely better that if you want to "revert" to a previous state, like git, your "revert" is an action that moves forward in time (i.e., a new migration).

Secondly, as a member of the Ethereum community who loves its distributed nature, leaving one contract creator with the keys to change the contract at all times seems like a re-centralization of power. A user, even on reviewing the contract, has no guarantee that the contract won't be spontaneously updated to different terms.

Can you explain this in more detail? You bring up a good point in that the Migrator contract needs an owner, and should only be updated by the owner, but don't believe that was your main point. In my proposal above, the creator of all contracts -- including the Migrator contract -- are owned by the person deploying them. You can already set the deployment address in Truffle. If you want your Migrator contract to be owned by multiple groups, you can edit the contract yourself to do so. In fact, you can even migrate the Migrator contract, but maybe we should add hooks to make that easier.

I guess this doesn't need to be an issue, as long as we keep the possibility for different types of migrations. Some migrations may require a vote of an approving board, which is essentially the premise of Boardroom by @SilentCicero.

The world is your oyster @FlySwatter. :) The migrator contract just holds migration state: i.e., the last migration that has been run. It doesn't specify what the migration actually is. For that, you must specify it in your migration code, which as stated above can be anything you want.

I guess I'm just making sure that the migration structure is open-ended enough that people could write their own migration contracts, which might require an asynchronous block step of waiting for a third party to sign off on a change.

Again, the Migrator contract is owned by you and you can edit it however you want, as long as it upholds a simple API. If you'd like to change it so that it has multiple owners, you can.

[danfinlay]

Cool, thanks for the thorough answer, @tcoulter!

I'm still on the fence re: up/down migrations, but it sounds like you're designing this migration path to be generic enough that it can be structured in whatever way a person wants, so it sounds good!

[chrisclark]

Would it make sense to address these constraints:

One that doesn't take constructor parameters, and one that's meant to exist as a singleton.

separately from updates/migrations? I don't see that they are inextricably related.

[tcoulter]

@chrisclark I'm not sure it makes sense to (or put differently, this is a way of addressing that). We could create a system that allows for that and works in addition to after_deploy scripts, but then what do you do down the line when you've had couple contracts in production for a few weeks and realize a bug needs to be fixed? The current deployment implementation won't support that. If we move to a migration infrastructure, you can deploy with constructor parameters as well as create a system for future deployments.

Note that, if it's not clear, the .new() function in MyContract.new() is where you specify constructor parameters if any exist.

[zmitton]

Great ideas. This addresses one of the biggest unanswered questions I've had with Ethereum in general: how to update imperfect code.

Having said that, I AM concerned with having the OPTION to keep the contract immutable (in an easy way). Could there be maybe a flag upon project initialization that somehow specifies this? Either by disabling the feature, or maybe automatically writing a migration contract with rules that it can't be updated?

[tcoulter]

Hi @zmitton. I'm not sure I understand. By definition contracts deployed to the Ethereum network are immutable, and there's no way to get around this. The closest option available is to create a "pointer" contract which contains an address of the actual contract meant to service a dapp. When you want to update the dapp's code, you deploy a completely new contract and then update the pointer (an example of this is the Hub and Spoke, above). But all of this is not a function of Truffle; the migrations above are just examples. If you want this pointer functionality you'll have to build it into your dapp yourself.

[redsquirrel]

Example: 1461005828324_initial_deploy.js. Here, the numbered prefix was produced by new Date().getTime(), but it can theoretically be any number.

At the risk of stating the obvious, I can't imagine not using a timestamp as the prefix. They have the double-benefit of sorting your migrations in sequence, as well as ensuring the migration filenames are unique.

[redsquirrel]

If the Migrator contract doesn't exist or is deleted by the user, Truffle will instead run all migrations in order, and will not save the migiration [sic] state upon completion.

It seems like in this situation, it should either auto-create the Migrator contract, or possibly prompt the user about whether a Migrator contract should be created. Otherwise, it seems like it's leaving the dapp in a bad state.

[tcoulter]

@redsquirrel

It seems like in this situation, it should either auto-create the Migrator contract, or possibly prompt the user about whether a Migrator contract should be created. Otherwise, it seems like it's leaving the dapp in a bad state.

For new projects (i.e., truffle init) I was thinking about making the Migrator contract deployed as the first migration, and Truffle would use it to record the migration state only if a contract called Migrator existed -- and it would check for the Migrator contract after each migration completed, in case the Migrator had been updated.

That said, "Migrator" should probably be renamed as it's not doing the migration. It's just recording the state. Perhaps MigrationState.

[thiagodelgado111]

That sounds awesome, @tcoulter. I'm really excited about that feature :)

[redsquirrel]

Lastly, there is currently no notion of migrating both "up" and "down" as provided in Rails...

I think this is fine. Being able to migrate "down" in Rails is nice, but I've never actually seen it used in production. I've only ever seen it used to revert/reset local schema changes while developing migrations and/or new features.

[tcoulter]

I agree @redsquirrel. Do we need that development benefit in our case, however? I was thinking about getting around that issue with something like this:

$ truffle migrate --all

This will just start from scratch. When used on the TestRPC, things will be quick. This assumes A) your migrations aren't doing heavy processing, and so won't take awhile (i.e., you won't be merging thousands of rows like in Rails); and B) you won't have hundreds of migrations. That said, the up/down interface could easily be added in later with little effort if ever it was needed.

[redsquirrel]

@tcoulter I imagine it would be nice to be able to re-run all migrations (assuming you're on TestRPC) to be able to get yourself out of a bad state, as well as ensuring that your migrations do actually play nice with each other. In general, though, I'd punt most of these questions until you get feedback from people using migrations in the wild. (In other words, keep it simple. 😄)

[tcoulter]

Ya, I'm in the same boat. On the verge of implementing it to get better feedback. So far response is positive, and so comes down to implementation.

[redsquirrel]

@tcoulter Is there a branch we can watch? 👀

[tcoulter]

Just pushed the branch migrations to master. No differences from master yet, but changes will be added there.

[tcoulter]

I've created a pull request that should make watching easier: #143

[tcoulter]

While developing the migrations feature, I’m finding that migrations can be made easier if there were an object to help you with the transactions. I’ve recently whipped up an example — not quite a prototype — where a migration could look something like this:

module.exports = function(accounts) {
  deployer.autolink();
  // and/or
  deployer.link(SomeLibrary, [
    SomeContract,
    AnotherContract
  ]);

  deployer.new(SomeContract);
  deployer.new(AnotherContract);

  deployer.then(function() {
    return SomeContract.deployed().callSomeFunction(5);
  });

  deployer.then(function() {
    // Do more complex things.
  });
};

Here, the deployer object is your helper, and it can provide functions such as “autolink” (link libraries automatically based on their dependencies), “link” (link specific libraries to specific contracts), “new” (deploy new version of the contract and ensure the address gets saved), and then more general, “then” (perform any arbitrary step in deployment). All objects like SomeContract are Pudding contract objects, and through the magic of promises your migration steps can be written synchronously (i.e., the function just creates one long promise chain stored in deployer and executes it later rather than executing each step in the function itself).

The point of all this is to provide a mechanism for people to write their own complex deployments without making a tool that's equally complex, while at the same time making deployments as easy as possible. Thoughts?

[thiagodelgado111]

Maybe we could follow Rails up and down idea? Would that be possible? I like the idea of the deployer maybe we could name it as helper or deployHelper something like this to make it more clear.

Can we have a fallback function to call in case of errors during migration?
e.g:

migration('Adding "users" mapping to contract"', function(args, helper, accounts) {

//to be used to change the value of a flag or bind a new contract address to a contract
 helper.up(function(done) { 
   let oldContract = Contract.at(args.address);
   oldContract.change
 })

//to be used when something is done and cannot be undone or at least easily undone
 helper.change(function(done) { 
   let oldContract = Contract.at(args.address);

   UpdatedContract.new()
   .then(function(_newContract) {
      return oldContract.upgradeTo(_newContract.address);
   })
   .then(function(txHash) {
      console.log('Contract upgraded!');
      done();
   })
   .catch(done);
 });

 //to be used to undo what "up" is doing
 helper.down(function(done) {

 });

 helper.onError(function() {
   //notify the user, fix something on the blockchain, create a log entry somewhere, something like that
 });
});

[tcoulter]

@thiagodelgado111 Couple things:

Having an up/down concept is a good one, in general, but undefined behavior will occur whenever a migration errors. For instance, imagine a migration that, when migrating up, makes 20 transactions (possibly an exaggeration). Say transaction 11 fails: This means the whole migration fails. Since we're dealing with a blockchain, you can't revert those transactions, which means you're in an unknown state. If you migrate down from there, it's likely that your down will fail as well since the down expects the environment to be in a specific state, which it is not. This is a similar problem with rollbacks: How do you rollback a migration that can't be reverted? The only way is human interaction, where you have to determine the best possible rollback yourself - and it's entirely situationally dependent. Which means - to recap - we could have a down and a rollback, but they're unlikely to work in most cases, and so in most cases are very useless. The only way around this that I can see is to have one transaction per migration, but this becomes untenable - though I can imagine some people going this route.

Also, in my experience with other migration systems (i.e., Rails) down and rollback were rarely used in production, and were only useful during development. Though I can see the benefit of having these features during development, we there are other ways to solve that problem, like testing/developing from a known blockchain state, which can be reverted. That said, you've made me think of a great feature: automatic reverts when using the TestRPC. The TestRPC supports state reverting, so when developing migrations, truffle can take advantage of this and automatically revert if there's an error during a migration.

Given the above, out of the gate I'm likely not going to add in support for down and rollback, but we can consider those features once more people use the first implementation and we get feedback from users.

Thanks, as always!

[thiagodelgado111]

Hi, @tcoulter! Thanks for reading my comment and answering it :)

When I thought about having a rollback/down function I thought that it could be useful for little things e.g: reverting a change I did in a contract state variable modified on this migration. The same applies to up, it would be used for little things as well like the one I mentioned before e.g: a single transaction changing the state of a variable, calling a constant function, maybe even things that are not strictly related to the blockchain like calling a service to get data from it, anything like that.

Anyway, I agree it might not be really useful. Would you still consider instead having:

• A setup function to be used to put up anything that's needed for the next function
• change that would be the place for the 20 txs to be executed
• And maybe one called onError, to be called if the migration fails (I could log an event on a bug tracking service, log it on a file, etc.)

var myAppData = require('myapp/libs/data');
migration('Adding "users" mapping to contract"', function(accounts) {

 helper.setup(function(done) { 
   migrationData.params = myAppData.getParams();
 })

 helper.change(function(done) { 
   let oldContract = Contract.at(args.address);
   var newContract;
   UpdatedContract.new()
   .then(function(_newContract) {
      newContract = _newContract;
      return oldContract.upgradeTo(_newContract.address);
   })
   .then(function() {
      return newContract.use(migrationData.params);
   })
   .then(function(txHash) {
      console.log('Contract upgraded!');
      done();
   })
   .catch(done);
 });

 helper.onError(function() {
   //notify the user, fix something on the blockchain, create a log entry somewhere, something like that
 });
});

Also, I'm glad I ended up helping you to have the automatic revert idea with TestRPC hahaha 😄
Again, thanks for reading and for your feedback :)

[tcoulter]

@thiagodelgado111 The deployer system is still being fleshed out, but currently it works like this:

A "migration" is a set of steps that need to be performed. In your migration, you build these steps and add them to the list - you don't actually execute the steps. The steps are executed behind the scenes after your migration has finished queueing them. Given that, your notion of setup and change (as far as I understand them) are just special versions of a more general idea: setup is just the first step, and change is a step somewhere in the middle. Here's what your example might look like in the implementation I have now:

var myAppData = require('myapp/libs/data');
module.exports = function(deployer) {
  // Get the existing contract address before the deploy starts.
  var existingAddress = Contract.address;

  // Replace the version of Contract. deploy() will do this automatically
  // if it has previously been deployed in an earlier migration.
  deployer.deploy(Contract);
  deployer.then(function() { 
    // Call the upgradeTo function on the existing contract.
    Contract.at(existingAddress).upgradeTo(Contract.address);
  }).then(function() {
    // Call the use() function on the new contract.
    // Remember: A new version was deployed in the previous step.
    Contract.deployed().use(myAppData.getParams());
  });
});

As you can see, the structure is very linear, and cleaner than typical Pudding though it's taking advantage of Pudding constructs considerably.

Note that I removed your setup step entirely as it was just a function call (getParams()) that could be added later on in the code. I should also add that Truffle is taking care of telling the deployer what environment it's working in; setting up the web3 providers, setting the deployment account, etc. So you don't have to deal with any of that within the migration.

The only feature I haven't implemented that you suggested is the onError feature. Good suggestion. I need to think about implementation, but I'll do my best to support your case. My hunch, however, is it might be best to have a CI tool take a look at the truffle migrate exit code (non-zero on errors) and send you the email instead.

[thiagodelgado111]

Cool, @tcoulter! I think you're right about leaving the task of sending an e-mail to the CI tool, I just thought it might be useful to have something like this! Thanks for answering! 😃

[tcoulter]

This has been added in Truffle 2.0. Closing! Thanks for everyone's feedback!