Prevent failures during and after release

[Changaco]

Yesterday we had trouble when deploying #2451 and #2485. Here's what happened:

After manually adding the content of branch.sql to schema.sql, I ran the following command:

heroku pg:psql -a gittip <branch.sql && git rm branch.sql && git add schema.sql && git commit -m 'merge branch.sql into schema.sql' && git push && ./release.sh 13.1.6

The two changes that ended up causing failures are:

1. The branch.sql from Add an is_closed attribute to Participant #2485 that I applied before running release.sh. Doing so shouldn't have caused any problem because the SQL only added a column to a table, which is a retro-compatible change. Unfortunately there's a bug in the libraries we use to interact with Postgresql, so most (if not all) HTTP requests failed in the time between the application of branch.sql and the completion of release.sh.
2. The TESTING environment created by Fix regression from #2379 #2451. I forgot to add it on Heroku, so release.sh worked but the newly deployed app crashed immediately.

The solutions are:

• Fix eliminate race condition when updating types liberapay/postgres.py#26.
• Add a pre-deployment check to ensure that the Heroku env contains all the required variables. We can probably add that easily in release.sh.

[Changaco]

I just merged and deployed #2352. To avoid new transient errors in sentry I ran heroku maintenance:on -a gittip before applying branch.sql.

[chadwhitacre]

Reprioritized from ★★★ to TeamX ★ since this is deployment-related instead of user-facing.

[Changaco]

@whit537 It very much affects users when requests fail or the app crashes.

[chadwhitacre]

Okay, changed back. :-)

[chadwhitacre]

@Changaco Where does this stand in light of #2830?

[Changaco]

@whit537 release.sh now asks you if the maintenance mode should be turned on when there is a branch.sql. liberapay/postgres.py#26 still needs to be fixed though.

[rohitpaulk]

liberapay/postgres.py#26 is fixed, closing this.