Release upgrade scenario tests

[vdemeester]

Related to latest release (0.5.2) that include few problems, with the one of interest here being #1075, a controller panic.
We decided during the post-mortem brainstorm, that we need to have some upgrade scenario tests, a least (or to start with) for the release.

In a gist this means at least the following scenarios

• Install the previous pipeline release, upgrade to the current release, validate that things are working (aka, controller, webhook are not failing, creating and running a pipeline and/or task works)
• Install the previous pipeline release, create and run pipelines and tasks, upgrade to the current release, validate things are working (same as above).

There is also two "side" of this (can be worked at different time)

1. having those scenario in some form (bash, go, …) and be able to run them (manually, on the ci)
2. having a Task that does that (with the assumption that we have a cluster available, see https://github.com/tektoncd/pipeline/issues/534)

This issue is here to track that work.

Note that for master, this would just use ko apply -f config/ to update to current master. But we may want to track that in a different issue

[houshengbo]

/assign @houshengbo

[vdemeester]

Updated the description a bit more.

[houshengbo]

Here is the plan:

1. Add a new flag called --upgrade-tests to the script plumbing/scripts/presubmit-tests.sh, so that
   presubmit-tests.sh supports this upgrade scenario. Then, implement the scenarios in the scripts.

Create a new prow job in plumbing to call the script ./test/presubmit-tests.sh --upgrade-tests, to run the upgrade tests.

2. Another way can be create a new script called ./test/presubmit-upgrade-tests.sh in pipeline repo. Use env variable to indicate whether it is upgrade or not. When ./test/presubmit-upgrade-tests.sh is called, we set to upgrade mode, so that we run the upgrade scenarios. If upgrade is not set, we run the old ./test/presubmit-tests.sh --integration-tests.

Create a new prow job in plumbing to call the script ./test/presubmit-upgrade-tests.sh, to run the upgrade tests.

Which one sounds better? @vdemeester

[vdemeester]

@houshengbo also I feel this should be a periodic job instead of sthg run on each PR. In that sense, I think have a separate script makes more sense 👼