Acceptance tests should attempt to download pack & lifecycle pre-requisites when not specified by environment variable

[simonjjones]

Description

When running make acceptance-all, there are a variety of pre-requisites that are expected to be present on my local system, with their locations specified by environment variable. This presents a challenging environment for contributors hoping to validate that changes continue to be compatible with previous versions of pack and lifecycle, as well as builders created with a previous version of pack. The current feedback loop requires either understanding this complex set of requirements, or creating a PR in order to trigger these tests in a GitHub Action

Proposed solution

It is possible to replicate the logic used defined in our GitHub Action workflow to identify which versions of each pre-requisite should be used, and then download these to a local system before continuing with our acceptance suite.

This proposal is to add functionality to our acceptance test suite that will download and consume any pre-requisites that have not been specified from the GitHub locations currently used in our workflows. These downloads should be communicated to the contributor very clearly, with instructions as to how to more explicitly configure them if preferred. The assets should be downloaded to a temporary folder and cleaned up after each run.

If the pre-requisites are locally specified by environment variable, there should be no change to the existing flow.

Describe alternatives you've considered

• Improved documentation for running combinatorial acceptance tests locally in the Contributing document
• Earlier, more explicit failure in acceptance tests, with a clear explanation of which configuration properties and guidelines as to how to manually download the required assets.

Additional context

• Our current workflow for identifying the latest lifecycle asset involves consuming the GitHub Actions Workflows API which appears to require an authenticated user session. This may mean we require a GitHub authentication token as part of our test configuration. It's not clear whether this is strictly a better solution than the existing setup of having contributors manually configure this, so we may be better off falling back on improved documentation & failure messaging. Getting the latest lifecycle is an existing pre-requisite for running make acceptance so contributors should be somewhat familiar with the process.

[jromero]

@natalieparellano based on our discussion:

Issues with current GH actions workflow for compatibility testing:

• Since we are testing against master branch of lifecycle, changes can cause unrelated failures to Pull Requests
  • Contributors should not have to debug something that has nothing to do with their PR
• Downloading artifacts requires some form of authentication. This adds UX complexity for automating the downloading of assets if we were to continue using the artifacts from lifecycle.
• We see the need to continue testing master pack against lifecycle master but these should not prevent a Pull Request from being merged

Reasons why we shouldn’t use previous pack’s DefaultLifecycleVersion:

• Short-lived previous versions (such as releases with major bugs) have longer term impact on our compatibility tests.
  • Alternatives:
    • SemVer
    • Use spec API version
  • Resolution: Ultimately we see the value of using the spec API versions but we don’t have an easy way of retrieving this information at this time so we will try to use SemVer.

Outcome:

• current lifecycle: latest released version of the lifecycle
• previous lifecycle: latest previously released minor version of the lifecycle (using SemVer)
• current pack: current build
• previous pack: latest released version of pack