feat(build): conditional ci

[david-leifker]

This PR introduces an action which determines which major component has changes (frontend, backend, ingestion, setup jobs). Based on the changes it will selectively run CI jobs.

In the unified workflow:

• images are pulled from head/head-slim and swapped with the expected image tags when needed just before smoke-tests are run
• the smoke test matrix is also dynamic based on the changed component
• when publishing is enabled everything is run to ensure images are built and tested

A new custom action local to this repository will maintain the mapping from file paths to major component. That action will then be used in any workflow which needs selective jobs/steps. Currently this means the primary unified image build and the build and test workflows.

For example: a frontend change would trigger the build of the frontend container and only run cypress tests.

Scenarios

No change to any major component

Frontend change only

• Build frontend image
• Runs cypress tests

Ingestion change only

• Build ingestion image
• Runs non-cypress tests

Backend change

• Build backend images (exclude frontend and ingestion images)
• Runs all smoke tests

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[hsheth2]

does it make sense to add an "ingestion-base" change detector?

we have a fair bit of logic around conditionally building the ingestion-base image, and I think it might make sense to refactor that into here or docker-unified's setup job

that said, this PR is already a huge improvement as it is and this is more of a nice to have and definitely not a blocker

[david-leifker]

Honestly I'll have to look at that logic again. There is an extra bit of complexity due to multiple stages plus full/slim. Since that was working ok as far as I know, I opted to just preserve that logic. In this PR it just kind of gets triggered all the time and its own logic maintains control over when and what to build.

[hsheth2]

this is going to be hugely helpful