feat: Add ability to filter GCP project ingestion based on project labels

[anaghshineh]

What Changed?

We've wanted to have more control over project ingestion at my organization. We've been maintaining a "whitelist" of projects we'd like to ingest, but this is becoming unmanageable as we create more projects. We'd like to leverage consistent project labels in GCP to drive ingestion. This PR takes a first pass at allowing folks to define project_labels in their BigQuery ingestion recipes. It uses the GCP Resource Manager search_projects method to query for projects with the given labels. This does not introduce any breaking changes, as ingestion will continue to favor project_ids & will respect any name patterns.

Example:

source:
    type: bigquery
    config:
        project_labels:
            - "team:data"
            - "domain:*"
        credential:
            project_id: '${BIGQUERY_INGESTION_PROJECT}'
            client_email: '${SERVICE_ACCOUNT_CLIENT_EMAIL}'
            private_key_id: '${BIGQUERY_PRIVATE_KEY_ID}'
            private_key: '${BIGQUERY_PRIVATE_KEY}'
            client_id: '${SERVICE_ACCOUNT_UNIQUE_ID}'
        project_id_pattern:
            deny:
                - '^sys-[0-9]{26}$'
sink:
    type: datahub-rest
    config:
        server: '${DATAHUB_BASE_URL}'
        token: '${DATAHUB_ACCESS_TOKEN}'
        max_threads: 20

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[treff7es]

Do you know if this client needs extra permission, or does it not need any new permission?
If it is needed, then can we make sure not to create this client if, in the config, it is not set to use it?

[anaghshineh]

Good question. I initially implemented this PR used the list_projects method but landed on the search_projects one due to its flexibility in searching for projects based on parents AND labels. Looking at GCP docs, seems like you'll need to ensure the SA has resourcemanager.projects.get on the projects you want to grab.

Is that your read, too?

[anshbansal]

@anaghshineh There is a easy way to test this out by creating a service account without this permission and try to run this code. If this does indeed require a new permission then please

• update the docs with the new roles/permissions required
• make this an optional thing disabled by default. We don't want every orgs ingestion to break due to this new permission. Some orgs have heavier checks on getting new permissions. We don't want them to get stuck on permissions when this gets in.
• Please add in https://github.com/datahub-project/datahub/blob/master/docs/how/updating-datahub.md that there is new option that requires new permissions

As this is open source we have to be careful w.r.t. not breaking other orgs ingestion as much as possible.

[anshbansal]

Please don't make changes in this file. This file is updated separately. This is mainly a caching mechanism for our builds.

[anshbansal]

Please put this under a config disabled by default. We don't want folks ingestions to break due to new permissions required as mentioned in other comment.

[hsheth2]

Closing as #11169 was merged