feat(ingestion/tableau): hidden asset handling

[haeniya]

Description

This PR enables the configuration of how to handle hidden assets in Tableau. Views and Dashboards can be hidden within Workbooks. Hidden assets can be identified by a missing/blank luid, as stated in the documentation: https://help.tableau.com/current/api/metadata_api/en-us/reference/view.doc.html

This PR introduces two new config properties tags_for_hidden_assets and ingest_hidden_assets to better control what to do with hidden assets. They can either be skipped completely or tagged.

Configuration in the recipe could look like this:

source:
  type: tableau
  config:
    # Coordinates
    connect_uri: https://tableau.example.com

    # Handling of hidden assets
    tags_for_hidden_assets: ['private', 'hidden']
    # or completely skip hidden assets with
    # ingest_hidden_assets: False   

    # Credentials
    username: "${TABLEAU_USER}"
    password: "${TABLEAU_PASSWORD}"

sink:
  type: "datahub-rest"
  config:
    server: "http://localhost:8080"

Closes #6421

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[haeniya]

There seemed to be a general issue with the tag ingestion. If all tags are removed in Tableau they wouldn't get removed in Datahub because tags are only saved if there are some tags present. Now, we basically always save the GlobalTags aspect even if there are no tags. But this allows us to properly reset the tags if they were removed from Tableau. I refactored this for all the assets with tags.

This resulted in quite a big changeset in the golden test files. Let me know if you see an issue with this. Thanks.

[haeniya]

Hi @hsheth2, would be great to get your feedback on this. Many thanks.

[mayurinehate]

Mostly good, requested minor change to avoid breaking change to default behaviour.

[haeniya]

@mayurinehate and @hsheth2 anything missing here?

[mayurinehate]

Merging the PR now.

[YuriyGavrilov]

Thank you a lot 🙏 @haeniya @mayurinehate

[mayurinehate]

@haeniya I would like to revisit and hear your thoughts on the need for config tags_for_hidden_assets as opposed to tagging hidden resources with a hard-coded "hidden" tag by default. In general - we are trying to minimize custom configs that do not yield much value in order to simplify the source configuration and future changes in code.

[haeniya]

@haeniya I would like to revisit and hear your thoughts on the need for config tags_for_hidden_assets as opposed to tagging hidden resources with a hard-coded "hidden" tag by default. In general - we are trying to minimize custom configs that do not yield much value in order to simplify the source configuration and future changes in code.

Hi @mayurinehate, so in our case we want two tags, "private" and "hidden", to be added to hidden assets. We have a custom view that filters out "private" assets from the "public" catalog. We could also change the definition of the custom view to also exclude "hidden" assets but I think it makes sense to have this property anyway. I can imagine, there might be users that want this for example in a different language so I probably wouldn't hardcode it to just "hidden".