Save manifest JSON files to the configured analytical database

[bashyroger]

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

While is is great that DBT generates JSON structured artifact / logging metadata as described here: https://docs.getdbt.com/reference/artifacts/dbt-artifacts/#notes
...they are of little use if they are not saved to the configured analytical database.

If this was done , then a package like the tails.com DBT artifacts package would not need the cumbersome steps indicated at the end: https://github.com/tailsdotcom/dbt_artifacts, the steps where you need to upload these files to your database...

This is even more problematic when you are solely on DBT cloud, as we are.

So while I realize that DBT is an T-L tool , I think an exception must be made for DBT related logging metadata that, if it is structured as JSON, can (for example) be loaded to a Snowflake variant data type with ease.

Describe alternatives you've considered

Doing this manually; not scalable / maintainable

Who will this benefit?

Everyone that wants to do analytics on DBT metadata or needs to store this logging metadata for compliance

Are you interested in contributing this feature?

No response

Anything else?

No response

[ChenyuLInx]

Hey @bashyroger, thanks for making this request! It is a great question and we have been thinking about it.

We believe the best way to support answering those questions must be via the dbt Cloud Metadata API. That’s the most reliable, resilient, least crufty way to access this info. We are aware that you still have to use the Admin API to get job ids to then pass over to Metadata API and it would be a separate step out of the normally scheduled dbt run. And actively looking for solutions to resolve it

At the same time:

• We understand why dbt users want to access this information right within their warehouse (dbt models on dbt models!) It’s the tooling that feels comfortable to them!
• It’s frustrating that dbt-core deployments via Airflow/Dagster/etc have this capability, whereas dbt Cloud users really don’t, outside of the PUT hack on Snowflake

So:

• We should continue to drill into what use cases/questions are motivating this ask — and pass that information back to the Metadata team as valuable user feedback. One thing we are thinking of is an automated tap for loading from the Metadata API to warehouses.
• We should look into the possibility of similar “for now” hacks on other databases, which are necessarily super specific to those databases. Niall (from Brooklyn Data Co, maintainer of the dbt_artifacts package) opened issues in dbt-redshift + dbt-bigquery. But this method has its own drawbacks, any failed step in a dbt Cloud job causes subsequent steps to be skipped. So the PUT hack on Snowflake (and comparable hacks on Redshift/BigQuery) would continue to have the limitation where if a job fails, its artifact won’t be uploaded.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[eozturkTF]

Hello
Any news on this please?

Cheers

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.