[CT-842] Retrieve Git Information from DBT, if available

[nbdaniels]

Hi there! I figured this might be better in a new ticket but I am copying from a comment I made recently on a closed thread.

I saw this thread and couldn't find much help anywhere else after searching for a bit. Is it possible to do something like this now with DBT's current version?

For some background, I'm looking to track changes made to data from PRs using Python. I'm looking to grab git information (like current branch and user who committed most recently to that branch) from a specific DBT run triggered from a PR. With this information I believe I can pull it into a Python script and run with FAL (https://github.com/fal-ai/fal) to understand what changes are planned to be made to production data and who is making the change.

Please let me know if this isn't the right place or if I'm missing something from other DBT resources!

[jeremyyeo]

At the top of my head, dbt can read environment variables so maybe something like this temporarily:

Edit: Looks like this was mentioned in that other issue :P

[nbdaniels]

Thanks for the quick response! Please excuse my lack of understanding and additional context here. I'd like to do something like this in dbtcloud but environment variables are set in the UI.

In this case, would we only be able to develop + run dbt locally (no dbtcloud) and only have this work if we set CURRENT_GIT_SHA manually each time? I tried doing this originally but I was having the same issues as the other thread: this has to be done manually for each run which adds a human element I'd like to avoid, if possible.

[jtcohen6]

@nbdaniels I'm curious to hear a bit more about your intended workflow. Is this part of your CI/CD process? I understand the desire to associate these pieces of information:

• who is proposing the change (git commit author)
• which models are changing in this commit / PR (the dbt way of doing this today: https://docs.getdbt.com/guides/legacy/understanding-state)
• diffs in underlying models between production vs. staging / CI build

[nbdaniels]

I'm so sorry for the delay here! Let me see if I can add some more context/detail.

Currently our process of managing our dbt models relies on a continuous deployment so we can move quickly on stakeholder requests (need new fields from source, add business logic, etc.). The vast majority of the team works on dbtcloud for cloud development and manages git branches/changes there, then submits a PR directly in the github UI that requires review before merging into production.

Since we're moving quickly on requests and we have such large amounts of data, we're looking for a better way to validate that our changes to data models will not significantly affect fields that we did not intend to change. In our current flow, this is mostly up to the PR author (using BI + data warehouse) to determine the changes manually, since the one who made the changes will have the most context on what should be changing. Once the changes are determined, they are posted in the PR to add context for the reviewer.

We're looking for a better way to:

• reduce overhead in determining changes without changing the workflow of dev --> PR (trying to avoid manual changes made before each PR, like the CURRENT_GIT_SHA recommendation above)
• store changes made by PR, author over time
• (ideally) have some sort of alert system that can tag a user (e.g. Slack) with a model diff summary or a best way to get to it

Ideal state:

1. Development: Author makes changes in dev and creates a PR
2. Diff Workflow: PR triggers some sort of workflow to compare PR schema to prod schema for designated tables
3. Diff Alert/Report: Diff workflow returns author, diff summary in an accessible place (e.g. Slack)
4. Review: Author can review diff summary to make changes or submit PR for review, providing summary for reviewers with extra context

Let me know if I can answer anything else or if I misunderstand your ask!

[lostmygithubaccount]

@nbdaniels would it be sufficient to include the git information in the metadata for a run? then you can use the API to get the repo, SHA, etc. as needed, or am I misunderstanding?

Otherwise, is there some concrete engineering work needed here? I'm inclined to convert this to a discussion -- in general we're very interested in these "GitOps" workflows but I'm not sure on the specific here.

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions[bot]]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

[github-actions[bot]]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

[sdrzymala]

Hi there,

I have just found that topic and would communicate that I would also benefit from that.

My idea was to create a new model in DBT that would hold the information about the current environment and it's version ie:

• environment name - can be done with target.name
• product version - commit she or ideally a git tag
• perhaps last dbt start time - can be done with get_dbt_start_datetime(), end time with some effort could also be added
• some other information describing the current environment
  I could then show the information on the report(s) to show the given environment name and the version of the product for troubleshooting and reference. When working with multiple environments and dealing with bugs/questions/etc from other developers and business this could be helpful.

Now, when I was using a self hosted DBT I could make it working easily by getting the git tag/commit sha in one step of the pipeline and then passing to variable defined in dbt project, but of course when using dbt cloud that seems to be not possible.

I can see that such information is available in dbt cloud in the job run so maybe this wouldn't be such difficult to pass that information further?

If anyone else could relate that maybe we could reopen and analyse again.

Thanks!

[dbeatty10]

I'm going to promote this issue to a Discussion.

[blarghmatey]

My use case for gaining access to the git information is to be able to automatically namespace the schema during development so that the schema name is e.g. staging_{{git_branch}} to avoid developers accidentally stepping on each other's work, and avoid accidentally running against the production schema during development. Currently what I will likely have to do is add a CLI wrapper for developers to use that automatically sets the current branch as an env variable and template that into the profile.yaml

[kyle-kent]

@dbeatty10 @jtcohen6 this might be a useful feature for a state style selection method. Something like -s state:modified.git, -s 'git:modified', or even -s 'git:diff<commit hash or branch>. The purpose would be CI/CD and testing changes in complex projects with many tables.