Use dbt incremental model to update records based on a condition #10598
Unanswered
giacomochiarella
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I cannot find a way using dbt incremental model to update records based on a condition. Shortly, I would like to leverage the ability of incremental model to process only new data BUT I want the resulting records replace the existing ones only if column1 (in the resulting record) > column1 (in the existing record), of course based on a unique key.
Example:
Let's say my model extracts per order id the one having the greatest total
Very first run (full-refresh):
Input
Output
and all are added to the model given it is the very first run
Second run (incremental):
Input
Output
But, given the same order ids exist with greater total except
order_id = 3
, onlyorder_id = 3
must be replaced.To implement this, I need to read the current data, but if I reference the model itself in the model dbt complains because it found a circular dependency. Can dbt do that?
I guess it could work if I could somehow instruct dbt to copy the model in a temporary table and reference this table to avoid the circular dependency. A bit hacky, would be possible in the pre_hook to execute a sql query to copy the current model in a temporary table in order to reference this model and avoid the circular dependency?
Beta Was this translation helpful? Give feedback.
All reactions