How do I "Materialise All" for assets that are in different code locations.

[bolinzzz]

Hi team, on a very high level, I have the following setup. I have two code locations A and B.

Code location A consists extract and load python assets (both partitioned and non partitioned).
Code location B consists of dbt assets, forming a network dependent on assets from code location A.

I wanted to trigger a materialise all, for all assets in a lineage from code location A. See screenshot:

However, I am prompt with

Seems like I need to form a connect graph in which root assets shared the same partitioning. How do I do that? In my pipeline, it is moving from partitioned extract and load python assets to non partitioned transformation dbt assets.

[OwenKephart]

Hi @bolinzzz, unfortunately this is not currently possible. That error text is confusing to unwind, but the correct parsing is:

"defined in the same code location" and ("share a partition space" or "form a connected graph in which root assets share the same partitioning")

[bolinzzz]

Thanks @OwenKephart! In that case, will it be possible to create a job that targets an arbitrary set of dbt assets during run time based on the dbt_select expression passed into the run config?

Have you seen anyone doing this before? We will like to have the ability to bulk select and full-refresh all downstream of a particular source, something like this

$ dbt run --select source:jaffle_shop.orders+ --full-refresh

The source resides in another code location, hence I can't just materialise all via the dagster UI

[jamiedemaria]

dynamically creating the contents of a job based on run config isn't really something we support. If you're running jobs with the same sets of dbt_assets repeatedly, though you could make an asset job using define_asset_job and the asset selection syntax https://docs.dagster.io/concepts/assets/asset-selection-syntax

[bolinzzz]

Thank you @jamiedemaria, that's exactly what I am doing at the moment. Is the support for "Materialise All" across different code locations part of the roadmap?

[jamiedemaria]

I don't believe so

[Giangblackk]

@bolinzzz I have the same problem as you, and I found out that Dagster has recently added a feature called Declarative Automation which allow developers to automatically materialize downstream assets even though they are defined in different code locations (that is how I interpret it, but it's propably more useful than that).
Here is the video introducing this feature. https://www.youtube.com/watch?v=Z77s50b_Sks
I tried and it works to me. Hope it helps you.