Be able to !include external yaml files

[whisperstream]

Describe the feature

Sometimes its useful to reuse yaml (especially the schema.yml sections relating to docs). In my case, I have tables in different schemas and sometimes I have a view to the same table (except with different permissions). Ultimately though the view is just a select * from table_xyz, so the docs are the same. I don't want to have to maintain the documentation twice, and would prefer just to link to it.

The quick way to do this would be to allow external file linking as described here (https://stackoverflow.com/questions/528281/how-can-i-include-a-yaml-file-inside-another)

This is transparent to dbt and handled by the yaml loader.

Describe alternatives you've considered

Duplicating the documentation

Additional context

Who will this benefit?

Anyone who wants to reuse parts of their yaml files

[drewbanin]

This is a cool idea, thanks @whisperstream! dbt does funky things with file paths, but I bet there's a way to make this work :)

[drewbanin]

For this particular use-case, check out docs blocks if you haven't already. You'll still need to enumerate the columns for each model, but you can reference the docs blocks instead of re-typing the table/column descriptions.

Additionally, you may be able to use yaml anchors if you're able to define the docs for both of these models in the same file.

[whisperstream]

have used the docs blocks already and have used yaml anchors before, but let me make a PR and test out the include and see how dbt gets on with it and you can decide to merge or not?

[drewbanin]

sure, sounds good! Full disclosure, we might not merge this in the near-term -- we in general try to be careful about allowing arbitrary file access in the dbt context. There are things we can do here that may assuage this concern, so sure, I'd love to see a PR if you're able to send one through :)

[laxjesse]

similar but different use case: I was hoping to be able to reference a non-schema.yml yaml file in a model.sql context to take advantage of the fromyaml context method.

I basically have a model that looks like this

{% set my_yml_str -%}
test1:
  - eligibility_start_criteria: some business logic
    eligibility_end_criteria: some more business logic
test2:
  - eligibility_start_criteria: more business logic
    eligibility_end_criteria: moar biz logic
goes on for a while ...
{%- endset %}

{% set my_dict = fromyaml(my_yml_str) %}
select
{% for item, criteria in my_dict.items() %}
  '{{ item }}' as group,
  {{ criteria[0]['eligibility_start_criteria'] }} as eligibility_start_criteria,
  {{ criteria[0]['eligibility_end_criteria'] }} as eligibility_end_criteria
{% endfor %}
from test

and was thinking it would be good to be able to maintain my_yml_str in a separate .yml file rather than in the model itself

[mrshu]

@drewbanin unless @whisperstream has already put a PR together, would you still be interested in seeing one?

I believe the ability to !include specific parts of .yml files could make them much more manageable (and in turn helps mitigate some of the standard complaints).

Thanks!

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[mrshu]

Well, it's been a while and it seems like this issue will get closed in a few days, so let me just summarize what I think could be done here.

• Much of YAML processing passes through safe_load, which in turn makes use of SafeLoader
• The SafeLoader could be extended to support !include similar to the answer in https://stackoverflow.com/a/9577670
• This would in turn make it so the fromyaml (but also basically any YAML string processed by dbt) gets the ability to !include other YAML files

Going by @drewbanin's comment above, I am still not sure to what extent does it make sense to try to implement this but if it sounds feasible, I might still take a stab at it at some point 🙂

Thanks!

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest; add a comment to notify the maintainers.

[mingfang]

please reopen

[databius]

It's useful to reuse yaml, please reopen

[ricklamps]

Please reopen! Being able to !include yaml files is very useful. The use case I'm thinking of is separating the source.yml file into many files, one per table, and !includeing them in the main source.yml file. If there's a better way to do this please let me know

[databius]

There is a new ticket with similar content: #9695