Enable jinja2 template inheritance

[fetanchaud]

Issue

Users cannot use jinja2 template inheritance to build models

Issue description

Given a jinja2 my_base_template.temp template file (extension is meaningless),
any model begining with {% extends "my_base_template.temp" %} fails.

Results

Error message : "no loader for this environment specified"

System information

DBT 0.12.2
python 3.7.1
windows 7

Feature

Enable jinja2 template inheritance

Feature description

Whilst jinja2 macros are of great use for code reuse, it seems to have cases when inheritance could be of great help, when one wants to reuse behavior instead.

Who will this benefit?

In my use case, I am trying to write generic models for each of the standard Data Vault entities (hub, link, satellite, etc...) on top of a persistent staging area, and inheriting from a template would be of great help not to repeat myself.

Best regards,

Fabrice Etanchaud

[drewbanin]

@fetanchaud thanks for the report! Your data vault entity use case is a really good one!

I'm unsure what's involved in supporting jinja templates, though I agree that they're a powerful construct worth supporting in dbt. I can have a look at what an implementation would entail. Is this something you'd be interested in working on if we can come up with a good plan?

[fetanchaud]

Hi @drewbanin,
why not ? I will be pleased to help !
The problem is my python is not very rich...
I can checkout the project and see how models are listed and parsed, it seems dbt is not using FileSystemLoader loader.

[drewbanin]

hmmm, ok! In thinking about this more, a feature like this would require significant changes to internals of dbt. I'd also be happy to work on this, but I'm unsure when I'll be able to tackle it.

Can you help by including some examples / test cases that we can use to validate the feature? They don't need to be realistic, but if you're able to comment with an example template and an example of a model that uses it, that would be really helpful!

[fetanchaud]

Hi @drewbanin,

sorry for the delay.
In my data vault experiment with dbt, I end up writing several materializations (psa, hub, sat, status tracking satellite, non historized link...). They are all based on the incremental materialization, sharing the same skeleton, differing only in a few places (config, full refresh and incremental sqls). So this could be the place to start : defining a generic incremental template, and redefining blocks in the inheriting templates.
As soon as possible, I will include one example to explain myself.

Best regards,
fabrice etanchaud

[gherolyants]

Hi! Is there any plan to include this feature into next releases? I'm working on a model consistency monitoring tool for our data warehouse and jinja template inheritance would save me 50% of lines of code not to mention DRY.

[drewbanin]

Hey @gherolyants - this isn't currently prioritized on our end, and I think it will be a pretty big change when it's eventually implemented! I think there are a lot of great patterns that a feature like this will facilitate.

Can you tell me a little bit about the model consistency monitoring tool you're looking to build? I intuitively feel that template inheritance will be a powerful feature in dbt, but I haven't been able to pinpoint any really great use-cases that aren't served by macros!

[anthonyhallsdc]

My use case is Snowflake has stored procedures written in javascript. It would be nice to be able to use the jinja "include" to bring in the javascript from a json file. That way when I am developing javascript in my IDE i get the benefits of all the javascript syntax highlighting and extensions.

[pwildenhain]

For my use case, we're creating a metric warehouse using dbt, and a key part of that is union'ing a bunch of raw metric tables together, based off a jinja dict of metric metadata (numerators, denominators, etc). I think a macro would make whats happening a little opaque, rather than being able to make use of the block and inherits tags (i.e creating different blocks for the different metric dicts).

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.