Decoupling dbt Adapters from dbt Core versions #9171
Replies: 4 comments 7 replies
-
|
what happens to the following things that currently live in Core?
|
Beta Was this translation helpful? Give feedback.
-
|
@MichelleArk I just came here to say that, as a dbt-netezza adapter maintainer, I am very excited about this change and what it will mean for our ability not to run several minor versions of dbt-core behind due to bandwidth constraints on our team contributions. |
Beta Was this translation helpful? Give feedback.
-
|
global project - this will move to the new dbt-adapter package (it technically has already been moved)! Similar to the base python interfaces, global project represents a collection of base interfaces and default implementations that adapters can / must override to be compatible with dbt-core. Changes to global project macros for core functionality are akin to extending the BaseAdapter interface and will be stable within major releases of dbt-adapter. postgres plugin - medium-term we plan to move this out of dbt-core as well, but it is not a blocker for decoupling the rest of the adapters from dbt-core releases, so it's a lower priority in this set of plans. |
Beta Was this translation helpful? Give feedback.
-
|
what effect (if any) will there be on adapters that inherit from other adapters? like dbt-redshift, dbt-databricks, and dbt-synapse |
Beta Was this translation helpful? Give feedback.
-
|
@dataders right now we (dbt-databricks) still depend a decent amount on dbt-spark. If it enables decoupled shipping, however, I think we would invest in decoupling that dependency. Further, I think the general changes being introduced here should reduce our need to depend on dbt-spark: part of why we do today is that the interplay between core and adapter isn't very straight forward, so we rely on dbt-spark to get that part correct, so that we can make modifications mostly at the leaves. |
Beta Was this translation helpful? Give feedback.
-
|
This will not impact the ability for adapters to depend on other adapters |
Beta Was this translation helpful? Give feedback.
-
|
Right now we have coupled shipping with dbt-spark, and for the most part it works because we know dbt-spark will have to be compatible with the next version of core before we are even asked to make progress on our adapter. But if spark and core decouple, are those same guarantees going to be there? Will dbt-spark be promptly updated to keep pace with new capabilities as they're developed? If not, its going to reduce the value in remaining dependent. |
Beta Was this translation helpful? Give feedback.
-
What I think is important to understand is that we're inverting the dependency relationship here: no longer is it a question of being compatible with core but core being compatible with the adapter. Ultimately the feature development you're concerned with will be driven out of dbt-adapters so the question is: will dbt-spark get updates to match dbt-adapters? Yes, as evidenced by our recent investment in dbt-spark CI improvements we plan on supporting it with the same care we give the other adapters we directly maintain |
Beta Was this translation helpful? Give feedback.
-
|
Hi all, what is the anticipated date for |
Beta Was this translation helpful? Give feedback.
-
|
Per @colin-rogers-dbt: Current plan is to release dbt-adapter 1.0 around when we release dbt-core 1.8 (April-ish). |
Beta Was this translation helpful? Give feedback.
-
|
Update: We've started to release initial alphas of dbt-adapters: https://pypi.org/project/dbt-adapters/ and have opened a discussion in the new repo with a migration guide: dbt-labs/dbt-adapters#87 |
Beta Was this translation helpful? Give feedback.
-
TL;DR
what’s this about
We’re already underway on an initiative to further decouple adapters from dbt Core. The goals of the work are to do the following:
how you can help
This discussion serves to both inform and collect feedback.
Background
(@dataders)
Let’s unpack the “Adapters & Artifacts” section of @jtcohen6 & @graciegoheen's November 2023 roadmap update:
The Problem
The Solution:
dbt-adapter==1.0.0To make this happen, we will introduce a new Python package published to PyPI,
dbt-adapter, upon which both dbt-core and adapters depend. The ‘adapter protocol’ must be stable, and versioned separately, so that we can ensure backward/forward compatibility for new versions of dbt-core and adapters.This will decouple adapter maintainers from the regular release cadence of dbt-core. Adapter maintainers can also start implementation work on adapter protocol additions, as soon as the protocol has been updated and released. These changes are opt-in, and a not a prerequisite for ongoing backwards-compatibility.
Impact
For end users
pip install dbt-postgrespip install dbt-core dbt-redshift(more below)But how does a user know they are installing the right version of dbt-core?
Thankfully both dbt-core and adapter implementations will have to declare their dependency on dbt-adapter and dbt-common so pip will be able to resolve to a correct version of both or throw an error that they have requested incompatible versions
For external adapter maintainers
Impact to integration testing
In the short term while the source implementations of adapters will no longer require a pinned dependency to a minor version of dbt-core, integration tests will still require an explicit dependency on dbt-core in order to test the combination of the dbt CLI installed and executed with particular adapter. More guidance/detail for adapter maintainers will be forthcoming on if/how testing will be impacted.
In the medium to long term we will be investigating ways to test adapters without relying on dbt-core along with new, easier to use, testing tools.
Timelines & Migration Plan
(@MichelleArk, @colin-rogers-dbt)
decouple & lift adapter protocol out of dbt-core
This work has us untangling some gnarly, deeply-baked technical debt from cyclic dependencies to shared global state across components that should ideally be independent. If that’s your thing and you’d like to follow along, we’re tracking that work in this epic!
Once the refactoring is completed in-place, we will begin lifting the adapter-specific and common components into new repositories/packages, setting up their releases, and documentation.
dbt-adapter 1.0 release, migration guides & tooling
Slated for January 2024, This will give adapter maintainers several months to migrate before the next minor release of dbt-core (1.8, in April 2024), which will require adapters to be compatible with dbt-adapter>=1.0.
The migration itself will require adapter implementations to replace their dependency on
dbt-core==1.*todbt-adapter>=1.0and (potentially, depending on their usage)dbt-common>=1.0. Any import path changes will need to be worked through, meaning nature of this migration will largely be find-and-replace of renamed import paths. For an example, check out the dbt-bigquery migration draftWe’ll also be providing a migration script to track progress and automate the trivial, but tedious parts of this migration.
dbt-core 1.8 release
This will be the first dbt-core release that will require a migration to
dbt-adapter>=1.0.0for compatibility. This is because this version of dbt-core will not include any of the base adapter interfaces or interfaces that dbt-* adapters have historically depended on.dbt-core 1.8+ releases
These will require no changes from adapter maintainers to ensure forward and backward compatibility between. Any new features that require adapter changes may not be supported, but this will be handled gracefully and existing interfaces will not
dbt-adapter 1.0+ releases
dbt-adapter can release minor versions completely independently from dbt-core. In this world we can start to improve adapter maintainer quality of life and address long-known pain points such as:
What changes would you like to see? Comment below!
Beta Was this translation helpful? Give feedback.
All reactions