[PLUS-1409] Introduce translator pattern for Fivetran #25557
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This stack of pull requests is managed by Graphite. Learn more about stacking. |
briantu
force-pushed
the
briantu/fivetran-translator
branch
from
dd863cd
to
69c2cce
Compare
benpankow
reviewed
Oct 28, 2024
python_modules/libraries/dagster-fivetran/dagster_fivetran/asset_defs.py
Outdated
Show resolved
Hide resolved
python_modules/libraries/dagster-fivetran/dagster_fivetran/asset_defs.py
Outdated
Show resolved
Hide resolved
benpankow
reviewed
Oct 28, 2024
python_modules/libraries/dagster-fivetran/dagster_fivetran/asset_defs.py
Outdated
Show resolved
Hide resolved
benpankow
reviewed
Oct 29, 2024
python_modules/libraries/dagster-fivetran/dagster_fivetran_tests/test_load_from_instance.py
Outdated
Show resolved
Hide resolved
python_modules/libraries/dagster-fivetran/dagster_fivetran_tests/test_load_from_instance.py
Outdated
Show resolved
Hide resolved
python_modules/libraries/dagster-fivetran/dagster_fivetran_tests/test_load_from_instance.py
Outdated
Show resolved
Hide resolved
briantu
force-pushed
the
briantu/fivetran-translator
branch
from
c2a4fb7
to
1fa4e78
Compare
briantu
changed the title
briantu
force-pushed
the
briantu/fivetran-translator
branch
from
177ef9e
to
34d0210
Compare
maximearmstrong
added a commit
that referenced
this pull request
Nov 8, 2024
…classes for rework (#25751) ## Summary & Motivation ~~Builds out a very barebones translator class for the new version of the Fivetran integration.~~ ~~The implementation for this translator will be inspired by the `DagsterFivetranTranslator` introduced in #25557, but a new implementation is required to leverage the workspace context and state-backed definitions, which is incompatible with the current translator and way of building assets.~~ Edit after Ben's comment [here](#25751 (comment)): Move things around under translator.py. This PR leverages the `DagsterFivetranTranslator` introduced in introduced in #25557. `FivetranWorkspaceData` will implement the method `to_fivetran_connector_table_props_data` in a subsequent PR, that will map raw connector and destination data fetched using the Fivetran API into `FivetranConnectorTableProps` objects, that are compatible with the translator. This process will match what we currently do. ## How I Tested These Changes Tests will be added in subsequent PRs.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
Our BI integrations (Power BI, Sigma, Tableau, Looker) and DBT integration use a translator pattern to allow users to customize the assets they load from the external tool. We want to do the same for Fivetran.
How I Tested These Changes
Changelog
[Fivetran] Introduced
DagsterFivetranTranslatorto customize assets loaded from Fivetran.