Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sling ELT Error when using Sling's env.yaml to manage connections #25925

Open
Westm7 opened this issue Nov 14, 2024 · 0 comments
Open

Sling ELT Error when using Sling's env.yaml to manage connections #25925

Westm7 opened this issue Nov 14, 2024 · 0 comments
Labels
integration: embedded-elt Related to dagster-embedded-elt which uses Sling and data Load Tool (dlt) type: bug Something isn't working

Comments

@Westm7
Copy link

Westm7 commented Nov 14, 2024

What's the issue?

I'm using sling's env.yaml file to manage my database connections, that way I can easily run the same sling commands outside of dagster when needed.

I just add a sling resource with no connections like below and everything worked well prior to version 1.8.12

sling_resource = {"sling": SlingResource(connections=[])}

Now I receive the following error when materializing my sling assets.

AttributeError: 'NoneType' object has no attribute 'get'
  File "venv/lib/python3.12/site-packages/dagster/_core/execution/plan/utils.py", line 54, in op_execution_error_boundary
    yield
  File "venv/lib/python3.12/site-packages/dagster/_utils/__init__.py", line 482, in iterate_with_context
    next_output = next(iterator)
                  ^^^^^^^^^^^^^^
  File "dagster_project/assets/sling_assets.py", line 41, in _assets
    yield from sling.replicate(context=context)
  File "venv/lib/python3.12/site-packages/dagster_embedded_elt/sling/sling_event_iterator.py", line 189, in __next__
    return next(self._inner_iterator)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "venv/lib/python3.12/site-packages/dagster_embedded_elt/sling/resources.py", line 407, in _replicate
    object_key = stream.get("config", {}).get("object")
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I've narrowed the error down to, "dagster_embedded_elt/sling/resources.py", line 407.
object_key = stream.get("config", {}).get("object")

The "config" key is set to None not null, so stream.get("config", {}) returns None instead of {} and the .get{"object") fails.

If you update line to 407 to:
object_key = (stream.get("config") or {}).get("object")
It works as expected.

What did you expect to happen?

Assets to be materialized without errors.

How to reproduce?

Reproduce by using sling's env.yaml to define your database connections and use a blank SlingResource() object with no connections in your dagster project.

sling_resource = {"sling": SlingResource(connections=[])}

and attempt to materialize an asset.

Dagster version

dagster, version 1.9.0

Deployment type

Local

Deployment details

No response

Additional information

If you update line to 407 of "dagster_embedded_elt/sling/resources.py" to:
object_key = (stream.get("config") or {}).get("object")
Assets materialize as expected.

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.
By submitting this issue, you agree to follow Dagster's Code of Conduct.

@Westm7 Westm7 added the type: bug Something isn't working label Nov 14, 2024
@garethbrickman garethbrickman added the integration: embedded-elt Related to dagster-embedded-elt which uses Sling and data Load Tool (dlt) label Nov 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
integration: embedded-elt Related to dagster-embedded-elt which uses Sling and data Load Tool (dlt) type: bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants