Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mapping over multiple parameters results in 1 task fewer than expected #23425

Closed
1 of 2 tasks
aspain opened this issue May 2, 2022 · 1 comment · Fixed by #23434
Closed
1 of 2 tasks

Mapping over multiple parameters results in 1 task fewer than expected #23425

aspain opened this issue May 2, 2022 · 1 comment · Fixed by #23434
Assignees
Labels
affected_version:2.3 Issues Reported for 2.3 area:core kind:bug This is a clearly a bug
Milestone

Comments

@aspain
Copy link
Contributor

aspain commented May 2, 2022

Apache Airflow version

2.3.0 (latest released)

What happened

While testing the example given for Mapping over multiple parameters I noticed only 5 tasks are being mapped rather than the expected 6.

task example from the doc:

    @task
    def add(x: int, y: int):
        return x + y


    added_values = add.expand(x=[2, 4, 8], y=[5, 10])

The doc mentions:

# This results in the add function being called with
# add(x=2, y=5)
# add(x=2, y=10)
# add(x=4, y=5)
# add(x=4, y=10)
# add(x=8, y=5)
# add(x=8, y=10)

But when I create a DAG with the example, only 5 tasks are mapped instead of 6:
image

What you think should happen instead

A task should be mapped for all 6 possible outcomes, rather than only 5

How to reproduce

Create a DAG using the example provided [here](Mapping over multiple parameters) and check the number of mapped instances:
image

Operating System

macOS 11.5.2

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==3.3.0
apache-airflow-providers-celery==2.1.4
apache-airflow-providers-cncf-kubernetes==4.0.1
apache-airflow-providers-databricks==2.6.0
apache-airflow-providers-elasticsearch==3.0.3
apache-airflow-providers-ftp==2.1.2
apache-airflow-providers-google==6.8.0
apache-airflow-providers-http==2.1.2
apache-airflow-providers-imap==2.2.3
apache-airflow-providers-microsoft-azure==3.8.0
apache-airflow-providers-postgres==4.1.0
apache-airflow-providers-redis==2.0.4
apache-airflow-providers-slack==4.2.3
apache-airflow-providers-snowflake==2.6.0
apache-airflow-providers-sqlite==2.1.3

Deployment

Astronomer

Deployment details

Localhost instance of Astronomer Runtime 5.0.0

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@aspain aspain added area:core kind:bug This is a clearly a bug labels May 2, 2022
@jedcunningham
Copy link
Member

Interesting. Looks like it's an issue with expanding on static lists, as this results in 6:

    @task
    def first():
        return [2, 4, 8]

    @task
    def second():
        return [5, 10]

    @task
    def add(x: int, y: int):
        return x + y

    added_values = add.expand(x=first(), y=second())

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affected_version:2.3 Issues Reported for 2.3 area:core kind:bug This is a clearly a bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants