Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable Enhanced Callback Support Across All Callback Types #253

Closed
jroach-astronomer opened this issue Oct 17, 2024 · 2 comments
Closed

Enable Enhanced Callback Support Across All Callback Types #253

jroach-astronomer opened this issue Oct 17, 2024 · 2 comments
Assignees
@jroach-astronomer

Comments

@jroach-astronomer
Copy link
Member

jroach-astronomer commented Oct 17, 2024 •
edited
Loading

Continuing the work done in #209, add enhanced callback support across all callback types, both at the DAG-level, as well as at the task-level.
@jroach-astronomer
Copy link
Member Author

@tatiana, @pankajastro, please assign this issue to me.

@jroach-astronomer
Copy link
Member Author

Clear line-of-sight forward on the work that is work to be done in order to apply this new pattern to all callback types.

@jroach-astronomer jroach-astronomer mentioned this issue Dec 17, 2024
Merged
@cmarteepants cmarteepants mentioned this issue Dec 18, 2024
Open
jroach-astronomer added a commit to jroach-astronomer/dag-factory that referenced this issue Dec 30, 2024
@jroach-astronomer
Added test-coverage for astronomer#253
c4a2342
pankajkoti pushed a commit that referenced this issue Jan 9, 2025
@jroach-astronomer
Add Robust Support for Callbacks at Task and TaskGroup Level (#322)
39c813e 
Pull request to address #253, which aims to provide enhanced support for
callback types at the **DAG**, `default_args`, **Task** and
**TaskGroup** level. There are now four different ways that callbacks
can be defined via `dag-factory`. Below, you'll also see an example of
each.

1. By passing a string that points to a callable.
2. By passing a file path and file name that points to a callable,
similar to above.
3. By providing a string that points to a callable, but with some
parameters being provided at runtime.
4. Using callbacks from providers (like Slack).

### Passing a string that points to a callable
``` yaml
...
  task_groups:
    task_group_1:
      default_args:
        on_success_callback: print_hello.print_hello_from_callback
      dependencies: [task_1, task_2]
...
```


### Passing a file path and file name that points to a callable
``` yaml
...
    task_2:
      operator: airflow.operators.bash_operator.BashOperator
      bash_command: "echo 2"
      on_success_callback_name: print_hello_from_callback
      on_success_callback_file: $CONFIG_ROOT_DIR/print_hello.py
      dependencies: [start]
...
```


### By providing a string that points to a callable, but with some
parameters being provided at runtime
``` yaml
...
  on_failure_callback:
    callback: customized.callbacks.custom_callbacks.output_message
    param1: param1
    param2: param2
...
```

### Using callbacks from providers
``` yaml
...
    on_failure_callback:
      callback: airflow.providers.slack.notifications.slack.send_slack_notification
      slack_conn_id: slack_conn_id
      text: |
        🔴 Task Failed.
        This task has failed and needs to be addressed.
        Please remediate this issue ASAP.
      channel: "#channel"
...
```

Unit-tests have been added and updated appropriately.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jroach-astronomer jroach-astronomer

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jroach-astronomer