diff --git a/docs/user_guide/advanced_composition/chaining_flyte_entities.md b/docs/user_guide/advanced_composition/chaining_flyte_entities.md index 29bc71e1bf..4fb12a4149 100644 --- a/docs/user_guide/advanced_composition/chaining_flyte_entities.md +++ b/docs/user_guide/advanced_composition/chaining_flyte_entities.md @@ -17,7 +17,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte Let's establish a sequence where `t1()` occurs after `t0()`, and `t2()` follows `t1()`. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/chain_entities.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/chain_entities.py :caption: advanced_composition/chain_entities.py :lines: 1-30 ``` @@ -27,7 +27,7 @@ Let's establish a sequence where `t1()` occurs after `t0()`, and `t2()` follows Just like tasks, you can chain {ref}`subworkflows `. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/chain_entities.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/chain_entities.py :caption: advanced_composition/chain_entities.py :lines: 34-49 ``` @@ -36,13 +36,13 @@ To run the provided workflows on the Flyte cluster, use the following commands: ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/chain_entities.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/chain_entities.py \ chain_tasks_wf ``` ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/chain_entities.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/chain_entities.py \ chain_workflows_wf ``` diff --git a/docs/user_guide/advanced_composition/conditionals.md b/docs/user_guide/advanced_composition/conditionals.md index e73b81e5ed..bbaf3dc27a 100644 --- a/docs/user_guide/advanced_composition/conditionals.md +++ b/docs/user_guide/advanced_composition/conditionals.md @@ -18,7 +18,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the necessary libraries. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/conditional.py :caption: advanced_composition/conditional.py :lines: 1-3 ``` @@ -29,7 +29,7 @@ In this example, we introduce two tasks, `calculate_circle_circumference` and `calculate_circle_area`. The workflow dynamically chooses between these tasks based on whether the input falls within the fraction range (0-1) or not. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/conditional.py :caption: advanced_composition/conditional.py :lines: 11-37 ``` @@ -40,7 +40,7 @@ We establish an `if` condition with multiple branches, which will result in a fa It's important to note that any `conditional` statement in Flyte is expected to be complete, meaning that all possible branches must be accounted for. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/conditional.py :caption: advanced_composition/conditional.py :pyobject: shape_properties_with_multiple_branches ``` @@ -55,7 +55,7 @@ a convention also observed in other libraries. ## Consuming the output of a conditional Here, we write a task that consumes the output returned by a `conditional`. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/conditional.py :caption: advanced_composition/conditional.py :lines: 66-81 ``` @@ -66,7 +66,7 @@ You can check if a boolean returned from the previous task is `True`, but unary operations are not supported directly. Instead, use the `is_true`, `is_false` and `is_none` methods on the result. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/conditional.py :caption: advanced_composition/conditional.py :lines: 89-119 ``` @@ -79,7 +79,7 @@ Inputs and outputs are automatically encapsulated in a special object known as { ## Using boolean workflow inputs in a conditional You can directly pass a boolean to a workflow. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/conditional.py :caption: advanced_composition/conditional.py :pyobject: boolean_input_wf ``` @@ -92,7 +92,7 @@ This special object enables it to exhibit additional behavior. You can run the workflows locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/conditional.py :caption: advanced_composition/conditional.py :lines: 129-135 ``` @@ -102,7 +102,7 @@ You can run the workflows locally as follows: You can nest conditional sections arbitrarily inside other conditional sections. However, these nested sections can only be in the `then` part of a `conditional` block. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/conditional.py :caption: advanced_composition/conditional.py :lines: 142-164 ``` @@ -112,14 +112,14 @@ However, these nested sections can only be in the `then` part of a `conditional` Let's write a fun workflow that triggers the `calculate_circle_circumference` task in the event of a "heads" outcome, and alternatively, runs the `calculate_circle_area` task in the event of a "tail" outcome. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/conditional.py :caption: advanced_composition/conditional.py :pyobject: consume_task_output ``` You can run the workflow locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/conditional.py :caption: advanced_composition/conditional.py :lines: 181-188 ``` @@ -130,43 +130,43 @@ To run the provided workflows on the Flyte cluster, use the following commands: ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/conditional.py \ shape_properties --radius 3.0 ``` ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/conditional.py \ shape_properties_with_multiple_branches --radius 11.0 ``` ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/conditional.py \ shape_properties_accept_conditional_output --radius 0.5 ``` ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/conditional.py \ boolean_wf ``` ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/conditional.py \ boolean_input_wf --boolean_input ``` ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/conditional.py \ nested_conditions --radius 0.7 ``` ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/conditional.py \ consume_task_output --radius 0.4 --seed 7 ``` diff --git a/docs/user_guide/advanced_composition/decorating_tasks.md b/docs/user_guide/advanced_composition/decorating_tasks.md index 87141d4cc0..6b39c9f363 100644 --- a/docs/user_guide/advanced_composition/decorating_tasks.md +++ b/docs/user_guide/advanced_composition/decorating_tasks.md @@ -17,14 +17,14 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the required dependencies. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/decorating_tasks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/decorating_tasks.py :caption: advanced_composition/decorating_tasks.py :lines: 1-4 ``` Create a logger to monitor the execution's progress. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/decorating_tasks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/decorating_tasks.py :caption: advanced_composition/decorating_tasks.py :lines: 7 ``` @@ -33,7 +33,7 @@ Create a logger to monitor the execution's progress. We define a decorator that logs the input and output details for a decorated task. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/decorating_tasks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/decorating_tasks.py :caption: advanced_composition/decorating_tasks.py :pyobject: log_io ``` @@ -44,7 +44,7 @@ We create a task named `t1` that is decorated with `log_io`. The order of invoking the decorators is important. `@task` should always be the outer-most decorator. ::: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/decorating_tasks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/decorating_tasks.py :caption: advanced_composition/decorating_tasks.py :pyobject: t1 ``` @@ -58,7 +58,7 @@ You can also stack multiple decorators on top of each other as long as `@task` i We define a decorator that verifies if the output from the decorated function is a positive number before it's returned. If this assumption is violated, it raises a `ValueError` exception. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/decorating_tasks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/decorating_tasks.py :caption: advanced_composition/decorating_tasks.py :pyobject: validate_output ``` @@ -69,14 +69,14 @@ The output of the `validate_output` task uses {py:func}`~functools.partial` to i We define a function that uses both the logging and validator decorators. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/decorating_tasks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/decorating_tasks.py :caption: advanced_composition/decorating_tasks.py :pyobject: t2 ``` Finally, we compose a workflow that calls `t1` and `t2`. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/decorating_tasks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/decorating_tasks.py :caption: advanced_composition/decorating_tasks.py :lines: 53-59 ``` @@ -87,7 +87,7 @@ To run the provided workflow on the Flyte cluster, use the following command: ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/decorating_tasks.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/decorating_tasks.py \ decorating_task_wf --x 10 ``` diff --git a/docs/user_guide/advanced_composition/decorating_workflows.md b/docs/user_guide/advanced_composition/decorating_workflows.md index 3cd4fda0a2..751cd6a95c 100644 --- a/docs/user_guide/advanced_composition/decorating_workflows.md +++ b/docs/user_guide/advanced_composition/decorating_workflows.md @@ -23,7 +23,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the necessary libraries. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/decorating_workflows.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/decorating_workflows.py :caption: advanced_composition/decorating_workflows.py :lines: 1-6 ``` @@ -32,7 +32,7 @@ Let's define the tasks we need for setup and teardown. In this example, we use t {py:class}`unittest.mock.MagicMock` class to create a fake external service that we want to initialize at the beginning of our workflow and finish at the end. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/decorating_workflows.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/decorating_workflows.py :caption: advanced_composition/decorating_workflows.py :lines: 9-21 ``` @@ -45,7 +45,7 @@ external service and Flyte. We create a decorator that we want to use to wrap our workflow function. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/decorating_workflows.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/decorating_workflows.py :caption: advanced_composition/decorating_workflows.py :pyobject: setup_teardown ``` @@ -66,14 +66,14 @@ There are a few key pieces to note in the `setup_teardown` decorator above: We define two tasks that will constitute the workflow. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/decorating_workflows.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/decorating_workflows.py :caption: advanced_composition/decorating_workflows.py :lines: 63-70 ``` And then create our decorated workflow: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/decorating_workflows.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/decorating_workflows.py :caption: advanced_composition/decorating_workflows.py :lines: 74-82 ``` @@ -84,7 +84,7 @@ To run the provided workflow on the Flyte cluster, use the following command: ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/decorating_workflows.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/decorating_workflows.py \ decorating_workflow --x 10.0 ``` diff --git a/docs/user_guide/advanced_composition/dynamic_workflows.md b/docs/user_guide/advanced_composition/dynamic_workflows.md index c39272dfdf..949d88a847 100644 --- a/docs/user_guide/advanced_composition/dynamic_workflows.md +++ b/docs/user_guide/advanced_composition/dynamic_workflows.md @@ -40,28 +40,28 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, we import the required libraries. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/dynamic_workflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/dynamic_workflow.py :caption: advanced_composition/dynamic_workflow.py :lines: 1 ``` We define a task that returns the index of a character, where A-Z/a-z is equivalent to 0-25. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/dynamic_workflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/dynamic_workflow.py :caption: advanced_composition/dynamic_workflow.py :pyobject: return_index ``` We also create a task that prepares a list of 26 characters by populating the frequency of each character. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/dynamic_workflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/dynamic_workflow.py :caption: advanced_composition/dynamic_workflow.py :pyobject: update_list ``` We define a task to calculate the number of common characters between the two strings. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/dynamic_workflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/dynamic_workflow.py :caption: advanced_composition/dynamic_workflow.py :pyobject: derive_count ``` @@ -75,7 +75,7 @@ We define a dynamic workflow to accomplish the following: The looping process is contingent on the number of characters in both strings, which is unknown until runtime. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/dynamic_workflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/dynamic_workflow.py :caption: advanced_composition/dynamic_workflow.py :pyobject: count_characters ``` @@ -97,14 +97,14 @@ Local execution works when a `@dynamic` decorator is used because Flytekit treat Define a workflow that triggers the dynamic workflow. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/dynamic_workflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/dynamic_workflow.py :caption: advanced_composition/dynamic_workflow.py :pyobject: dynamic_wf ``` You can run the workflow locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/dynamic_workflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/dynamic_workflow.py :caption: advanced_composition/dynamic_workflow.py :lines: 78-79 ``` @@ -214,13 +214,13 @@ To run the provided workflows on the Flyte cluster, you can use the following co ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/dynamic_workflow.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/dynamic_workflow.py \ dynamic_wf --s1 "Pear" --s2 "Earth" ``` ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/dynamic_workflow.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/dynamic_workflow.py \ merge_sort --numbers '[1813, 3105, 3260, 2634, 383, 7037, 3291, 2403, 315, 7164]' --numbers_count 10 ``` diff --git a/docs/user_guide/advanced_composition/eager_workflows.md b/docs/user_guide/advanced_composition/eager_workflows.md index 480374413b..9bf3e019c8 100644 --- a/docs/user_guide/advanced_composition/eager_workflows.md +++ b/docs/user_guide/advanced_composition/eager_workflows.md @@ -45,7 +45,7 @@ using the `@eager` decorator. To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/eager_workflows.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/eager_workflows.py :caption: advanced_composition/eager_workflows.py :lines: 1-21 ``` @@ -116,7 +116,7 @@ One of the biggest benefits of eager workflows is that you can now materialize task and subworkflow outputs as Python values and do operations on them just like you would in any other Python function. Let's look at another example: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/eager_workflows.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/eager_workflows.py :caption: advanced_composition/eager_workflows.py :pyobject: another_eager_workflow ``` @@ -131,7 +131,7 @@ As you saw in the `simple_eager_workflow` workflow above, you can use regular Python conditionals in your eager workflows. Let's look at a more complicated example: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/eager_workflows.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/eager_workflows.py :caption: advanced_composition/eager_workflows.py :lines: 36-53 ``` @@ -144,7 +144,7 @@ to check if `out` is negative, but we're also using the `gt_100` task in the You can also gather the outputs of multiple tasks or subworkflows into a list: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/eager_workflows.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/eager_workflows.py :caption: advanced_composition/eager_workflows.py :lines: 58-69 ``` @@ -153,7 +153,7 @@ You can also gather the outputs of multiple tasks or subworkflows into a list: You can also invoke static workflows from within an eager workflow: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/eager_workflows.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/eager_workflows.py :caption: advanced_composition/eager_workflows.py :lines: 74-84 ``` @@ -162,7 +162,7 @@ You can also invoke static workflows from within an eager workflow: You can have nest eager subworkflows inside a parent eager workflow: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/eager_workflows.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/eager_workflows.py :caption: advanced_composition/eager_workflows.py :lines: 89-97 ``` @@ -171,7 +171,7 @@ You can have nest eager subworkflows inside a parent eager workflow: You can also catch exceptions in eager workflows through `EagerException`: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/eager_workflows.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/eager_workflows.py :caption: advanced_composition/eager_workflows.py :lines: 102-117 ``` @@ -195,7 +195,7 @@ and remotely. You can execute eager workflows locally by simply calling them like a regular `async` function: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/eager_workflows.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/eager_workflows.py :caption: advanced_composition/eager_workflows.py :lines: 123-125 ``` @@ -244,7 +244,7 @@ When using a sandbox cluster started with `flytectl demo start`, however, the `client_secret_group` and `client_secret_key` are not required, since the default sandbox configuration does not require key-based authentication. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/eager_workflows.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/eager_workflows.py :caption: advanced_composition/eager_workflows.py :lines: 130-145 ``` diff --git a/docs/user_guide/advanced_composition/intratask_checkpoints.md b/docs/user_guide/advanced_composition/intratask_checkpoints.md index 8c83eb154b..d856a45714 100644 --- a/docs/user_guide/advanced_composition/intratask_checkpoints.md +++ b/docs/user_guide/advanced_composition/intratask_checkpoints.md @@ -51,14 +51,14 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the necessary libraries and set the number of task retries to `3`: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/checkpoint.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/checkpoint.py :caption: advanced_composition/checkpoint.py :lines: 1-4 ``` We define a task to iterate precisely `n_iterations`, checkpoint its state, and recover from simulated failures: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/checkpoint.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/checkpoint.py :caption: advanced_composition/checkpoint.py :pyobject: use_checkpoint ``` @@ -69,14 +69,14 @@ The checkpoint system offers additional APIs, documented in the code accessible Create a workflow that invokes the task: The task will automatically undergo retries in the event of a {ref}`FlyteRecoverableException `. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/checkpoint.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/checkpoint.py :caption: advanced_composition/checkpoint.py :pyobject: checkpointing_example ``` The local checkpoint is not utilized here because retries are not supported: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/checkpoint.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/checkpoint.py :caption: advanced_composition/checkpoint.py :lines: 37-42 ``` @@ -87,7 +87,7 @@ To run the provided workflow on the Flyte cluster, use the following command: ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/checkpoint.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/checkpoint.py \ checkpointing_example --n_iterations 10 ``` diff --git a/docs/user_guide/advanced_composition/map_tasks.md b/docs/user_guide/advanced_composition/map_tasks.md index bae49faa82..bdd339b954 100644 --- a/docs/user_guide/advanced_composition/map_tasks.md +++ b/docs/user_guide/advanced_composition/map_tasks.md @@ -23,14 +23,14 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the required libraries: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/map_task.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/map_task.py :caption: advanced_composition/map_task.py :lines: 1 ``` Here's a simple workflow that uses {py:func}`map_task `: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/map_task.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/map_task.py :caption: advanced_composition/map_task.py :lines: 4-19 ``` @@ -78,7 +78,7 @@ When defining a map task, avoid calling other tasks in it. Flyte can't accuratel In this example, the map task `suboptimal_mappable_task` would not give you the best performance: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/map_task.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/map_task.py :caption: advanced_composition/map_task.py :lines: 31-40 ``` @@ -94,7 +94,7 @@ You might need to map a task with multiple inputs. For instance, consider a task that requires three inputs: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/map_task.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/map_task.py :caption: advanced_composition/map_task.py :pyobject: multi_input_task ``` @@ -103,21 +103,21 @@ You may want to map this task with only the ``quantity`` input, while keeping th Since a map task accepts only one input, you can achieve this by partially binding values to the map task. This can be done using the {py:func}`functools.partial` function: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/map_task.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/map_task.py :caption: advanced_composition/map_task.py :lines: 52-58 ``` Another possibility is to bind the outputs of a task to partials: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/map_task.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/map_task.py :caption: advanced_composition/map_task.py :lines: 63-72 ``` You can also provide multiple lists as input to a `map_task`: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/map_task.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/map_task.py :caption: advanced_composition/map_task.py :pyobject: map_workflow_with_lists ``` @@ -132,31 +132,31 @@ To run the provided workflows on the Flyte cluster, use the following commands: ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/map_task.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/map_task.py \ map_workflow ``` ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/map_task.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/map_task.py \ map_workflow_with_additional_params ``` ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/map_task.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/map_task.py \ multiple_inputs_map_workflow ``` ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/map_task.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/map_task.py \ map_workflow_partial_with_task_output ``` ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/map_task.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/map_task.py \ map_workflow_with_lists ``` diff --git a/docs/user_guide/advanced_composition/subworkflows.md b/docs/user_guide/advanced_composition/subworkflows.md index 8c4971e853..08a4bbb8d4 100644 --- a/docs/user_guide/advanced_composition/subworkflows.md +++ b/docs/user_guide/advanced_composition/subworkflows.md @@ -24,7 +24,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte Here's an example illustrating the calculation of slope, intercept and the corresponding y-value: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/subworkflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/subworkflow.py :caption: advanced_composition/subworkflow.py :lines: 1-35 ``` @@ -34,7 +34,7 @@ Subsequently, the `regression_line_wf` triggers `slope_intercept_wf` and then co To execute the workflow locally, use the following: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/subworkflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/subworkflow.py :caption: advanced_composition/subworkflow.py :lines: 39-40 ``` @@ -43,14 +43,14 @@ It's possible to nest a workflow that contains a subworkflow within another work Workflows can be easily constructed from other workflows, even if they function as standalone entities. Each workflow in this module has the capability to exist and run independently: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/subworkflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/subworkflow.py :caption: advanced_composition/subworkflow.py :pyobject: nested_regression_line_wf ``` You can run the nested workflow locally as well: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/subworkflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/subworkflow.py :caption: advanced_composition/subworkflow.py :lines: 52-53 ``` @@ -71,7 +71,7 @@ external workflows may offer a way to distribute the workload of a workflow acro Here's an example that illustrates the concept of external workflows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/subworkflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/subworkflow.py :caption: advanced_composition/subworkflow.py :lines: 61-71 ``` @@ -85,7 +85,7 @@ In the console screenshot above, note that the launch plan execution ID differs You can run a workflow containing an external workflow locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/subworkflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/subworkflow.py :caption: advanced_composition/subworkflow.py :lines: 75-76 ``` @@ -96,19 +96,19 @@ To run the provided workflows on a Flyte cluster, use the following commands: ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/subworkflow.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/subworkflow.py \ regression_line_wf ``` ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/subworkflow.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/subworkflow.py \ nested_regression_line_wf ``` ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/subworkflow.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/subworkflow.py \ nested_regression_line_lp ``` diff --git a/docs/user_guide/advanced_composition/waiting_for_external_inputs.md b/docs/user_guide/advanced_composition/waiting_for_external_inputs.md index 6af1782c12..0d3a2aae28 100644 --- a/docs/user_guide/advanced_composition/waiting_for_external_inputs.md +++ b/docs/user_guide/advanced_composition/waiting_for_external_inputs.md @@ -43,7 +43,7 @@ your workflow to mock out the behavior of some long-running computation. To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/waiting_for_external_inputs.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/waiting_for_external_inputs.py :caption: advanced_composition/waiting_for_external_inputs.py :lines: 1-20 ``` @@ -75,7 +75,7 @@ but before publishing it you want to give it a custom title. You can achieve this by defining a `wait_for_input` node that takes a `str` input and finalizes the report: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/waiting_for_external_inputs.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/waiting_for_external_inputs.py :caption: advanced_composition/waiting_for_external_inputs.py :lines: 24-49 ``` @@ -107,7 +107,7 @@ an explicit approval signal before continuing execution. Going back to our report-publishing use case, suppose that we want to block the publishing of a report for some reason (e.g. if they don't appear to be valid): -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/waiting_for_external_inputs.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/waiting_for_external_inputs.py :caption: advanced_composition/waiting_for_external_inputs.py :lines: 53-64 ``` @@ -120,7 +120,7 @@ You can also use the output of the `approve` function as a promise, feeding it to a subsequent task. Let's create a version of our report-publishing workflow where the approval happens after `create_report`: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/waiting_for_external_inputs.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/waiting_for_external_inputs.py :caption: advanced_composition/waiting_for_external_inputs.py :pyobject: approval_as_promise_wf ``` @@ -133,7 +133,7 @@ useful when we combine them with other Flyte constructs, like {ref}`conditionals To illustrate this, let's extend the report-publishing use case so that we produce an "invalid report" output in case we don't approve the final report: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/waiting_for_external_inputs.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/waiting_for_external_inputs.py :caption: advanced_composition/waiting_for_external_inputs.py :lines: 88-114 ``` diff --git a/docs/user_guide/basics/documenting_workflows.md b/docs/user_guide/basics/documenting_workflows.md index 3a29f9e562..9f5e20d5fb 100644 --- a/docs/user_guide/basics/documenting_workflows.md +++ b/docs/user_guide/basics/documenting_workflows.md @@ -15,14 +15,14 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the relevant libraries: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/documenting_workflows.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/documenting_workflows.py :caption: basics/documenting_workflows.py :lines: 1-3 ``` We import the `slope` and `intercept` tasks from the `workflow.py` file. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/documenting_workflows.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/documenting_workflows.py :caption: basics/documenting_workflows.py :lines: 6 ``` @@ -35,7 +35,7 @@ The initial section of the docstring provides a concise overview of the workflow The subsequent section provides a comprehensive explanation. The last part of the docstring outlines the parameters and return type. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/documenting_workflows.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/documenting_workflows.py :caption: basics/documenting_workflows.py :pyobject: sphinx_docstring_wf ``` @@ -49,7 +49,7 @@ The next section offers a comprehensive description. The third section of the docstring details all parameters along with their respective data types. The final section of the docstring explains the return type and its associated data type. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/documenting_workflows.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/documenting_workflows.py :caption: basics/documenting_workflows.py :pyobject: numpy_docstring_wf ``` @@ -63,7 +63,7 @@ The subsequent section of the docstring provides an extensive explanation. The third segment of the docstring outlines the parameters and return type, including their respective data types. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/documenting_workflows.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/documenting_workflows.py :caption: basics/documenting_workflows.py :pyobject: google_docstring_wf ``` diff --git a/docs/user_guide/basics/hello_world.md b/docs/user_guide/basics/hello_world.md index 63b830c010..a63b175fbe 100644 --- a/docs/user_guide/basics/hello_world.md +++ b/docs/user_guide/basics/hello_world.md @@ -17,7 +17,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import `task` and `workflow` from the `flytekit` library: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/hello_world.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/hello_world.py :caption: basics/hello_world.py :lines: 1 ``` @@ -25,7 +25,7 @@ To begin, import `task` and `workflow` from the `flytekit` library: Define a task that produces the string "Hello, World!". Simply using the `@task` decorator to annotate the Python function: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/hello_world.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/hello_world.py :caption: basics/hello_world.py :pyobject: say_hello ``` @@ -33,14 +33,14 @@ Simply using the `@task` decorator to annotate the Python function: You can handle the output of a task in the same way you would with a regular Python function. Store the output in a variable and use it as a return value for a Flyte workflow: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/hello_world.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/hello_world.py :caption: basics/hello_world.py :pyobject: hello_world_wf ``` Run the workflow by simply calling it like a Python function: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/hello_world.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/hello_world.py :caption: basics/hello_world.py :lines: 19-20 ``` diff --git a/docs/user_guide/basics/imperative_workflows.md b/docs/user_guide/basics/imperative_workflows.md index 562685f32a..db0e3a5ee0 100644 --- a/docs/user_guide/basics/imperative_workflows.md +++ b/docs/user_guide/basics/imperative_workflows.md @@ -23,28 +23,28 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the necessary dependencies: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/imperative_workflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/imperative_workflow.py :caption: basics/imperative_workflow.py :lines: 1 ``` We import the `slope` and `intercept` tasks from the `workflow.py` file: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/imperative_workflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/imperative_workflow.py :caption: basics/imperative_workflow.py :lines: 4 ``` Create an imperative workflow: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/imperative_workflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/imperative_workflow.py :caption: basics/imperative_workflow.py :lines: 7 ``` Add the workflow inputs to the imperative workflow: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/imperative_workflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/imperative_workflow.py :caption: basics/imperative_workflow.py :lines: 11-12 ``` @@ -56,21 +56,21 @@ you can create a {ref}`launch plan `. Add the tasks that need to be triggered from within the workflow: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/imperative_workflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/imperative_workflow.py :caption: basics/imperative_workflow.py :lines: 16-19 ``` Lastly, add the workflow output: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/imperative_workflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/imperative_workflow.py :caption: basics/imperative_workflow.py :lines: 23 ``` You can execute the workflow locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/imperative_workflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/imperative_workflow.py :caption: basics/imperative_workflow.py :lines: 27-28 ``` diff --git a/docs/user_guide/basics/launch_plans.md b/docs/user_guide/basics/launch_plans.md index 7ef61e4d2f..63ace47a70 100644 --- a/docs/user_guide/basics/launch_plans.md +++ b/docs/user_guide/basics/launch_plans.md @@ -27,56 +27,56 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the necessary libraries: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/launch_plan.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/launch_plan.py :caption: basics/launch_plan.py :lines: 1 ``` We import the workflow from the `workflow.py` file for which we're going to create a launch plan: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/launch_plan.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/launch_plan.py :caption: basics/launch_plan.py :lines: 5 ``` Create a default launch plan with no inputs during serialization: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/launch_plan.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/launch_plan.py :caption: basics/launch_plan.py :lines: 8 ``` You can run the launch plan locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/launch_plan.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/launch_plan.py :caption: basics/launch_plan.py :lines: 11 ``` Create a launch plan and specify the default inputs: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/launch_plan.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/launch_plan.py :caption: basics/launch_plan.py :lines: 14-16 ``` You can trigger the launch plan locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/launch_plan.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/launch_plan.py :caption: basics/launch_plan.py :lines: 19 ``` You can override the defaults as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/launch_plan.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/launch_plan.py :caption: basics/launch_plan.py :lines: 22 ``` It's possible to lock launch plan inputs, preventing them from being overridden during execution: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/launch_plan.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/launch_plan.py :caption: basics/launch_plan.py :lines: 25-27 ``` diff --git a/docs/user_guide/basics/named_outputs.md b/docs/user_guide/basics/named_outputs.md index 2e02678822..00b9160997 100644 --- a/docs/user_guide/basics/named_outputs.md +++ b/docs/user_guide/basics/named_outputs.md @@ -22,21 +22,21 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the required dependencies: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/named_outputs.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/named_outputs.py :caption: basics/named_outputs.py :lines: 1-3 ``` Define a `NamedTuple` and assign it as an output to a task: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/named_outputs.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/named_outputs.py :caption: basics/named_outputs.py :lines: 6-14 ``` Likewise, assign a `NamedTuple` to the output of `intercept` task: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/named_outputs.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/named_outputs.py :caption: basics/named_outputs.py :lines: 18-26 ``` @@ -59,14 +59,14 @@ Remember that we are extracting individual task execution outputs by dereferenci This is necessary because `NamedTuple`s function as tuples and require this dereferencing: ::: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/named_outputs.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/named_outputs.py :caption: basics/named_outputs.py :lines: 32-39 ``` You can run the workflow locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/named_outputs.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/named_outputs.py :caption: basics/named_outputs.py :lines: 43-44 ``` diff --git a/docs/user_guide/basics/shell_tasks.md b/docs/user_guide/basics/shell_tasks.md index 95df9a90d8..8680b87f5d 100644 --- a/docs/user_guide/basics/shell_tasks.md +++ b/docs/user_guide/basics/shell_tasks.md @@ -15,7 +15,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte First, import the necessary libraries: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/shell_task.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/shell_task.py :caption: basics/shell_task.py :lines: 1-8 ``` @@ -24,7 +24,7 @@ With the required imports in place, you can proceed to define a shell task. To create a shell task, provide a name for it, specify the bash script to be executed, and define inputs and outputs if needed: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/shell_task.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/shell_task.py :caption: basics/shell_task.py :lines: 13-55 ``` @@ -40,21 +40,21 @@ Here's a breakdown of the parameters of the `ShellTask`: We define a task to instantiate `FlyteFile` and `FlyteDirectory`. A `.gitkeep` file is created in the FlyteDirectory as a placeholder to ensure the directory exists: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/shell_task.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/shell_task.py :caption: basics/shell_task.py :pyobject: create_entities ``` We create a workflow to define the dependencies between the tasks: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/shell_task.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/shell_task.py :caption: basics/shell_task.py :pyobject: shell_task_wf ``` You can run the workflow locally: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/shell_task.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/shell_task.py :caption: basics/shell_task.py :lines: 85-86 ``` diff --git a/docs/user_guide/basics/tasks.md b/docs/user_guide/basics/tasks.md index 8c059e8d02..acedcad51e 100644 --- a/docs/user_guide/basics/tasks.md +++ b/docs/user_guide/basics/tasks.md @@ -34,7 +34,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import `task` from the `flytekit` library: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/task.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/task.py :caption: basics/task.py :lines: 1 ``` @@ -45,7 +45,7 @@ Learn more about the supported types in the {ref}`type-system section `out0, out1, out2, ...`. You can execute a Flyte task just like any regular Python function: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/task.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/task.py :caption: basics/task.py :lines: 14-15 ``` @@ -73,7 +73,7 @@ the values for the corresponding parameters. To run it locally, you can use the following `pyflyte run` command: ``` pyflyte run \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/task.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/task.py \ slope --x '[-3,0,3]' --y '[7,4,-2]' ``` @@ -81,7 +81,7 @@ If you want to run it remotely on the Flyte cluster, simply add the `--remote flag` to the `pyflyte run` command: ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/task.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/task.py \ slope --x '[-3,0,3]' --y '[7,4,-2]' ``` diff --git a/docs/user_guide/basics/workflows.md b/docs/user_guide/basics/workflows.md index 85b6db1b8e..a78890fa46 100644 --- a/docs/user_guide/basics/workflows.md +++ b/docs/user_guide/basics/workflows.md @@ -25,7 +25,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import {py:func}`~flytekit.task` and {py:func}`~flytekit.workflow` from the flytekit library: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/workflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/workflow.py :caption: basics/workflow.py :lines: 1 ``` @@ -33,7 +33,7 @@ To begin, import {py:func}`~flytekit.task` and {py:func}`~flytekit.workflow` fro We define `slope` and `intercept` tasks to compute the slope and intercept of the regression line, respectively: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/workflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/workflow.py :caption: basics/workflow.py :lines: 6-19 ``` @@ -41,7 +41,7 @@ intercept of the regression line, respectively: Define a workflow to establish the task dependencies. Just like a task, a workflow is also strongly typed: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/workflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/workflow.py :caption: basics/workflow.py :pyobject: simple_wf ``` @@ -68,7 +68,7 @@ Bear in mind that a workflow can have tasks, other workflows and dynamic workflo You can run a workflow by calling it as you would with a Python function and providing the necessary inputs: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/workflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/workflow.py :caption: basics/workflow.py :lines: 33-34 ``` @@ -77,7 +77,7 @@ To run the workflow locally, you can use the following `pyflyte run` command: ``` pyflyte run \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/workflow.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/workflow.py \ simple_wf --x '[-3,0,3]' --y '[7,4,-2]' ``` @@ -86,7 +86,7 @@ simply add the `--remote flag` to the `pyflyte run` command: ``` pyflyte run --remote \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/workflow.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/workflow.py \ simple_wf --x '[-3,0,3]' --y '[7,4,-2]' ``` @@ -101,7 +101,7 @@ without the confines of a workflow, offers a convenient approach for iterating o You can use the {py:func}`functools.partial` function to assign default or constant values to the parameters of your tasks. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/workflow.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/workflow.py :caption: basics/workflow.py :lines: 39-45 ``` diff --git a/docs/user_guide/customizing_dependencies/imagespec.md b/docs/user_guide/customizing_dependencies/imagespec.md index 586bfe2580..7718c1646a 100644 --- a/docs/user_guide/customizing_dependencies/imagespec.md +++ b/docs/user_guide/customizing_dependencies/imagespec.md @@ -26,7 +26,7 @@ Before building the image, Flytekit checks the container registry first to see i To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/customizing_dependencies/customizing_dependencies/image_spec.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/customizing_dependencies/customizing_dependencies/image_spec.py :caption: customizing_dependencies/image_spec.py :lines: 1-4 ``` @@ -45,7 +45,7 @@ More specifically, flytekit invokes [DefaultImages.default_image()](https://gith This function determines and returns the default image based on the Python version and flytekit version. For example, if you are using python 3.8 and flytekit 0.16.0, the default image assigned will be `ghcr.io/flyteorg/flytekit:py3.8-1.6.0`. If desired, you can also override the default image by providing a custom `base_image` parameter when using the `ImageSpec`. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/customizing_dependencies/customizing_dependencies/image_spec.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/customizing_dependencies/customizing_dependencies/image_spec.py :caption: customizing_dependencies/image_spec.py :lines: 6-19 ``` @@ -59,14 +59,14 @@ To upload the image to the local registry in the demo cluster, indicate the regi If the task is indeed using the image built from the `ImageSpec`, it will then import Tensorflow. This approach helps minimize module loading time and prevents unnecessary dependency installation within a single image. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/customizing_dependencies/customizing_dependencies/image_spec.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/customizing_dependencies/customizing_dependencies/image_spec.py :caption: customizing_dependencies/image_spec.py :lines: 21-22 ``` To enable tasks to utilize the images built with `ImageSpec`, you can specify the `container_image` parameter for those tasks. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/customizing_dependencies/customizing_dependencies/image_spec.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/customizing_dependencies/customizing_dependencies/image_spec.py :caption: customizing_dependencies/image_spec.py :lines: 27-56 ``` diff --git a/docs/user_guide/customizing_dependencies/multiple_images_in_a_workflow.md b/docs/user_guide/customizing_dependencies/multiple_images_in_a_workflow.md index f4411fac65..c999a06620 100644 --- a/docs/user_guide/customizing_dependencies/multiple_images_in_a_workflow.md +++ b/docs/user_guide/customizing_dependencies/multiple_images_in_a_workflow.md @@ -18,7 +18,7 @@ If the Docker image is not available publicly, refer to {ref}`Pulling Private Im To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/customizing_dependencies/customizing_dependencies/multi_images.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/customizing_dependencies/customizing_dependencies/multi_images.py :caption: customizing_dependencies/multi_images.py :lines: 1-24 ``` diff --git a/docs/user_guide/customizing_dependencies/raw_containers.md b/docs/user_guide/customizing_dependencies/raw_containers.md index 72a2acd71f..ae72b652f3 100644 --- a/docs/user_guide/customizing_dependencies/raw_containers.md +++ b/docs/user_guide/customizing_dependencies/raw_containers.md @@ -15,7 +15,7 @@ Refer to the raw protocol to understand how to leverage this. To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/customizing_dependencies/customizing_dependencies/raw_container.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/customizing_dependencies/customizing_dependencies/raw_container.py :caption: customizing_dependencies/raw_container.py :lines: 1-5 ``` @@ -31,7 +31,7 @@ is `calculate_ellipse_area_shell`. This name has to be unique in the entire proj `inputs` and `outputs` specify the interface for the task; thus it should be an ordered dictionary of typed input and output variables. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/customizing_dependencies/customizing_dependencies/raw_container.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/customizing_dependencies/customizing_dependencies/raw_container.py :caption: customizing_dependencies/raw_container.py :lines: 15-112 ``` @@ -39,7 +39,7 @@ output variables. As can be seen in this example, `ContainerTask`s can be interacted with like normal Python functions, whose inputs correspond to the declared input variables. All data returned by the tasks are consumed and logged by a Flyte task. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/customizing_dependencies/customizing_dependencies/raw_container.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/customizing_dependencies/customizing_dependencies/raw_container.py :caption: customizing_dependencies/raw_container.py :pyobject: wf ``` diff --git a/docs/user_guide/data_types_and_io/accessing_attributes.md b/docs/user_guide/data_types_and_io/accessing_attributes.md index 4c4a01483f..8df9aaed4d 100644 --- a/docs/user_guide/data_types_and_io/accessing_attributes.md +++ b/docs/user_guide/data_types_and_io/accessing_attributes.md @@ -15,7 +15,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the required dependencies and define a common task for subsequent use: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/attribute_access.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/attribute_access.py :caption: data_types_and_io/attribute_access.py :lines: 1-10 ``` @@ -27,7 +27,7 @@ You can access an output list using index notation. Flyte currently does not support output promise access through list slicing. ::: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/attribute_access.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/attribute_access.py :caption: data_types_and_io/attribute_access.py :lines: 14-23 ``` @@ -35,7 +35,7 @@ Flyte currently does not support output promise access through list slicing. ## Dictionary Access the output dictionary by specifying the key. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/attribute_access.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/attribute_access.py :caption: data_types_and_io/attribute_access.py :lines: 27-35 ``` @@ -43,7 +43,7 @@ Access the output dictionary by specifying the key. ## Data class Directly access an attribute of a dataclass. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/attribute_access.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/attribute_access.py :caption: data_types_and_io/attribute_access.py :lines: 39-53 ``` @@ -51,14 +51,14 @@ Directly access an attribute of a dataclass. ## Complex type Combinations of list, dict and dataclass also work effectively. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/attribute_access.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/attribute_access.py :caption: data_types_and_io/attribute_access.py :lines: 57-80 ``` You can run all the workflows locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/attribute_access.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/attribute_access.py :caption: data_types_and_io/attribute_access.py :lines: 84-88 ``` diff --git a/docs/user_guide/data_types_and_io/dataclass.md b/docs/user_guide/data_types_and_io/dataclass.md index 4c7704d73a..6ff4f08a1f 100644 --- a/docs/user_guide/data_types_and_io/dataclass.md +++ b/docs/user_guide/data_types_and_io/dataclass.md @@ -25,7 +25,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the necessary dependencies: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/dataclass.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/dataclass.py :caption: data_types_and_io/dataclass.py :lines: 1-10 ``` @@ -33,7 +33,7 @@ To begin, import the necessary dependencies: ## Python types We define a `dataclass` with `int`, `str` and `dict` as the data types. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/dataclass.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/dataclass.py :caption: data_types_and_io/dataclass.py :pyobject: Datum ``` @@ -46,7 +46,7 @@ All variables in a data class should be **annotated with their type**. Failure t Once declared, a dataclass can be returned as an output or accepted as an input. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/dataclass.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/dataclass.py :caption: data_types_and_io/dataclass.py :lines: 28-43 ``` @@ -55,7 +55,7 @@ Once declared, a dataclass can be returned as an output or accepted as an input. We also define a data class that accepts {std:ref}`StructuredDataset `, {std:ref}`FlyteFile ` and {std:ref}`FlyteDirectory `. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/dataclass.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/dataclass.py :caption: data_types_and_io/dataclass.py :lines: 47-84 ``` @@ -65,14 +65,14 @@ flyte file, flyte directory and structured dataset. We define a workflow that calls the tasks created above. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/dataclass.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/dataclass.py :caption: data_types_and_io/dataclass.py :pyobject: dataclass_wf ``` You can run the workflow locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/dataclass.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/dataclass.py :caption: data_types_and_io/dataclass.py :lines: 97-98 ``` @@ -80,7 +80,7 @@ You can run the workflow locally as follows: To trigger a task that accepts a dataclass as an input with `pyflyte run`, you can provide a JSON file as an input: ``` pyflyte run \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/dataclass.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/dataclass.py \ add --x dataclass_input.json --y dataclass_input.json ``` diff --git a/docs/user_guide/data_types_and_io/enum_type.md b/docs/user_guide/data_types_and_io/enum_type.md index 58142c93ea..b8e9011921 100644 --- a/docs/user_guide/data_types_and_io/enum_type.md +++ b/docs/user_guide/data_types_and_io/enum_type.md @@ -22,7 +22,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the dependencies: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/enum_type.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/enum_type.py :caption: data_types_and_io/enum_type.py :lines: 1-3 ``` @@ -30,14 +30,14 @@ To begin, import the dependencies: We define an enum and a simple coffee maker workflow that accepts an order and brews coffee ☕️ accordingly. The assumption is that the coffee maker only understands enum inputs: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/enum_type.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/enum_type.py :caption: data_types_and_io/enum_type.py :lines: 9-35 ``` The workflow can also accept an enum value: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/enum_type.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/enum_type.py :caption: data_types_and_io/enum_type.py :pyobject: coffee_maker_enum ``` @@ -45,13 +45,13 @@ The workflow can also accept an enum value: You can send a string to the `coffee_maker_enum` workflow during its execution, like this: ``` pyflyte run \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/enum_type.py \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/enum_type.py \ coffee_maker_enum --coffee_enum="latte" ``` You can run the workflows locally: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/enum_type.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/enum_type.py :caption: data_types_and_io/enum_type.py :lines: 44-46 ``` diff --git a/docs/user_guide/data_types_and_io/flytedirectory.md b/docs/user_guide/data_types_and_io/flytedirectory.md index b3700f7249..121a7d9b67 100644 --- a/docs/user_guide/data_types_and_io/flytedirectory.md +++ b/docs/user_guide/data_types_and_io/flytedirectory.md @@ -16,7 +16,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the libraries: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/folder.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/folder.py :caption: data_types_and_io/folder.py :lines: 1-10 ``` @@ -27,7 +27,7 @@ let's continue by considering the normalization of columns in a CSV file. The following task downloads a list of URLs pointing to CSV files and returns the folder path in a `FlyteDirectory` object. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/folder.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/folder.py :caption: data_types_and_io/folder.py :pyobject: download_files ``` @@ -57,7 +57,7 @@ demonstrates how Flyte tasks are simply entrypoints of execution, which can them other functions and routines that are written in pure Python. ::: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/folder.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/folder.py :caption: data_types_and_io/folder.py :pyobject: normalize_columns ``` @@ -65,7 +65,7 @@ other functions and routines that are written in pure Python. We then define a task that accepts the previously downloaded folder, along with some metadata about the column names of each file in the directory and the column names that we want to normalize. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/folder.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/folder.py :caption: data_types_and_io/folder.py :pyobject: normalize_all_files ``` @@ -74,14 +74,14 @@ Compose all of the above tasks into a workflow. This workflow accepts a list of URL strings pointing to a remote location containing a CSV file, a list of column names associated with each CSV file, and a list of columns that we want to normalize. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/folder.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/folder.py :caption: data_types_and_io/folder.py :pyobject: download_and_normalize_csv_files ``` You can run the workflow locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/folder.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/folder.py :caption: data_types_and_io/folder.py :lines: 94-114 ``` diff --git a/docs/user_guide/data_types_and_io/flytefile.md b/docs/user_guide/data_types_and_io/flytefile.md index 8bc4471559..e9c02e2132 100644 --- a/docs/user_guide/data_types_and_io/flytefile.md +++ b/docs/user_guide/data_types_and_io/flytefile.md @@ -23,7 +23,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte First, import the libraries: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/file.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/file.py :caption: data_types_and_io/file.py :lines: 1-8 ``` @@ -43,7 +43,7 @@ Predefined aliases for commonly used flyte file formats are also available. You can find them [here](https://github.com/flyteorg/flytekit/blob/master/flytekit/types/file/__init__.py). ::: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/file.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/file.py :caption: data_types_and_io/file.py :pyobject: normalize_columns ``` diff --git a/docs/user_guide/data_types_and_io/pickle_type.md b/docs/user_guide/data_types_and_io/pickle_type.md index 19987d6288..301ff95f9f 100644 --- a/docs/user_guide/data_types_and_io/pickle_type.md +++ b/docs/user_guide/data_types_and_io/pickle_type.md @@ -27,7 +27,7 @@ This example demonstrates how you can utilize custom objects without registering To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/pickle_type.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/pickle_type.py :caption: data_types_and_io/pickle_type.py :lines: 1 ``` @@ -40,7 +40,7 @@ Alternatively, you can {ref}`turn this object into a dataclass ` for We have used a simple object here for demonstration purposes. ::: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/pickle_type.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/pickle_type.py :caption: data_types_and_io/pickle_type.py :lines: 7-26 ``` @@ -53,7 +53,7 @@ or significant list elements, you can specify a batch size. This feature allows for the processing of each batch as a separate pickle file. The following example demonstrates how to set the batch size. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/pickle_type.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/pickle_type.py :caption: data_types_and_io/pickle_type.py :lines: 35-58 ``` @@ -64,7 +64,7 @@ The `welcome_superheroes` task will generate two pickle files: one containing tw You can run the workflows locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/pickle_type.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/pickle_type.py :caption: data_types_and_io/pickle_type.py :lines: 62-64 ``` diff --git a/docs/user_guide/data_types_and_io/pytorch_type.md b/docs/user_guide/data_types_and_io/pytorch_type.md index b224b0f7b5..24696f6a75 100644 --- a/docs/user_guide/data_types_and_io/pytorch_type.md +++ b/docs/user_guide/data_types_and_io/pytorch_type.md @@ -18,7 +18,7 @@ At times, you may find the need to pass tensors and modules (models) within your To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/pytorch_type.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/pytorch_type.py :caption: data_types_and_io/pytorch_type.py :lines: 5-50 ``` @@ -36,7 +36,7 @@ According to the PyTorch [docs](https://pytorch.org/tutorials/beginner/saving_lo it's recommended to store the module's `state_dict` rather than the module itself, although the serialization should work in either case. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/pytorch_type.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/pytorch_type.py :caption: data_types_and_io/pytorch_type.py :lines: 63-117 ``` diff --git a/docs/user_guide/data_types_and_io/structureddataset.md b/docs/user_guide/data_types_and_io/structureddataset.md index 1ba165717f..e4eed0a956 100644 --- a/docs/user_guide/data_types_and_io/structureddataset.md +++ b/docs/user_guide/data_types_and_io/structureddataset.md @@ -37,14 +37,14 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the dependencies for the example: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :lines: 1-18 ``` Define a task that returns a Pandas DataFrame. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :pyobject: generate_pandas_df ``` @@ -66,7 +66,7 @@ you can just specify the column names and their types in the structured dataset First, initialize column types you want to extract from the `StructuredDataset`. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :lines: 30-31 ``` @@ -76,7 +76,7 @@ When you invoke `all()` with ``pandas.DataFrame``, the Flyte engine downloads th Keep in mind that you can invoke ``open()`` with any dataframe type that's supported or added to structured dataset. For instance, you can use ``pa.Table`` to convert the Pandas DataFrame to a PyArrow table. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :lines: 41-51 ``` @@ -89,7 +89,7 @@ You can use a custom serialization format to serialize your dataframes. Here's how you can register the Pandas to CSV handler, which is already available, and enable the CSV serialization by annotating the structured dataset with the CSV format: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :lines: 57-71 ``` @@ -198,7 +198,7 @@ enabling the use of a 2D NumPy array as a valid type within structured datasets. Extend `StructuredDatasetEncoder` and implement the `encode` function. The `encode` function converts NumPy array to an intermediate format (parquet file format in this case). -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :pyobject: NumpyEncodingHandler ``` @@ -208,7 +208,7 @@ The `encode` function converts NumPy array to an intermediate format (parquet fi Extend {py:class}`StructuredDatasetDecoder` and implement the {py:meth}`~StructuredDatasetDecoder.decode` function. The {py:meth}`~StructuredDatasetDecoder.decode` function converts the parquet file to a `numpy.ndarray`. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :pyobject: NumpyDecodingHandler ``` @@ -218,7 +218,7 @@ The {py:meth}`~StructuredDatasetDecoder.decode` function converts the parquet fi Create a default renderer for numpy array, then Flytekit will use this renderer to display schema of NumPy array on the Flyte deck. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :pyobject: NumpyRenderer ``` @@ -228,14 +228,14 @@ Specify the Python type you want to register this encoder with (`np.ndarray`), the storage engine to register this against (if not specified, it is assumed to work for all the storage backends), and the byte format, which in this case is `PARQUET`. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :lines: 127-129 ``` You can now use `numpy.ndarray` to deserialize the parquet file to NumPy and serialize a task's output (NumPy array) to a parquet file. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :lines: 134-149 ``` @@ -246,7 +246,7 @@ You can now use `numpy.ndarray` to deserialize the parquet file to NumPy and ser You can run the code locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :lines: 153-157 ``` @@ -259,7 +259,7 @@ Like most storage formats (e.g. Avro, Parquet, and BigQuery), StructuredDataset Nested field StructuredDataset should be run when flytekit version > 1.11.0. ::: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :lines: 159-270 ``` diff --git a/docs/user_guide/development_lifecycle/cache_serializing.md b/docs/user_guide/development_lifecycle/cache_serializing.md index 6552b90d30..1445de13cf 100644 --- a/docs/user_guide/development_lifecycle/cache_serializing.md +++ b/docs/user_guide/development_lifecycle/cache_serializing.md @@ -17,7 +17,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte For any {py:func}`flytekit.task` in Flyte, there is always one required import, which is: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/task_cache_serialize.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/task_cache_serialize.py :caption: development_lifecycle/task_cache_serialize.py :lines: 1 ``` @@ -27,7 +27,7 @@ Task cache serializing is disabled by default to avoid unexpected behavior for t This operation is only useful for cacheable tasks, where one may reuse output from a previous execution. Flyte requires implicitly enabling the `cache` parameter on all cache serializable tasks. Cache key definitions follow the same rules as non-serialized cache tasks. It is important to understand the implications of the task signature and `cache_version` parameter in defining cached results. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/task_cache_serialize.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/task_cache_serialize.py :caption: development_lifecycle/task_cache_serialize.py :pyobject: square ``` diff --git a/docs/user_guide/development_lifecycle/caching.md b/docs/user_guide/development_lifecycle/caching.md index 8419992f89..c14ec76563 100644 --- a/docs/user_guide/development_lifecycle/caching.md +++ b/docs/user_guide/development_lifecycle/caching.md @@ -25,14 +25,14 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte Import the necessary libraries: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/task_cache.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/task_cache.py :caption: development_lifecycle/task_cache.py :lines: 1-3 ``` For any {py:func}`flytekit.task` in Flyte, there is always one required import, which is: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/task_cache.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/task_cache.py :caption: development_lifecycle/task_cache.py :lines: 8-10 ``` @@ -43,7 +43,7 @@ Task caching is disabled by default to avoid unintended consequences of caching Bumping the `cache_version` is akin to invalidating the cache. You can manually update this version and Flyte caches the next execution instead of relying on the old cache. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/task_cache.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/task_cache.py :caption: development_lifecycle/task_cache.py :pyobject: square ``` @@ -103,7 +103,7 @@ The format used by the store is opaque and not meant to be inspectable. The default behavior displayed by Flyte's memoization feature might not match the user intuition. For example, this code makes use of pandas dataframes: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/task_cache.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/task_cache.py :caption: development_lifecycle/task_cache.py :lines: 39-54 ``` @@ -112,7 +112,7 @@ If run twice with the same inputs, one would expect that `bar` would trigger a c However, with release 1.2.0, Flyte provides a new way to control memoization behavior of literals. This is done via a `typing.Annotated` call on the task signature. For example, in order to cache the result of calls to `bar`, you can rewrite the code above like this: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/task_cache.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/task_cache.py :caption: development_lifecycle/task_cache.py :lines: 64-85 ``` @@ -128,7 +128,7 @@ This feature also works in local execution. Here's a complete example of the feature: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/task_cache.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/task_cache.py :caption: development_lifecycle/task_cache.py :lines: 97-134 ``` diff --git a/docs/user_guide/development_lifecycle/decks.md b/docs/user_guide/development_lifecycle/decks.md index b944183049..c59cbd1945 100644 --- a/docs/user_guide/development_lifecycle/decks.md +++ b/docs/user_guide/development_lifecycle/decks.md @@ -28,7 +28,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the dependencies: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/decks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py :lines: 1-4 ``` @@ -39,7 +39,7 @@ We create a new deck named `pca` and render Markdown content along with a You can begin by initializing an {ref}`ImageSpec ` object to encompass all the necessary dependencies. This approach automatically triggers a Docker build, alleviating the need for you to manually create a Docker image. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/decks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py :lines: 15-19 ``` @@ -51,7 +51,7 @@ To upload the image to the local registry in the demo cluster, indicate the regi Note the usage of `append` to append the Plotly deck to the Markdown deck. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/decks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py :pyobject: pca_plot ``` @@ -96,7 +96,7 @@ When the task connected with a deck object is executed, these objects employ ren Creates a profile report from a Pandas DataFrame. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/decks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py :lines: 44-51 ``` @@ -113,7 +113,7 @@ Creates a profile report from a Pandas DataFrame. Renders DataFrame as an HTML table. This renderer doesn't necessitate plugin installation since it's accessible within the flytekit library. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/decks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py :lines: 57-64 ``` @@ -127,7 +127,7 @@ This renderer doesn't necessitate plugin installation since it's accessible with Converts a Markdown string into HTML, producing HTML as a Unicode string. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/decks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py :pyobject: markdown_renderer ``` @@ -147,7 +147,7 @@ The median (Q2) is indicated by a line within the box. Typically, the whiskers extend to the edges of the box, plus or minus 1.5 times the interquartile range (IQR: Q3-Q1). -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/decks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py :lines: 85-91 ``` @@ -162,7 +162,7 @@ plus or minus 1.5 times the interquartile range (IQR: Q3-Q1). Converts a {ref}`FlyteFile ` or `PIL.Image.Image` object into an HTML string, where the image data is encoded as a base64 string. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/decks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py :lines: 97-111 ``` @@ -176,7 +176,7 @@ where the image data is encoded as a base64 string. Converts a Pandas dataframe into an HTML table. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/decks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py :lines: 115-123 ``` diff --git a/docs/user_guide/development_lifecycle/failure_node.md b/docs/user_guide/development_lifecycle/failure_node.md index a3bd1ae328..cd4c225d76 100644 --- a/docs/user_guide/development_lifecycle/failure_node.md +++ b/docs/user_guide/development_lifecycle/failure_node.md @@ -15,21 +15,21 @@ To address this issue, you can add a failure node into your workflow. This ensur To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/failure_node.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/failure_node.py :caption: development_lifecycle/failure_node.py :lines: 1-6 ``` Create a task that will fail during execution: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/failure_node.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/failure_node.py :caption: development_lifecycle/failure_node.py :lines: 10-18 ``` Create a task that will be executed if any of the tasks in the workflow fail: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/failure_node.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/failure_node.py :caption: development_lifecycle/failure_node.py :pyobject: clean_up ``` @@ -40,21 +40,21 @@ Specify the `on_failure` to a cleanup task. This task will be executed if any of The input of `clean_up` should be the exact same as the input of the workflow. ::: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/failure_node.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/failure_node.py :caption: development_lifecycle/failure_node.py :pyobject: subwf ``` By setting the failure policy to `FAIL_AFTER_EXECUTABLE_NODES_COMPLETE` to ensure that the `wf1` is executed even if the subworkflow fails. In this case, both parent and child workflows will fail, resulting in the `clean_up` task being executed twice: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/failure_node.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/failure_node.py :caption: development_lifecycle/failure_node.py :lines: 42-53 ``` You can also set the `on_failure` to a workflow. This workflow will be executed if any of the tasks in the workflow fail: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/development_lifecycle/development_lifecycle/failure_node.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/failure_node.py :caption: development_lifecycle/failure_node.py :pyobject: wf2 ``` diff --git a/docs/user_guide/extending/user_container_task_plugins.md b/docs/user_guide/extending/user_container_task_plugins.md index 96ed6fb310..99a3adf155 100644 --- a/docs/user_guide/extending/user_container_task_plugins.md +++ b/docs/user_guide/extending/user_container_task_plugins.md @@ -32,7 +32,7 @@ def wait_and_run(path: str) -> int: return do_next(path=path) ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/extending/extending/user_container.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/extending/extending/user_container.py :caption: extending/user_container.py :lines: 1-6 ``` @@ -42,7 +42,7 @@ def wait_and_run(path: str) -> int: As illustrated above, to achieve this structure we need to create a class named `WaitForObjectStoreFile`, which derives from {py:class}`flytekit.PythonFunctionTask` as follows. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/extending/extending/user_container.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/extending/extending/user_container.py :caption: extending/user_container.py :pyobject: WaitForObjectStoreFile ``` @@ -68,14 +68,14 @@ Refer to the [spark plugin](https://github.com/flyteorg/flytekit/tree/master/plu ### Actual usage -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/extending/extending/user_container.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/extending/extending/user_container.py :caption: extending/user_container.py :lines: 54-69 ``` And of course, you can run the workflow locally using your own new shiny plugin! -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/extending/extending/user_container.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/extending/extending/user_container.py :caption: extending/user_container.py :lines: 73-78 ``` diff --git a/docs/user_guide/productionizing/customizing_task_resources.md b/docs/user_guide/productionizing/customizing_task_resources.md index 1f95d26219..6ba07a604b 100644 --- a/docs/user_guide/productionizing/customizing_task_resources.md +++ b/docs/user_guide/productionizing/customizing_task_resources.md @@ -36,35 +36,35 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte Import the dependencies: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/customizing_resources.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/customizing_resources.py :caption: productionizing/customizing_resources.py :lines: 1-3 ``` Define a task and configure the resources to be allocated to it: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/customizing_resources.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/customizing_resources.py :caption: productionizing/customizing_resources.py :pyobject: count_unique_numbers ``` Define a task that computes the square of a number: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/customizing_resources.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/customizing_resources.py :caption: productionizing/customizing_resources.py :pyobject: square ``` You can use the tasks decorated with memory and storage hints like regular tasks in a workflow. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/customizing_resources.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/customizing_resources.py :caption: productionizing/customizing_resources.py :pyobject: my_workflow ``` You can execute the workflow locally. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/customizing_resources.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/customizing_resources.py :caption: productionizing/customizing_resources.py :lines: 32-34 ``` @@ -82,7 +82,7 @@ Let's understand how the resources can be initialized with an example. Import the dependencies. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/customizing_resources.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/customizing_resources.py :caption: productionizing/customizing_resources.py :lines: 38-40 ``` @@ -90,28 +90,28 @@ Import the dependencies. Define a task and configure the resources to be allocated to it. You can use tasks decorated with memory and storage hints like regular tasks in a workflow. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/customizing_resources.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/customizing_resources.py :caption: productionizing/customizing_resources.py :pyobject: count_unique_numbers ``` Define a task that computes the square of a number: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/customizing_resources.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/customizing_resources.py :caption: productionizing/customizing_resources.py :pyobject: square_1 ``` The `with_overrides` method overrides the old resource allocations: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/customizing_resources.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/customizing_resources.py :caption: productionizing/customizing_resources.py :pyobject: my_pipeline ``` You can execute the workflow locally: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/customizing_resources.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/customizing_resources.py :caption: productionizing/customizing_resources.py :lines: 65-67 ``` diff --git a/docs/user_guide/productionizing/notifications.md b/docs/user_guide/productionizing/notifications.md index 6837a7b632..d33199a177 100644 --- a/docs/user_guide/productionizing/notifications.md +++ b/docs/user_guide/productionizing/notifications.md @@ -21,14 +21,14 @@ When a workflow reaches a specified [terminal workflow execution phase](https:// To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/lp_notifications.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/lp_notifications.py :caption: productionizing/lp_notifications.py :lines: 1 ``` Consider the following example workflow: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/lp_notifications.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/lp_notifications.py :caption: productionizing/lp_notifications.py :lines: 3-14 ``` @@ -37,21 +37,21 @@ Here are three scenarios that can help deepen your understanding of how notifica 1. Launch Plan triggers email notifications when the workflow execution reaches the `SUCCEEDED` phase. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/lp_notifications.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/lp_notifications.py :caption: productionizing/lp_notifications.py :lines: 20-30 ``` 2. Notifications shine when used for scheduled workflows to alert for failures. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/lp_notifications.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/lp_notifications.py :caption: productionizing/lp_notifications.py :lines: 33-44 ``` 3. Notifications can be combined with different permutations of terminal phases and recipient targets. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/lp_notifications.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/lp_notifications.py :caption: productionizing/lp_notifications.py :lines: 48-70 ``` diff --git a/docs/user_guide/productionizing/reference_launch_plans.md b/docs/user_guide/productionizing/reference_launch_plans.md index b089e1838e..bce75e4681 100644 --- a/docs/user_guide/productionizing/reference_launch_plans.md +++ b/docs/user_guide/productionizing/reference_launch_plans.md @@ -16,7 +16,7 @@ Reference launch plans cannot be run locally. You must mock them out. To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/reference_launch_plan.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/reference_launch_plan.py :caption: productionizing/reference_launch_plan.py :lines: 1-36 ``` diff --git a/docs/user_guide/productionizing/reference_tasks.md b/docs/user_guide/productionizing/reference_tasks.md index 83dfe9e2dc..d91ecd4bbc 100644 --- a/docs/user_guide/productionizing/reference_tasks.md +++ b/docs/user_guide/productionizing/reference_tasks.md @@ -16,7 +16,7 @@ Reference tasks cannot be run locally. You must mock them out. To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/reference_task.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/reference_task.py :caption: productionizing/reference_task.py :lines: 1-36 ``` diff --git a/docs/user_guide/productionizing/schedules.md b/docs/user_guide/productionizing/schedules.md index f361bb7a57..1b72fd8bec 100644 --- a/docs/user_guide/productionizing/schedules.md +++ b/docs/user_guide/productionizing/schedules.md @@ -25,7 +25,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte Consider the following example workflow: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/lp_schedules.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/lp_schedules.py :caption: productionizing/lp_schedules.py :lines: 1-14 ``` @@ -39,7 +39,7 @@ The `date_formatter_wf` workflow can be scheduled using either the `CronSchedule [Cron](https://en.wikipedia.org/wiki/Cron) expression strings use this {ref}`syntax `. An incorrect cron schedule expression would lead to failure in triggering the schedule. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/lp_schedules.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/lp_schedules.py :caption: productionizing/lp_schedules.py :lines: 17-29 ``` @@ -54,7 +54,7 @@ If you prefer to use an interval rather than a cron scheduler to schedule your w Here's an example: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/lp_schedules.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/lp_schedules.py :caption: productionizing/lp_schedules.py :lines: 34-57 ``` diff --git a/docs/user_guide/productionizing/secrets.md b/docs/user_guide/productionizing/secrets.md index 9957f5cdaf..7eba15e653 100644 --- a/docs/user_guide/productionizing/secrets.md +++ b/docs/user_guide/productionizing/secrets.md @@ -62,7 +62,7 @@ Once you've defined a secret on the Flyte backend, `flytekit` exposes a class called {py:class}`~flytekit.Secret`s, which allows you to request a secret from the configured secret manager: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/use_secrets.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/use_secrets.py :caption: productionizing/use_secrets.py :lines: 1-6, 49-53 ``` @@ -76,7 +76,7 @@ In the code below we specify two variables, `SECRET_GROUP` and `SECRET_NAME`, which maps onto the `user-info` secret that we created with `kubectl` above, with a key called `user_secret`. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/use_secrets.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/use_secrets.py :caption: productionizing/use_secrets.py :lines: 66-67 ``` @@ -92,7 +92,7 @@ invoking the {py:func}`flytekit.current_context` function, as shown below. At runtime, flytekit looks inside the task pod for an environment variable or a mounted file with a predefined name/path and loads the value. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/use_secrets.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/use_secrets.py :caption: productionizing/use_secrets.py :pyobject: secret_task ``` @@ -127,14 +127,14 @@ the same secret: In this case, the secret group will be `user-info`, with three available secret keys: `user_secret`, `username`, and `password`: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/use_secrets.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/use_secrets.py :caption: productionizing/use_secrets.py :lines: 107-108 ``` The Secret structure allows passing two fields, matching the key and the group, as previously described: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/use_secrets.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/use_secrets.py :caption: productionizing/use_secrets.py :lines: 113-124 ``` @@ -155,14 +155,14 @@ In these scenarios you can specify the `mount_requirement=Secret.MountType.FILE` In the following example we force the mounting to be an environment variable: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/use_secrets.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/use_secrets.py :caption: productionizing/use_secrets.py :lines: 139-158 ``` These tasks can be used in your workflow as usual -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/use_secrets.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/use_secrets.py :caption: productionizing/use_secrets.py :pyobject: my_secret_workflow ``` @@ -172,7 +172,7 @@ These tasks can be used in your workflow as usual The simplest way to test secret accessibility is to export the secret as an environment variable. There are some helper methods available to do so: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/productionizing/productionizing/use_secrets.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/use_secrets.py :caption: productionizing/use_secrets.py :lines: 172-182 ``` diff --git a/docs/user_guide/testing/mocking_tasks.md b/docs/user_guide/testing/mocking_tasks.md index b22db0eb51..b95af69b14 100644 --- a/docs/user_guide/testing/mocking_tasks.md +++ b/docs/user_guide/testing/mocking_tasks.md @@ -6,42 +6,42 @@ A lot of the tasks that you write you can run locally, but some of them you will To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/testing/testing/mocking.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/testing/testing/mocking.py :caption: testing/mocking.py :lines: 1-6 ``` This is a generic SQL task (and is by default not hooked up to any datastore nor handled by any plugin), and must be mocked: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/testing/testing/mocking.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/testing/testing/mocking.py :caption: testing/mocking.py :lines: 10-16 ``` This is a task that can run locally: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/testing/testing/mocking.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/testing/testing/mocking.py :caption: testing/mocking.py :pyobject: t1 ``` Declare a workflow that chains these two tasks together. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/testing/testing/mocking.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/testing/testing/mocking.py :caption: testing/mocking.py :pyobject: my_wf ``` Without a mock, calling the workflow would typically raise an exception, but with the `task_mock` construct, which returns a `MagicMock` object, we can override the return value. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/testing/testing/mocking.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/testing/testing/mocking.py :caption: testing/mocking.py :pyobject: main_1 ``` There is another utility as well called `patch` which offers the same functionality, but in the traditional Python patching style, where the first argument is the `MagicMock` object. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/testing/testing/mocking.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/testing/testing/mocking.py :caption: testing/mocking.py :lines: 45-56 ```