Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add single task execution docs #354

Merged
merged 4 commits into from
Jun 22, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions rsts/user/features/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,4 @@ Flyte Features
launchplans
task_cache
roles
single_task_execution
90 changes: 90 additions & 0 deletions rsts/user/features/single_task_execution.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
.. _features-singletaskexec:

Single Task Excution
####################

What are single task executions?
================================

Tasks are the most atomic unit of execution in Flyte. Although workflows are traditionally composed of multiple tasks with dependencies
defined by shared inputs and outputs it can be helpful to execute a single task during the process of iterating on its definition.
It can be tedious to write a new workflow definition every time you want to excecute a single task under development but single task
executions can be used to easily iterate on task logic.

Launch a single task
====================

After you've built an image with your updated task code, create an execution using launch:

.. code-block:: python

@inputs(plant=Types.String)
@outputs(out=Types.String)
@python_task
def my_task(wf_params, plant, out)
...


my_single_task_execution = my_task.launch(project="my_flyte_projext", domain="development", inputs={'plant': 'ficus'})
print("Created {}".format(my_single_task_execution.id))

Just like workflow executions, you can optionally pass a user-defined name, labels, annotations, and/or notifications when launching a single task.

The type of ``my_single_task_execution`` is `SdkWorkflowExecution <https://github.com/lyft/flytekit/blob/1926b1285591ae941d7fc9bd4c2e4391c5c1b21b/flytekit/common/workflow_execution.py#L14>`_
and has the full set of methods and functionality available for conventional WorkflowExecutions.


Fetch and launch a single task
==============================

Single task executions aren't limited to just tasks you've defined in your code. You can reference previously registered tasks and launch a single task execution like so:

.. code-block:: python

from flytekit.common.tasks import task as _task

my_task = _task.SdkTask.fetch("my_flyte_project", "production", "workflows.my_task", "abc123") # project, domain, name, version

my_task_exec = my_task.launch(project="my_other_project", domain="development", inputs={'plant': 'philodendron'})
my_task_exec.wait_for_completion()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can Admin withstand potentially tens of thousands of these polling?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think we should move to websockets?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following up offline

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why websockets? These should be converted to long poll if needed. But we don’t need it at all I feel. This is a usecase only for testing



Launch a single task from the commandline
=========================================

Previously registered tasks can also be launched from the command-line using :ref:`flyte-cli <features-flytecli>`

.. code-block:: console

$ flyte-cli -h example.com -p my_flyte_project -d development launch-task \
-u tsk:my_flyte_project:production:my_complicated_task:abc123 -- an_input=hi \
other_input=123 more_input=qwerty


Monitoring single task executions in the Flyte console
======================================================

Single task executions don't yet have native support in the Flyte console but they're accessible using the same URLs as ordinary workflow executions.

For example, for a console hosted example.com you can visit ``example.com/console/projects/<my_project>/domains/<my_domain>/executions/<execution_name>`` to track the progress of
your execution. Log links and status changes will be available as your execution progresses.


Registering and launching a single task
=======================================

A certain category of tasks don't rely on custom containers with registered images to run. Therefore, you may find it convenient to use
``register_and_launch`` on a task definition to immediately launch a single task execution, like so:

.. code-block:: python

containerless_task = SdkPrestoTask(
task_inputs=inputs(ds=Types.String, count=Types.Integer, rg=Types.String),
statement="SELECT * FROM flyte.widgets WHERE ds = '{{ .Inputs.ds}}' LIMIT '{{ .Inputs.count}}'",
output_schema=Types.Schema([("a", Types.String), ("b", Types.Integer)]),
routing_group="{{ .Inputs.rg }}",
)

my_single_task_execution = containerless_task.register_and_launch(project="my_flyte_projext", domain="development",
inputs={'ds': '2020-02-29', 'count': 10, 'rg': 'my_routing_group'})