cookbook

Back to Snacks Menu

📚 Flytekit Cookbook

This is a collection of short "how to" articles demonstrating the various capabilities and idiomatic usage of Flytekit. Note this currently does not include articles/tid-bits on how to use the Flyte platform at large, though in the future we may expand it to that.

🍛 Contents

1. Write and Execute a Task
2. Create a workflow from Tasks
3. Launch Plans
4. Multiple Schedules for a Workflow
5. Executing pre-created tasks & workflows
6. Interact with Past Workflow / Task Executions
7. Working with Types
8. Composing a Workflow from shared tasks and workflows
9. Map/Array Tasks
10. Compose a Workflow from other workflows
11. Dynamically Generate a Workflow at Runtime
12. Dynamic Tasks
13. Tasks without flytekit or Using arbitrary containers
14. Jupyter notebook as a task
15. Different container per task
16. Customize Offloaded Data Location

Each example is organized into a separate folder at this layer, and each has the Readme file linked to in the Contents, as well as supporting .py and .ipynb files, some of which contain information in a lot more depth.

Setup

Careful care has been taken to ensure that all the .py files contained in this guide are fully usable and compilable into Flyte entities. Readers that want to run these workflows for themselves should

1. Follow instructions in the main Flyte documentation to set up a local Flyte cluster. All the commands in this book assume that you are using Docker Desktop. If you are using minikube or another K8s deployment, the commands will need to be modified.
2. Please also ensure that you have a Python virtual environment installed and activated, and have pip installed the requirements.
3. Use flyte-cli to create a project named flytesnacks (the name the Makefile is set up to use).

Using the Cookbook

The examples written in this cookbook are meant to used in two ways, as a reference to read, and a live project to iterate and test on your local (or EKS/GCP/etc K8s cluster). To make some simple changes to the code in this cookbook to just try things out, or to just see it run on your local cluster, the iteration cycle should be

1. Make your changes and commit (the steps below require a clean git tree).
2. Activate your Python virtual environment with the requirements installed.
3. make docker_build This will build the image tagged with just flytecookbook:<sha>, no registry will be prefixed.
4. make register_sandbox This will register the Flyte entities in this cookbook against your local installation.

If you are just iterating locally, there is no need to push your Docker image. For Docker for Desktop at least, locally built images will be available for use in its K8s cluster.

If you would like to later push your image to a registry (Dockerhub, ECR, etc.), you can

REGISTRY=docker.io/corp make docker_push