execution environments (#164)

* added ExecutionEnvironment protos Signed-off-by: Daniel Rammer <[email protected]> * updated naming Signed-off-by: Daniel Rammer <[email protected]> * Agents protocol refactor (#4874) Signed-off-by: Haytham Abuelfutuh <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Haytham Abuelfutuh <[email protected]> * bringing ExecutionEnvAssignments from flytekit to propeller ExecutionConfig Signed-off-by: Daniel Rammer <[email protected]> * fasttask environment extant working e2e Signed-off-by: Daniel Rammer <[email protected]> * using InMemoryEnvironmentBuilder for fasttask Signed-off-by: Daniel Rammer <[email protected]> * removed some complexity from ExecutionEnvClient interface Signed-off-by: Daniel Rammer <[email protected]> * copied fasttask repo here Signed-off-by: Daniel Rammer <[email protected]> * building fasttask plugin into propeller Signed-off-by: Daniel Rammer <[email protected]> * moved ExecutionEnvClient Signed-off-by: Daniel Rammer <[email protected]> * reverting Makefile skipping flyteconsole change Signed-off-by: Daniel Rammer <[email protected]> * docs Signed-off-by: Daniel Rammer <[email protected]> * fixing flyteplugins import Signed-off-by: Daniel Rammer <[email protected]> * fixed unit tests Signed-off-by: Daniel Rammer <[email protected]> * fixed linter Signed-off-by: Daniel Rammer <[email protected]> * fixed codespell issues Signed-off-by: Daniel Rammer <[email protected]> * codespell attempt 2 Signed-off-by: Daniel Rammer <[email protected]> * copying fasttask to single binary Signed-off-by: Daniel Rammer <[email protected]> --------- Signed-off-by: Daniel Rammer <[email protected]> Signed-off-by: Haytham Abuelfutuh <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: Haytham Abuelfutuh <[email protected]>
flyteorg · Apr 12, 2024 · f3c0f19 · f3c0f19
1 parent 090bd75
commit f3c0f19
Show file tree

Hide file tree

Showing 129 changed files with 8,819 additions and 1,102 deletions.
diff --git a/.codespellrc b/.codespellrc
@@ -3,4 +3,4 @@ skip = .git,*.pdf,*.svg,go.sum,go.mod,*requirements.txt,gen
 # some strings with unicodes, constructs like [o]utput
 ignore-regex = ".*\\0[0-9][0-9].*"|json:"[^"]*"|\b[a-z]*\[[a-z]\][a-z]*\b
 # some ad-hoc variable names etc
-ignore-words-list = te,nd,querys,ser
+ignore-words-list = crate,te,nd,querys,ser
diff --git a/Dockerfile b/Dockerfile
@@ -12,6 +12,7 @@ WORKDIR /flyteorg/build
 
 COPY cacheservice cacheservice
 COPY datacatalog datacatalog
+COPY fasttask fasttask
 COPY flyteadmin flyteadmin
 COPY flytecopilot flytecopilot
 COPY flyteidl flyteidl

diff --git a/Dockerfile.flytepropeller b/Dockerfile.flytepropeller
@@ -15,6 +15,7 @@ WORKDIR /go/src/github.com/flyteorg/flytepropeller
 
 COPY cacheservice ../cacheservice
 COPY datacatalog ../datacatalog
+COPY fasttask ../fasttask
 COPY flyteadmin ../flyteadmin
 COPY flytecopilot ../flytecopilot
 COPY flyteidl ../flyteidl

diff --git a/Dockerfile.sandbox-lite b/Dockerfile.sandbox-lite
@@ -17,6 +17,7 @@ WORKDIR /app/flyte
 
 COPY cacheservice cacheservice
 COPY datacatalog datacatalog
+COPY fasttask fastask
 COPY flyteadmin flyteadmin
 COPY flytecopilot flytecopilot
 COPY flyteidl flyteidl

diff --git a/fasttask/README.md b/fasttask/README.md
@@ -0,0 +1,17 @@
+# fast task
+FastTask is a Flyte plugin to execute tasks quickly using a persistent, external worker; reducing the overhead of cold-starting a k8s Pod as an execution environment. This enables Flyte to execute short tasks in parallel, without the overhead of starting a new Pod for each task; instead sharing a pool of workers across multiple tasks and workflow executions.
+
+## development setup
+    # get flytekit in terminal
+    source ~/Development/flytekit/.venv-union/bin/activate
+
+    # create temporary python dir for fast registration
+    mkdir /tmp/fasttask-test
+    export PYTHONPATH=$PYTHONPATH:/tmp/fasttask-test
+
+    # allow flytekit to connect to minio cluster
+    export FLYTE_AWS_ACCESS_KEY_ID=minio
+    export FLYTE_AWS_SECRET_ACCESS_KEY=miniostorage
+    export FLYTE_AWS_ENDPOINT=http://localhost:30084
+## build fast task worker image
+    docker build -t hamersaw/fasttask:<tag> -f Dockerfile .
diff --git a/fasttask/buf.gen.yaml b/fasttask/buf.gen.yaml
@@ -0,0 +1,10 @@
+version: v1
+plugins:
+  - plugin: buf.build/protocolbuffers/go:v1.28.1
+    out: plugin
+  - plugin: buf.build/grpc/go:v1.3.0
+    out: plugin
+  - plugin: buf.build/community/neoeinstein-prost:v0.2.3
+    out: worker/src/pb
+  - plugin: buf.build/community/neoeinstein-tonic:v0.3.0
+    out: worker/src/pb
diff --git a/fasttask/buf.work.yaml b/fasttask/buf.work.yaml
@@ -0,0 +1,3 @@
+version: v1
+directories:
+  - protos
diff --git a/fasttask/generate_protos.sh b/fasttask/generate_protos.sh
@@ -0,0 +1,2 @@
+#!/bin/bash
+docker run --rm -u $(id -u):$(id -g) -e "BUF_CACHE_DIR=/tmp/cache" --volume "$(pwd):/workspace" --workdir /workspace bufbuild/buf generate
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		#!/bin/bash
		docker run --rm -u $(id -u):$(id -g) -e "BUF_CACHE_DIR=/tmp/cache" --volume "$(pwd):/workspace" --workdir /workspace bufbuild/buf generate