Tracker branch for Milestone v0.2.0

.github/workflows/tests.yml

@@ -23,4 +23,3 @@ jobs:
         go-version: ${{ matrix.go-version }}
     - name: Run end-to-end tests
       run: make end2end
-

CHANGELOG/CHANGELOG-v0.2.0.md

@@ -0,0 +1,32 @@
+#Flyte V0.2.0
+
+Changes since v0.1.0
+
+- Go Mod support
+- Go 1.13 upgrade
+- OAuth Support (CLI, UI, Admin)
+- Standardized pull request templates & OSS process standardization
+- PluginMachinery and flexible statemachine for Arbitrary plugins
+- Improved Engine performance, reliability and write requirements to etcD (scalability)
+- Backoff support
+- Timeouts
+- Dynamic workflow support
+- Resource manager per project/domain
+- RawOutputDirectories created in FlytePropeller
+- Improve visibility and observability
+- User/System error differentiation
+- Optional interruptible tasks (lets use spot instances, reduce cost)
+- Caps on queue time for workflows
+- Multi cluster improvements
+- Visibility into execution cluster for the execution
+- Add descriptions to Identifiers (Projects, Workflows etc)
+- Configuration improvements throughout - You can configure Flyte for your environment
+- Multi Cluster placement manager
+
+Flytekit (SDK improvements)
+- Subworkflow support
+- Collaboration of Workflows support (Workflow fetch)
+- Three-legged OAuth and basic authentication support
+- Fetch the latest version of a task instead of having to specify a version
+
+And ofcourse Bug fixes throughout!

CHANGELOG/README.md

@@ -0,0 +1,3 @@
+Change Logs
+
+[Changes between v0.1.0 -> v0.2.0](CHANGELOG-v0.2.0.md)

README.md

@@ -1,21 +1,85 @@
-# Flyte
+![Flyte Logo](images/flyte_lockup_gradient_on_light.png "Flyte Logo")
 
-Flyte is an open source, K8s-native extensible orchestration engine that manages the core machine learning pipelines at Lyft: ETAs, pricing, incentives, mapping, vision, and more. 
+[![Current Release](https://img.shields.io/github/release/lyft/flyte.svg)](https://github.com/lyft/flyte/releases/latest)
+[![Build Status](https://travis-ci.org/lyft/flyte.svg?branch=master)](https://travis-ci.org/lyft/flyte)
+[![License](https://img.shields.io/badge/LICENSE-Apache2.0-ff69b4.svg)](http://www.apache.org/licenses/LICENSE-2.0.html)
+![Commit activity](https://img.shields.io/github/commit-activity/w/lyft/flyte.svg?style=plastic)
+![Commit since last release](https://img.shields.io/github/commits-since/lyft/flyte/latest.svg?style=plastic)
+![GitHub milestones Completed](https://img.shields.io/github/milestones/closed/lyft/flyte?style=plastic)
+![GitHub next milestone percentage](https://img.shields.io/github/milestones/progress-percent/lyft/flyte/3?style=plastic)
+![Twitter Follow](https://img.shields.io/twitter/follow/flyteorg?label=Follow&style=social)
+[![Slack Status](https://img.shields.io/badge/slack-join_chat-white.svg?logo=slack&style=social)](https://docs.google.com/forms/d/e/1FAIpQLSf8bNuyhy7rkm77cOXPHIzCm3ApfL7Tdo7NUs6Ej2NOGQ1PYw/viewform?pli=1)
 
-# Community
-Home:  https://flyte.org
+Flyte is a container-native, type-safe workflow and pipelines platform optimized for large scale processing and machine learning written in Golang. Workflows can be written in any language, with out of the box support for Python. 
 
-Docs:  https://lyft.github.io/flyte
+# Homepage
+https://flyte.org
+Docs: https://lyft.github.io/flyte
 
-Slack:  [https://flyte-org.slack.com](https://docs.google.com/forms/d/e/1FAIpQLSf8bNuyhy7rkm77cOXPHIzCm3ApfL7Tdo7NUs6Ej2NOGQ1PYw/viewform)
+# Introduction
+Flyte is a fabric that connects disparate computation backends using a type safe data dependency graph. It records all changes to a pipeline, making it possible to rewind time. It also stores
+a history of all executions and provides an intuitive UI, CLI and REST/gRPC API to interact with the computation.
 
-Twitter: https://twitter.com/flyteorg
+Flyte is more than a workflow engine, it provides workflows as a core concepts, but it also provides a single unit of execution - tasks, as a top level concept. Multiple tasks arranged in a data
+producer-consumer order creates a workflow. Flyte workflows are pure specification and can be created using any language. Every task can also by any language. We do provide first class support for
+python, making it perfect for modern Machine Learning and Data processing pipelines.
 
-# Repos 
+# Features
+ - Used at Scale in production by 500+ users at Lyft with more than 400k workflows a month and more than 20+ million container executions per month
+ - Centralized Inventory of Tasks, Workflows and Executions
+ - gRPC / REST interface to define and executes tasks and workflows
+ - Type safe construction of pipelines, each task has an interface which is characterized by its input and outputs. Thus illegal construction of pipelines fails during declaration rather than at
+   runtime
+ - Types that help in creating machine learning and data processing pipelines like - Blobs (images, arbitrary files), Directories, Schema (columnar structured data), collections, maps etc
+ - Memoization and Lineage tracking
+ - Workflows features
+  * Multiple Schedules for every workflow
+  * Parallel step execution
+  * Extensible Backend to add customized plugin experiences
+  * Arbitrary container execution
+  * Branching
+  * Inline Subworkflows (a workflow can be embeded within one node of the top level workflow)
+  * Distributed Remote Child workflows (a remote workflow can be triggered and statically verified at compile time)
+  * Array Tasks (map some function over a large dataset, controlled execution of 1000's of containers)
+  * Dynamic Workflow creation and execution - with runtime type safety
+  * Container side plugins with first class support in python
+ - Maintain an inventory of tasks and workflows
+ - Record history of all executions and executions (as long as they follow convention) are completely repeatable
+ - Multi Cloud support (AWS, GCP and others)
+ - Extensible core
+ - Modularized
+ - Automated notifications to Slack, Email, Pagerduty
+ - Deep observability
+ - Multi K8s cluster support
+ - Comes with many system supported out of the box on K8s like Spark etc.
+ - Snappy Console
+ - Python CLI
+ - Written in Golang and optimized for performance
 
+## Coming Soon
+ - Single Task Execution support
+ - Reactive pipelines
+ - More integrations
+
+
+# Available Plugins
+ - Containers
+ - K8s Pods
+ - AWS Batch Arrays
+ - K8s Pod arrays
+ - K8s Spark (native pyspark and java/scala)
+ - Qubole Hive
+ - Presto Queries
+
+# Current Usage 
+- Lyft Rideshare
+- Lyft L5 autonomous
+- Juno
+
+# Component Repos 
 Repo | Language | Purpose
 --- | --- | ---
-[flyte](https://github.com/lyft/flyte) | RST | home, documentation, issues
+[flyte](https://github.com/lyft/flyte) | Kustomize,RST | deployment, documentation, issues
 [flyteidl](https://github.com/lyft/flyteidl) | Protobuf | interface definitions
 [flytepropeller](https://github.com/lyft/flytepropeller) | Go | execution engine
 [flyteadmin](https://github.com/lyft/flyteadmin) | Go | control plane

deployment/sandbox/flyte_generated.yaml

@@ -883,6 +883,8 @@ spec:
         prometheus.io/scrape: "true"
       labels:
         app: datacatalog
+        app.kubernetes.io/name: datacatalog
+        app.kubernetes.io/version: 0.2.1
     spec:
       containers:
       - command:
@@ -891,7 +893,7 @@ spec:
         - --config
         - /etc/datacatalog/config/datacatalog_config.yaml
         - serve
-        image: docker.io/lyft/datacatalog:v0.1.2
+        image: docker.io/lyft/datacatalog:v0.2.1
         imagePullPolicy: IfNotPresent
         name: datacatalog
         ports:
@@ -923,7 +925,7 @@ spec:
         - /etc/datacatalog/config/datacatalog_config.yaml
         - migrate
         - run
-        image: docker.io/lyft/datacatalog:v0.1.2
+        image: docker.io/lyft/datacatalog:v0.2.1
         imagePullPolicy: IfNotPresent
         name: run-migrations
         volumeMounts:
@@ -957,6 +959,8 @@ spec:
         prometheus.io/scrape: "true"
       labels:
         app: flyteadmin
+        app.kubernetes.io/name: flyteadmin
+        app.kubernetes.io/version: 0.2.4
     spec:
       containers:
       - command:
@@ -965,7 +969,7 @@ spec:
         - --config
         - /etc/flyte/config/flyteadmin_config.yaml
         - serve
-        image: docker.io/lyft/flyteadmin:v0.2.3
+        image: docker.io/lyft/flyteadmin:v0.2.4
         imagePullPolicy: IfNotPresent
         name: flyteadmin
         ports:
@@ -1016,7 +1020,7 @@ spec:
         - /etc/flyte/config/flyteadmin_config.yaml
         - migrate
         - run
-        image: docker.io/lyft/flyteadmin:v0.2.3
+        image: docker.io/lyft/flyteadmin:v0.2.4
         imagePullPolicy: IfNotPresent
         name: run-migrations
         volumeMounts:
@@ -1032,7 +1036,7 @@ spec:
         - flytesnacks
         - flytetester
         - flyteexamples
-        image: docker.io/lyft/flyteadmin:v0.2.3
+        image: docker.io/lyft/flyteadmin:v0.2.4
         imagePullPolicy: IfNotPresent
         name: seed-projects
         volumeMounts:
@@ -1045,7 +1049,7 @@ spec:
         - /etc/flyte/config/flyteadmin_config.yaml
         - clusterresource
         - sync
-        image: docker.io/lyft/flyteadmin:v0.2.3
+        image: docker.io/lyft/flyteadmin:v0.2.4
         imagePullPolicy: IfNotPresent
         name: sync-cluster-resources
         volumeMounts:
@@ -1080,12 +1084,14 @@ spec:
     metadata:
       labels:
         app: flyteconsole
+        app.kubernetes.io/name: flyteconsole
+        app.kubernetes.io/version: 0.4.0
     spec:
       containers:
       - envFrom:
         - configMapRef:
             name: flyte-console-config
-        image: docker.io/lyft/flyteconsole:v0.3.2
+        image: docker.io/lyft/flyteconsole:v0.4.0
         name: flyteconsole
         ports:
         - containerPort: 8080
@@ -1121,7 +1127,7 @@ spec:
       labels:
         app: flytepropeller
         app.kubernetes.io/name: flytepropeller
-        app.kubernetes.io/version: 0.2.13
+        app.kubernetes.io/version: 0.2.20
     spec:
       containers:
       - args:
@@ -1136,7 +1142,7 @@ spec:
           valueFrom:
             fieldRef:
               fieldPath: metadata.name
-        image: docker.io/lyft/flytepropeller:v0.2.13
+        image: docker.io/lyft/flytepropeller:v0.2.20
         imagePullPolicy: IfNotPresent
         name: flytepropeller
         ports:
@@ -1441,7 +1447,7 @@ spec:
             - /etc/flyte/config/flyteadmin_config.yaml
             - clusterresource
             - sync
-            image: docker.io/lyft/flyteadmin:v0.2.1
+            image: docker.io/lyft/flyteadmin:v0.2.4
             imagePullPolicy: IfNotPresent
             name: sync-cluster-resources
             volumeMounts:

deployment/test/flyte_generated.yaml

@@ -626,6 +626,8 @@ spec:
         prometheus.io/scrape: "true"
       labels:
         app: flyteadmin
+        app.kubernetes.io/name: flyteadmin
+        app.kubernetes.io/version: 0.2.4
     spec:
       containers:
       - command:
@@ -634,7 +636,7 @@ spec:
         - --config
         - /etc/flyte/config/flyteadmin_config.yaml
         - serve
-        image: docker.io/lyft/flyteadmin:v0.2.3
+        image: docker.io/lyft/flyteadmin:v0.2.4
         imagePullPolicy: IfNotPresent
         name: flyteadmin
         ports:
@@ -685,7 +687,7 @@ spec:
         - /etc/flyte/config/flyteadmin_config.yaml
         - migrate
         - run
-        image: docker.io/lyft/flyteadmin:v0.2.3
+        image: docker.io/lyft/flyteadmin:v0.2.4
         imagePullPolicy: IfNotPresent
         name: run-migrations
         volumeMounts:
@@ -701,7 +703,7 @@ spec:
         - flytesnacks
         - flytetester
         - flyteexamples
-        image: docker.io/lyft/flyteadmin:v0.2.3
+        image: docker.io/lyft/flyteadmin:v0.2.4
         imagePullPolicy: IfNotPresent
         name: seed-projects
         volumeMounts:
@@ -714,7 +716,7 @@ spec:
         - /etc/flyte/config/flyteadmin_config.yaml
         - clusterresource
         - sync
-        image: docker.io/lyft/flyteadmin:v0.2.3
+        image: docker.io/lyft/flyteadmin:v0.2.4
         imagePullPolicy: IfNotPresent
         name: sync-cluster-resources
         volumeMounts:
@@ -753,7 +755,7 @@ spec:
       labels:
         app: flytepropeller
         app.kubernetes.io/name: flytepropeller
-        app.kubernetes.io/version: 0.2.13
+        app.kubernetes.io/version: 0.2.20
     spec:
       containers:
       - args:
@@ -766,7 +768,7 @@ spec:
           valueFrom:
             fieldRef:
               fieldPath: metadata.name
-        image: docker.io/lyft/flytepropeller:v0.2.13
+        image: docker.io/lyft/flytepropeller:v0.2.20
         imagePullPolicy: IfNotPresent
         name: flytepropeller
         ports:
@@ -1012,7 +1014,7 @@ spec:
             - /etc/flyte/config/flyteadmin_config.yaml
             - clusterresource
             - sync
-            image: docker.io/lyft/flyteadmin:v0.2.1
+            image: docker.io/lyft/flyteadmin:v0.2.4
             imagePullPolicy: IfNotPresent
             name: sync-cluster-resources
             volumeMounts:

end2end/tests/endtoend.yaml

@@ -11,7 +11,7 @@ spec:
     command:
     - bash
     - -c
-    image: docker.io/lyft/flytetester:v0.1.2_flytekitv0.6.0b1
+    image: docker.io/lyft/flytetester:v0.1.4_flytekitv0.6.0
     imagePullPolicy: IfNotPresent
     name: flytetester
     resources:

kustomize/base/admindeployment/deployment.yaml

@@ -15,6 +15,8 @@ spec:
     metadata:
       labels:
         app: flyteadmin
+        app.kubernetes.io/name: flyteadmin
+        app.kubernetes.io/version: 0.2.4
       annotations:
         prometheus.io/scrape: "true"
         prometheus.io/port: "10254"
@@ -29,15 +31,15 @@ spec:
           name: flyte-admin-config
       initContainers:
       - name: run-migrations
-        image: docker.io/lyft/flyteadmin:v0.2.3
+        image: docker.io/lyft/flyteadmin:v0.2.4
         imagePullPolicy: IfNotPresent
         command: ["flyteadmin", "--logtostderr", "--config", "/etc/flyte/config/flyteadmin_config.yaml", "migrate", "run"]
         volumeMounts:
         - name: config-volume
           mountPath: /etc/flyte/config
       containers:
       - name: flyteadmin
-        image: docker.io/lyft/flyteadmin:v0.2.3
+        image: docker.io/lyft/flyteadmin:v0.2.4
         imagePullPolicy: IfNotPresent
         command: ["flyteadmin", "--logtostderr", "--config", "/etc/flyte/config/flyteadmin_config.yaml", "serve"]
         ports:

kustomize/base/console/deployment.yaml

@@ -15,13 +15,15 @@ spec:
     metadata:
       labels:
         app: flyteconsole
+        app.kubernetes.io/name: flyteconsole
+        app.kubernetes.io/version: 0.4.0
     spec:
       volumes:
       - name: shared-data
         emptyDir: {}
       containers:
       - name: flyteconsole
-        image: docker.io/lyft/flyteconsole:v0.3.2
+        image: docker.io/lyft/flyteconsole:v0.4.0
         # args: []
         ports:
         - containerPort: 8080