From 9f5aa51ceac5f1d1291465c8cd0595a172eac1e7 Mon Sep 17 00:00:00 2001 From: Samhita Alla Date: Tue, 26 Oct 2021 20:57:53 +0530 Subject: [PATCH] Plugin READMEs (#723) Signed-off-by: Robert Everson --- plugins/flytekit-aws-athena/README.md | 11 +++++++++++ plugins/flytekit-aws-sagemaker/README.md | 13 +++++++++++++ plugins/flytekit-data-fsspec/README.md | 9 ++++----- plugins/flytekit-dolt/README.md | 19 +++++++++++++++++++ plugins/flytekit-greatexpectations/README.md | 4 +++- plugins/flytekit-hive/README.md | 11 +++++++++++ plugins/flytekit-k8s-pod/README.md | 13 +++++++++++++ plugins/flytekit-kf-pytorch/README.md | 13 +++++++++++++ plugins/flytekit-kf-tensorflow/README.md | 11 +++++++++++ plugins/flytekit-modin/README.md | 11 +++++++++++ plugins/flytekit-pandera/README.md | 11 +++++++++++ plugins/flytekit-papermill/README.md | 11 +++++++++++ plugins/flytekit-snowflake/README.md | 13 +++++++++++++ plugins/flytekit-spark/README.md | 13 +++++++++++++ plugins/flytekit-sqlalchemy/README.md | 11 +++++++++++ 15 files changed, 168 insertions(+), 6 deletions(-) create mode 100644 plugins/flytekit-aws-athena/README.md create mode 100644 plugins/flytekit-dolt/README.md create mode 100644 plugins/flytekit-hive/README.md create mode 100644 plugins/flytekit-k8s-pod/README.md create mode 100644 plugins/flytekit-kf-pytorch/README.md create mode 100644 plugins/flytekit-kf-tensorflow/README.md create mode 100644 plugins/flytekit-modin/README.md create mode 100644 plugins/flytekit-pandera/README.md create mode 100644 plugins/flytekit-papermill/README.md create mode 100644 plugins/flytekit-snowflake/README.md create mode 100644 plugins/flytekit-spark/README.md create mode 100644 plugins/flytekit-sqlalchemy/README.md diff --git a/plugins/flytekit-aws-athena/README.md b/plugins/flytekit-aws-athena/README.md new file mode 100644 index 0000000000..99b0a5ebc9 --- /dev/null +++ b/plugins/flytekit-aws-athena/README.md @@ -0,0 +1,11 @@ +# Flytekit AWS Athena Plugin + +Flyte backend can be connected with Athena. Once enabled, it allows you to query AWS Athena service (Presto + ANSI SQL Support) and retrieve typed schema (optionally). + +To install the plugin, run the following command: + +```bash +pip install flytekitplugins-athena +``` + +An [example](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/aws/athena/athena.html#sphx-glr-auto-integrations-aws-athena-athena-py) can be found in the documentation. diff --git a/plugins/flytekit-aws-sagemaker/README.md b/plugins/flytekit-aws-sagemaker/README.md index e69de29bb2..0974da52c5 100644 --- a/plugins/flytekit-aws-sagemaker/README.md +++ b/plugins/flytekit-aws-sagemaker/README.md @@ -0,0 +1,13 @@ +# Flytekit AWS Sagemaker Plugin + +Amazon SageMaker provides several built-in machine learning algorithms that you can use for a variety of problem types. Flyte Sagemaker plugin intends to greatly simplify using Sagemaker for training. We have tried to distill the API into a meaningful subset that makes it easier for users to adopt and run with Sagemaker. + +To install the plugin, run the following command: + +```bash +pip install flytekitplugins-awssagemaker +``` + +To install Sagemaker in the Flyte deployment's backend, go through the [prerequisites](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/aws/sagemaker_training/index.html#prerequisites). + +[Built-in sagemaker](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/aws/sagemaker_training/sagemaker_builtin_algo_training.html#sphx-glr-auto-integrations-aws-sagemaker-training-sagemaker-builtin-algo-training-py) and [custom sagemaker](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/aws/sagemaker_training/sagemaker_custom_training.html#sphx-glr-auto-integrations-aws-sagemaker-training-sagemaker-custom-training-py) training models can be found in the documentation. diff --git a/plugins/flytekit-data-fsspec/README.md b/plugins/flytekit-data-fsspec/README.md index 7d79b71bf7..e7962f1b3b 100644 --- a/plugins/flytekit-data-fsspec/README.md +++ b/plugins/flytekit-data-fsspec/README.md @@ -1,6 +1,5 @@ -fsspec data plugin for flytekit - Experimental -================================================= +# fsspec data plugin for Flytekit — Experimental -This plugin provides an implementation of the data persistence layer in flytekit, that uses fsspec. Once this plugin -is installed, it overrides all default implementation of dataplugins and provides ones supported by fsspec. this plugin -will only install the fsspec core. To actually install all fsspec plugins, please follow fsspec documentation. +This plugin provides an implementation of the data persistence layer in Flytekit that uses fsspec. Once this plugin +is installed, it overrides all default implementations of the data plugins and provides the ones supported by fsspec. This plugin +will only install the fsspec core. To install all fsspec plugins, please follow the [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/). diff --git a/plugins/flytekit-dolt/README.md b/plugins/flytekit-dolt/README.md new file mode 100644 index 0000000000..6aac2c130c --- /dev/null +++ b/plugins/flytekit-dolt/README.md @@ -0,0 +1,19 @@ +# Flytekit Dolt Plugin + +The DoltTable plugin is a wrapper that uses [Dolt](https://github.com/dolthub/dolt) to move data between pandas.DataFrame’s at execution time and database tables at rest. + +The Dolt plugin and Dolt command-line tool can be installed as follows: + +```bash +pip install flytekitplugins.dolt +sudo bash -c 'curl -L https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash' +``` + +Dolt requires a user configuration to run init: + +``` +dolt config --global --add user.email +dolt config --global --add user.name +``` + +All the [examples](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/flytekit_plugins/dolt/index.html) can be found in the documentation. diff --git a/plugins/flytekit-greatexpectations/README.md b/plugins/flytekit-greatexpectations/README.md index 2ba6476155..f23ba44226 100644 --- a/plugins/flytekit-greatexpectations/README.md +++ b/plugins/flytekit-greatexpectations/README.md @@ -1,6 +1,6 @@ # Flytekit Great Expectations Plugin -Great Expectations' helps enforce data quality. The plugin supports the usage of Great Expectations as task and type. +Great Expectations helps enforce data quality. The plugin supports the usage of Great Expectations as task and type. To install the plugin, run the following command: @@ -72,3 +72,5 @@ def simple_task( def simple_wf(directory: str = "my_assets") -> str: return simple_task(directory=directory) ``` + +[More examples](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/flytekit_plugins/greatexpectations/index.html) can be found in the documentation. diff --git a/plugins/flytekit-hive/README.md b/plugins/flytekit-hive/README.md new file mode 100644 index 0000000000..d63d41bede --- /dev/null +++ b/plugins/flytekit-hive/README.md @@ -0,0 +1,11 @@ +# Flytekit Hive Plugin + +Flyte backend can be connected with various Hive services. Once enabled, it allows you to query a Hive service (e.g., Qubole) and retrieve typed schema (optionally). + +To install the plugin, run the following command: + +```bash +pip install flytekitplugins-hive +``` + +An [example](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/external_services/hive/hive.html#sphx-glr-auto-integrations-external-services-hive-hive-py) can be found in the documentation. diff --git a/plugins/flytekit-k8s-pod/README.md b/plugins/flytekit-k8s-pod/README.md new file mode 100644 index 0000000000..c74c514231 --- /dev/null +++ b/plugins/flytekit-k8s-pod/README.md @@ -0,0 +1,13 @@ +# Flytekit Kubernetes Pod Plugin + +By default, Flyte tasks decorated with `@task` are essentially single functions that are loaded in one container. But often, there is a need to run a job with more than one container. + +In this case, a regular task is not enough. Hence, Flyte provides a Kubernetes pod abstraction to execute multiple containers, which can be accomplished using Pod's `task_config`. The `task_config` can be leveraged to fully customize the pod spec used to run the task. + +To install the plugin, run the following command: + +```bash +pip install flytekitplugins-pod +``` + +An [example](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/pod/pod.html) can be found in the documentation. diff --git a/plugins/flytekit-kf-pytorch/README.md b/plugins/flytekit-kf-pytorch/README.md new file mode 100644 index 0000000000..280fe687b6 --- /dev/null +++ b/plugins/flytekit-kf-pytorch/README.md @@ -0,0 +1,13 @@ +# Flytekit Kubeflow PyTorch Plugin + +This plugin uses the Kubeflow PyTorch Operator and provides an extremely simplified interface for executing distributed training using various PyTorch backends. + +To install the plugin, run the following command: + +```bash +pip install flytekitplugins-kfpytorch +``` + +To set up PyTorch operator in the Flyte deployment's backend, follow the [PyTorch Operator Setup](https://docs.flyte.org/en/latest/deployment/plugin_setup/pytorch_operator.html) guide. + +An [example](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/kfpytorch/pytorch_mnist.html#sphx-glr-auto-integrations-kubernetes-kfpytorch-pytorch-mnist-py) showcasing PyTorch operator can be found in the documentation. diff --git a/plugins/flytekit-kf-tensorflow/README.md b/plugins/flytekit-kf-tensorflow/README.md new file mode 100644 index 0000000000..9e4c26fa70 --- /dev/null +++ b/plugins/flytekit-kf-tensorflow/README.md @@ -0,0 +1,11 @@ +# Flytekit Kubeflow TensorFlow Plugin + +This plugin uses the Kubeflow TensorFlow Operator and provides an extremely simplified interface for executing distributed training using various TensorFlow backends. + +To install the plugin, run the following command: + +```bash +pip install flytekitplugins-kftensorflow +``` + +_Example coming soon!_ diff --git a/plugins/flytekit-modin/README.md b/plugins/flytekit-modin/README.md new file mode 100644 index 0000000000..a1f93989ad --- /dev/null +++ b/plugins/flytekit-modin/README.md @@ -0,0 +1,11 @@ +# Flytekit Modin Plugin + +Modin is an emerging drop-in replacement or rather extension of Pandas. This plugin could be helpful to use Modin as a data type. + +To install the plugin, run the following command: + +```bash +pip install flytekitplugins-modin +``` + +_Example coming soon!_ diff --git a/plugins/flytekit-pandera/README.md b/plugins/flytekit-pandera/README.md new file mode 100644 index 0000000000..63fc1d338c --- /dev/null +++ b/plugins/flytekit-pandera/README.md @@ -0,0 +1,11 @@ +# Flytekit Pandera Plugin + +Flytekit python natively supports many data types, including a FlyteSchema type for type-annotating Pandas DataFrames. The Flytekit Pandera plugin provides an alternative for defining DataFrame schemas by integrating with Pandera, which is a runtime data validation tool for Pandas DataFrames. + +To install the plugin, run the following command: + +```bash +pip install flytekitplugins-pandera +``` + +All [examples](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/flytekit_plugins/pandera_examples/index.html) can be found in the documentation. diff --git a/plugins/flytekit-papermill/README.md b/plugins/flytekit-papermill/README.md new file mode 100644 index 0000000000..fc990a1ef1 --- /dev/null +++ b/plugins/flytekit-papermill/README.md @@ -0,0 +1,11 @@ +# Flytekit Papermill Plugin + +It is possible to run a Jupyter notebook as a Flyte task using Papermill. Papermill executes the notebook as a whole, so before using this plugin, it is essential to construct your notebook as recommended by Papermill. + +To install the plugin, run the following command: + +```bash +pip install flytekitplugins-papermill +``` + +An [example](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/flytekit_plugins/papermilltasks/simple.html#sphx-glr-auto-integrations-flytekit-plugins-papermilltasks-simple-py) can be found in the documentation. We also have a [tutorial](https://docs.flyte.org/projects/cookbook/en/latest/auto/case_studies/feature_engineering/eda/index.html) showcasing the various ways in which you can leverage the Papermill plugin. diff --git a/plugins/flytekit-snowflake/README.md b/plugins/flytekit-snowflake/README.md new file mode 100644 index 0000000000..d31a4f08a5 --- /dev/null +++ b/plugins/flytekit-snowflake/README.md @@ -0,0 +1,13 @@ +# Flytekit Snowflake Plugin + +Snowflake enables us to build data-intensive applications without operational burden. Flyte backend can be connected with the Snowflake service. Once enabled, it can allow you to query a Snowflake service. + +To install the plugin, run the following command: + +```bash +pip install flytekitplugins-snowflake +``` + +To configure Snowflake in the Flyte deployment's backend, follow the [configuration guide](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/external_services/snowflake/index.html#configuring-the-backend-to-get-snowflake-working). + +An [example](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/external_services/snowflake/snowflake.html#sphx-glr-auto-integrations-external-services-snowflake-snowflake-py) showcasing Snowflake service can be found in the documentation. diff --git a/plugins/flytekit-spark/README.md b/plugins/flytekit-spark/README.md new file mode 100644 index 0000000000..f067c84c48 --- /dev/null +++ b/plugins/flytekit-spark/README.md @@ -0,0 +1,13 @@ +# Flytekit Spark Plugin + +Flyte can execute Spark jobs natively on a Kubernetes Cluster, which manages a virtual cluster’s lifecycle, spin-up, and tear down. It leverages the open-sourced Spark On K8s Operator and can be enabled without signing up for any service. This is like running a transient spark cluster — a type of cluster spun up for a specific Spark job and torn down after completion. + +To install the plugin, run the following command: + +```bash +pip install flytekitplugins-spark +``` + +To configure Spark in the Flyte deployment's backend, follow [Step 1](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_spark/index.html#step-1-deploy-spark-plugin-in-the-flyte-backend), [2](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_spark/index.html#step-2-environment-setup), and [3](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_spark/index.html#step-3-optionally-setup-visibility). + +All [examples](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_spark/index.html) showcasing execution of Spark jobs using the plugin can be found in the documentation. diff --git a/plugins/flytekit-sqlalchemy/README.md b/plugins/flytekit-sqlalchemy/README.md new file mode 100644 index 0000000000..8ae1094210 --- /dev/null +++ b/plugins/flytekit-sqlalchemy/README.md @@ -0,0 +1,11 @@ +# Flytekit SQLAlchemy Plugin + +SQLAlchemy is the Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL. Flyte provides an easy-to-use interface to utilize SQLAlchemy to connect to various SQL Databases. + +To install the plugin, run the following command: + +```bash +pip install flytekitplugins-sqlalchemy +``` + +An [example](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/flytekit_plugins/sql/sql-alchemy.html) can be found in the documentation.