Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restructured Kubeflow Pipelines docs #3737

Merged
merged 35 commits into from
Jun 20, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
810f813
Restructured Kubeflow Pipelines docs
hbelmiro May 17, 2024
85598ce
Fixed broken links
hbelmiro May 21, 2024
77abd75
Moved new redirects to the end of the file
hbelmiro May 22, 2024
f092ea6
Renamed How-to/User guides to User guides
hbelmiro May 22, 2024
05a9805
Fixed redirects
hbelmiro May 30, 2024
8632993
Removed double slash typo
hbelmiro May 30, 2024
d430d0a
Fixed "Legacy v1" title
hbelmiro May 30, 2024
bbfccca
Added "Operator Guides" section
hbelmiro May 30, 2024
00103fd
Added brackets to Legacy page
hbelmiro May 31, 2024
58297c8
Moved Introduction and getting-started.md to the root
hbelmiro May 31, 2024
f4be4d3
Reverted changes to other components
hbelmiro May 31, 2024
566ab75
Removed catch-all redirects and moved new redirects to the bottom of …
hbelmiro May 31, 2024
d6f6692
Renamed the how-to directory to user-guides
hbelmiro May 31, 2024
350dde8
Regrouped user guides
hbelmiro May 31, 2024
9ea2e53
Moved "Concepts" to Pipelines
hbelmiro May 31, 2024
4745c0e
Moved component-spec.md
hbelmiro Jun 4, 2024
40c788e
Moved interfaces.md
hbelmiro Jun 4, 2024
c9b762d
Moved pipeline-root.md
hbelmiro Jun 4, 2024
775bf70
Reverted changes outside of pipelines
hbelmiro Jun 4, 2024
84a164a
Update content/en/docs/components/pipelines/reference/sdk.md
hbelmiro Jun 4, 2024
82bc5b1
Removed warning
hbelmiro Jun 4, 2024
b2aba66
Fixed /docs/components/pipelines/installation/ broken links
hbelmiro Jun 10, 2024
266dea8
Removed beta status flag from pipeline-root.md
hbelmiro Jun 10, 2024
9665cfe
Fixed broken links
hbelmiro Jun 11, 2024
bb40231
Fixed broken links
hbelmiro Jun 11, 2024
1b3b0fd
Update content/en/_redirects
hbelmiro Jun 12, 2024
35bad8a
Update content/en/_redirects
hbelmiro Jun 12, 2024
82b1895
Update content/en/docs/components/pipelines/concepts/component.md
hbelmiro Jun 12, 2024
1f4da3b
Update content/en/docs/components/pipelines/legacy-v1/sdk/manipulate-…
hbelmiro Jun 12, 2024
68f6ef4
Update content/en/docs/components/pipelines/legacy-v1/sdk/python-func…
hbelmiro Jun 12, 2024
ed58c4d
Update content/en/docs/components/pipelines/legacy-v1/sdk/python-func…
hbelmiro Jun 12, 2024
63721a8
Update content/en/docs/components/pipelines/legacy-v1/tutorials/api-p…
hbelmiro Jun 12, 2024
ae0eb54
Update content/en/docs/components/pipelines/legacy-v1/tutorials/build…
hbelmiro Jun 12, 2024
7be77f3
Fixed redirects
hbelmiro Jun 13, 2024
ceac4fd
Fixed redirects
hbelmiro Jun 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion assets/icons/pipelines.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions content/en/_index.html
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ <h3 class="section-head text-center">Kubeflow Components</h3>
<div class="container">
<div class="card-deck">
<div class="card border-primary-dark">
<a href="/docs/components/pipelines/v2/introduction/" target="_blank" rel="noopener" >
<a href="/docs/components/pipelines/overview/" target="_blank" rel="noopener" >
<img
src="/docs/images/logos/kubeflow.png"
class="card-img-top"
Expand All @@ -53,7 +53,7 @@ <h3 class="section-head text-center">Kubeflow Components</h3>
<div class="card-body bg-primary-dark">
<h5 class="card-title text-white section-head">Pipelines</h5>
<p class="card-text text-white">
<a target="_blank" rel="noopener" href="/docs/components/pipelines/v2/introduction/">Kubeflow Pipelines</a> (KFP) is a platform for building then deploying portable and scalable machine learning workflows using Kubernetes.
<a target="_blank" rel="noopener" href="/docs/components/pipelines/overview/">Kubeflow Pipelines</a> (KFP) is a platform for building then deploying portable and scalable machine learning workflows using Kubernetes.
</p>
</div>
</div>
Expand Down
164 changes: 123 additions & 41 deletions content/en/_redirects

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -54,5 +54,5 @@ weight = 40
## Next steps

- See a [simple example](https://github.com/kubeflow/examples/tree/master/pipelines/simple-notebook-pipeline) of creating Kubeflow pipelines in a Jupyter notebook.
- Build machine-learning pipelines with the [Kubeflow Pipelines SDK](/docs/components/pipelines/sdk/sdk-overview/).
- Build machine-learning pipelines with the [Kubeflow Pipelines SDK](/docs/components/pipelines/legacy-v1/sdk/sdk-overview/).
- Learn the advanced features available from a Kubeflow notebook, such as [submitting Kubernetes resources](/docs/components/notebooks/submit-kubernetes/) or [building Docker images](/docs/components/notebooks/custom-notebook/).
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
+++
title = "Concepts"
description = "Concepts used in Kubeflow Pipelines"
weight = 30
weight = 4
+++
Original file line number Diff line number Diff line change
Expand Up @@ -57,11 +57,11 @@ deserialize the data for use in the downstream component.

## Next steps

* Read an [overview of Kubeflow Pipelines](/docs/components/pipelines/introduction/).
* Follow the [pipelines quickstart guide](/docs/components/pipelines/overview/quickstart/)
* Read an [overview of Kubeflow Pipelines](/docs/components/pipelines/overview/).
* Follow the [pipelines quickstart guide](/docs/components/pipelines/getting-started/)
to deploy Kubeflow and run a sample pipeline directly from the Kubeflow
Pipelines UI.
* Build your own
[component and pipeline](/docs/components/pipelines/sdk/build-component/).
* Build a [reusable component](/docs/components/pipelines/sdk/component-development/) for
[component and pipeline](/docs/components/pipelines/legacy-v1/sdk/component-development/).
* Build a [reusable component](/docs/components/pipelines/legacy-v1/sdk/component-development/) for
sharing in multiple pipelines.
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ groups. Experiments can contain arbitrary runs, including

## Next steps

* Read an [overview of Kubeflow Pipelines](/docs/components/pipelines/introduction/).
* Follow the [pipelines quickstart guide](/docs/components/pipelines/overview/quickstart/)
* Read an [overview of Kubeflow Pipelines](/docs/components/pipelines/overview/).
* Follow the [pipelines quickstart guide](/docs/components/pipelines/getting-started/)
to deploy Kubeflow and run a sample pipeline directly from the Kubeflow
Pipelines UI.
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ parent contains a conditional clause.)

## Next steps

* Read an [overview of Kubeflow Pipelines](/docs/components/pipelines/introduction/).
* Follow the [pipelines quickstart guide](/docs/components/pipelines/overview/quickstart/)
* Read an [overview of Kubeflow Pipelines](/docs/components/pipelines/overview/).
* Follow the [pipelines quickstart guide](/docs/components/pipelines/getting-started/)
to deploy Kubeflow and run a sample pipeline directly from the Kubeflow
Pipelines UI.
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,11 @@ data to rich interactive visualizations.

## Next steps

* Read an [overview of Kubeflow Pipelines](/docs/components/pipelines/introduction/).
* Follow the [pipelines quickstart guide](/docs/components/pipelines/overview/quickstart/)
* Read an [overview of Kubeflow Pipelines](/docs/components/pipelines/overview/).
* Follow the [pipelines quickstart guide](/docs/components/pipelines/getting-started/)
to deploy Kubeflow and run a sample pipeline directly from the Kubeflow
Pipelines UI.
* Read more about the available
[output viewers](/docs/components/pipelines/sdk/output-viewer/)
[output viewers](/docs/components/pipelines/legacy-v1/sdk/output-viewer/)
and how to provide the metadata to make use of the visualizations
that the output viewers provide.
Original file line number Diff line number Diff line change
@@ -1,18 +1,16 @@
+++
title = "Pipeline Root"
description = "Getting started with Kubeflow Pipelines pipeline root"
weight = 50
weight = 15

+++
{{% beta-status
feedbacklink="https://github.com/kubeflow/pipelines/issues" %}}

Starting from [Kubeflow Pipelines SDK v2](https://www.kubeflow.org/docs/components/pipelines/sdk-v2/) and Kubeflow Pipelines 1.7.0, Kubeflow Pipelines supports a new intermediate artifact repository feature: pipeline root in both [standalone deployment](https://www.kubeflow.org/docs/components/pipelines/installation/standalone-deployment/) and [AI Platform Pipelines](https://cloud.google.com/ai-platform/pipelines/docs).
Starting from [Kubeflow Pipelines SDK v2](https://www.kubeflow.org/docs/components/pipelines/legacy-v1/sdk-v2/) and Kubeflow Pipelines 1.7.0, Kubeflow Pipelines supports a new intermediate artifact repository feature: pipeline root in both [standalone deployment](/docs/components/pipelines/legacy-v1/installation/standalone-deployment/) and [AI Platform Pipelines](https://cloud.google.com/ai-platform/pipelines/docs).

## Before you start
This guide tells you the basic concepts of Kubeflow Pipelines pipeline root and how to use it.
This guide assumes that you already have Kubeflow Pipelines installed, or want to use standalone or AI Platform Pipelines options in the [Kubeflow Pipelines deployment
guide](/docs/components/pipelines/installation/) to deploy Kubeflow Pipelines.
guide](/docs/components/pipelines/operator-guides/installation/) to deploy Kubeflow Pipelines.

## What is pipeline root?

Expand Down Expand Up @@ -56,7 +54,7 @@ kubectl edit configMap kfp-launcher -n ${namespace}
This pipeline root will be the default pipeline root for all pipelines running in the Kubernetes namespace unless you override it using one of the following options:

#### Via Building Pipelines
You can configure a pipeline root through the `kfp.dsl.pipeline` annotation when [building pipelines](https://www.kubeflow.org/docs/components/pipelines/sdk-v2/build-pipeline/#build-your-pipeline)
You can configure a pipeline root through the `kfp.dsl.pipeline` annotation when [building pipelines](/docs/components/pipelines/legacy-v1/sdk/build-pipeline/#build-your-pipeline)

#### Via Submitting a Pipeline through SDK
You can configure pipeline root via `pipeline_root` argument when you submit a Pipeline using one of the following:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ start Docker containers, and the containers in turn start your programs.
After developing your pipeline, you can upload your pipeline using the Kubeflow Pipelines UI or the Kubeflow Pipelines SDK.

## Next steps
* Read an [overview of Kubeflow Pipelines](/docs/components/pipelines/introduction/).
* Follow the [pipelines quickstart guide](/docs/components/pipelines/overview/quickstart/)
* Read an [overview of Kubeflow Pipelines](/docs/components/pipelines/overview/).
* Follow the [pipelines quickstart guide](/docs/components/pipelines/getting-started/)
to deploy Kubeflow and run a sample pipeline directly from the Kubeflow
Pipelines UI.
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ available:

## Next steps

* Read an [overview of Kubeflow Pipelines](/docs/components/pipelines/introduction/).
* Follow the [pipelines quickstart guide](/docs/components/pipelines/overview/quickstart/)
* Read an [overview of Kubeflow Pipelines](/docs/components/pipelines/overview/).
* Follow the [pipelines quickstart guide](/docs/components/pipelines/getting-started/)
to deploy Kubeflow and run a sample pipeline directly from the Kubeflow
Pipelines UI.
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ triggered to run frequently.

## Next steps

* Read an [overview of Kubeflow Pipelines](/docs/components/pipelines/introduction/).
* Follow the [pipelines quickstart guide](/docs/components/pipelines/overview/quickstart/)
* Read an [overview of Kubeflow Pipelines](/docs/components/pipelines/overview/).
* Follow the [pipelines quickstart guide](/docs/components/pipelines/getting-started/)
to deploy Kubeflow and run a sample pipeline directly from the Kubeflow
Pipelines UI.
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ an if/else like clause in the pipeline code.

## Next steps

* Read an [overview of Kubeflow Pipelines](/docs/components/pipelines/introduction/).
* Follow the [pipelines quickstart guide](/docs/components/pipelines/overview/quickstart/)
* Read an [overview of Kubeflow Pipelines](/docs/components/pipelines/overview/).
* Follow the [pipelines quickstart guide](/docs/components/pipelines/getting-started/)
to deploy Kubeflow and run a sample pipeline directly from the Kubeflow
Pipelines UI.
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
+++
title = "Hello World Pipeline"
title = "Getting started"
description = "Create your first pipeline"
weight = 3
weight = 2
+++

{{% kfp-v2-keywords %}}
Expand Down Expand Up @@ -57,10 +57,10 @@ The client will print a link to view the pipeline execution graph and logs in th

In the next few sections, you'll learn more about the core concepts of authoring pipelines and how to create more expressive, useful pipelines.

[installation]: /docs/components/pipelines/v2/installation/
[installation]: /docs/components/pipelines/operator-guides/installation/
[client]: https://kubeflow-pipelines.readthedocs.io/en/stable/source/client.html#kfp.client.Client
[compiler]: https://kubeflow-pipelines.readthedocs.io/en/stable/source/compiler.html#kfp.compiler.Compiler
[ir-yaml]: /docs/components/pipelines/v2/compile-a-pipeline#ir-yaml
[compile-a-pipeline]: /docs/components/pipelines/v2/compile-a-pipeline/
[ir-yaml]: /docs/components/pipelines/user-guides/core-functions/compile-a-pipeline#ir-yaml
[compile-a-pipeline]: /docs/components/pipelines/user-guides/core-functions/compile-a-pipeline/
[dsl-pipeline]: https://kubeflow-pipelines.readthedocs.io/en/stable/source/dsl.html#kfp.dsl.pipeline
[dsl-component]: https://kubeflow-pipelines.readthedocs.io/en/stable/source/dsl.html#kfp.dsl.component
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
+++
title = "Pipelines Interfaces"
title = "Interfaces"
description = "The ways you can interact with the Kubeflow Pipelines system"
weight = 20
weight = 3

+++

Expand All @@ -21,7 +21,7 @@ From the Kubeflow Pipelines UI you can perform the following tasks:
* Run one or more of the preloaded samples to try out pipelines quickly.
* Upload a pipeline as a compressed file. The pipeline can be one that you
have built (see how to [build a
pipeline](/docs/components/pipelines/sdk/build-pipeline/)) or one
pipeline](/docs/components/pipelines/legacy-v1/sdk/build-pipeline/)) or one
that someone has shared with you.
* Create an *experiment* to group one or more of your pipeline runs.
See the [definition of an
Expand All @@ -33,21 +33,21 @@ From the Kubeflow Pipelines UI you can perform the following tasks:
* Compare the results of one or more runs within an experiment.
* Schedule runs by creating a recurring run.

See the [quickstart guide](/docs/components/pipelines/overview/quickstart/) for more
See the [quickstart guide](/docs/components/pipelines/legacy-v1/overview/quickstart/) for more
information about accessing the Kubeflow Pipelines UI and running the samples.

When building a pipeline component, you can write out information for display
in the UI. See the guides to [exporting
metrics](/docs/components/pipelines/sdk/pipelines-metrics/) and [visualizing results in
the UI](/docs/components/pipelines/sdk/output-viewer/).
metrics](/docs/components/pipelines/legacy-v1/sdk/pipelines-metrics/) and [visualizing results in
the UI](/docs/components/pipelines/legacy-v1/sdk/output-viewer/).

## Python SDK

The Kubeflow Pipelines SDK provides a set of Python packages that you can use to
specify and run your ML workflows.

See the [introduction to the Kubeflow Pipelines
SDK](/docs/components/pipelines/sdk/sdk-overview/) for an overview of the ways you can
SDK](/docs/components/pipelines/legacy-v1/sdk/sdk-overview/) for an overview of the ways you can
use the SDK to build pipeline components and pipelines.

## REST API
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
+++
title = "v1"
title = "Legacy (v1)"
description = "Kubeflow Pipelines v1 Documentation"
weight = 15
weight = 999
+++
Original file line number Diff line number Diff line change
Expand Up @@ -32,15 +32,15 @@ improvements can make it the default executor that most people should use going
* Security: more secure
* No `privileged` access.
* Cannot escape the privileges of the pod's service account.
* Migration: `command` must be specified in [Kubeflow Pipelines component specification](https://www.kubeflow.org/docs/components/pipelines/reference/component-spec/).
* Migration: `command` must be specified in [Kubeflow Pipelines component specification](/docs/components/pipelines/reference/component-spec/).

Note, the same migration requirement is required by [Kubeflow Pipelines v2 compatible mode](https://www.kubeflow.org/docs/components/pipelines/sdk-v2/v2-compatibility/), refer to
Note, the same migration requirement is required by [Kubeflow Pipelines v2 compatible mode](/docs/components/pipelines/legacy-v1/sdk/v2-compatibility/), refer to
[known caveats & breaking changes](https://github.com/kubeflow/pipelines/issues/6133).

#### Migrate to Emissary Executor

Prerequisite: emissary executor is only available in Kubeflow Pipelines backend version 1.7+.
To upgrade, refer to [upgrading Kubeflow Pipelines](/docs/components/pipelines/upgrade/).
To upgrade, refer to [upgrading Kubeflow Pipelines](/docs/components/pipelines/legacy-v1/installation/upgrade//).

##### Configure an existing Kubeflow Pipelines cluster to use emissary executor

Expand Down Expand Up @@ -92,7 +92,7 @@ To upgrade, refer to [upgrading Kubeflow Pipelines](/docs/components/pipelines/u
For [AI Platform Pipelines](https://cloud.google.com/ai-platform/pipelines/docs), check the "Use emissary executor" checkbox during installation.
For [Kubeflow Pipelines Standalone](https://www.kubeflow.org/docs/components/pipelines/installation/standalone-deployment/), install `env/platform-agnostic-emissary`:
For [Kubeflow Pipelines Standalone](/docs/components/pipelines/legacy-v1/installation/standalone-deployment/), install `env/platform-agnostic-emissary`:
```bash
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/platform-agnostic-emissary?ref=$PIPELINE_VERSION"
Expand All @@ -105,7 +105,7 @@ existing clusters.
##### Migrate pipeline components to run on emissary executor
Some pipeline components require manual updates to run on emissary executor.
For [Kubeflow Pipelines component specification](https://www.kubeflow.org/docs/components/pipelines/reference/component-spec/) YAML,
For [Kubeflow Pipelines component specification](/docs/components/pipelines/reference/component-spec/) YAML,
the `command` field must be specified.
Step by step component migration tutorial:
Expand Down Expand Up @@ -152,7 +152,7 @@ Step by step component migration tutorial:
1. The updated component can run on emissary executor now.
Note: Kubeflow Pipelines SDK compiler always specifies a command for
[python function based components](https://www.kubeflow.org/docs/components/pipelines/sdk/python-function-components/).
[python function based components](/docs/components/pipelines/legacy-v1/sdk/python-function-components/).
Therefore, these components will continue to work on emissary executor without
modifications.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Such deployment methods can be part of your local environment using the supplied
kustomize manifests for test purposes. This guide is an alternative to

[Deploying Kubeflow Pipelines
(KFP)](/docs/started/getting-started/#installing-kubeflow).
(KFP)](/docs/started/#installing-kubeflow).

## Before you get started

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ portable installation that only includes Kubeflow Pipelines.
* Kubeflow Pipelines as [part of a full Kubeflow deployment](#full-kubeflow-deployment) provides
all Kubeflow components and more integration with each platform.
* **Beta**: [Google Cloud AI Platform Pipelines](#google-cloud-ai-platform-pipelines) makes it easier to install and use Kubeflow Pipelines on Google Cloud by providing a management UI on [Google Cloud Console](https://console.cloud.google.com/ai-platform/pipelines/clusters).
* A [local](/docs/components/pipelines/installation/localcluster-deployment) Kubeflow Pipelines deployment for testing purposes.
* A [local](/docs/components/pipelines/legacy-v1/installation/localcluster-deployment) Kubeflow Pipelines deployment for testing purposes.

## Choosing an installation option

Expand All @@ -23,7 +23,7 @@ all Kubeflow components and more integration with each platform.
If yes, choose the [full Kubeflow deployment](#full-kubeflow-deployment).
1. Can you use a cloud/on-prem Kubernetes cluster?

If you can't, you should try using Kubeflow Pipelines on a local Kubernetes cluster for learning and testing purposes by following the steps in [Deploying Kubeflow Pipelines on a local cluster](/docs/components/pipelines/installation/localcluster-deployment).
If you can't, you should try using Kubeflow Pipelines on a local Kubernetes cluster for learning and testing purposes by following the steps in [Deploying Kubeflow Pipelines on a local cluster](/docs/components/pipelines/legacy-v1/installation/localcluster-deployment).
1. Do you want to use Kubeflow Pipelines with [multi-user support](https://github.com/kubeflow/pipelines/issues/1223)?

If yes, choose the [full Kubeflow deployment](#full-kubeflow-deployment) with version >= v1.1.
Expand Down Expand Up @@ -56,7 +56,7 @@ Kubeflow Pipelines into an existing Kubernetes cluster.

Installation guide
: [Kubeflow Pipelines Standalone deployment
guide](/docs/components/pipelines/installation/standalone-deployment/)
guide](/docs/components/pipelines/legacy-v1/installation/standalone-deployment/)

Interfaces
:
Expand All @@ -74,7 +74,7 @@ Release Schedule
You will have access to the latest features.

Upgrade Support (**Beta**)
: [Upgrading Kubeflow Pipelines Standalone](/docs/components/pipelines/installation/standalone-deployment/#upgrading-kubeflow-pipelines) introduces how to upgrade
: [Upgrading Kubeflow Pipelines Standalone](/docs/components/pipelines/legacy-v1/installation/standalone-deployment/#upgrading-kubeflow-pipelines) introduces how to upgrade
in place.

Google Cloud Integrations
Expand All @@ -101,7 +101,7 @@ Use this option to deploy Kubeflow Pipelines to your local machine, on-premises,
or to a cloud, as part of a full Kubeflow installation.

Installation guide
: [Kubeflow installation guide](/docs/started/getting-started/)
: [Kubeflow installation guide](/docs/started/)

Interfaces
:
Expand Down
Loading