Skip to content

Commit

Permalink
Merge pull request #17 from viadee-internal/feat-doc
Browse files Browse the repository at this point in the history 
Documentation
  • Loading branch information
@astrothmann
astrothmann authored Mar 22, 2024
2 parents c779e98 + 927b337 commit 673948a
Show file tree
Hide file tree
Showing 17 changed files with 364 additions and 91 deletions.
302 changes: 262 additions & 40 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
# Kubeflow Connector

A custom Camunda 8 outbound connector to communicate with the [Kubeflow](https://www.kubeflow.org/) API. It supports version 1 aqnd version 2 of the Kubeflow API and supports the functions mentioned below in the [API section](#api-section).
A custom Camunda 8 outbound connector to communicate with the [Kubeflow](https://www.kubeflow.org/) API. It supports the multi-user setup with profiles and namespaces as well as version 1 and 2 of the Kubeflow Pipeline API. It currently supports the functions mentioned below in the [API section](#api-section).

## Build
# Build and Run

## Build a deployable jar file
You can package the Connector by running the following command:

```bash
Expand All @@ -15,67 +16,300 @@ This will create the following artifacts:
- A thin JAR without dependencies.
- An fat JAR containing all dependencies, potentially shaded to avoid classpath conflicts. This will not include the SDK artifacts since those are in scope `provided` and will be brought along by the respective Connector Runtime executing the Connector.

## <a id="api-section"></a> API
Currenty this connector supports the following methods from the Kubeflow Pipeline API.
### Shading dependencies

You can use the `maven-shade-plugin` defined in the [Maven configuration](./pom.xml) to relocate common dependencies
that are used in other Connectors and the [Connector Runtime](https://github.com/camunda-community-hub/spring-zeebe/tree/master/connector-runtime#building-connector-runtime-bundles).
This helps to avoid classpath conflicts when the Connector is executed.

Use the `relocations` configuration in the Maven Shade plugin to define the dependencies that should be shaded.
The [Maven Shade documentation](https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html)
provides more details on relocations.

## Test locally

Run unit tests

```bash
mvn clean verify
```

## Test with local runtime
In your IDE you can also simply navigate to the `LocalContainerRuntime` class in test scope and run it via your IDE.
If necessary, you can adjust `application.properties` in test scope.

### Configure a local environment for the Connector
The example bpmn models provided in the bpmn folder of the repository are set up in a way that they rely on environment variables being defined for the kubeflow and authentication configuration. We provided an [example.env](example.env) file that is tailored to the local development environment. You can copy this file to *.env* so that it is considered when you start the connector runtime locally.


# <a id="api-section"></a> API
Currenty this connector supports the following methods from the Kubeflow Pipeline API in both API versions 1 and 2.

- Get Pipelines
- Get Experiments
- Get Experiment by ID
- Get Runs
- Get Run by ID
- Get Run by Name
- Start Run
- Start Run and Monitor
- Create Experiment

### Get Pipelines
The inputs describes the parameters that can be set in the modeler for the operation.
The output is the complete output you will get written into the variable you enter under Output mapping in the result variable. In the result expression you can more specifically extract the data as required to limit the output.

In every operation additional HTTP headers can be set under Kubeflow API -> HTTP Headers if necessary.

## Get Pipelines

### Input

#### Parameter Filter (optional)

API Version 1:

```json
{ "predicates":
[
{
"op":"IS_SUBSTRING",
"key": "name",
"string_value": "Control"
}
]
}
```

API Version 2:
```json
{ "predicates":
[
{
"operation":"IS_SUBSTRING",
"key": "name",
"string_value": "Control"
}
]
}
```
For details on filter structure and options check the proto-buffer files:

Version 1: [https://github.com/kubeflow/pipelines/blob/master/backend/api/v1beta1/filter.proto](https://github.com/kubeflow/pipelines/blob/master/backend/api/v1beta1/filter.proto)

Version 2: [https://github.com/kubeflow/pipelines/blob/master/backend/api/v2beta1/filter.proto](https://github.com/kubeflow/pipelines/blob/master/backend/api/v2beta1/filter.proto)

#### Parameter Namespace (optional)

| Name | Description | Example | Notes |
|----------|------------------|-------------------|----------------------------------------------------------------------------|
| filter | define the filter to apply to the call | `alice` | Has no effect on the function call outcome. |
| token | Mock token value | `my-secret-token` | Has no effect on the function call outcome. |
| message | Mock message | `Hello World` | Echoed back in the output. If starts with 'fail', an error will be thrown. |
API Version 1 and 2: If omitted, shared pipelines will also be returned.

### Output

API Version 1: [https://www.kubeflow.org/docs/components/pipelines/v1/reference/api/kubeflow-pipeline-api-spec/#/definitions/apiListPipelinesResponse](https://www.kubeflow.org/docs/components/pipelines/v1/reference/api/kubeflow-pipeline-api-spec/#/definitions/apiListPipelinesResponse)

API Version 2: [https://www.kubeflow.org/docs/components/pipelines/v2/reference/api/kubeflow-pipeline-api-spec/#operation--apis-v2beta1-pipelines-get](https://www.kubeflow.org/docs/components/pipelines/v2/reference/api/kubeflow-pipeline-api-spec/#operation--apis-v2beta1-pipelines-get)

## Get Experiments

### Input

#### Parameter Filter

API Version 1:
```json
{
"result": {
"myProperty": "Message received: ..."
}
{ "predicates":
[
{
"op":"IS_SUBSTRING",
"key": "name",
"string_value": "test"
}
]
}
```

### Error codes
API Version 2:
```json
{ "predicates":
[
{
"operation":"IS_SUBSTRING",
"key": "name",
"string_value": "test"
}
]
}
```

| Code | Description |
|------|--------------------------------------------|
| FAIL | Message starts with 'fail' (ignoring case) |
#### Parameter Namespace (required in multiuser deployment)

## Test locally
API Version 1 and 2: namespace where the experiments should be retrieved from.

Run unit tests
### Output

```bash
mvn clean verify
API Version 1: [https://www.kubeflow.org/docs/components/pipelines/v1/reference/api/kubeflow-pipeline-api-spec/#/definitions/apiListExperimentsResponse](https://www.kubeflow.org/docs/components/pipelines/v1/reference/api/kubeflow-pipeline-api-spec/#/definitions/apiListExperimentsResponse)

API Version 2: [https://www.kubeflow.org/docs/components/pipelines/v2/reference/api/kubeflow-pipeline-api-spec/#/definitions/v2beta1ListExperimentsResponse](https://www.kubeflow.org/docs/components/pipelines/v2/reference/api/kubeflow-pipeline-api-spec/#/definitions/v2beta1ListExperimentsResponse)

## Get Experiment By ID

### Input

#### Parameter Experiment ID
The ID of the experiment that should be retrieved.

### Output

API Version 1: [https://www.kubeflow.org/docs/components/pipelines/v1/reference/api/kubeflow-pipeline-api-spec/#/definitions/apiExperiment](https://www.kubeflow.org/docs/components/pipelines/v1/reference/api/kubeflow-pipeline-api-spec/#/definitions/apiExperiment)

API Version 2: [https://www.kubeflow.org/docs/components/pipelines/v2/reference/api/kubeflow-pipeline-api-spec/#/definitions/v2beta1Experiment](https://www.kubeflow.org/docs/components/pipelines/v2/reference/api/kubeflow-pipeline-api-spec/#/definitions/v2beta1Experiment)

## <a id="get-runs"></a> Get Runs

### Input

#### Parameter Filter

API Version 1:
```json
{ "predicates":
[
{
"op":"IS_SUBSTRING",
"key": "name",
"string_value": "test"
}
]
}
```

API Version 2:
```json
{ "predicates":
[
{
"operation":"IS_SUBSTRING",
"key": "name",
"string_value": "test"
}
]
}
```

### Test with local runtime
#### Parameter Namespace (required in multiuser deployment)

Use the [Camunda Connector Runtime](https://github.com/camunda-community-hub/spring-zeebe/tree/master/connector-runtime#building-connector-runtime-bundles) to run your function as a local Java application.
API Version 1 and 2: namespace where the runs should be retrieved from.

In your IDE you can also simply navigate to the `LocalContainerRuntime` class in test scope and run it via your IDE.
If necessary, you can adjust `application.properties` in test scope.
### Output

API Version 1: [https://www.kubeflow.org/docs/components/pipelines/v1/reference/api/kubeflow-pipeline-api-spec/#/definitions/apiListRunsResponse](https://www.kubeflow.org/docs/components/pipelines/v1/reference/api/kubeflow-pipeline-api-spec/#/definitions/apiListRunsResponse)

API Version 2: [https://www.kubeflow.org/docs/components/pipelines/v2/reference/api/kubeflow-pipeline-api-spec/#/definitions/v2beta1ListRunsResponse](https://www.kubeflow.org/docs/components/pipelines/v2/reference/api/kubeflow-pipeline-api-spec/#/definitions/v2beta1ListRunsResponse)


## Get Run By ID

### Input

#### Parameter Run ID
The ID of the run that should be retrieved.

### Output

API Version 1: [https://www.kubeflow.org/docs/components/pipelines/v1/reference/api/kubeflow-pipeline-api-spec/#/definitions/apiRunDetail](https://www.kubeflow.org/docs/components/pipelines/v1/reference/api/kubeflow-pipeline-api-spec/#/definitions/apiRunDetail)

API Version 2: [https://www.kubeflow.org/docs/components/pipelines/v2/reference/api/kubeflow-pipeline-api-spec/#/definitions/v2beta1Run](https://www.kubeflow.org/docs/components/pipelines/v2/reference/api/kubeflow-pipeline-api-spec/#/definitions/v2beta1Run)

## Get Run By Name
This request tries to retrieve a single run by looking for a run that contains the name defined in the name parameter.

### Input

#### Parameter Name
The name to look for in the runs. This looks for the name to be equal.

#### Parameter Namespace
The namespace where to look for runs.

### Output
equal to responses of [Get Runs](#get-runs)


## <a id="start-run"></a> Start Run
This operation starts a run and continues the process without waiting for the pipeline to finish.

### Input

#### Parameter Pipeline ID
The ID of the pipeline that should be run.

#### Parameter Experiment ID
The ID of the experiment in which the run should be started.

#### Parameter Suffix of run name
The name of the triggered run will be created as such: <Camunda_process_instance_id>_<suffix_of_run_name>.

#### Parameter Run parameters (optional)
If the pipeline requires parameters to run they can be set here.

### Output

API Version 1: [https://www.kubeflow.org/docs/components/pipelines/v1/reference/api/kubeflow-pipeline-api-spec/#/definitions/apiRunDetail](https://www.kubeflow.org/docs/components/pipelines/v1/reference/api/kubeflow-pipeline-api-spec/#/definitions/apiRunDetail)

API Version 2: [https://www.kubeflow.org/docs/components/pipelines/v2/reference/api/kubeflow-pipeline-api-spec/#/definitions/v2beta1Run](https://www.kubeflow.org/docs/components/pipelines/v2/reference/api/kubeflow-pipeline-api-spec/#/definitions/v2beta1Run)

## Start Run and Monitor
This operation starts a run and monitors it until it finishes are throws an error. If the connector crashes during the run, the connector will pick up the already started run and monitor it after receiving the job again from Zeebe.

### Input

#### Parameter Pipeline ID
The ID of the pipeline that should be run.

#### Parameter Experiment ID
The ID of the experiment in which the run should be started.

#### Polling interval
You can define the interval in which the connector should regularly check the state of the run using ISO 8601 format. E. g. PT10S means every 10 seconds.

#### Parameter Suffix of run name
The name of the triggered run will be created as such: <Camunda_process_instance_id>_<suffix_of_run_name>.

#### Parameter Run parameters (optional)
If the pipeline requires parameters to run they can be set here.

### Output
see [Start Run](#start-run)

## Create Experiment
This operation allows to create an experiment, which can e. g. be used in a subsequent run.

### Input

#### Parameter Experiment Name
The name of the experiment to create.

#### Parameter Experiment Description (optional)
The description of the experiment.

#### Parameter Namespace
The namespace where to create the experiment in.

### Output

API Version 1: [https://www.kubeflow.org/docs/components/pipelines/v1/reference/api/kubeflow-pipeline-api-spec/#/definitions/apiExperiment](https://www.kubeflow.org/docs/components/pipelines/v1/reference/api/kubeflow-pipeline-api-spec/#/definitions/apiExperiment)

API Version 2: [https://www.kubeflow.org/docs/components/pipelines/v2/reference/api/kubeflow-pipeline-api-spec/#/definitions/v2beta1Experiment](https://www.kubeflow.org/docs/components/pipelines/v2/reference/api/kubeflow-pipeline-api-spec/#/definitions/v2beta1Experiment)

## Error handling

In case a return code >= 400 is returned by the API calls, the connector will raise an exception.

## Element Template
The element templates can be found in the [kubeflow-connector.json](./element-templates/kubeflow-connector.json) file.

# Local development environment setup

Start a local kubernets cluster e.g. with mimikube or kind.
Start a local kubernets cluster e.g. with minikube or kind.

Run the following command from the root folder in order to deploy the dev environment into the kubernetes cluster.
```bash
Expand Down Expand Up @@ -112,16 +346,4 @@ client-secret: Jq09L1liFa0UiaXnL3pcnXzlqOKXaoOW
```

## Contact Information
For any queries and further support, please drop us a mail at [camunda8-connector-su-aaaamkuzw6jcci2hm7rscvue7y@viadee.slack.com](mailto:camunda8-connector-su-aaaamkuzw6jcci2hm7rscvue7y@viadee.slack.com)

# OLD ==========

### Shading dependencies

You can use the `maven-shade-plugin` defined in the [Maven configuration](./pom.xml) to relocate common dependencies
that are used in other Connectors and the [Connector Runtime](https://github.com/camunda-community-hub/spring-zeebe/tree/master/connector-runtime#building-connector-runtime-bundles).
This helps to avoid classpath conflicts when the Connector is executed.

Use the `relocations` configuration in the Maven Shade plugin to define the dependencies that should be shaded.
The [Maven Shade documentation](https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html)
provides more details on relocations.
For any queries and further support, please drop us a mail at [camunda8-connector-su-aaaamkuzw6jcci2hm7rscvue7y@viadee.slack.com](mailto:camunda8-connector-su-aaaamkuzw6jcci2hm7rscvue7y@viadee.slack.com)
11 changes: 5 additions & 6 deletions bpmn/create_experiment.bpmn
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="UTF-8"?>
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlns:dc="http://www.omg.org/spec/DD/20100524/DC" xmlns:zeebe="http://camunda.org/schema/zeebe/1.0" xmlns:di="http://www.omg.org/spec/DD/20100524/DI" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:modeler="http://camunda.org/schema/modeler/1.0" id="Definitions_16bytap" targetNamespace="http://bpmn.io/schema/bpmn" exporter="Camunda Modeler" exporterVersion="5.19.0" modeler:executionPlatform="Camunda Cloud" modeler:executionPlatformVersion="8.3.0">
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlns:dc="http://www.omg.org/spec/DD/20100524/DC" xmlns:zeebe="http://camunda.org/schema/zeebe/1.0" xmlns:di="http://www.omg.org/spec/DD/20100524/DI" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:modeler="http://camunda.org/schema/modeler/1.0" id="Definitions_16bytap" targetNamespace="http://bpmn.io/schema/bpmn" exporter="Camunda Modeler" exporterVersion="5.21.0" modeler:executionPlatform="Camunda Cloud" modeler:executionPlatformVersion="8.3.0">
<bpmn:process id="create_experiment" isExecutable="true">
<bpmn:startEvent id="Event_04cpbtp">
<bpmn:outgoing>Flow_0ar4k28</bpmn:outgoing>
Expand All @@ -13,14 +13,13 @@
<bpmn:extensionElements>
<zeebe:taskDefinition type="de.viadee.bpm.camunda:connector-kubeflow:1" />
<zeebe:ioMapping>
<zeebe:input source="20" target="connectionTimeoutInSeconds" />
<zeebe:input source="http://localhost:32357" target="configuration.kubeflowUrl" />
<zeebe:input source="kubeflow-user-example-com" target="configuration.multiusernamespace" />
<zeebe:input source="noAuth" target="authentication.type" />
<zeebe:input source="http://localhost:30000" target="configuration.kubeflowUrl" />
<zeebe:input source="multiUserMode" target="configuration.typeOfUserMode" />
<zeebe:input source="environment" target="authentication.type" />
<zeebe:input source="pipelinesV1" target="kubeflowapi.api" />
<zeebe:input source="create_experiment" target="kubeflowapi.operation" />
<zeebe:input source="kubeflow-user-example-com" target="kubeflowapi.namespace" />
<zeebe:input source="camunda2" target="kubeflowapi.experimentName" />
<zeebe:input source="={&#34;Cookie&#34;: &#34;authservice_session=MTcwNzc1MzcxMnxOd3dBTkRkRFRqTk9Vek5CVFVzMFREWXpNbFEwVmswM1RVeEdSRlZTVEZkWE16ZFNTa3hEUTBaVFQweFBVMEpCUmxWTlJrZFFNMEU9fPPT2hwktXE4xoZY4whPwUYKVsTMq1fmzbCHs6DBtX2e&#34;}" target="kubeflowapi.httpHeaders" />
<zeebe:input source="20" target="timeout.connectionTimeoutInSeconds" />
</zeebe:ioMapping>
<zeebe:taskHeaders>
Expand Down
Loading

0 comments on commit 673948a

Please sign in to comment.