Discussion on helm-based JITServer operator

[chrisc66]

As a continuous part of the JITServer on-cloud deployment discussion, we would like to implement an OpenJ9 JITServer operator to support OpenShift users.

The initial idea is to take advantage of the existing JITServer helm chart and implement a helm-based operator, then publish it as a community operator (without RedHat certification).

This issue will be populated as more details become available. A few items to confirm:

• Explore the possibility of hosting operator on github repo, or we might have to host it on quay.io.
• Find out and document what are the required steps before users can install this operator (operator-source.yaml).
• Maintainance for version update or functionality updates.

In the future, we might want to extend the capacity of the operator and implement a go-based operator, but this is not our focus right now.

FYI: @mpirvu @keithc-ca @EmanElsaban

[chrisc66]

Additional information on operator deployment and registry. There are three approaches to deploy operators with increasing complexity.

1. Not managed by OLM, scripts with pure kubectl commands

   Operators are yaml files, so they can be deployed via kubectl commands. Smaller-scale operators sometimes include install/uninstall scripts. Users can run the scripts to deploy operators, however, these operators are not managed by Operator Lifecycle Manager (OLM) and thus don't have UI.

   This is the simplest approach since OLM isn't involved. The existing operator yaml files are enough without requiring additional OLM manifests.

2. Managed by OLM, creating our own operator-source.yaml

   To enable UI and OLM, we need to generate olm-catelog directory using operator-sdk in the source code. It contains package.yaml, OperatorSource, ClusterServiceVersion, that can be recognized and managed by the OLM. Users can either install operator-source.yaml manually (this approach), or obtain the source from a registry (next approach).

   This approach will require us to use operator-courier to push operator (image and manifests) to quay.io. Users then need to install operator-source.yaml on their cluster, which points to the operator on quay.

3. Managed by OLM, source from community operator registry

   This is the most complex approach for us, since the operator needs to go through a list of manifest verification CICD checks defined by the operator registry. OperatorHub is a popular community operator registry managed by RedHat. Once the operator is published, users can see it on their cluster or OperatorHub directly.

For each of the above approaches, we need to find an image registry (and account) to host operator images. It could be dockerhub, quay.io, etc. Unfortunately, Github is not an ideal location to host operator since we need to host operator images with multiple versions.

My suggestion is to take the second approach. In the future, we can explore if it is worth publishing to operator registry or more powerful operator.

[chrisc66]

Explore the possibility of hosting operator on github repo, or we might have to host it on quay.io.

Github is not capable of hosting operator images, a image repository is needed. If taking the second approach above, we need to host operator image and manifest on quay.io. Any suggestion on which account to use? They also accept SSO with RedHat, Github and google accounts. Is there such OpenJ9 organization account available?

Find out and document what are the required steps before users can install this operator

If taking the second approach, users only needs to install operator-source.yaml and then the operator is visible on the cluster OLM.
i.e. kubectl apply -f GITHUB_RAW_URL/operator-source.yaml

Maintainance for version update or functionality updates

If taking the second approach, for every helm chart update

• double deliver helm chart change to operator/helm-chart
• rebuild operator image with updated helm chart
• generate new manifests (OperatorSource, ClusterServiceServer, package.yaml, ...) and append to olm-catalog (directory that stores all past versions)
• push operator image and manifests to quay.io

[mpirvu]

Thanks @chrisc66. Solution 2 (Managed by OLM, creating our own operator-source.yaml) looks like a sensible approach to start with. I don't have yet an answer about the account for quay.io. It appears that going forward quay.io also needs a RedHat account: https://www.openshift.com/blog/quay.io-will-require-a-red-hat-account-after-june-30-2021...wait-what

[chrisc66]

As discussed, we are going to follow the second approach for deploying the JITServer operator (deployment is managed by OLM without community operator registry certification).

Summarizing a to-do list for going forward.

1. Use operator-sdk to generate helm-based operator

   operator-sdk init --plugins helm --helm-chart PATH_TO_CHART --helm-chart-repo https://raw.githubusercontent.com/eclipse/openj9-utils/master/helm-chart/openj9-jitserver-chart
   

2. Generate operator manifest (further research required)

   Operator manifest includes content in olm-catalog directory, e.g. ClusterServiceVersion, OperatorSource, etc. The file name and format could change due to operator-sdk version upgrade. This part needs further research on what does the new manifest include, and how to generate new manifest.

   Once the manifest is generated, verify the operator manifests using operator-courier verify.

3. Create a repository on Quay.io

   Make sure to create an "application repository". Possibly host the application repository under an official OpenJ9 account.

4. Build and push operator image to quay

   Build operator image from Dockerfile and push to application repository using operator-courier push.

5. Testing and maintenance

   Verify that the operator can successfully deploy a JITServer pod, the operator RBAC is set up properly, and JITServer operator shows up under cluster operator hub after applying operator source.

Two useful tools from operator-framework that help with building JITServer operator and worth further investigation.

• operator-sdk generates and writes operator easier with high-level abstractions.
• operator-courier builds, validates and pushes operator artifacts.