Add OpenShift KFDef

[vpavlin]

Which issue is resolved by this Pull Request:
Avoid running oc commands when deploying Kubeflow on OpenShift

Description of your changes:
The proper way to extend deployment targets list for Kubeflow is to come up with kfdef configuration. So far we have resorted to deploying KF to OpenShift by running a long set of oc commands which leads to diverting from a normal KF deployment process.

This PR is trying to align the KF on OpenShift deployment with a standard KF deployment process.

NOTE: It is still work in progress, so the KFDef file only deploys few components, but we can merge it soon and extend it with other PRs to make the testing and deploying simpler.

How to try

Checkout the PR

sed  -i  's#uri: .*#uri: '$PWD'#' ./kfdef/kfctl_openshift.yaml 
kfctl build --file=kfdef/kfctl_openshift.yaml
kfctl apply --file=./kfdef/kfctl_openshift.yaml

You should see non-commented components in the kfctl_openshift.yaml running successfully

[crobby]

Trying this out: when I do kfctl build --file=kfdef/kfctl_openshift.yaml, I get...

couldn't generate KfApp: (kubeflow.error): Code 500 with message: could not sync cache. Error: (kubeflow.error): Code 400 with message: couldn't download URI /home/vpavlin/devel/github.com/vpavlin/manifests Error stat /home/vpavlin/devel/github.com/vpavlin/manifests: no such file or directory

[crobby]

Looks like it has a reference to your local filesystem:

uri: /home/vpavlin/devel/github.com/vpavlin/manifests

[crobby]

Changed the uri to my local filesystem and did:
kfctl build --file=/kfctl_openshift.yaml
kfctl apply --file=/kfctl_openshift.yaml
Seems to be doing something now, but I am seeing the following warning:

WARN[0047] Encountered error during apply: (kubeflow.error): Code 500 with message: Apply.Run Error unable to recognize "/tmp/kout122983657": no matches for kind "Application" in version "app.k8s.io/v1beta1" filename="kustomize/kustomize.go:183"

[vpavlin]

Ah, right

uri: /home/vpavlin/devel/github.com/vpavlin/manifests

I updated the description to account for this

WARN[0047] Encountered error during apply: (kubeflow.error): Code 500 with message: Apply.Run Error unable to recognize "/tmp/kout122983657": no matches for kind "Application" in version "app.k8s.io/v1beta1" filename="kustomize/kustomize.go:183"

Damn:) Yeah, this is because of the overlays applicaiton, I think we can remove those from OpenShift KFDef as we do not really have a use for that CR

[crobby]

Still seeing the following with the latest version and a fresh cluster.
WARN[0042] Encountered error during apply: (kubeflow.error): Code 500 with message: Apply.Run Error unable to recognize "/tmp/kout757664611": no matches for kind "Application" in version "app.k8s.io/v1beta1" filename="kustomize/kustomize.go:183"

[crobby]

Back to trying to get this working: I actually went ahead and added in the applicaiton-crd stuff (since other things will require it....tf-serving does) and I got farther than before. But now I wind-up seeing: Deployment.apps "metadata-ui" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"metadata-ui", "kustomize.component":"metadata"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable]
Not sure what is happening just yet.

[crobby]

Hmm...I think I just answered my own question...sort of. Seems like the second running against the existing deployment caused the issue. I blew away the existing deployment and it seems to have worked. That fixes my issue, but might be a problem if others run into a "partially successful" install like I was dealing with at first. If I re-run against my fully successful install, it seems to be fine.

[crobby]

Seems like metadata-db is still failing: log message from the pod is:

mkdir: cannot create directory '/var/lib/mysql': Permission denied

[crobby]

Ok, after further review, my problem with metadata-db is related to my use of CRC and this issue: crc-org/crc#814 The sort of cheap workaround for this is to change the metadata-db deployment to use emptyDir: {} instead of persistentVolumeClaim. That seems to yield a filesystem that allows the database to start up.
A deployment of Openshift elsewhere, like AWS, should not have this issue to deal with.

[vpavlin]

We will now be creating 2 SCCs for OpenShift deployment:

1. kubeflow-anyuid-istio - for Istio related SAs (guess this will be made obsolete by Integrating RH Service Mesh with Kubeflow #8)
2. kubeflow-anyuid-$(NAMESPACE) - for SAs in KF namespace - to make sure KF can be deployed to a namespace which is not named kubeflow and also potentially into multiple namespaces at the same time

I also added an SA metadatadb for metadata-db, so that we don't need anyuid for default SA

[crobby]

Ok, I was able to run this and everything I poked at seems happy. (the only thing I tweaked besides the path was the Volume definition for metadata-db due to the CRC bug).
Ideally, we come up with a solution so that your personal directory structure doesn't exist in what's committed, but I think this is in a good/merge-able place for our current purposes.

[vpavlin]

Just hit this issue: kubeflow/kubeflow#4642

It should not block us from merging, but it is something to keep in mind as the repo download will probably not work when people try it out with the proper URI

[crobby]

New update lgtm. I will merge this.