diff --git a/README.md b/README.md index 85be597b8..43ef025eb 100644 --- a/README.md +++ b/README.md @@ -44,11 +44,17 @@ The Pravega Operator manages Pravega clusters deployed to Kubernetes and automat ## Quickstart +We recommend using our [helm charts](charts) for all installation and upgrades (but not for rollbacks at the moment since helm rollbacks are still experimental). The helm charts for pravega operator (version 0.4.5 onwards) and pravega cluster (version 0.5.0 onwards) are published in [https://charts.pravega.io](https://charts.pravega.io/). To add this repository to your Helm repos, use the following command +``` +helm repo add pravega https://charts.pravega.io +``` +There are manual deployment, upgrade and rollback options available as well. + ### Install the Operator > Note: If you are running on Google Kubernetes Engine (GKE), please [check this first](doc/development.md#installation-on-google-kubernetes-engine). -We recommend using helm to deploy a Pravega Operator. Check out the [helm installation](charts/pravega-operator/README.md) document for instructions. +To understand how to deploy a Pravega Operator using helm, refer to [this](charts/pravega-operator#installing-the-chart). #### Deploying in Test Mode The Operator can be run in a "test" mode if we want to create pravega on minikube or on a cluster with very limited resources by enabling `testmode: true` in `values.yaml` file. Operator running in test mode skips minimum replica requirement checks on Pravega components. "Test" mode ensures a bare minimum setup of pravega and is not recommended to be used in production environments. @@ -79,22 +85,9 @@ $ kubectl create -f ./example/pvc-tier2.yaml #### Install a Pravega cluster -Use Helm to install a sample Pravega cluster with release name `bar`. - -``` -$ helm install bar charts/pravega --set zookeeperUri=[ZOOKEEPER_HOST] --set bookkeeperUri=[BOOKKEEPER_SVC] --set storage.longtermStorage.filesystem.pvc=[TIER2_NAME] -``` - -where: - -- **[ZOOKEEPER_HOST]** is the host or IP address of your Zookeeper deployment (e.g. `zookeeper-client:2181`). Multiple Zookeeper URIs can be specified, use a comma-separated list and DO NOT leave any spaces in between (e.g. `zookeeper-0:2181,zookeeper-1:2181,zookeeper-2:2181`). -- **[BOOKKEEPER_SVC]** is the is the name of the headless service of your Bookkeeper deployment (e.g. `bookkeeper-bookie-0.bookkeeper-bookie-headless.default.svc.cluster.local:3181,bookkeeper-bookie-1.bookkeeper-bookie-headless.default.svc.cluster.local:3181,bookkeeper-bookie-2.bookkeeper-bookie-headless.default.svc.cluster.local:3181`). -- **[TIER2_NAME]** is the longtermStorage `PersistentVolumeClaim` name. `pravega-tier2` if you created the PVC above. - - -Check out the [Pravega Helm Chart](charts/pravega) for the complete list of configurable parameters. +To understand how to deploy a pravega cluster using helm, refer to [this](charts/pravega#installing-the-chart). -Verify that the cluster instances and its components are being created. +Once the pravega cluster with release name `bar` has been created, use the following command to verify that the cluster instances and its components are being created. ``` $ kubectl get PravegaCluster @@ -132,27 +125,27 @@ statefulset.apps/bar-pravega-segmentstore 3 3 2m By default, a `PravegaCluster` instance is only accessible within the cluster through the Controller `ClusterIP` service. From within the Kubernetes cluster, a client can connect to Pravega at: ``` -tcp://-pravega-controller.:9090 +tcp://[CLUSTER_NAME]-pravega-controller.[NAMESPACE]:9090 ``` And the `REST` management interface is available at: ``` -http://-pravega-controller.:10080/ +http://[CLUSTER_NAME]-pravega-controller.[NAMESPACE]:10080/ ``` Check out the [external access documentation](doc/external-access.md) if your clients need to connect to Pravega from outside Kubernetes. -Check out the [exposing Segmentstore service on single IP address ](https://github.com/pravega/pravega-operator/blob/4aa88641c3d5a1d5afbb2b9e628846639fd13290/doc/external-access.md#exposing-segmentstore-service-on-single-ip-address-and-different-ports) if your clients need to connect to Pravega Segment store on the same IP address from outside Kubernetes. +Check out the [exposing Segmentstore service on single IP address](https://github.com/pravega/pravega-operator/blob/4aa88641c3d5a1d5afbb2b9e628846639fd13290/doc/external-access.md#exposing-segmentstore-service-on-single-ip-address-and-different-ports) if your clients need to connect to Pravega Segment store on the same IP address from outside Kubernetes. ### Scale a Pravega cluster -You can scale Pravega components independently by modifying their corresponding field in the Pravega resource spec. You can either `kubectl edit` the cluster or `kubectl patch` it. If you edit it, update the number of replicas for BookKeeper, Controller, and/or Segment Store and save the updated spec. +You can scale Pravega components independently by modifying their corresponding field in the Pravega resource spec. You can either `kubectl edit` the cluster or `kubectl patch` it. If you edit it, update the number of replicas for Controller, and/or Segment Store and save the updated spec. Example of patching the Pravega resource to scale the Segment Store instances to 4. ``` -kubectl patch PravegaCluster --type='json' -p='[{"op": "replace", "path": "/spec/pravega/segmentStoreReplicas", "value": 4}]' +kubectl patch PravegaCluster [CLUSTER_NAME] --type='json' -p='[{"op": "replace", "path": "/spec/pravega/segmentStoreReplicas", "value": 4}]' ``` ### Upgrade a Pravega cluster @@ -162,7 +155,7 @@ Check out the [upgrade guide](doc/upgrade-cluster.md). ### Uninstall the Pravega cluster ``` -$ helm uninstall bar +$ helm uninstall [PRAVEGA_RELEASE_NAME] $ kubectl delete -f ./example/pvc-tier2.yaml ``` @@ -171,7 +164,7 @@ $ kubectl delete -f ./example/pvc-tier2.yaml > Note that the Pravega clusters managed by the Pravega operator will NOT be deleted even if the operator is uninstalled. ``` -$ helm uninstall foo +$ helm uninstall [PRAVEGA_OPERATOR_RELEASE_NAME] ``` If you want to delete the Pravega clusters, make sure to do it before uninstalling the operator. diff --git a/charts/pravega-operator/README.md b/charts/pravega-operator/README.md index 9e3d9eecd..ed78a9a70 100644 --- a/charts/pravega-operator/README.md +++ b/charts/pravega-operator/README.md @@ -13,35 +13,41 @@ This chart bootstraps a [pravega-operator](https://github.com/pravega/pravega-op - An existing Apache Bookkeeper 4.9.2 cluster. This can be easily deployed using our [BookKeeper Operator](https://github.com/pravega/bookkeeper-operator) - Cert-Manager v0.15.0+ or some other certificate management solution in order to manage the webhook service certificates. This can be easily deployed by referring to [this](https://cert-manager.io/docs/installation/kubernetes/) - An Issuer and a Certificate (either self-signed or CA signed) in the same namespace that the Pravega Operator will be installed (refer to [this](https://github.com/pravega/pravega-operator/blob/master/deploy/certificate.yaml) manifest to create a self-signed certificate in the default namespace) - > The name of the certificate (*webhookCert.certName*), the name of the secret created by this certificate (*webhookCert.secretName*), the tls.crt (*webhookCert.crt*) and tls.key (*webhookCert.key*) need to be specified against the corresponding fields in the values.yaml file, or can be provided with the install command as shown [here](#installing-the-chart). - The values *tls.crt* and *tls.key* are contained in the secret which is created by the certificate and can be obtained using the following command - ``` - kubectl get secret -o yaml | grep tls. - ``` ## Installing the Chart -To install the chart with the release name `my-release`: +To install the pravega-operator chart, use the following commands: ``` -$ helm install my-release pravega-operator --set webhookCert.crt= --set webhookCert.generate=false --set webhookCert.certName= --set webhookCert.secretName= +$ helm repo add pravega https://charts.pravega.io +$ helm repo update +$ helm install [RELEASE_NAME] pravega/pravega-operator --version=[VERSION] --set webhookCert.crt=[TLS_CRT] --set webhookCert.generate=false --set webhookCert.certName=[CERT_NAME] --set webhookCert.secretName=[SECRET_NAME] ``` +where: +- **[RELEASE_NAME]** is the release name for the pravega-operator chart. +- **[DEPLOYMENT_NAME]** is the name of the pravega-operator deployment so created. (If [RELEASE_NAME] contains the string `pravega-operator`, `[DEPLOYMENT_NAME] = [RELEASE_NAME]`, else `[DEPLOYMENT_NAME] = [RELEASE_NAME]-pravega-operator`. The [DEPLOYMENT_NAME] can however be overridden by providing `--set fullnameOverride=[DEPLOYMENT_NAME]` along with the helm install command) +- **[VERSION]** can be any stable release version for pravega-operator from 0.5.0 onwards. +- **[CERT_NAME]** is the name of the certificate created in the previous step +- **[SECRET_NAME]** is the name of the secret created by the above certificate +- **[TLS_CRT]** is contained in the above secret and can be obtained using the command `kubectl get secret [SECRET_NAME] -o yaml | grep tls.crt` -The command deploys pravega-operator on the Kubernetes cluster in the default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation. +This command deploys a pravega-operator on the Kubernetes cluster in its default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation. + +>Note: If the pravega-operator version is 0.4.5, webhookCert.generate, webhookCert.crt, webhookCert.certName and webhookCert.secretName should not be set. Also in this case, bookkeeper operator, cert-manager and the certificate/issuer do not need to be deployed as prerequisites. ## Uninstalling the Chart -To uninstall/delete the `my-release` deployment: +To uninstall/delete the pravega-operator chart, use the following command: ``` -$ helm uninstall my-release +$ helm uninstall [RELEASE_NAME] ``` -The command removes all the Kubernetes components associated with the chart and deletes the release. +This command removes all the Kubernetes components associated with the chart and deletes the release. ## Configuration -The following table lists the configurable parameters of the Pravega operator chart and their default values. +The following table lists the configurable parameters of the pravega-operator chart and their default values. | Parameter | Description | Default | | ----- | ----------- | ------ | diff --git a/charts/pravega-operator/templates/webhook.yaml b/charts/pravega-operator/templates/webhook.yaml index f481057bf..406531dfa 100644 --- a/charts/pravega-operator/templates/webhook.yaml +++ b/charts/pravega-operator/templates/webhook.yaml @@ -23,7 +23,7 @@ metadata: labels: {{ include "pravega-operator.commonLabels" . | indent 4 }} annotations: - cert-manager.io/inject-ca-from: default/selfsigned-cert + cert-manager.io/inject-ca-from: {{ .Release.Namespace }}/{{ .Values.certName }} webhooks: - clientConfig: service: diff --git a/charts/pravega/README.md b/charts/pravega/README.md index 12074a2d3..b3dbc5455 100644 --- a/charts/pravega/README.md +++ b/charts/pravega/README.md @@ -16,27 +16,39 @@ This chart creates a [Pravega](https://github.com/pravega/pravega) cluster in [K ## Installing the Chart -To install the chart with the release name `my-release`: +To install the pravega chart, use the following commands: ``` -$ helm install my-release pravega +$ helm repo add pravega https://charts.pravega.io +$ helm repo update +$ helm install [RELEASE_NAME] pravega/pravega --version=[VERSION] --set zookeeperUri=[ZOOKEEPER_HOST] --set bookkeeperUri=[BOOKKEEPER_SVC] --set storage.longtermStorage.filesystem.pvc=[TIER2_NAME] ``` +where: +- **[RELEASE_NAME]** is the release name for the pravega chart. +- **[CLUSTER_NAME]** is the name of the pravega cluster so created. (If [RELEASE_NAME] contains the string `pravega`, `[CLUSTER_NAME] = [RELEASE_NAME]`, else `[CLUSTER_NAME] = [RELEASE_NAME]-pravega`. The [CLUSTER_NAME] can however be overridden by providing `--set fullnameOverride=[CLUSTER_NAME]` along with the helm install command) + **Note:** You need to ensure that the [CLUSTER_NAME] is the same value as that provided in the [bookkeeper chart configuration](https://github.com/pravega/bookkeeper-operator/tree/master/charts/bookkeeper#configuration), the default value for which is `pravega` and can be achieved by either providing the `[RELEASE_NAME] = pravega` or by providing `--set fullnameOverride=pravega` at the time of installing the pravega chart. On the contrary, the default value of [CLUSTER_NAME] in the bookkeeper charts can also be overridden by providing `--set pravegaClusterName=[CLUSTER_NAME]` at the time of installing the bookkeeper chart) +- **[VERSION]** can be any stable release version for pravega from 0.5.0 onwards. +- **[ZOOKEEPER_HOST]** is the host or IP address of your Zookeeper deployment (e.g. `zookeeper-client:2181`). Multiple Zookeeper URIs can be specified, use a comma-separated list and DO NOT leave any spaces in between (e.g. `zookeeper-0:2181,zookeeper-1:2181,zookeeper-2:2181`). +- **[BOOKKEEPER_SVC]** is the is the name of the headless service of your Bookkeeper deployment (e.g. `bookkeeper-bookie-0.bookkeeper-bookie-headless.default.svc.cluster.local:3181,bookkeeper-bookie-1.bookkeeper-bookie-headless.default.svc.cluster.local:3181,bookkeeper-bookie-2.bookkeeper-bookie-headless.default.svc.cluster.local:3181`). +- **[TIER2_NAME]** is the longtermStorage `PersistentVolumeClaim` name (`pravega-tier2` if you created the PVC using the manifest provided). -The command deploys pravega on the Kubernetes cluster in the default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation. +This command deploys pravega on the Kubernetes cluster in its default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation. + +>Note: If the underlying pravega operator version is 0.4.5, bookkeeperUri should not be set, and the pravega-bk chart should be used instead of the pravega chart ## Uninstalling the Chart -To uninstall/delete the `my-release` deployment: +To uninstall/delete the pravega chart, use the following command: ``` -$ helm uninstall my-release +$ helm uninstall [RELEASE_NAME] ``` -The command removes all the Kubernetes components associated with the chart and deletes the release. +This command removes all the Kubernetes components associated with the chart and deletes the release. ## Configuration -The following table lists the configurable parameters of the Pravega chart and their default values. +The following table lists the configurable parameters of the pravega chart and their default values. | Parameter | Description | Default | | ----- | ----------- | ------ | @@ -70,8 +82,8 @@ The following table lists the configurable parameters of the Pravega chart and t | `segmentStore.resources.limits.memory` | Memory limits for segmentStore | `4Gi` | | `segmentStore.service.type` | Override the segmentStore service type, if external access is enabled (LoadBalancer/NodePort) | | | `segmentStore.service.annotations` | Annotations to add to the segmentStore service, if external access is enabled | `{}` | -| `segmentStore.service.segmentStoreLoadBalancerIP` |It is used to provide a LoadBalancerIP | | -| `segmentStore.service.segmentStoreExternalTrafficPolicy` | It is used to provide segmentStoreExternalTrafficPolicy | | +| `segmentStore.service.loadBalancerIP` |It is used to provide a LoadBalancerIP for the segmentStore service | | +| `segmentStore.service.externalTrafficPolicy` | It is used to provide ExternalTrafficPolicy for the segmentStore service | | | `segmentStore.jvmOptions` | JVM Options for segmentStore | `[]` | | `storage.longtermStorage.type` | Type of long term storage backend to be used (filesystem/ecs/hdfs) | `filesystem` | | `storage.longtermStorage.filesystem.pvc` | Name of the pre-created PVC, if long term storage type is filesystem | `pravega-tier2` | diff --git a/charts/pravega/templates/pravega.yaml b/charts/pravega/templates/pravega.yaml index ddc80f1a9..4c224d195 100644 --- a/charts/pravega/templates/pravega.yaml +++ b/charts/pravega/templates/pravega.yaml @@ -9,9 +9,15 @@ spec: {{- if .Values.tls }} tls: static: + {{- if .Values.tls.secret.controller }} controllerSecret: {{ .Values.tls.secret.controller }} + {{- end }} + {{- if .Values.tls.secret.segmentStore }} segmentStoreSecret: {{ .Values.tls.secret.segmentStore }} + {{- end }} + {{- if .Values.tls.caBundle }} caBundle: {{ .Values.tls.caBundle }} + {{- end }} {{- end }} authentication: enabled: {{ .Values.authentication.enabled }} @@ -33,10 +39,12 @@ spec: {{- if .Values.externalAccess.enabled }} controllerServiceAccountName: {{ .Values.serviceAccount.name }} segmentStoreServiceAccountName: {{ .Values.serviceAccount.name }} - {{- if .Values.segmentStore.service.segmentStoreLoadBalancerIP }} - segmentStoreLoadBalancerIP: {{ .Values.segmentStore.service.segmentStoreLoadBalancerIP }} + {{- if .Values.segmentStore.service.loadBalancerIP }} + segmentStoreLoadBalancerIP: {{ .Values.segmentStore.service.loadBalancerIP }} + {{- end }} + {{- if .Values.segmentStore.service.externalTrafficPolicy }} + segmentStoreExternalTrafficPolicy: {{ .Values.segmentStore.service.externalTrafficPolicy }} {{- end }} - segmentStoreExternalTrafficPolicy: {{ .Values.segmentStore.service.segmentStoreExternalTrafficPolicy }} {{- if .Values.controller.service.type }} controllerExtServiceType: {{ .Values.controller.service.type }} {{- end }} @@ -70,11 +78,9 @@ spec: {{- end }} {{- if .Values.segmentStore.secret }} segmentStoreSecret: - {{- if .Values.segmentStore.secret.name }} - secret: {{ .Values.segmentStore.name }} - {{- end }} + secret: {{ .Values.segmentStore.secret.name }} {{- if .Values.segmentStore.secret.path }} - mountPath: {{ .Values.segmentStore.path }} + mountPath: {{ .Values.segmentStore.secret.path }} {{- end }} {{- end }} {{- if .Values.controller.jvmOptions }} @@ -86,20 +92,26 @@ spec: {{ toYaml .Values.segmentStore.jvmOptions | indent 6 }} {{- end }} debugLogging: {{ .Values.debugLogging }} + {{- if .Values.storage.cache }} cacheVolumeClaimTemplate: accessModes: [ "ReadWriteOnce" ] + {{- if .Values.storage.cache.className }} storageClassName: {{ .Values.storage.cache.className }} + {{- end }} + {{- if .Values.storage.cache.size }} resources: requests: storage: {{ .Values.storage.cache.size }} + {{- end }} + {{- end }} longtermStorage: - {{- if eq .Values.storage.longtermStorage.type "ecs"}} + {{- if eq (default .Values.storage.longtermStorage.type "filesystem") "ecs" }} ecs: configUri: {{ .Values.storage.longtermStorage.ecs.configUri }} bucket: {{ .Values.storage.longtermStorage.ecs.bucket }} prefix: {{ .Values.storage.longtermStorage.ecs.prefix }} credentials: {{ .Values.storage.longtermStorage.ecs.credentials }} - {{- else if eq .Values.storage.longtermStorage.type "hdfs"}} + {{- else if eq (default .Values.storage.longtermStorage.type "filesystem") "hdfs"}} hdfs: uri: {{ .Values.storage.longtermStorage.hdfs.uri }} root: {{ .Values.storage.longtermStorage.hdfs.root }} @@ -107,7 +119,7 @@ spec: {{- else }} filesystem: persistentVolumeClaim: - claimName: {{ .Values.storage.longtermStorage.filesystem.pvc }} + claimName: {{ default "pravega-tier2" .Values.storage.longtermStorage.filesystem.pvc }} {{- end }} {{- if .Values.options }} options: diff --git a/doc/operator-upgrade.md b/doc/operator-upgrade.md index 7513cd49f..6d8dd01f0 100644 --- a/doc/operator-upgrade.md +++ b/doc/operator-upgrade.md @@ -7,24 +7,26 @@ Starting Operator version `0.4.3` we also support major version upgrades for Pra `0.4.4 --> 0.5.0` + `0.4.5 --> 0.5.0` + # Upgrade Guide -## Upgrading till 0.4.4 +## Upgrading till 0.4.5 or from 0.5.0 to above ### Trigger the upgrade via helm -Pravega operator can be upgraded using the following command +Pravega operator can be upgraded to a version **[VERSION]** using the following command ``` -$ helm upgrade +$ helm upgrade [PRAVEGA_OPERATOR_RELEASE_NAME] pravega/pravega-operator --version=[VERSION] ``` ### Trigger the upgrade manually -Pravega operator can be upgraded manually by modifying the image tag using kubectl edit, patch or apply +The pravega operator with deployment name **[DEPLOYMENT_NAME]** can be upgraded manually by modifying the image tag using kubectl edit, patch or apply ``` -$ kubectl edit +$ kubectl edit [DEPLOYMENT_NAME] ``` The upgrade is handled as a [rolling update](https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/) by Kubernetes and results in a new operator pod being created and the old one being terminated. @@ -37,7 +39,7 @@ Here is a list of changes and their impact: 1. The Pravega Cluster CRD -Till Operator `0.4.4`, the Pravega CR (version `v1alpha1`) includes Bookkeeper. +Till Operator `0.4.5`, the Pravega CR (version `v1alpha1`) includes Bookkeeper. Starting version `0.5.0`, Pravega CR does **not** include Bookkeeper. Bookkeeper is moved out and is now a prerequisite for Pravega deployment. It can be installed separately either using [Bookkeeper-Operator](https://github.com/pravega/bookkeeper-operator) or some other means. @@ -97,14 +99,12 @@ To install cert-manager check [this](https://cert-manager.io/docs/installation/k 3. [Bookkeeper Operator](https://github.com/pravega/bookkeeper-operator/tree/master/charts/bookkeeper-operator) must be deployed in the same namespace as Pravega Operator, prior to triggering the upgrade. Also, Bookkeeper operator version map should contain the bookkeeper versions of the installed bookkeeper. 4. Install an Issuer and a Certificate (either self-signed or CA signed) in the same namespace as the Pravega Operator (refer to [this](https://github.com/pravega/pravega-operator/blob/master/deploy/certificate.yaml) manifest to create a self-signed certificate in the default namespace). -> The name of the certificate (*webhookCert.certName*), the name of the secret created by this certificate (*webhookCert.secretName*), the tls.crt (*webhookCert.crt*) and tls.key (*webhookCert.key*) need to be specified against the corresponding fields in the values.yaml file, or can be provided with the upgrade command as shown [here](#triggering-the-upgrade). -The values *tls.crt* and *tls.key* are contained in the secret which is created by the certificate and can be obtained using the following command + +5. Execute the script `pre-upgrade.sh` inside the [scripts](https://github.com/pravega/pravega-operator/blob/master/scripts) folder. This script patches the `pravega-webhook-svc` with the required annotations and labels. The format of the command is ``` -kubectl get secret -o yaml | grep tls. +./pre-upgrade.sh [PRAVEGA_OPERATOR_RELEASE_NAME][PRAVEGA_OPERATOR_NAMESPACE] ``` -5. Execute the script `pre-upgrade.sh` inside the [scripts](https://github.com/pravega/pravega-operator/blob/master/scripts) folder. This script patches the `pravega-webhook-svc` with the required annotations and labels. - ### Triggering the upgrade @@ -112,25 +112,30 @@ kubectl get secret -o yaml | grep tls. The upgrade to Operator 0.5.0 can be triggered using the following command ``` -helm upgrade --set webhookCert.crt= --set webhookCert.generate=false --set webhookCert.certName= --set webhookCert.secretName= +helm upgrade [PRAVEGA_OPERATOR_RELEASE_NAME] pravega/pravega-operator --version=0.5.0 --set webhookCert.crt=[TLS_CRT] --set webhookCert.generate=false --set webhookCert.certName=[CERT_NAME] --set webhookCert.secretName=[SECRET_NAME] ``` +where: +- `[CERT_NAME]` is the name of the certificate that has been created +- `[SECRET_NAME]` is the name of the secret created by the above certificate +- `[TLS_CRT]` is contained in the above secret and can be obtained using the command `kubectl get secret [SECRET_NAME] -o yaml | grep tls.crt` + -Once the upgrade completes and the following command starts returning a response instead of throwing an error message (this might take around 7 to 10 minutes after the operator upgrade has been done) +Wait for the upgrade to complete (which can be determined once the following command starts returning a response instead of throwing an error message). This might take around 7 to 10 minutes after the operator upgrade has been done. ``` -kubectl describe PravegaCluster +kubectl describe PravegaCluster [CLUSTER_NAME] ``` -Execute the script `post-upgrade.sh` inside the [scripts](https://github.com/pravega/pravega-operator/blob/master/scripts) folder. The format of the command is +Next, execute the script `post-upgrade.sh` inside the [scripts](https://github.com/pravega/pravega-operator/blob/master/scripts) folder. The format of the command is ``` -./post-upgrade.sh +./post-upgrade.sh [CLUSTER_NAME] [PRAVEGA_RELEASE_NAME] [BOOKKEEPER_RELEASE_NAME] [VERSION] [NAMESPACE] [ZOOKEEPER_SVC_NAME] [BOOKKEEPER_REPLICA_COUNT] ``` -This script patches the `PravegaCluster` and `BookkeeperCluster` resources with the required annotations and labels, and updates their corresponding helm releases. This script needs the following arguments -1. Name of the PravegaCluster or BookkeeperCluster resource (check the output of `kubectl get PravegaCluster` to obtain this name). -2. Name of the release that has been created for the v1alpha1 PravegaCluster resource (check the output of `helm ls` to obtain this name). -3. Name of the release that needs to be created for the BookkeeperCluster resource. -4. Version of the PravegaCluster or BookkeeperCluster resources (check the output of `kubectl get PravegaCluster` to obtain the version number). -5. Namespace in which PravegaCluster and BookkeeperCluster resources are deployed (this is an optional parameter and its default value is `default`). -6. Name of the zookeeper client service (this is an optional parameter and its default value is `zookeeper-client`). -7. Number of replicas in the BookkeeperCluster (this is an optional parameter and its default value is 3). +This script patches the `PravegaCluster` and the newly created `BookkeeperCluster` resources with the required annotations and labels, and updates their corresponding helm releases. This script needs the following arguments +1. **[CLUSTER_NAME]** is the name of the PravegaCluster or BookkeeperCluster (check the output of `kubectl get PravegaCluster` to obtain this name). +2. **[PRAVEGA_RELEASE_NAME]** is the release name corresponding to the v1alpha1 PravegaCluster chart (check the output of `helm ls` to obtain this name). +3. **[BOOKKEEPER_RELEASE_NAME]** is the name of the release that needs to be created for the BookkeeperCluster chart. +4. **[VERSION]** is the version of the PravegaCluster or BookkeeperCluster resources (check the output of `kubectl get PravegaCluster` to obtain the version number). +5. **[NAMESPACE]** is the namespace in which PravegaCluster and BookkeeperCluster resources are deployed (this is an optional parameter and its default value is `default`). +6. **[ZOOKEEPER_SVC_NAME]** is the name of the zookeeper client service (this is an optional parameter and its default value is `zookeeper-client`). +7. **[BOOKKEEPER_REPLICA_COUNT]** is the number of replicas in the BookkeeperCluster (this is an optional parameter and its default value is `3`). #### Upgrade manually diff --git a/doc/rollback-cluster.md b/doc/rollback-cluster.md index bbe0c37e8..c6a775402 100644 --- a/doc/rollback-cluster.md +++ b/doc/rollback-cluster.md @@ -24,10 +24,10 @@ Status: True Reason: UpgradeFailed Message:
``` -After an Upgrade Failure the output of `kubectl describe pravegacluster pravega` would look like this: +After an Upgrade Failure the output of `kubectl describe pravegacluster [CLUSTER_NAME]` would look like this: ``` -$> kubectl describe pravegacluster pravega +$> kubectl describe pravegacluster bar-pravega . . . Spec: . . . @@ -46,7 +46,7 @@ Conditions: Type: PodsReady Last Transition Time: 2019-09-06T09:00:13Z Last Update Time: 2019-09-06T09:00:13Z - Message: failed to sync segmentstore version. pod pravega-pravega-segmentstore-0 update failed because of ImagePullBackOff + Message: failed to sync segmentstore version. pod bar-pravega-pravega-segmentstore-0 update failed because of ImagePullBackOff Reason: UpgradeFailed Status: True Type: Error @@ -60,7 +60,7 @@ where `0.6.0-2252.b6f6512` is the version we tried upgrading to and `0.6.0-2239. ## Manual Rollback Trigger -A Rollback is triggered when a Pravega Cluster is in `UpgradeFailed` Error State and a user manually updates version feild in the PravegaCluster spec to point to the last stable cluster version. +A Rollback is triggered when a Pravega Cluster is in `UpgradeFailed` Error State and a user manually updates version field in the PravegaCluster spec to point to the last stable cluster version. A Rollback involves moving all components in the cluster back to the last stable cluster version. As with upgrades, the operator rolls back one component at a time and one pod at a time to preserve high-availability. @@ -68,27 +68,28 @@ Note: 1. A Rollback to only the last stable cluster version is supported at this point. 2. Changing the cluster spec version to the previous cluster version, when cluster is not in `UpgradeFailed` state, will not trigger a rollback. -## Rollback via Helm +## Rollback via Helm (Experimental) The following command prints the historical revisions of a particular helm release ``` -$ helm history +$ helm history [PRAVEGA_RELEASE_NAME] ``` Rollback can be triggered via helm using the following command ``` -$ helm rollback +$ helm rollback [PRAVEGA_RELEASE_NAME] [REVISION_NUMBER] --wait --timeout 600s ``` -Rollback will be successfully triggered only if the previous revision number is provided. +Rollback will be successfully triggered only if a [REVISION_NUMBER] corresponding to the last stable cluster version is provided. +>Note: Helm rollbacks are still an experimental feature and are not encouraged. We strongly recommend using manual rollbacks. ## Rollback Implementation When Rollback is triggered the cluster moves into ClusterCondition `RollbackInProgress`. Once the Rollback completes, this condition is set to false. -During a Rollback, the Cluster Status should look something like: +During a Rollback, the Cluster Status should look something like this: ``` -$> kubectl describe pravegacluster pravega +$> kubectl describe pravegacluster bar-pravega . . . Status: Conditions: @@ -102,7 +103,7 @@ Status: Type: PodsReady Last Transition Time: 2019-09-20T10:41:10Z Last Update Time: 2019-09-20T10:41:10Z - Message: failed to sync segmentstore version. pod pravega-pravega-segmentstore-0 update failed because of ImagePullBackOff + Message: failed to sync segmentstore version. pod bar-pravega-pravega-segmentstore-0 update failed because of ImagePullBackOff Reason: UpgradeFailed Status: True Type: Error @@ -114,9 +115,9 @@ Status: . . . ``` Here the `RollbackInProgress` condition being `true` indicates that a Rollback is in Progress. -Also `Reason` and `Message` feilds of this condition indicate the component being rolled back and number of updated replicas respectively. +Also `Reason` and `Message` fields of this condition indicate the component being rolled back and number of updated replicas respectively. -The operator rolls back components following the reverse upgrade order : +The operator rolls back components following the reverse upgrade order (only if number of segmentstore replicas is greater than 1): 1. Pravega Controller 2. Pravega Segment Store @@ -180,7 +181,7 @@ Status: Type: PodsReady Last Transition Time: 2019-09-20T09:46:24Z Last Update Time: 2019-09-20T09:50:57Z - Message: failed to sync segmentstore version. pod pravega-pravega-segmentstore-0 update failed because of ImagePullBackOff + Message: failed to sync segmentstore version. pod bar-pravega-pravega-segmentstore-0 update failed because of ImagePullBackOff Reason: RollbackFailed Status: True Type: Error diff --git a/doc/upgrade-cluster.md b/doc/upgrade-cluster.md index 274f0b8d5..185cfe55d 100644 --- a/doc/upgrade-cluster.md +++ b/doc/upgrade-cluster.md @@ -22,8 +22,8 @@ Your Pravega cluster should be in a healthy state. You can check your cluster he ``` $ kubectl get PravegaCluster -NAME VERSION DESIRED MEMBERS READY MEMBERS AGE -pravega 0.4.0 7 7 11m +NAME VERSION DESIRED MEMBERS READY MEMBERS AGE +bar-pravega 0.4.0 7 7 11m ``` ## Valid Upgrade Paths @@ -34,18 +34,18 @@ To understand the valid upgrade paths for a pravega cluster, refer to the [versi ### Upgrading via Helm -The upgrade can be triggered via helm using the following command +The upgrade of the pravega cluster from a version **[OLD_VERSION]** to **[NEW_VERSION]** can be triggered via helm using the following command ``` -$ helm upgrade --reuse-values --timeout 600s +$ helm upgrade [PRAVEGA_RELEASE_NAME] pravega/pravega --version=[NEW_VERSION] --set version=[NEW_VERSION] --reuse-values --timeout 600s ``` -By specifying the `--reuse-values` option, the values of all parameters are retained across upgrades. However if some values need to be modified during the upgrade, the `--set` flag can be used to specify the new values of these parameters. +**Note:** By specifying the `--reuse-values` option, the configuration of all parameters are retained across upgrades. However if some values need to be modified during the upgrade, the `--set` flag can be used to specify the new configuration for these parameters. Also, by skipping the `reuse-values` flag, the values of all parameters are reset to the default configuration that has been specified in the published charts for version [NEW_VERSION]. ### Upgrading manually To initiate the upgrade process manually, a user has to update the `spec.version` field on the `PravegaCluster` custom resource. This can be done in three different ways using the `kubectl` command. -1. `kubectl edit PravegaCluster `, modify the `version` value in the YAML resource, save, and exit. +1. `kubectl edit PravegaCluster [CLUSTER_NAME]`, modify the `version` value in the YAML resource, save, and exit. 2. If you have the custom resource defined in a local YAML file, e.g. `pravega.yaml`, you can modify the `version` value, and reapply the resource with `kubectl apply -f pravega.yaml`. -3. `kubectl patch PravegaCluster --type='json' -p='[{"op": "replace", "path": "/spec/version", "value": "X.Y.Z"}]'`. +3. `kubectl patch PravegaCluster [CLUSTER_NAME] --type='json' -p='[{"op": "replace", "path": "/spec/version", "value": "X.Y.Z"}]'`. After the `version` field is updated, the operator will detect the version change and it will trigger the upgrade process. @@ -146,21 +146,21 @@ You can monitor your upgrade process by listing the Pravega clusters. If a desir ``` $ kubectl get PravegaCluster -NAME VERSION DESIRED VERSION DESIRED MEMBERS READY MEMBERS AGE -pravega 0.4.0 0.5.0 5 4 1h +NAME VERSION DESIRED VERSION DESIRED MEMBERS READY MEMBERS AGE +bar-pravega 0.4.0 0.5.0 5 4 1h ``` When the upgrade process has finished, the version will be updated. ``` $ kubectl get PravegaCluster -NAME VERSION DESIRED MEMBERS READY MEMBERS AGE -pravega 0.5.0 5 5 1h +NAME VERSION DESIRED MEMBERS READY MEMBERS AGE +bar-pravega 0.5.0 5 5 1h ``` The command `kubectl describe` can be used to track progress of the upgrade. ``` -$ kubectl describe PravegaCluster pravega +$ kubectl describe PravegaCluster bar-pravega ... Status: Conditions: @@ -184,7 +184,7 @@ The `Reason` field in Upgrading Condition shows the component currently being up If upgrade has failed, please check the `Status` section to understand the reason for failure. ``` -$ kubectl describe PravegaCluster pravega +$ kubectl describe PravegaCluster bar-pravega ... Status: Conditions: @@ -196,7 +196,7 @@ Status: Type: PodsReady Last Transition Time: 2019-04-01T19:43:08+02:00 Last Update Time: 2019-04-01T19:43:08+02:00 - Message: failed to sync segmentstore version. pod pravega-pravega-segmentstore-0 is restarting + Message: failed to sync segmentstore version. pod bar-pravega-pravega-segmentstore-0 is restarting Reason: UpgradeFailed Status: True Type: Error @@ -204,12 +204,12 @@ Status: Current Version: 0.4.0 Members: Ready: - pravega-pravega-controller-64ff87fc49-kqp9k - pravega-pravega-segmentstore-1 - pravega-pravega-segmentstore-2 - pravega-pravega-segmentstore-3 + bar-pravega-pravega-controller-64ff87fc49-kqp9k + bar-pravega-pravega-segmentstore-1 + bar-pravega-pravega-segmentstore-2 + bar-pravega-pravega-segmentstore-3 Unready: - pravega-pravega-segmentstore-0 + bar-pravega-pravega-segmentstore-0 Ready Replicas: 4 Replicas: 5 ``` @@ -219,16 +219,15 @@ You can also find useful information at the operator logs. ``` ... INFO[5884] syncing cluster version from 0.4.0 to 0.5.0-1 -INFO[5885] Reconciling PravegaCluster default/pravega -INFO[5886] updating statefulset (pravega-pravega-segmentstore) template image to 'adrianmo/pravega:0.5.0-1' -INFO[5896] Reconciling PravegaCluster default/pravega -INFO[5897] statefulset (pravega-bookie) status: 0 updated, 4 ready, 4 target -INFO[5897] upgrading pod: pravega-pravega-segmentstore-0 -INFO[5899] Reconciling PravegaCluster default/pravega -INFO[5900] statefulset (pravega-pravega-segmentstore) status: 1 updated, 3 ready, 4 target -INFO[5929] Reconciling PravegaCluster default/pravega -INFO[5930] statefulset (pravega-pravega-segmentstore) status: 1 updated, 3 ready, 4 target -INFO[5930] error syncing cluster version, upgrade failed. failed to sync segmentstore version. pod pravega-pravega-segmentstore-0 is restarting +INFO[5885] Reconciling PravegaCluster default/bar-pravega +INFO[5886] updating statefulset (bar-pravega-pravega-segmentstore) template image to 'pravega/pravega:0.5.0-1' +INFO[5896] Reconciling PravegaCluster default/bar-pravega +INFO[5897] upgrading pod: bar-pravega-pravega-segmentstore-0 +INFO[5899] Reconciling PravegaCluster default/bar-pravega +INFO[5900] statefulset (bar-pravega-pravega-segmentstore) status: 1 updated, 3 ready, 4 target +INFO[5929] Reconciling PravegaCluster default/bar-pravega +INFO[5930] statefulset (bar-pravega-pravega-segmentstore) status: 1 updated, 3 ready, 4 target +INFO[5930] error syncing cluster version, upgrade failed. failed to sync segmentstore version. pod bar-pravega-pravega-segmentstore-0 is restarting ... ```