diff --git a/datahub-kubernetes/README.md b/datahub-kubernetes/README.md index 15b7338f198702..0e59211fae4b77 100644 --- a/datahub-kubernetes/README.md +++ b/datahub-kubernetes/README.md @@ -1,135 +1,14 @@ ---- -title: "Deploying with Kubernetes" ---- - -# Deploying Datahub with Kubernetes - -## Introduction -[This directory](https://github.com/linkedin/datahub/tree/master/datahub-kubernetes) provides -the Kubernetes [Helm](https://helm.sh/) charts for deploying [Datahub](https://github.com/linkedin/datahub/tree/master/datahub-kubernetes/datahub) and it's [dependencies](https://github.com/linkedin/datahub/tree/master/datahub-kubernetes/prerequisites) -(Elasticsearch, optionally Neo4j, MySQL, and Kafka) on a Kubernetes cluster. - -## Setup -1. Set up a kubernetes cluster - - In a cloud platform of choice like [Amazon EKS](https://aws.amazon.com/eks), - [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine), - and [Azure Kubernetes Service](https://azure.microsoft.com/en-us/services/kubernetes-service/) OR - - In local environment using [Minikube](https://minikube.sigs.k8s.io/docs/). - Note, more than 7GB of RAM is required to run Datahub and it's dependencies -2. Install the following tools: - - [kubectl](https://kubernetes.io/docs/tasks/tools/) to manage kubernetes resources - - [helm](https://helm.sh/docs/intro/install/) to deploy the resources based on helm charts. - Note, we only support Helm 3. - -## Components -Datahub consists of 4 main components: [GMS](https://datahubproject.io/docs/gms), -[MAE Consumer](https://datahubproject.io/docs/metadata-jobs/mae-consumer-job), -[MCE Consumer](https://datahubproject.io/docs/metadata-jobs/mce-consumer-job), and -[Frontend](https://datahubproject.io/docs/datahub-frontend). Kubernetes deployment -for each of the components are defined as subcharts under the main -[Datahub](https://github.com/linkedin/datahub/tree/master/datahub-kubernetes/datahub) -helm chart. - -The main components are powered by 4 external dependencies: -- Kafka -- Local DB (MySQL, Postgres, MariaDB) -- Search Index (Elasticsearch) -- Graph Index (Supports either Neo4j or Elasticsearch) - -The dependencies must be deployed before deploying Datahub. We created a separate -[chart](https://github.com/linkedin/datahub/tree/master/datahub-kubernetes/prerequisites) -for deploying the dependencies with example configuration. They could also be deployed -separately on-prem or leveraged as managed services. To remove your dependency on Neo4j, -set enabled to false in the `datahub-kubernetes/prerequisites/values.yaml` file. -Then, override the `graph_service_impl` field in `datahub-kubernetes/datahub/values.yaml` to -have the value `elasticsearch` instead of `neo4j`. - -## Quickstart -Assuming kubectl context points to the correct kubernetes cluster, first create kubernetes secrets that contain MySQL and Neo4j passwords. - -```(shell) -kubectl create secret generic mysql-secrets --from-literal=mysql-root-password=datahub -kubectl create secret generic neo4j-secrets --from-literal=neo4j-password=datahub -``` - -The above commands sets the passwords to "datahub" as an example. Change to any password of choice. - -Second, deploy the dependencies by running the following - -```(shell) -(cd prerequisites && helm dep update) -helm install prerequisites prerequisites/ -``` - -Note, after changing the configurations in the values.yaml file, you can run - -```(shell) -helm upgrade prerequisites prerequisites/ -``` - -To just redeploy the dependencies impacted by the change. - -Run `kubectl get pods` to check whether all the pods for the dependencies are running. -You should get a result similar to below. - -``` -NAME READY STATUS RESTARTS AGE -elasticsearch-master-0 1/1 Running 0 62m -elasticsearch-master-1 1/1 Running 0 62m -elasticsearch-master-2 1/1 Running 0 62m -prerequisites-cp-schema-registry-cf79bfccf-kvjtv 2/2 Running 1 63m -prerequisites-kafka-0 1/1 Running 2 62m -prerequisites-mysql-0 1/1 Running 1 62m -prerequisites-neo4j-community-0 1/1 Running 0 52m -prerequisites-zookeeper-0 1/1 Running 0 62m -``` - -deploy Datahub by running the following - -```(shell) -helm install datahub datahub/ -``` - -Values in [values.yaml](https://github.com/linkedin/datahub/tree/master/datahub-kubernetes/datahub/values.yaml) -have been preset to point to the dependencies deployed using the [prerequisites](https://github.com/linkedin/datahub/tree/master/datahub-kubernetes/prerequisites) -chart with release name "prerequisites". If you deployed the helm chart using a different release name, update the quickstart-values.yaml file accordingly before installing. - -Run `kubectl get pods` to check whether all the datahub pods are running. You should get a result similar to below. - -``` -NAME READY STATUS RESTARTS AGE -datahub-datahub-frontend-84c58df9f7-5bgwx 1/1 Running 0 4m2s -datahub-datahub-gms-58b676f77c-c6pfx 1/1 Running 0 4m2s -datahub-datahub-mae-consumer-7b98bf65d-tjbwx 1/1 Running 0 4m3s -datahub-datahub-mce-consumer-8c57d8587-vjv9m 1/1 Running 0 4m2s -datahub-elasticsearch-setup-job-8dz6b 0/1 Completed 0 4m50s -datahub-kafka-setup-job-6blcj 0/1 Completed 0 4m40s -datahub-mysql-setup-job-b57kc 0/1 Completed 0 4m7s -elasticsearch-master-0 1/1 Running 0 97m -elasticsearch-master-1 1/1 Running 0 97m -elasticsearch-master-2 1/1 Running 0 97m -prerequisites-cp-schema-registry-cf79bfccf-kvjtv 2/2 Running 1 99m -prerequisites-kafka-0 1/1 Running 2 97m -prerequisites-mysql-0 1/1 Running 1 97m -prerequisites-neo4j-community-0 1/1 Running 0 88m -prerequisites-zookeeper-0 1/1 Running 0 97m -``` - -You can run the following to expose the frontend locally. Note, you can find the pod name using the command above. -In this case, the datahub-frontend pod name was `datahub-datahub-frontend-84c58df9f7-5bgwx`. - -```(shell) -kubectl port-forward 9002:9002 -``` - -You should be able to access the frontend via http://localhost:9002. - -Once you confirm that the pods are running well, you can set up ingress for datahub-frontend -to expose the 9002 port to the public. -## Other useful commands - -| Command | Description | -|-----|------| -| helm uninstall datahub | Remove DataHub | -| helm ls | List of Helm charts | -| helm history | Fetch a release history | +DataHub Helm Charts +============================================================================== + +> **Notice**: As of July 2021, we have migrated the helm charts to a separate [repo](https://github.com/acryldata/datahub-helm) +> to allow automatic releases and better control over the discoverability of the charts. +> +> We have also published the helm charts to `https://helm.datahubproject.io`. You can deploy without referring to the codebase by running the following commands. +> +> ``` +> helm repo add datahub https://helm.datahubproject.io +> helm install datahub datahub/datahub +>``` +> +> Please contribute any changes to the new repo. diff --git a/datahub-kubernetes/datahub/.helmignore b/datahub-kubernetes/datahub/.helmignore deleted file mode 100644 index bf01528c256950..00000000000000 --- a/datahub-kubernetes/datahub/.helmignore +++ /dev/null @@ -1,24 +0,0 @@ -# Patterns to ignore when building packages. -# This supports shell glob matching, relative path matching, and -# negation (prefixed with !). Only one pattern per line. -.DS_Store -# Common VCS dirs -.git/ -.gitignore -.bzr/ -.bzrignore -.hg/ -.hgignore -.svn/ -# Common backup files -*.swp -*.bak -*.tmp -*.orig -*~ -# Various IDEs -.project -.idea/ -*.tmproj -.vscode/ -charts/*.tgz \ No newline at end of file diff --git a/datahub-kubernetes/datahub/Chart.yaml b/datahub-kubernetes/datahub/Chart.yaml deleted file mode 100644 index 7afac46ee7741e..00000000000000 --- a/datahub-kubernetes/datahub/Chart.yaml +++ /dev/null @@ -1,31 +0,0 @@ -apiVersion: v2 -name: datahub -description: A Helm chart for LinkedIn DataHub -type: application -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -version: 0.2.4 -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. -appVersion: 0.8.5 #0.3.1 -dependencies: - - name: datahub-gms - version: 0.2.2 - repository: file://./charts/datahub-gms - condition: datahub-gms.enabled - - name: datahub-frontend - version: 0.2.1 - repository: file://./charts/datahub-frontend - condition: datahub-frontend.enabled - - name: datahub-mae-consumer - version: 0.2.1 - repository: file://./charts/datahub-mae-consumer - condition: global.datahub_standalone_consumers_enabled - - name: datahub-mce-consumer - version: 0.2.1 - repository: file://./charts/datahub-mce-consumer - condition: global.datahub_standalone_consumers_enabled - - name: datahub-ingestion-cron - version: 0.2.1 - repository: file://./charts/datahub-ingestion-cron - condition: datahub-ingestion-cron.enabled \ No newline at end of file diff --git a/datahub-kubernetes/datahub/README.md b/datahub-kubernetes/datahub/README.md deleted file mode 100644 index 359464af9b7e91..00000000000000 --- a/datahub-kubernetes/datahub/README.md +++ /dev/null @@ -1,73 +0,0 @@ -datahub -======= -A Helm chart for LinkedIn DataHub - -Current chart version is `0.1.2` - -## Install DataHub -Navigate to the current directory and run the below command. Update the `datahub/values.yaml` file with valid hostname/IP address configuration for elasticsearch, neo4j, schema-registry, broker & mysql. - -`` -helm install datahub datahub/ -`` - -## Chart Values - -| Key | Type | Default | Description | -|-----|------|---------|-------------| -| datahub-frontend.enabled | bool | `true` | Enable Datahub Front-end | -| datahub-frontend.image.repository | string | `"linkedin/datahub-frontend-react"` | Image repository for datahub-frontend | -| datahub-frontend.image.tag | string | `"v0.8.5"` | Image tag for datahub-frontend | -| datahub-gms.enabled | bool | `true` | Enable GMS | -| datahub-gms.image.repository | string | `"linkedin/datahub-gms"` | Image repository for datahub-gms | -| datahub-gms.image.tag | string | `"v0.8.5"` | Image tag for datahub-gms | -| datahub-mae-consumer.enabled | bool | `true` | Enable MAE Consumer | -| datahub-mae-consumer.image.repository | string | `"linkedin/datahub-mae-consumer"` | Image repository for datahub-mae-consumer | -| datahub-mae-consumer.image.tag | string | `"v0.8.5"` | Image tag for datahub-mae-consumer | -| datahub-mce-consumer.enabled | bool | `true` | Enable MCE Consumer | -| datahub-mce-consumer.image.repository | string | `"linkedin/datahub-mce-consumer"` | Image repository for datahub-mce-consumer | -| datahub-mce-consumer.image.tag | string | `"v0.8.5"` | Image tag for datahub-mce-consumer | -| datahub-ingestion-cron.enabled | bool | `false` | Enable cronjob for periodic ingestion | -| elasticsearchSetupJob.enabled | bool | `true` | Enable setup job for elasicsearch | -| elasticsearchSetupJob.image.repository | string | `"linkedin/datahub-elasticsearch-setup"` | Image repository for elasticsearchSetupJob | -| elasticsearchSetupJob.image.tag | string | `"v0.8.5"` | Image repository for elasticsearchSetupJob | -| kafkaSetupJob.enabled | bool | `true` | Enable setup job for kafka | -| kafkaSetupJob.image.repository | string | `"linkedin/datahub-kafka-setup"` | Image repository for kafkaSetupJob | -| kafkaSetupJob.image.tag | string | `"v0.8.5"` | Image repository for kafkaSetupJob | -| mysqlSetupJob.enabled | bool | `false` | Enable setup job for mysql | -| mysqlSetupJob.image.repository | string | `""` | Image repository for mysqlSetupJob | -| mysqlSetupJob.image.tag | string | `""` | Image repository for mysqlSetupJob | -| global.datahub_analytics_enabled | boolean | true | Enable datahub usage analytics | -| global.datahub.appVersion | string | `"1.0"` | App version for annotation | -| global.datahub.gms.port | string | `"8080"` | Port of GMS service | -| global.elasticsearch.host | string | `"elasticsearch-master"` | Elasticsearch host name (endpoint) | -| global.elasticsearch.port | string | `"9200"` | Elasticsearch port | -| global.kafka.bootstrap.server | string | `"prerequisites-broker:9092"` | Kafka bootstrap servers (with port) | -| global.kafka.zookeeper.server | string | `"prerequisites-zookeeper:2181"` | Kafka zookeeper servers (with port) | -| global.kafka.schemaregistry.url | string | `"http://prerequisites-cp-schema-registry:8081"` | URL to kafka schema registry | -| global.neo4j.host | string | `"prerequisites-neo4j:7474"` | Neo4j host address (with port) | -| global.neo4j.uri | string | `"bolt://prerequisites-neo4j"` | Neo4j URI | -| global.neo4j.username | string | `"neo4j"` | Neo4j user name | -| global.neo4j.password.secretRef | string | `"neo4j-secrets"` | Secret that contains the Neo4j password | -| global.neo4j.password.secretKey | string | `"neo4j-password"` | Secret key that contains the Neo4j password | -| global.sql.datasource.driver | string | `"com.mysql.jdbc.Driver"` | Driver for the SQL database | -| global.sql.datasource.host | string | `"prerequisites-mysql:3306"` | SQL database host (with port) | -| global.sql.datasource.hostForMysqlClient | string | `"prerequisites-mysql"` | SQL database host (without port) | -| global.sql.datasource.url | string | `"jdbc:mysql://prerequisites-mysql:3306/datahub?verifyServerCertificate=false\u0026useSSL=true"` | URL to access SQL database | -| global.sql.datasource.username | string | `"root"` | SQL user name | -| global.sql.datasource.password.secretRef | string | `"mysql-secrets"` | Secret that contains the MySQL password | -| global.sql.datasource.password.secretKey | string | `"mysql-password"` | Secret key that contains the MySQL password | -| global.graph_service_impl | string | `neo4j` | One of `neo4j` or `elasticsearch`. Determines which backend to use for the GMS graph service. Elastic is recommended for a simplified deployment. Neo4j will be the default for now to maintain backwards compatibility. - -## Optional Chart Values - -| Key | Type | Default | Description | -|-----|------|---------|-------------| -| global.credentialsAndCertsSecrets.name | string | `""` | Name of the secret that holds SSL certificates (keystores, truststores) | -| global.credentialsAndCertsSecrets.path | string | `"/mnt/certs"` | Path to mount the SSL certificates | -| global.credentialsAndCertsSecrets.secureEnv | map | `{}` | Map of SSL config name and the corresponding value in the secret | -| global.springKafkaConfigurationOverrides | map | `{}` | Map of configuration overrides for accessing kafka | -| global.elasticsearch.useSSL | bool | `false` | Whether to enable SSL for accessing elasticsearch | -| global.elasticsearch.auth.username | string | `""` | Elasticsearch username | -| global.elasticsearch.auth.password.secretRef | string | `""` | Secret that contains the elasticsearch password | -| global.elasticsearch.auth.password.secretKey | string | `""` | Secret key that contains the elasticsearch password | diff --git a/datahub-kubernetes/datahub/charts/datahub-frontend/.helmignore b/datahub-kubernetes/datahub/charts/datahub-frontend/.helmignore deleted file mode 100644 index 0e8a0eb36f4ca2..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-frontend/.helmignore +++ /dev/null @@ -1,23 +0,0 @@ -# Patterns to ignore when building packages. -# This supports shell glob matching, relative path matching, and -# negation (prefixed with !). Only one pattern per line. -.DS_Store -# Common VCS dirs -.git/ -.gitignore -.bzr/ -.bzrignore -.hg/ -.hgignore -.svn/ -# Common backup files -*.swp -*.bak -*.tmp -*.orig -*~ -# Various IDEs -.project -.idea/ -*.tmproj -.vscode/ diff --git a/datahub-kubernetes/datahub/charts/datahub-frontend/Chart.yaml b/datahub-kubernetes/datahub/charts/datahub-frontend/Chart.yaml deleted file mode 100644 index 0006f88f5500dd..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-frontend/Chart.yaml +++ /dev/null @@ -1,21 +0,0 @@ -apiVersion: v2 -name: datahub-frontend -description: A Helm chart for Kubernetes - -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. -type: application - -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -version: 0.2.1 - -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. -appVersion: 0.3.1 diff --git a/datahub-kubernetes/datahub/charts/datahub-frontend/README.md b/datahub-kubernetes/datahub/charts/datahub-frontend/README.md deleted file mode 100644 index e697f7bdba7f62..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-frontend/README.md +++ /dev/null @@ -1,52 +0,0 @@ -datahub-frontend -================ -A Helm chart for datahub-frontend - -Current chart version is `0.2.0` - -## Chart Values - -| Key | Type | Default | Description | -|-----|------|---------|-------------| -| affinity | object | `{}` | | -| datahub.play.mem.buffer.size | string | `"10MB"` | | -| existingGmsSecret | object | {} | Reference to GMS secret if already exists | -| exporters.jmx.enabled | boolean | false | | -| extraEnvs | Extra [environment variables][] which will be appended to the `env:` definition for the container | `[]` | -| extraVolumes | Templatable string of additional `volumes` to be passed to the `tpl` function | "" | -| extraVolumeMounts | Templatable string of additional `volumeMounts` to be passed to the `tpl` function | "" | -| fullnameOverride | string | `"datahub-frontend"` | | -| global.datahub_analytics_enabled | boolean | true | | -| global.datahub.gms.port | string | `"8080"` | | -| image.pullPolicy | string | `"IfNotPresent"` | | -| image.repository | string | `"linkedin/datahub-frontend-react"` | | -| image.tag | string | `"v0.8.5"` | | -| imagePullSecrets | list | `[]` | | -| ingress.annotations | object | `{}` | | -| ingress.enabled | bool | `false` | | -| ingress.hosts[0].host | string | `"chart-example.local"` | | -| ingress.hosts[0].paths | list | `[]` | | -| ingress.hosts[0].redirectPaths | list | `[]` | | -| ingress.tls | list | `[]` | | -| livenessProbe.initialDelaySeconds | int | `60` | | -| livenessProbe.periodSeconds | int | `30` | | -| livenessProbe.failureThreshold | int | `4` | | -| nameOverride | string | `""` | | -| nodeSelector | object | `{}` | | -| podAnnotations | object | `{}` | | -| podSecurityContext | object | `{}` | | -| readinessProbe.initialDelaySeconds | int | `60` | | -| readinessProbe.periodSeconds | int | `30` | | -| readinessProbe.failureThreshold | int | `4` | | -| replicaCount | int | `1` | | -| resources | object | `{}` | | -| securityContext | object | `{}` | | -| service.port | int | `9001` | | -| service.type | string | `"LoadBalancer"` | | -| serviceAccount.annotations | object | `{}` | | -| serviceAccount.create | bool | `true` | | -| serviceAccount.name | string | `nil` | | -| tolerations | list | `[]` | | -| global.elasticsearch.host | string | `"elasticsearch"` | | -| global.elasticsearch.port | string | `"9200"` | | -| global.kafka.bootstrap.server | string | `"broker:9092"` | | diff --git a/datahub-kubernetes/datahub/charts/datahub-frontend/templates/NOTES.txt b/datahub-kubernetes/datahub/charts/datahub-frontend/templates/NOTES.txt deleted file mode 100644 index dfc2ba5e86a9a6..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-frontend/templates/NOTES.txt +++ /dev/null @@ -1,21 +0,0 @@ -1. Get the application URL by running these commands: -{{- if .Values.ingress.enabled }} -{{- range $host := .Values.ingress.hosts }} - {{- range .paths }} - http{{ if $.Values.ingress.tls }}s{{ end }}://{{ $host.host }}{{ . }} - {{- end }} -{{- end }} -{{- else if contains "NodePort" .Values.service.type }} - export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "datahub-frontend.fullname" . }}) - export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}") - echo http://$NODE_IP:$NODE_PORT -{{- else if contains "LoadBalancer" .Values.service.type }} - NOTE: It may take a few minutes for the LoadBalancer IP to be available. - You can watch the status of by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "datahub-frontend.fullname" . }}' - export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "datahub-frontend.fullname" . }} --template "{{"{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}"}}") - echo http://$SERVICE_IP:{{ .Values.service.port }} -{{- else if contains "ClusterIP" .Values.service.type }} - export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "datahub-frontend.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}") - echo "Visit http://127.0.0.1:8080 to use your application" - kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8080:80 -{{- end }} diff --git a/datahub-kubernetes/datahub/charts/datahub-frontend/templates/_helpers.tpl b/datahub-kubernetes/datahub/charts/datahub-frontend/templates/_helpers.tpl deleted file mode 100644 index 8d0341f5921131..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-frontend/templates/_helpers.tpl +++ /dev/null @@ -1,63 +0,0 @@ -{{/* vim: set filetype=mustache: */}} -{{/* -Expand the name of the chart. -*/}} -{{- define "datahub-frontend.name" -}} -{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} -{{- end -}} - -{{/* -Create a default fully qualified app name. -We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). -If release name contains chart name it will be used as a full name. -*/}} -{{- define "datahub-frontend.fullname" -}} -{{- if .Values.fullnameOverride -}} -{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} -{{- else -}} -{{- $name := default .Chart.Name .Values.nameOverride -}} -{{- if contains $name .Release.Name -}} -{{- .Release.Name | trunc 63 | trimSuffix "-" -}} -{{- else -}} -{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} -{{- end -}} -{{- end -}} -{{- end -}} - -{{/* -Create chart name and version as used by the chart label. -*/}} -{{- define "datahub-frontend.chart" -}} -{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} -{{- end -}} - -{{/* -Common labels -*/}} -{{- define "datahub-frontend.labels" -}} -helm.sh/chart: {{ include "datahub-frontend.chart" . }} -{{ include "datahub-frontend.selectorLabels" . }} -{{- if .Chart.AppVersion }} -app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} -{{- end }} -app.kubernetes.io/managed-by: {{ .Release.Service }} -{{- end -}} - -{{/* -Selector labels -*/}} -{{- define "datahub-frontend.selectorLabels" -}} -app.kubernetes.io/name: {{ include "datahub-frontend.name" . }} -app.kubernetes.io/instance: {{ .Release.Name }} -{{- end -}} - -{{/* -Create the name of the service account to use -*/}} -{{- define "datahub-frontend.serviceAccountName" -}} -{{- if .Values.serviceAccount.create -}} - {{ default (include "datahub-frontend.fullname" .) .Values.serviceAccount.name }} -{{- else -}} - {{ default "default" .Values.serviceAccount.name }} -{{- end -}} -{{- end -}} diff --git a/datahub-kubernetes/datahub/charts/datahub-frontend/templates/config-jmx-exporter.yaml b/datahub-kubernetes/datahub/charts/datahub-frontend/templates/config-jmx-exporter.yaml deleted file mode 100644 index f4304f6a7b370c..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-frontend/templates/config-jmx-exporter.yaml +++ /dev/null @@ -1,17 +0,0 @@ -{{- if .Values.exporters.jmx.enabled }} -apiVersion: v1 -kind: ConfigMap -metadata: - name: {{ include "datahub-frontend.fullname" . }}-config-jmx-exporter - labels: - {{- include "datahub-frontend.labels" . | nindent 4 }} -data: - config.yml: |- - hostPort: localhost:{{ .Values.env.JMXPORT }} - lowercaseOutputName: {{ .Values.exporters.jmx.config.lowercaseOutputName }} - lowercaseOutputLabelNames: {{ .Values.exporters.jmx.config.lowercaseOutputLabelNames }} - rules: -{{ .Values.exporters.jmx.config.rules | toYaml | indent 6 }} - ssl: false - startDelaySeconds: {{ .Values.exporters.jmx.config.startDelaySeconds }} -{{- end }} diff --git a/datahub-kubernetes/datahub/charts/datahub-frontend/templates/deployment.yaml b/datahub-kubernetes/datahub/charts/datahub-frontend/templates/deployment.yaml deleted file mode 100644 index e3b531f451831c..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-frontend/templates/deployment.yaml +++ /dev/null @@ -1,163 +0,0 @@ -apiVersion: apps/v1 -kind: Deployment -metadata: - name: {{ include "datahub-frontend.fullname" . }} - labels: - {{- include "datahub-frontend.labels" . | nindent 4 }} -spec: - replicas: {{ .Values.replicaCount }} - selector: - matchLabels: - {{- include "datahub-frontend.selectorLabels" . | nindent 6 }} - template: - metadata: - {{- with .Values.podAnnotations }} - annotations: - {{- toYaml . | nindent 8 }} - {{- end }} - labels: - {{- include "datahub-frontend.selectorLabels" . | nindent 8 }} - spec: - {{- with .Values.imagePullSecrets }} - imagePullSecrets: - {{- toYaml . | nindent 8 }} - {{- end }} - serviceAccountName: {{ include "datahub-frontend.serviceAccountName" . }} - securityContext: - {{- toYaml .Values.podSecurityContext | nindent 8 }} - volumes: - {{- with .Values.global.credentialsAndCertsSecrets }} - - name: datahub-certs-dir - secret: - defaultMode: 0444 - secretName: {{ .name }} - {{- end }} - {{- if .Values.exporters.jmx.enabled }} - - name: config-jmx-exporter - configMap: - name: {{ include "datahub-gms.fullname" . }}-config-jmx-exporter - {{- end }} - {{- with .Values.extraVolumes }} - {{- toYaml . | nindent 8 }} - {{- end }} - initContainers: - {{- with .Values.extraInitContainers }} - {{- toYaml . | nindent 8 }} - {{- end }} - containers: - - name: {{ .Chart.Name }} - securityContext: - {{- toYaml .Values.securityContext | nindent 12 }} - image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" - imagePullPolicy: {{ .Values.image.pullPolicy }} - ports: - - name: http - containerPort: 9002 - protocol: TCP - livenessProbe: - httpGet: - path: /admin - port: http - initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }} - periodSeconds: {{ .Values.livenessProbe.periodSeconds }} - failureThreshold: {{ .Values.livenessProbe.failureThreshold }} - readinessProbe: - httpGet: - path: /admin - port: http - initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }} - periodSeconds: {{ .Values.readinessProbe.periodSeconds }} - failureThreshold: {{ .Values.readinessProbe.failureThreshold }} - env: - - name: DATAHUB_GMS_HOST - value: {{ printf "%s-%s" .Release.Name "datahub-gms" }} - - name: DATAHUB_GMS_PORT - value: "{{ .Values.global.datahub.gms.port }}" - - name: DATAHUB_SECRET - valueFrom: - {{- if .Values.existingGmsSecret }} - secretKeyRef: - name: "{{ .Values.existingGmsSecret.name }}" - key: "{{ .Values.existingGmsSecret.key }}" - {{- else }} - secretKeyRef: - name: {{ printf "%s-gms-secret" .Release.Name }} - key: datahub.gms.secret - {{- end }} - - name: DATAHUB_APP_VERSION - value: "{{ .Values.global.datahub.appVersion }}" - - name: DATAHUB_PLAY_MEM_BUFFER_SIZE - value: "{{ .Values.datahub.play.mem.buffer.size }}" - - name: DATAHUB_ANALYTICS_ENABLED - value: "{{ .Values.global.datahub_analytics_enabled }}" - {{- if .Values.global.datahub_analytics_enabled }} - - name: KAFKA_BOOTSTRAP_SERVER - value: "{{ .Values.global.kafka.bootstrap.server }}" - {{- if .Values.global.springKafkaConfigurationOverrides }} - {{- range $configName, $configValue := .Values.global.springKafkaConfigurationOverrides }} - - name: KAFKA_PROPERTIES_{{ $configName | replace "." "_" | upper }} - value: {{ $configValue }} - {{- end }} - {{- end }} - {{- if .Values.global.credentialsAndCertsSecrets }} - {{- range $envVarName, $envVarValue := .Values.global.credentialsAndCertsSecrets.secureEnv }} - - name: KAFKA_PROPERTIES_{{ $envVarName | replace "." "_" | upper }} - valueFrom: - secretKeyRef: - name: {{ $.Values.global.credentialsAndCertsSecrets.name }} - key: {{ $envVarValue }} - {{- end }} - {{- end }} - - name: ELASTIC_CLIENT_HOST - value: "{{ .Values.global.elasticsearch.host }}" - - name: ELASTIC_CLIENT_PORT - value: "{{ .Values.global.elasticsearch.port }}" - {{- with .Values.global.elasticsearch.useSSL }} - - name: ELASTIC_CLIENT_USE_SSL - value: {{ . | quote }} - {{- end }} - {{- with .Values.global.elasticsearch.auth }} - - name: ELASTIC_CLIENT_USERNAME - value: {{ .username }} - - name: ELASTIC_CLIENT_PASSWORD - valueFrom: - secretKeyRef: - name: "{{ .password.secretRef }}" - key: "{{ .password.secretKey }}" - {{- end }} - {{- if .Values.global.kafka.topics }} - - name: DATAHUB_TRACKING_TOPIC - value: {{ .Values.global.kafka.topics.datahub_usage_event_name}} - {{- else }} - - name: DATAHUB_TRACKING_TOPIC - value: "DataHubUsageEvent_v1" - {{- end }} - {{- end }} - {{- with .Values.extraEnvs }} - {{- toYaml . | nindent 12 }} - {{- end }} - volumeMounts: - {{- with .Values.global.credentialsAndCertsSecrets }} - - name: datahub-certs-dir - mountPath: {{ .path | default "/mnt/certs" }} - {{- end }} - {{- with .Values.extraVolumeMounts }} - {{- toYaml . | nindent 12 }} - {{- end }} - resources: - {{- toYaml .Values.resources | nindent 12 }} - - {{- include "datahub-jmxexporter.container" . }} - - {{- with .Values.nodeSelector }} - nodeSelector: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.affinity }} - affinity: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.tolerations }} - tolerations: - {{- toYaml . | nindent 8 }} - {{- end }} diff --git a/datahub-kubernetes/datahub/charts/datahub-frontend/templates/ingress.yaml b/datahub-kubernetes/datahub/charts/datahub-frontend/templates/ingress.yaml deleted file mode 100644 index 2b0b4700d5e1d6..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-frontend/templates/ingress.yaml +++ /dev/null @@ -1,47 +0,0 @@ -{{- if .Values.ingress.enabled -}} -{{- $fullName := include "datahub-frontend.fullname" . -}} -{{- $svcPort := .Values.service.port -}} -{{- if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}} -apiVersion: networking.k8s.io/v1beta1 -{{- else -}} -apiVersion: extensions/v1beta1 -{{- end }} -kind: Ingress -metadata: - name: {{ $fullName }} - labels: - {{- include "datahub-frontend.labels" . | nindent 4 }} - {{- with .Values.ingress.annotations }} - annotations: - {{- toYaml . | nindent 4 }} - {{- end }} -spec: -{{- if .Values.ingress.tls }} - tls: - {{- range .Values.ingress.tls }} - - hosts: - {{- range .hosts }} - - {{ . | quote }} - {{- end }} - secretName: {{ .secretName }} - {{- end }} -{{- end }} - rules: - {{- range .Values.ingress.hosts }} - - host: {{ .host | quote }} - http: - paths: - {{- range .redirectPaths }} - - path: {{ .path }} - backend: - serviceName: {{ .name }} - servicePort: {{ .port }} - {{- end }} - {{- range .paths }} - - path: {{ . }} - backend: - serviceName: {{ $fullName }} - servicePort: {{ $svcPort }} - {{- end }} - {{- end }} -{{- end }} diff --git a/datahub-kubernetes/datahub/charts/datahub-frontend/templates/secrets.yaml b/datahub-kubernetes/datahub/charts/datahub-frontend/templates/secrets.yaml deleted file mode 100644 index 9faeb32e073f26..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-frontend/templates/secrets.yaml +++ /dev/null @@ -1,11 +0,0 @@ -{{- if not .Values.existingGmsSecret -}} -apiVersion: v1 -kind: Secret -metadata: - name: {{ printf "%s-gms-secret" .Release.Name }} - labels: - {{- include "datahub-frontend.labels" . | nindent 4 }} -type: Opaque -data: - datahub.gms.secret: {{ randAlphaNum 10 | b64enc | quote }} -{{- end -}} \ No newline at end of file diff --git a/datahub-kubernetes/datahub/charts/datahub-frontend/templates/service.yaml b/datahub-kubernetes/datahub/charts/datahub-frontend/templates/service.yaml deleted file mode 100644 index 68c318410ec558..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-frontend/templates/service.yaml +++ /dev/null @@ -1,19 +0,0 @@ -apiVersion: v1 -kind: Service -metadata: - name: {{ include "datahub-frontend.fullname" . }} - labels: - {{- include "datahub-frontend.labels" . | nindent 4 }} - {{- with .Values.service.annotations }} - annotations: - {{- toYaml . | nindent 4 }} - {{- end }} -spec: - type: {{ .Values.service.type }} - ports: - - port: {{ .Values.service.port }} - targetPort: http - protocol: TCP - name: http - selector: - {{- include "datahub-frontend.selectorLabels" . | nindent 4 }} diff --git a/datahub-kubernetes/datahub/charts/datahub-frontend/templates/serviceaccount.yaml b/datahub-kubernetes/datahub/charts/datahub-frontend/templates/serviceaccount.yaml deleted file mode 100644 index 7f9580adac5dbc..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-frontend/templates/serviceaccount.yaml +++ /dev/null @@ -1,12 +0,0 @@ -{{- if .Values.serviceAccount.create -}} -apiVersion: v1 -kind: ServiceAccount -metadata: - name: {{ include "datahub-frontend.serviceAccountName" . }} - labels: - {{- include "datahub-frontend.labels" . | nindent 4 }} - {{- with .Values.serviceAccount.annotations }} - annotations: - {{- toYaml . | nindent 4 }} - {{- end }} -{{- end -}} diff --git a/datahub-kubernetes/datahub/charts/datahub-frontend/values.yaml b/datahub-kubernetes/datahub/charts/datahub-frontend/values.yaml deleted file mode 100644 index 8108bc5974abdb..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-frontend/values.yaml +++ /dev/null @@ -1,172 +0,0 @@ -# Default values for datahub-frontend. -# This is a YAML-formatted file. -# Declare variables to be passed into your templates. - -replicaCount: 1 - -image: - repository: linkedin/datahub-frontend-react - tag: "v0.8.5" - pullPolicy: Always - -imagePullSecrets: [] -nameOverride: "" -fullnameOverride: "" - -serviceAccount: - # Specifies whether a service account should be created - create: true - # Annotations to add to the service account - annotations: {} - # The name of the service account to use. - # If not set and create is true, a name is generated using the fullname template - name: - -podAnnotations: {} - # co.elastic.logs/enabled: "true" - -podSecurityContext: {} - # fsGroup: 2000 - -securityContext: {} - # capabilities: - # drop: - # - ALL - # readOnlyRootFilesystem: true - # runAsNonRoot: true - # runAsUser: 1000 - -service: - type: LoadBalancer - port: 9002 - # Annotations to add to the service, this will help in adding - # Internal load balancer or various other annotation support in AWS - annotations: {} - # service.beta.kubernetes.io/aws-load-balancer-internal: "true" - -ingress: - enabled: false - annotations: {} - # kubernetes.io/ingress.class: nginx - # kubernetes.io/tls-acme: "true" - hosts: - - host: chart-example.local - paths: [] - redirectPaths: [] - tls: [] - # - secretName: chart-example-tls - # hosts: - # - chart-example.local - -# Extra environment variables -# This will be appended to the current 'env:' key. You can use any of the kubernetes env -# syntax here -extraEnvs: [] - # - name: MY_ENVIRONMENT_VAR - # value: the_value_goes_here - -extraVolumes: [] - # - name: extras - # emptyDir: {} - -extraVolumeMounts: [] - # - name: extras - # mountPath: /usr/share/extras - # readOnly: true - -extraInitContainers: [] - -resources: {} - # We usually recommend not to specify default resources and to leave this as a conscious - # choice for the user. This also increases chances charts run on environments with little - # resources, such as Minikube. If you do want to specify resources, uncomment the following - # lines, adjust them as necessary, and remove the curly braces after 'resources:'. - # limits: - # cpu: 100m - # memory: 128Mi - # requests: - # cpu: 100m - # memory: 128Mi - -nodeSelector: {} - -tolerations: [] - -affinity: {} - -env: - JMXPORT: 1099 - -exporters: - jmx: - enabled: false -# image: -# repository: bitnami/jmx-exporter -# tag: 0.15.0 -# pullPolicy: IfNotPresent -# config: -# lowercaseOutputName: true -# lowercaseOutputLabelNames: true -# rules: -# - pattern: ".*" -# startDelaySeconds: 30 -# env: {} -# resources: {} -# path: /metrics -# ports: -# jmxxp: -# containerPort: 5556 -# protocol: TCP -# livenessProbe: -# httpGet: -# path: /metrics -# port: jmxxp -# initialDelaySeconds: 30 -# periodSeconds: 15 -# timeoutSeconds: 60 -# failureThreshold: 8 -# successThreshold: 1 -# readinessProbe: -# httpGet: -# path: /metrics -# port: jmxxp -# initialDelaySeconds: 30 -# periodSeconds: 15 -# timeoutSeconds: 60 -# failureThreshold: 8 -# successThreshold: 1 -# serviceMonitor: -# interval: 30s -# scrapeTimeout: 30s -# scheme: http - -livenessProbe: - initialDelaySeconds: 60 - periodSeconds: 30 - failureThreshold: 4 - -readinessProbe: - initialDelaySeconds: 60 - periodSeconds: 30 - failureThreshold: 4 - -datahub: - play: - mem: - buffer: - size: "10MB" -global: - datahub_analytics_enabled: true - - elasticsearch: - host: "elasticsearch" - port: "9200" - - kafka: - bootstrap: - server: "broker:9092" - - datahub: - gms: - port: "8080" - appVersion: "1.0" \ No newline at end of file diff --git a/datahub-kubernetes/datahub/charts/datahub-gms/.helmignore b/datahub-kubernetes/datahub/charts/datahub-gms/.helmignore deleted file mode 100644 index 0e8a0eb36f4ca2..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-gms/.helmignore +++ /dev/null @@ -1,23 +0,0 @@ -# Patterns to ignore when building packages. -# This supports shell glob matching, relative path matching, and -# negation (prefixed with !). Only one pattern per line. -.DS_Store -# Common VCS dirs -.git/ -.gitignore -.bzr/ -.bzrignore -.hg/ -.hgignore -.svn/ -# Common backup files -*.swp -*.bak -*.tmp -*.orig -*~ -# Various IDEs -.project -.idea/ -*.tmproj -.vscode/ diff --git a/datahub-kubernetes/datahub/charts/datahub-gms/Chart.yaml b/datahub-kubernetes/datahub/charts/datahub-gms/Chart.yaml deleted file mode 100644 index 640adb09f802b6..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-gms/Chart.yaml +++ /dev/null @@ -1,21 +0,0 @@ -apiVersion: v2 -name: datahub-gms -description: A Helm chart for LinkedIn DataHub's datahub-gms component - -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. -type: application - -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -version: 0.2.2 - -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. -appVersion: 0.3.1 diff --git a/datahub-kubernetes/datahub/charts/datahub-gms/README.md b/datahub-kubernetes/datahub/charts/datahub-gms/README.md deleted file mode 100644 index 436d28b97e9f6a..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-gms/README.md +++ /dev/null @@ -1,67 +0,0 @@ -datahub-gms -=========== -A Helm chart for LinkedIn DataHub's datahub-gms component - -Current chart version is `0.2.0` - -## Chart Values - -| Key | Type | Default | Description | -|-----|------|---------|-------------| -| affinity | object | `{}` | | -| exporters.jmx.enabled | boolean | false | | -| extraEnvs | Extra [environment variables][] which will be appended to the `env:` definition for the container | `[]` | -| extraVolumes | Templatable string of additional `volumes` to be passed to the `tpl` function | "" | -| extraVolumeMounts | Templatable string of additional `volumeMounts` to be passed to the `tpl` function | "" | -| fullnameOverride | string | `"datahub-gms-deployment"` | | -| global.datahub.appVersion | string | `"1.0"` | | -| global.datahub.gms.port | string | `"8080"` | | -| global.elasticsearch.host | string | `"elasticsearch"` | | -| global.elasticsearch.port | string | `"9200"` | | -| global.hostAliases[0].hostnames[0] | string | `"broker"` | | -| global.hostAliases[0].hostnames[1] | string | `"mysql"` | | -| global.hostAliases[0].hostnames[2] | string | `"elasticsearch"` | | -| global.hostAliases[0].hostnames[3] | string | `"neo4j"` | | -| global.hostAliases[0].ip | string | `"192.168.0.104"` | | -| global.kafka.bootstrap.server | string | `"broker:9092"` | | -| global.kafka.schemaregistry.url | string | `"http://schema-registry:8081"` | | -| global.neo4j.host | string | `"neo4j:7474"` | | -| global.neo4j.uri | string | `"bolt://neo4j"` | | -| global.neo4j.username | string | `"neo4j"` | | -| global.neo4j.password.secretRef | string | `"neo4j-secrets"` | | -| global.neo4j.password.secretKey | string | `"neo4j-password"` | | -| global.sql.datasource.driver | string | `"com.mysql.jdbc.Driver"` | | -| global.sql.datasource.host | string | `"mysql"` | | -| global.sql.datasource.url | string | `"jdbc:mysql://mysql:3306/datahub?verifyServerCertificate=false\u0026useSSL=true"` | | -| global.sql.datasource.username | string | `"datahub"` | | -| global.sql.datasource.password.secretRef | string | `"mysql-secrets"` | | -| global.sql.datasource.password.secretKey | string | `"mysql-password"` | | -| global.graph_service_impl | string | `neo4j` | One of `neo4j` or `elasticsearch`. Determines which backend to use for the GMS graph service. Elastic is recommended for a simplified deployment. Neo4j will be the default for now to maintain backwards compatibility. -| image.pullPolicy | string | `"IfNotPresent"` | | -| image.repository | string | `"linkedin/datahub-gms"` | | -| image.tag | string | `"v0.8.5"` | | -| imagePullSecrets | list | `[]` | | -| ingress.annotations | object | `{}` | | -| ingress.enabled | bool | `false` | | -| ingress.hosts[0].host | string | `"chart-example.local"` | | -| ingress.hosts[0].paths | list | `[]` | | -| ingress.tls | list | `[]` | | -| livenessProbe.initialDelaySeconds | int | `60` | | -| livenessProbe.periodSeconds | int | `30` | | -| livenessProbe.failureThreshold | int | `8` | | -| nameOverride | string | `""` | | -| nodeSelector | object | `{}` | | -| podAnnotations | object | `{}` | | -| podSecurityContext | object | `{}` | | -| readinessProbe.initialDelaySeconds | int | `60` | | -| readinessProbe.periodSeconds | int | `30` | | -| readinessProbe.failureThreshold | int | `8` | | -| replicaCount | int | `1` | | -| resources | object | `{}` | | -| securityContext | object | `{}` | | -| service.port | int | `8080` | | -| service.type | string | `"LoadBalancer"` | | -| serviceAccount.annotations | object | `{}` | | -| serviceAccount.create | bool | `true` | | -| serviceAccount.name | string | `nil` | | -| tolerations | list | `[]` | | diff --git a/datahub-kubernetes/datahub/charts/datahub-gms/templates/NOTES.txt b/datahub-kubernetes/datahub/charts/datahub-gms/templates/NOTES.txt deleted file mode 100644 index 079e85070f12f4..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-gms/templates/NOTES.txt +++ /dev/null @@ -1,21 +0,0 @@ -1. Get the application URL by running these commands: -{{- if .Values.ingress.enabled }} -{{- range $host := .Values.ingress.hosts }} - {{- range .paths }} - http{{ if $.Values.ingress.tls }}s{{ end }}://{{ $host.host }}{{ . }} - {{- end }} -{{- end }} -{{- else if contains "NodePort" .Values.service.type }} - export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "datahub-gms.fullname" . }}) - export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}") - echo http://$NODE_IP:$NODE_PORT -{{- else if contains "LoadBalancer" .Values.service.type }} - NOTE: It may take a few minutes for the LoadBalancer IP to be available. - You can watch the status of by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "datahub-gms.fullname" . }}' - export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "datahub-gms.fullname" . }} --template "{{"{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}"}}") - echo http://$SERVICE_IP:{{ .Values.service.port }} -{{- else if contains "ClusterIP" .Values.service.type }} - export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "datahub-gms.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}") - echo "Visit http://127.0.0.1:8080 to use your application" - kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8080:80 -{{- end }} diff --git a/datahub-kubernetes/datahub/charts/datahub-gms/templates/_helpers.tpl b/datahub-kubernetes/datahub/charts/datahub-gms/templates/_helpers.tpl deleted file mode 100644 index 0d292aa2d57edf..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-gms/templates/_helpers.tpl +++ /dev/null @@ -1,63 +0,0 @@ -{{/* vim: set filetype=mustache: */}} -{{/* -Expand the name of the chart. -*/}} -{{- define "datahub-gms.name" -}} -{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} -{{- end -}} - -{{/* -Create a default fully qualified app name. -We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). -If release name contains chart name it will be used as a full name. -*/}} -{{- define "datahub-gms.fullname" -}} -{{- if .Values.fullnameOverride -}} -{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} -{{- else -}} -{{- $name := default .Chart.Name .Values.nameOverride -}} -{{- if contains $name .Release.Name -}} -{{- .Release.Name | trunc 63 | trimSuffix "-" -}} -{{- else -}} -{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} -{{- end -}} -{{- end -}} -{{- end -}} - -{{/* -Create chart name and version as used by the chart label. -*/}} -{{- define "datahub-gms.chart" -}} -{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} -{{- end -}} - -{{/* -Common labels -*/}} -{{- define "datahub-gms.labels" -}} -helm.sh/chart: {{ include "datahub-gms.chart" . }} -{{ include "datahub-gms.selectorLabels" . }} -{{- if .Chart.AppVersion }} -app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} -{{- end }} -app.kubernetes.io/managed-by: {{ .Release.Service }} -{{- end -}} - -{{/* -Selector labels -*/}} -{{- define "datahub-gms.selectorLabels" -}} -app.kubernetes.io/name: {{ include "datahub-gms.name" . }} -app.kubernetes.io/instance: {{ .Release.Name }} -{{- end -}} - -{{/* -Create the name of the service account to use -*/}} -{{- define "datahub-gms.serviceAccountName" -}} -{{- if .Values.serviceAccount.create -}} - {{ default (include "datahub-gms.fullname" .) .Values.serviceAccount.name }} -{{- else -}} - {{ default "default" .Values.serviceAccount.name }} -{{- end -}} -{{- end -}} diff --git a/datahub-kubernetes/datahub/charts/datahub-gms/templates/config-jmx-exporter.yaml b/datahub-kubernetes/datahub/charts/datahub-gms/templates/config-jmx-exporter.yaml deleted file mode 100644 index 9e7c23392c0a0a..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-gms/templates/config-jmx-exporter.yaml +++ /dev/null @@ -1,17 +0,0 @@ -{{- if .Values.exporters.jmx.enabled }} -apiVersion: v1 -kind: ConfigMap -metadata: - name: {{ include "datahub-gms.fullname" . }}-config-jmx-exporter - labels: - {{- include "datahub-gms.labels" . | nindent 4 }} -data: - config.yml: |- - hostPort: localhost:{{ .Values.env.JMXPORT }} - lowercaseOutputName: {{ .Values.exporters.jmx.config.lowercaseOutputName }} - lowercaseOutputLabelNames: {{ .Values.exporters.jmx.config.lowercaseOutputLabelNames }} - rules: -{{ .Values.exporters.jmx.config.rules | toYaml | indent 6 }} - ssl: false - startDelaySeconds: {{ .Values.exporters.jmx.config.startDelaySeconds }} -{{- end }} diff --git a/datahub-kubernetes/datahub/charts/datahub-gms/templates/deployment.yaml b/datahub-kubernetes/datahub/charts/datahub-gms/templates/deployment.yaml deleted file mode 100644 index ed42fd3acbed6e..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-gms/templates/deployment.yaml +++ /dev/null @@ -1,175 +0,0 @@ -apiVersion: apps/v1 -kind: Deployment -metadata: - name: {{ include "datahub-gms.fullname" . }} - labels: - {{- include "datahub-gms.labels" . | nindent 4 }} -spec: - replicas: {{ .Values.replicaCount }} - selector: - matchLabels: - {{- include "datahub-gms.selectorLabels" . | nindent 6 }} - template: - metadata: - {{- with .Values.podAnnotations }} - annotations: - {{- toYaml . | nindent 8 }} - {{- end }} - labels: - {{- include "datahub-gms.selectorLabels" . | nindent 8 }} - spec: - {{- with .Values.global.hostAliases }} - hostAliases: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.imagePullSecrets }} - imagePullSecrets: - {{- toYaml . | nindent 8 }} - {{- end }} - serviceAccountName: {{ include "datahub-gms.serviceAccountName" . }} - securityContext: - {{- toYaml .Values.podSecurityContext | nindent 8 }} - volumes: - {{- with .Values.global.credentialsAndCertsSecrets }} - - name: datahub-certs-dir - secret: - defaultMode: 0444 - secretName: {{ .name }} - {{- end }} - {{- if .Values.exporters.jmx.enabled }} - - name: config-jmx-exporter - configMap: - name: {{ include "datahub-gms.fullname" . }}-config-jmx-exporter - {{- end }} - {{- with .Values.extraVolumes }} - {{- toYaml . | nindent 8 }} - {{- end }} - initContainers: - {{- with .Values.extraInitContainers }} - {{- toYaml . | nindent 8 }} - {{- end }} - containers: - - name: {{ .Chart.Name }} - securityContext: - {{- toYaml .Values.securityContext | nindent 12 }} - image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" - imagePullPolicy: {{ .Values.image.pullPolicy }} - ports: - - name: http - containerPort: 8080 - protocol: TCP - livenessProbe: - httpGet: - path: /health - port: http - initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }} - periodSeconds: {{ .Values.livenessProbe.periodSeconds }} - failureThreshold: {{ .Values.livenessProbe.failureThreshold }} - readinessProbe: - httpGet: - path: /health - port: http - initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }} - periodSeconds: {{ .Values.readinessProbe.periodSeconds }} - failureThreshold: {{ .Values.readinessProbe.failureThreshold }} - env: - {{- if not .Values.global.datahub_standalone_consumers_enabled }} - - name: MCE_CONSUMER_ENABLED - value: "true" - - name: MAE_CONSUMER_ENABLED - value: "true" - {{- end }} - - name: DATAHUB_ANALYTICS_ENABLED - value: "{{ .Values.global.datahub_analytics_enabled }}" - - name: EBEAN_DATASOURCE_USERNAME - value: "{{ .Values.global.sql.datasource.username }}" - - name: EBEAN_DATASOURCE_PASSWORD - valueFrom: - secretKeyRef: - name: "{{ .Values.global.sql.datasource.password.secretRef }}" - key: "{{ .Values.global.sql.datasource.password.secretKey }}" - - name: EBEAN_DATASOURCE_HOST - value: "{{ .Values.global.sql.datasource.host }}" - - name: EBEAN_DATASOURCE_URL - value: "{{ .Values.global.sql.datasource.url }}" - - name: EBEAN_DATASOURCE_DRIVER - value: "{{ .Values.global.sql.datasource.driver }}" - - name: KAFKA_BOOTSTRAP_SERVER - value: "{{ .Values.global.kafka.bootstrap.server }}" - - name: KAFKA_SCHEMAREGISTRY_URL - value: "{{ .Values.global.kafka.schemaregistry.url }}" - - name: ELASTICSEARCH_HOST - value: "{{ .Values.global.elasticsearch.host }}" - - name: ELASTICSEARCH_PORT - value: "{{ .Values.global.elasticsearch.port }}" - {{- with .Values.global.elasticsearch.useSSL }} - - name: ELASTICSEARCH_USE_SSL - value: {{ . | quote }} - {{- end }} - {{- with .Values.global.elasticsearch.auth }} - - name: ELASTICSEARCH_USERNAME - value: {{ .username }} - - name: ELASTICSEARCH_PASSWORD - valueFrom: - secretKeyRef: - name: "{{ .password.secretRef }}" - key: "{{ .password.secretKey }}" - {{- end }} - - name: GRAPH_SERVICE_IMPL - value: {{ .Values.global.graph_service_impl }} - {{- if eq .Values.global.graph_service_impl "neo4j" }} - - name: NEO4J_HOST - value: "{{ .Values.global.neo4j.host }}" - - name: NEO4J_URI - value: "{{ .Values.global.neo4j.uri }}" - - name: NEO4J_USERNAME - value: "{{ .Values.global.neo4j.username }}" - - name: NEO4J_PASSWORD - valueFrom: - secretKeyRef: - name: "{{ .Values.global.neo4j.password.secretRef }}" - key: "{{ .Values.global.neo4j.password.secretKey }}" - {{- end }} - {{- if .Values.global.springKafkaConfigurationOverrides }} - {{- range $configName, $configValue := .Values.global.springKafkaConfigurationOverrides }} - - name: SPRING_KAFKA_PROPERTIES_{{ $configName | replace "." "_" | upper }} - value: {{ $configValue }} - {{- end }} - {{- end }} - {{- if .Values.global.credentialsAndCertsSecrets }} - {{- range $envVarName, $envVarValue := .Values.global.credentialsAndCertsSecrets.secureEnv }} - - name: SPRING_KAFKA_PROPERTIES_{{ $envVarName | replace "." "_" | upper }} - valueFrom: - secretKeyRef: - name: {{ $.Values.global.credentialsAndCertsSecrets.name }} - key: {{ $envVarValue }} - {{- end }} - {{- end }} - {{- with .Values.extraEnvs }} - {{- toYaml . | nindent 12 }} - {{- end }} - volumeMounts: - {{- with .Values.global.credentialsAndCertsSecrets }} - - name: datahub-certs-dir - mountPath: {{ .path | default "/mnt/certs" }} - {{- end }} - {{- with .Values.extraVolumeMounts }} - {{- toYaml . | nindent 12 }} - {{- end }} - resources: - {{- toYaml .Values.resources | nindent 12 }} - - {{- include "datahub-jmxexporter.container" . }} - - {{- with .Values.nodeSelector }} - nodeSelector: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.affinity }} - affinity: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.tolerations }} - tolerations: - {{- toYaml . | nindent 8 }} - {{- end }} diff --git a/datahub-kubernetes/datahub/charts/datahub-gms/templates/ingress.yaml b/datahub-kubernetes/datahub/charts/datahub-gms/templates/ingress.yaml deleted file mode 100644 index b7547f09ff50f3..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-gms/templates/ingress.yaml +++ /dev/null @@ -1,47 +0,0 @@ -{{- if .Values.ingress.enabled -}} -{{- $fullName := printf "%s-%s" .Release.Name "datahub-gms"}} -{{- $svcPort := .Values.global.datahub.gms.port -}} -{{- if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}} -apiVersion: networking.k8s.io/v1beta1 -{{- else -}} -apiVersion: extensions/v1beta1 -{{- end }} -kind: Ingress -metadata: - name: {{ $fullName}} - labels: - {{- include "datahub-gms.labels" . | nindent 4 }} - {{- with .Values.ingress.annotations }} - annotations: - {{- toYaml . | nindent 4 }} - {{- end }} -spec: -{{- if .Values.ingress.tls }} - tls: - {{- range .Values.ingress.tls }} - - hosts: - {{- range .hosts }} - - {{ . | quote }} - {{- end }} - secretName: {{ .secretName }} - {{- end }} -{{- end }} - rules: - {{- range .Values.ingress.hosts }} - - host: {{ .host | quote }} - http: - paths: - {{- range .redirectPaths }} - - path: {{ .path }} - backend: - serviceName: {{ .name }} - servicePort: {{ .port }} - {{- end }} - {{- range .paths }} - - path: {{ . }} - backend: - serviceName: {{ $fullName }} - servicePort: {{ $svcPort }} - {{- end }} - {{- end }} -{{- end }} diff --git a/datahub-kubernetes/datahub/charts/datahub-gms/templates/service.yaml b/datahub-kubernetes/datahub/charts/datahub-gms/templates/service.yaml deleted file mode 100644 index 6bda3cca8dfbfa..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-gms/templates/service.yaml +++ /dev/null @@ -1,19 +0,0 @@ -apiVersion: v1 -kind: Service -metadata: - name: {{ printf "%s-%s" .Release.Name "datahub-gms" }} - labels: - {{- include "datahub-gms.labels" . | nindent 4 }} - {{- with .Values.service.annotations }} - annotations: - {{- toYaml . | nindent 4 }} - {{- end }} -spec: - type: {{ .Values.service.type }} - ports: - - port: {{ .Values.global.datahub.gms.port }} - targetPort: http - protocol: TCP - name: http - selector: - {{- include "datahub-gms.selectorLabels" . | nindent 4 }} diff --git a/datahub-kubernetes/datahub/charts/datahub-gms/templates/serviceaccount.yaml b/datahub-kubernetes/datahub/charts/datahub-gms/templates/serviceaccount.yaml deleted file mode 100644 index 6a390fb651b479..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-gms/templates/serviceaccount.yaml +++ /dev/null @@ -1,12 +0,0 @@ -{{- if .Values.serviceAccount.create -}} -apiVersion: v1 -kind: ServiceAccount -metadata: - name: {{ include "datahub-gms.serviceAccountName" . }} - labels: - {{- include "datahub-gms.labels" . | nindent 4 }} - {{- with .Values.serviceAccount.annotations }} - annotations: - {{- toYaml . | nindent 4 }} - {{- end }} -{{- end -}} diff --git a/datahub-kubernetes/datahub/charts/datahub-gms/values.yaml b/datahub-kubernetes/datahub/charts/datahub-gms/values.yaml deleted file mode 100644 index d7ed10ce768b44..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-gms/values.yaml +++ /dev/null @@ -1,196 +0,0 @@ -# Default values for datahub-gms. -# This is a YAML-formatted file. -# Declare variables to be passed into your templates. - -replicaCount: 1 - -image: - repository: linkedin/datahub-gms - pullPolicy: Always - tag: v0.8.5 - -imagePullSecrets: [] -nameOverride: "" -fullnameOverride: "" - -serviceAccount: - # Specifies whether a service account should be created - create: true - # Annotations to add to the service account - annotations: {} - # The name of the service account to use. - # If not set and create is true, a name is generated using the fullname template - name: - -podAnnotations: {} - # co.elastic.logs/enabled: "true" - -podSecurityContext: {} - # fsGroup: 2000 - -securityContext: {} - # capabilities: - # drop: - # - ALL - # readOnlyRootFilesystem: true - # runAsNonRoot: true - # runAsUser: 1000 - -service: - type: LoadBalancer #ClusterIP - # Annotations to add to the service, this will help in adding - # Internal load balancer or various other annotation support in AWS - annotations: {} - # service.beta.kubernetes.io/aws-load-balancer-internal: "true" - -ingress: - enabled: false - annotations: {} - # kubernetes.io/ingress.class: nginx - # kubernetes.io/tls-acme: "true" - hosts: - - host: chart-example.local - paths: [] - redirectPaths: [] - tls: [] - # - secretName: chart-example-tls - # hosts: - # - chart-example.local - -# Extra environment variables -# This will be appended to the current 'env:' key. You can use any of the kubernetes env -# syntax here -extraEnvs: [] - # - name: MY_ENVIRONMENT_VAR - # value: the_value_goes_here - -extraVolumes: [] - # - name: extras - # emptyDir: {} - -extraVolumeMounts: [] - # - name: extras - # mountPath: /usr/share/extras - # readOnly: true - -extraInitContainers: [] - -resources: {} - # We usually recommend not to specify default resources and to leave this as a conscious - # choice for the user. This also increases chances charts run on environments with little - # resources, such as Minikube. If you do want to specify resources, uncomment the following - # lines, adjust them as necessary, and remove the curly braces after 'resources:'. - # limits: - # cpu: 100m - # memory: 128Mi - # requests: - # cpu: 100m - # memory: 128Mi - -nodeSelector: {} - -tolerations: [] - -affinity: {} - -env: - JMXPORT: 1099 - -exporters: - jmx: - enabled: false -# image: -# repository: bitnami/jmx-exporter -# tag: 0.15.0 -# pullPolicy: IfNotPresent -# config: -# lowercaseOutputName: true -# rules: -# - pattern: ".*" -# startDelaySeconds: 30 -# env: {} -# resources: {} -# path: /metrics -# ports: -# jmxxp: -# containerPort: 5556 -# protocol: TCP -# livenessProbe: -# httpGet: -# path: /metrics -# port: jmxxp -# initialDelaySeconds: 30 -# periodSeconds: 15 -# timeoutSeconds: 60 -# failureThreshold: 8 -# successThreshold: 1 -# readinessProbe: -# httpGet: -# path: /metrics -# port: jmxxp -# initialDelaySeconds: 30 -# periodSeconds: 15 -# timeoutSeconds: 60 -# failureThreshold: 8 -# successThreshold: 1 -# serviceMonitor: -# interval: 30s -# scrapeTimeout: 30s -# scheme: http - -livenessProbe: - initialDelaySeconds: 60 - periodSeconds: 30 - failureThreshold: 8 - -readinessProbe: - initialDelaySeconds: 60 - periodSeconds: 30 - failureThreshold: 8 - -#This section is useful if we are installing this chart separately for testing -# helm install datahub-gms datahub-gms/ -global: - datahub_analytics_enabled: true - graph_service_impl: neo4j - - elasticsearch: - host: "elasticsearch" - port: "9200" - - kafka: - bootstrap: - server: "broker:9092" - schemaregistry: - url: "http://schema-registry:8081" - - neo4j: - host: "neo4j:7474" - uri: "bolt://neo4j" - username: "neo4j" - password: - secretRef: "neo4j-secrets" - secretKey: "neo4j-password" - - sql: - datasource: - host: "mysql:3306" - url: "jdbc:mysql://mysql:3306/datahub?verifyServerCertificate=false&useSSL=true" - driver: "com.mysql.jdbc.Driver" - username: "datahub" - password: - secretRef: "mysql-secrets" - secretKey: "mysql-password" - - datahub: - gms: - port: "8080" - appVersion: "1.0" - - hostAliases: - - ip: "192.168.0.104" - hostnames: - - "broker" - - "mysql" - - "elasticsearch" - - "neo4j" diff --git a/datahub-kubernetes/datahub/charts/datahub-ingestion-cron/Chart.yaml b/datahub-kubernetes/datahub/charts/datahub-ingestion-cron/Chart.yaml deleted file mode 100644 index 63d51995234f5a..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-ingestion-cron/Chart.yaml +++ /dev/null @@ -1,21 +0,0 @@ -apiVersion: v2 -name: datahub-ingestion-cron -description: A Helm chart for Kubernetes - -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. -type: application - -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -version: 0.2.2 - -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. -appVersion: 0.3.1 diff --git a/datahub-kubernetes/datahub/charts/datahub-ingestion-cron/README.md b/datahub-kubernetes/datahub/charts/datahub-ingestion-cron/README.md deleted file mode 100644 index d931ee10fe5bf2..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-ingestion-cron/README.md +++ /dev/null @@ -1,27 +0,0 @@ -datahub-ingestion-cron -================ -A Helm chart for datahub's metadata-ingestion framework with kerberos authentication. - -## Chart Values - -| Key | Type | Default | Description | -|-----|------|---------|-------------| -| image.pullPolicy | string | `"Always"` | Image pull policy | -| image.repository | string | `"linkedin/datahub-ingestion"` | DataHub Ingestion image repository | -| image.tag | string | `"v0.8.5"` | DataHub Ingestion image tag | -| imagePullSecrets | array | `[]` (does not add image pull secrets to deployed pods) | Docker registry secret names as an array | -| labels | string | `{}` | Metadata labels to be added to each crawling cron job | -| crons | type | `{}` | A map of crawling parameters per different technology being crawler, the key in the object will be used as the name for the new cron job | -| crons.schedule | string | `"0 0 * * *"` | Cron expression (default is daily at midnight) for crawler jobs | -| crons.recipe | object | `{}` | Recipe configuration to be executed (required) | -| crons.recipe.configmapName | string | `""` | Name of configmap to be mounted containing recipe to be executed | -| crons.recipe.fileName | string | `""` | Name of property within configMap referenced by `recipe.configName` with the concrete recipe definition | -| crons.command | array | `["/bin/sh", "-c", "datahub ingest -c /etc/recipe/"]` | Array of strings denoting the crawling command to be invoked in the cron job. By default it will execute the recipe defined in the `crons.recipe` object. Cron crawling customization is possible by having extra volumes with custom logic to be executed. | -| crons.hostAliases | array | `[]` | host aliases | -| crons.env | object | `{}` | Environment variables to add to the cronjob container | -| crons.envFromSecrets | object | `{}` | Environment variables from secrets to the cronjob container | -| crons.envFromSecrets*.secret | string | | secretKeyRef.name used for environment variable | -| crons.envFromSecrets*.key | string | | secretKeyRef.key used for environment variable | -| crons.extraVolumes | array | `[]` | Additional volumes to add to the pods | -| crons.extraVolumeMounts | array | `[]` | Additional volume mounts to add to the pods | -| crons.extraInitContainers | object | `{}` | Init containers to add to the cronjob container | diff --git a/datahub-kubernetes/datahub/charts/datahub-ingestion-cron/templates/_helpers.tpl b/datahub-kubernetes/datahub/charts/datahub-ingestion-cron/templates/_helpers.tpl deleted file mode 100644 index bb6dbcf4a308d3..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-ingestion-cron/templates/_helpers.tpl +++ /dev/null @@ -1,63 +0,0 @@ -{{/* vim: set filetype=mustache: */}} -{{/* -Expand the name of the chart. -*/}} -{{- define "datahub-ingestion-cron.name" -}} -{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} -{{- end -}} - -{{/* -Create a default fully qualified app name. -We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). -If release name contains chart name it will be used as a full name. -*/}} -{{- define "datahub-ingestion-cron.fullname" -}} -{{- if .Values.fullnameOverride -}} -{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} -{{- else -}} -{{- $name := default .Chart.Name .Values.nameOverride -}} -{{- if contains $name .Release.Name -}} -{{- .Release.Name | trunc 63 | trimSuffix "-" -}} -{{- else -}} -{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} -{{- end -}} -{{- end -}} -{{- end -}} - -{{/* -Create chart name and version as used by the chart label. -*/}} -{{- define "datahub-ingestion-cron.chart" -}} -{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} -{{- end -}} - -{{/* -Common labels -*/}} -{{- define "datahub-ingestion-cron.labels" -}} -helm.sh/chart: {{ include "datahub-ingestion-cron.chart" . }} -{{ include "datahub-ingestion-cron.selectorLabels" . }} -{{- if .Chart.AppVersion }} -app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} -{{- end }} -app.kubernetes.io/managed-by: {{ .Release.Service }} -{{- end -}} - -{{/* -Selector labels -*/}} -{{- define "datahub-ingestion-cron.selectorLabels" -}} -app.kubernetes.io/name: {{ include "datahub-ingestion-cron.name" . }} -app.kubernetes.io/instance: {{ .Release.Name }} -{{- end -}} - -{{/* -Create the name of the service account to use -*/}} -{{- define "datahub-ingestion-cron.serviceAccountName" -}} -{{- if .Values.serviceAccount.create -}} - {{ default (include "datahub-ingestion-cron.fullname" .) .Values.serviceAccount.name }} -{{- else -}} - {{ default "default" .Values.serviceAccount.name }} -{{- end -}} -{{- end -}} diff --git a/datahub-kubernetes/datahub/charts/datahub-ingestion-cron/templates/cron.yaml b/datahub-kubernetes/datahub/charts/datahub-ingestion-cron/templates/cron.yaml deleted file mode 100644 index e0dadb63d11d78..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-ingestion-cron/templates/cron.yaml +++ /dev/null @@ -1,63 +0,0 @@ -{{- $baseName := include "datahub-ingestion-cron.fullname" .}} -{{- $labels := include "datahub-ingestion-cron.labels" .}} -{{- range $jobName, $val := .Values.crons }} -{{- $defaultCommand := printf "datahub ingest -c /etc/recipe/%s" $val.recipe.fileName }} -apiVersion: batch/v1beta1 -kind: CronJob -metadata: - name: "{{ $baseName }}-{{ $jobName }}" - labels: {{- $labels | nindent 4 }} -spec: - schedule: {{ default "0 0 * * *" .schedule | quote}} - jobTemplate: - spec: - template: - spec: - {{- with $.Values.imagePullSecrets }} - imagePullSecrets: - {{- toYaml . | nindent 12 }} - {{- end }} - {{- if .extraInitContainers }} - initContainers: - {{- toYaml .extraInitContainers | nindent 12 }} - {{- end }} - {{- if .hostAliases }} - hostAliases: {{- include "common.tplvalues.render" (dict "value" .hostAliases "context" $) | nindent 10 }} - {{- end }} - containers: - - name: {{ $jobName }}-crawler - image: "{{ $.Values.image.repository }}:{{ $.Values.image.tag }}" - imagePullPolicy: {{ $.Values.image.pullPolicy }} - volumeMounts: - - name: recipe - mountPath: /etc/recipe - {{- if .extraVolumeMounts }} - {{- toYaml .extraVolumeMounts | nindent 14 }} - {{- end }} - command: ["/bin/sh", "-c", {{ default $defaultCommand .command }} ] - env: - {{- if .env }} - {{- range $key,$value := .env }} - - name: {{ $key | quote}} - value: {{ $value | quote}} - {{- end }} - {{- end }} - {{- if .envFromSecrets }} - {{- range $key,$value := .envFromSecrets }} - - name: {{ $key | quote}} - valueFrom: - secretKeyRef: - name: {{ $value.secret | quote}} - key: {{ $value.key | quote}} - {{- end }} - {{- end }} - restartPolicy: OnFailure - volumes: - - name: recipe - configMap: - name: {{ required "A valid .recipe.configmapName entry is required!" $val.recipe.configmapName }} - {{- if .extraVolumes }} - {{- toYaml .extraVolumes | nindent 12 }} - {{- end }} ---- -{{- end }} \ No newline at end of file diff --git a/datahub-kubernetes/datahub/charts/datahub-ingestion-cron/values.yaml b/datahub-kubernetes/datahub/charts/datahub-ingestion-cron/values.yaml deleted file mode 100644 index dab5fdaa5f244c..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-ingestion-cron/values.yaml +++ /dev/null @@ -1,52 +0,0 @@ -# Default values for datahub-ingestion-cron. -# This is a YAML-formatted file. -# Declare variables to be passed into your templates. - -image: - repository: linkedin/datahub-ingestion - tag: v0.8.5 - pullPolicy: Always - -imagePullSecrets: [] - -crons: {} - #### Example data - #hive: - ## Daily at midnight (we may want to offset this to not conflict with other processes) - #schedule: "0 0 * * *" - - #recipe: - # configmapName: - # fileName: - - ## Command to be executed - #command: ["/bin/sh", "-c", "datahub ingest -c "] - - ## Deployment pod host aliases - ## https://kubernetes.io/docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/ - ## - #hostAliases: [] - - ## Environment variables. - #env: {} - - ## Environment variables from Secret resources. - #envFromSecrets: {} - - ## Additional primary volume mounts - ## - #extraVolumeMounts: - #- name: configmap-volume - # mountPath: config.yml - # subPath: config.yml - - ## Additional primary volumes - ## - #extraVolumes: - #- name: configmap-volume - # configMap: - # name: crawler-config - - ## Add your own init container or uncomment and modify the given example. - ## - #extraInitContainers: {} diff --git a/datahub-kubernetes/datahub/charts/datahub-jmxexporter/Chart.yaml b/datahub-kubernetes/datahub/charts/datahub-jmxexporter/Chart.yaml deleted file mode 100644 index cc96e4b071b4de..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-jmxexporter/Chart.yaml +++ /dev/null @@ -1,21 +0,0 @@ -apiVersion: v2 -name: datahub-jmxexporter -description: A Helm chart for Kubernetes - -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. -type: library - -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -version: 0.2.1 - -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. -appVersion: 0.3.1 diff --git a/datahub-kubernetes/datahub/charts/datahub-jmxexporter/templates/_container.tpl b/datahub-kubernetes/datahub/charts/datahub-jmxexporter/templates/_container.tpl deleted file mode 100644 index 47ee834190af4a..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-jmxexporter/templates/_container.tpl +++ /dev/null @@ -1,32 +0,0 @@ -{{- define "datahub-jmxexporter.container" -}} -{{- if .Values.exporters.jmx.enabled }} - - name: jmx-exporter - image: "{{ .Values.exporters.jmx.image.repository }}:{{ .Values.exporters.jmx.image.tag }}" - imagePullPolicy: {{ .Values.exporters.jmx.image.pullPolicy }} - args: ["{{ .Values.exporters.jmx.ports.jmxxp.containerPort}}", "/opt/jmx_exporter/config.yml"] - ports: - {{- range $key, $port := .Values.exporters.jmx.ports }} - - name: {{ $key }} -{{ toYaml $port | indent 14 }} - {{- end }} - livenessProbe: -{{ toYaml .Values.exporters.jmx.livenessProbe | indent 12 }} - readinessProbe: -{{ toYaml .Values.exporters.jmx.readinessProbe | indent 12 }} - env: - - name: SERVICE_PORT - value: {{ .Values.exporters.jmx.ports.jmxxp.containerPort | quote }} - {{- with .Values.exporters.jmx.env }} - {{- range $key, $value := . }} - - name: {{ $key | upper | replace "." "_" }} - value: {{ $value | quote }} - {{- end }} - {{- end }} - resources: -{{ toYaml .Values.exporters.jmx.resources | indent 12 }} - volumeMounts: - - name: config-jmx-exporter - mountPath: /opt/jmx_exporter/config.yml - subPath: config.yml -{{- end }} -{{- end }} diff --git a/datahub-kubernetes/datahub/charts/datahub-mae-consumer/.helmignore b/datahub-kubernetes/datahub/charts/datahub-mae-consumer/.helmignore deleted file mode 100644 index 0e8a0eb36f4ca2..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-mae-consumer/.helmignore +++ /dev/null @@ -1,23 +0,0 @@ -# Patterns to ignore when building packages. -# This supports shell glob matching, relative path matching, and -# negation (prefixed with !). Only one pattern per line. -.DS_Store -# Common VCS dirs -.git/ -.gitignore -.bzr/ -.bzrignore -.hg/ -.hgignore -.svn/ -# Common backup files -*.swp -*.bak -*.tmp -*.orig -*~ -# Various IDEs -.project -.idea/ -*.tmproj -.vscode/ diff --git a/datahub-kubernetes/datahub/charts/datahub-mae-consumer/Chart.yaml b/datahub-kubernetes/datahub/charts/datahub-mae-consumer/Chart.yaml deleted file mode 100644 index 4e5116696dc366..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-mae-consumer/Chart.yaml +++ /dev/null @@ -1,21 +0,0 @@ -apiVersion: v2 -name: datahub-mae-consumer -description: A Helm chart for Kubernetes - -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. -type: application - -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -version: 0.2.1 - -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. -appVersion: 0.3.1 diff --git a/datahub-kubernetes/datahub/charts/datahub-mae-consumer/README.md b/datahub-kubernetes/datahub/charts/datahub-mae-consumer/README.md deleted file mode 100644 index c5c85ac4bb245a..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-mae-consumer/README.md +++ /dev/null @@ -1,60 +0,0 @@ -datahub-mae-consumer -==================== -A Helm chart for datahub-mae-consumer - -Current chart version is `0.2.0` - -## Chart Values - -| Key | Type | Default | Description | -|-----|------|---------|-------------| -| affinity | object | `{}` | | -| exporters.jmx.enabled | boolean | false | | -| extraEnvs | Extra [environment variables][] which will be appended to the `env:` definition for the container | `[]` | -| extraVolumes | Templatable string of additional `volumes` to be passed to the `tpl` function | "" | -| extraVolumeMounts | Templatable string of additional `volumeMounts` to be passed to the `tpl` function | "" | -| fullnameOverride | string | `"datahub-mae-consumer"` | | -| global.datahub_analytics_enabled | boolean | true | | -| global.elasticsearch.host | string | `"elasticsearch"` | | -| global.elasticsearch.port | string | `"9200"` | | -| global.kafka.bootstrap.server | string | `"broker:9092"` | | -| global.kafka.schemaregistry.url | string | `"http://schema-registry:8081"` | | -| global.neo4j.host | string | `"neo4j:7474"` | | -| global.neo4j.uri | string | `"bolt://neo4j"` | | -| global.neo4j.username | string | `"neo4j"` | | -| global.neo4j.password.secretRef | string | `"neo4j-secrets"` | | -| global.neo4j.password.secretKey | string | `"neo4j-password"` | | -| global.hostAliases[0].hostnames[0] | string | `"broker"` | | -| global.hostAliases[0].hostnames[1] | string | `"mysql"` | | -| global.hostAliases[0].hostnames[2] | string | `"elasticsearch"` | | -| global.hostAliases[0].hostnames[3] | string | `"neo4j"` | | -| global.hostAliases[0].ip | string | `"192.168.0.104"` | | -| global.graph_service_impl | string | `neo4j` | One of `neo4j` or `elasticsearch`. Determines which backend to use for the GMS graph service. Elastic is recommended for a simplified deployment. Neo4j will be the default for now to maintain backwards compatibility. -| image.pullPolicy | string | `"IfNotPresent"` | | -| image.repository | string | `"linkedin/datahub-mae-consumer"` | | -| image.tag | string | `"v0.8.5"` | | -| imagePullSecrets | list | `[]` | | -| ingress.annotations | object | `{}` | | -| ingress.enabled | bool | `false` | | -| ingress.hosts[0].host | string | `"chart-example.local"` | | -| ingress.hosts[0].paths | list | `[]` | | -| ingress.tls | list | `[]` | | -| livenessProbe.initialDelaySeconds | int | `60` | | -| livenessProbe.periodSeconds | int | `30` | | -| livenessProbe.failureThreshold | int | `8` | | -| nameOverride | string | `""` | | -| nodeSelector | object | `{}` | | -| podAnnotations | object | `{}` | | -| podSecurityContext | object | `{}` | | -| readinessProbe.initialDelaySeconds | int | `60` | | -| readinessProbe.periodSeconds | int | `30` | | -| readinessProbe.failureThreshold | int | `8` | | -| replicaCount | int | `1` | | -| resources | object | `{}` | | -| securityContext | object | `{}` | | -| service.port | int | `80` | | -| service.type | string | `"ClusterIP"` | | -| serviceAccount.annotations | object | `{}` | | -| serviceAccount.create | bool | `true` | | -| serviceAccount.name | string | `nil` | | -| tolerations | list | `[]` | | diff --git a/datahub-kubernetes/datahub/charts/datahub-mae-consumer/templates/NOTES.txt b/datahub-kubernetes/datahub/charts/datahub-mae-consumer/templates/NOTES.txt deleted file mode 100644 index 6f2f355847e9d0..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-mae-consumer/templates/NOTES.txt +++ /dev/null @@ -1,21 +0,0 @@ -1. Get the application URL by running these commands: -{{- if .Values.ingress.enabled }} -{{- range $host := .Values.ingress.hosts }} - {{- range .paths }} - http{{ if $.Values.ingress.tls }}s{{ end }}://{{ $host.host }}{{ . }} - {{- end }} -{{- end }} -{{- else if contains "NodePort" .Values.service.type }} - export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "datahub-mae-consumer.fullname" . }}) - export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}") - echo http://$NODE_IP:$NODE_PORT -{{- else if contains "LoadBalancer" .Values.service.type }} - NOTE: It may take a few minutes for the LoadBalancer IP to be available. - You can watch the status of by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "datahub-mae-consumer.fullname" . }}' - export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "datahub-mae-consumer.fullname" . }} --template "{{"{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}"}}") - echo http://$SERVICE_IP:{{ .Values.service.port }} -{{- else if contains "ClusterIP" .Values.service.type }} - export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "datahub-mae-consumer.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}") - echo "Visit http://127.0.0.1:8080 to use your application" - kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8080:80 -{{- end }} diff --git a/datahub-kubernetes/datahub/charts/datahub-mae-consumer/templates/_helpers.tpl b/datahub-kubernetes/datahub/charts/datahub-mae-consumer/templates/_helpers.tpl deleted file mode 100644 index e677e252a6270c..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-mae-consumer/templates/_helpers.tpl +++ /dev/null @@ -1,63 +0,0 @@ -{{/* vim: set filetype=mustache: */}} -{{/* -Expand the name of the chart. -*/}} -{{- define "datahub-mae-consumer.name" -}} -{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} -{{- end -}} - -{{/* -Create a default fully qualified app name. -We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). -If release name contains chart name it will be used as a full name. -*/}} -{{- define "datahub-mae-consumer.fullname" -}} -{{- if .Values.fullnameOverride -}} -{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} -{{- else -}} -{{- $name := default .Chart.Name .Values.nameOverride -}} -{{- if contains $name .Release.Name -}} -{{- .Release.Name | trunc 63 | trimSuffix "-" -}} -{{- else -}} -{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} -{{- end -}} -{{- end -}} -{{- end -}} - -{{/* -Create chart name and version as used by the chart label. -*/}} -{{- define "datahub-mae-consumer.chart" -}} -{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} -{{- end -}} - -{{/* -Common labels -*/}} -{{- define "datahub-mae-consumer.labels" -}} -helm.sh/chart: {{ include "datahub-mae-consumer.chart" . }} -{{ include "datahub-mae-consumer.selectorLabels" . }} -{{- if .Chart.AppVersion }} -app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} -{{- end }} -app.kubernetes.io/managed-by: {{ .Release.Service }} -{{- end -}} - -{{/* -Selector labels -*/}} -{{- define "datahub-mae-consumer.selectorLabels" -}} -app.kubernetes.io/name: {{ include "datahub-mae-consumer.name" . }} -app.kubernetes.io/instance: {{ .Release.Name }} -{{- end -}} - -{{/* -Create the name of the service account to use -*/}} -{{- define "datahub-mae-consumer.serviceAccountName" -}} -{{- if .Values.serviceAccount.create -}} - {{ default (include "datahub-mae-consumer.fullname" .) .Values.serviceAccount.name }} -{{- else -}} - {{ default "default" .Values.serviceAccount.name }} -{{- end -}} -{{- end -}} diff --git a/datahub-kubernetes/datahub/charts/datahub-mae-consumer/templates/config-jmx-exporter.yaml b/datahub-kubernetes/datahub/charts/datahub-mae-consumer/templates/config-jmx-exporter.yaml deleted file mode 100644 index 72431b713e836e..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-mae-consumer/templates/config-jmx-exporter.yaml +++ /dev/null @@ -1,17 +0,0 @@ -{{- if .Values.exporters.jmx.enabled }} -apiVersion: v1 -kind: ConfigMap -metadata: - name: {{ include "datahub-mae-consumer.fullname" . }}-config-jmx-exporter - labels: - {{- include "datahub-mae-consumer.labels" . | nindent 4 }} -data: - config.yml: |- - hostPort: localhost:{{ .Values.env.JMXPORT }} - lowercaseOutputName: {{ .Values.exporters.jmx.config.lowercaseOutputName }} - lowercaseOutputLabelNames: {{ .Values.exporters.jmx.config.lowercaseOutputLabelNames }} - rules: -{{ .Values.exporters.jmx.config.rules | toYaml | indent 6 }} - ssl: false - startDelaySeconds: {{ .Values.exporters.jmx.config.startDelaySeconds }} -{{- end }} diff --git a/datahub-kubernetes/datahub/charts/datahub-mae-consumer/templates/deployment.yaml b/datahub-kubernetes/datahub/charts/datahub-mae-consumer/templates/deployment.yaml deleted file mode 100644 index a8b928512c3c03..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-mae-consumer/templates/deployment.yaml +++ /dev/null @@ -1,162 +0,0 @@ -apiVersion: apps/v1 -kind: Deployment -metadata: - name: {{ include "datahub-mae-consumer.fullname" . }} - labels: - {{- include "datahub-mae-consumer.labels" . | nindent 4 }} -spec: - replicas: {{ .Values.replicaCount }} - selector: - matchLabels: - {{- include "datahub-mae-consumer.selectorLabels" . | nindent 6 }} - template: - metadata: - {{- with .Values.podAnnotations }} - annotations: - {{- toYaml . | nindent 8 }} - {{- end }} - labels: - {{- include "datahub-mae-consumer.selectorLabels" . | nindent 8 }} - spec: - {{- with .Values.global.hostAliases }} - hostAliases: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.imagePullSecrets }} - imagePullSecrets: - {{- toYaml . | nindent 8 }} - {{- end }} - serviceAccountName: {{ include "datahub-mae-consumer.serviceAccountName" . }} - securityContext: - {{- toYaml .Values.podSecurityContext | nindent 8 }} - volumes: - {{- with .Values.global.credentialsAndCertsSecrets }} - - name: datahub-certs-dir - secret: - defaultMode: 0444 - secretName: {{ .name }} - {{- end }} - {{- if .Values.exporters.jmx.enabled }} - - name: config-jmx-exporter - configMap: - name: {{ include "datahub-mae-consumer.fullname" . }}-config-jmx-exporter - {{- end }} - {{- with .Values.extraVolumes }} - {{- toYaml . | nindent 8 }} - {{- end }} - initContainers: - {{- with .Values.extraInitContainers }} - {{- toYaml . | nindent 8 }} - {{- end }} - containers: - - name: {{ .Chart.Name }} - securityContext: - {{- toYaml .Values.securityContext | nindent 12 }} - image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" - imagePullPolicy: {{ .Values.image.pullPolicy }} - ports: - - name: http - containerPort: 9091 - protocol: TCP - livenessProbe: - httpGet: - path: /actuator/health - port: http - initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }} - periodSeconds: {{ .Values.livenessProbe.periodSeconds }} - failureThreshold: {{ .Values.livenessProbe.failureThreshold }} - readinessProbe: - httpGet: - path: /actuator/health - port: http - initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }} - periodSeconds: {{ .Values.readinessProbe.periodSeconds }} - failureThreshold: {{ .Values.readinessProbe.failureThreshold }} - env: - - name: MAE_CONSUMER_ENABLED - value: "true" - - name: GMS_HOST - value: {{ printf "%s-%s" .Release.Name "datahub-gms" }} - - name: GMS_PORT - value: "{{ .Values.global.datahub.gms.port }}" - - name: KAFKA_BOOTSTRAP_SERVER - value: "{{ .Values.global.kafka.bootstrap.server }}" - - name: KAFKA_SCHEMAREGISTRY_URL - value: "{{ .Values.global.kafka.schemaregistry.url }}" - - name: ELASTICSEARCH_HOST - value: "{{ .Values.global.elasticsearch.host }}" - - name: ELASTICSEARCH_PORT - value: "{{ .Values.global.elasticsearch.port }}" - {{- with .Values.global.elasticsearch.useSSL }} - - name: ELASTICSEARCH_USE_SSL - value: {{ . | quote }} - {{- end }} - {{- with .Values.global.elasticsearch.auth }} - - name: ELASTICSEARCH_USERNAME - value: {{ .username }} - - name: ELASTICSEARCH_PASSWORD - valueFrom: - secretKeyRef: - name: "{{ .password.secretRef }}" - key: "{{ .password.secretKey }}" - {{- end }} - - name: GRAPH_SERVICE_IMPL - value: {{ .Values.global.graph_service_impl }} - {{- if eq .Values.global.graph_service_impl "neo4j" }} - - name: NEO4J_HOST - value: "{{ .Values.global.neo4j.host }}" - - name: NEO4J_URI - value: "{{ .Values.global.neo4j.uri }}" - - name: NEO4J_USERNAME - value: "{{ .Values.global.neo4j.username }}" - - name: NEO4J_PASSWORD - valueFrom: - secretKeyRef: - name: "{{ .Values.global.neo4j.password.secretRef }}" - key: "{{ .Values.global.neo4j.password.secretKey }}" - {{- end }} - - name: DATAHUB_ANALYTICS_ENABLED - value: "{{ .Values.global.datahub_analytics_enabled }}" - {{- if .Values.global.springKafkaConfigurationOverrides }} - {{- range $configName, $configValue := .Values.global.springKafkaConfigurationOverrides }} - - name: SPRING_KAFKA_PROPERTIES_{{ $configName | replace "." "_" | upper }} - value: {{ $configValue }} - {{- end }} - {{- end }} - {{- if .Values.global.credentialsAndCertsSecrets }} - {{- range $envVarName, $envVarValue := .Values.global.credentialsAndCertsSecrets.secureEnv }} - - name: SPRING_KAFKA_PROPERTIES_{{ $envVarName | replace "." "_" | upper }} - valueFrom: - secretKeyRef: - name: {{ $.Values.global.credentialsAndCertsSecrets.name }} - key: {{ $envVarValue }} - {{- end }} - {{- end }} - {{- with .Values.extraEnvs }} - {{- toYaml . | nindent 12 }} - {{- end }} - volumeMounts: - {{- with .Values.global.credentialsAndCertsSecrets }} - - name: datahub-certs-dir - mountPath: {{ .path | default "/mnt/certs" }} - {{- end }} - {{- with .Values.extraVolumeMounts }} - {{- toYaml . | nindent 12 }} - {{- end }} - resources: - {{- toYaml .Values.resources | nindent 12 }} - - {{- include "datahub-jmxexporter.container" . }} - - {{- with .Values.nodeSelector }} - nodeSelector: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.affinity }} - affinity: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.tolerations }} - tolerations: - {{- toYaml . | nindent 8 }} - {{- end }} diff --git a/datahub-kubernetes/datahub/charts/datahub-mae-consumer/templates/service.yaml b/datahub-kubernetes/datahub/charts/datahub-mae-consumer/templates/service.yaml deleted file mode 100644 index 4a2178ff52e86a..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-mae-consumer/templates/service.yaml +++ /dev/null @@ -1,15 +0,0 @@ -apiVersion: v1 -kind: Service -metadata: - name: {{ printf "%s-%s" .Release.Name "datahub-mae-consumer" }} - labels: - {{- include "datahub-mae-consumer.labels" . | nindent 4 }} -spec: - type: {{ .Values.service.type }} - ports: - - port: {{ .Values.global.datahub.mae_consumer.port }} - targetPort: http - protocol: TCP - name: http - selector: - {{- include "datahub-mae-consumer.selectorLabels" . | nindent 4 }} diff --git a/datahub-kubernetes/datahub/charts/datahub-mae-consumer/templates/serviceaccount.yaml b/datahub-kubernetes/datahub/charts/datahub-mae-consumer/templates/serviceaccount.yaml deleted file mode 100644 index 14e6c5f6ced0ae..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-mae-consumer/templates/serviceaccount.yaml +++ /dev/null @@ -1,12 +0,0 @@ -{{- if .Values.serviceAccount.create -}} -apiVersion: v1 -kind: ServiceAccount -metadata: - name: {{ include "datahub-mae-consumer.serviceAccountName" . }} - labels: - {{- include "datahub-mae-consumer.labels" . | nindent 4 }} - {{- with .Values.serviceAccount.annotations }} - annotations: - {{- toYaml . | nindent 4 }} - {{- end }} -{{- end -}} diff --git a/datahub-kubernetes/datahub/charts/datahub-mae-consumer/values.yaml b/datahub-kubernetes/datahub/charts/datahub-mae-consumer/values.yaml deleted file mode 100644 index b2496e9f301068..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-mae-consumer/values.yaml +++ /dev/null @@ -1,179 +0,0 @@ -# Default values for datahub-mae-consumer. -# This is a YAML-formatted file. -# Declare variables to be passed into your templates. - -replicaCount: 1 - -image: - repository: linkedin/datahub-mae-consumer - pullPolicy: Always - tag: v0.8.5 - -imagePullSecrets: [] -nameOverride: "" -fullnameOverride: "" - -serviceAccount: - # Specifies whether a service account should be created - create: true - # Annotations to add to the service account - annotations: {} - # The name of the service account to use. - # If not set and create is true, a name is generated using the fullname template - name: - -podAnnotations: {} - # co.elastic.logs/enabled: "true" - -podSecurityContext: {} - # fsGroup: 2000 - -securityContext: {} - # capabilities: - # drop: - # - ALL - # readOnlyRootFilesystem: true - # runAsNonRoot: true - # runAsUser: 1000 - -service: - type: ClusterIP - port: 80 - -ingress: - enabled: false - annotations: {} - # kubernetes.io/ingress.class: nginx - # kubernetes.io/tls-acme: "true" - hosts: - - host: chart-example.local - paths: [] - tls: [] - # - secretName: chart-example-tls - # hosts: - # - chart-example.local - -# Extra environment variables -# This will be appended to the current 'env:' key. You can use any of the kubernetes env -# syntax here -extraEnvs: [] - # - name: MY_ENVIRONMENT_VAR - # value: the_value_goes_here - -extraVolumes: [] - # - name: extras - # emptyDir: {} - -extraVolumeMounts: [] - # - name: extras - # mountPath: /usr/share/extras - # readOnly: true - -extraInitContainers: [] - -resources: {} - # We usually recommend not to specify default resources and to leave this as a conscious - # choice for the user. This also increases chances charts run on environments with little - # resources, such as Minikube. If you do want to specify resources, uncomment the following - # lines, adjust them as necessary, and remove the curly braces after 'resources:'. - # limits: - # cpu: 100m - # memory: 128Mi - # requests: - # cpu: 100m - # memory: 128Mi - -nodeSelector: {} - -tolerations: [] - -affinity: {} - -env: - JMXPORT: 1099 - -exporters: - jmx: - enabled: false -# image: -# repository: bitnami/jmx-exporter -# tag: 0.15.0 -# pullPolicy: IfNotPresent -# config: -# lowercaseOutputName: true -# rules: -# - pattern: ".*" -# startDelaySeconds: 30 -# env: {} -# resources: {} -# path: /metrics -# ports: -# jmxxp: -# containerPort: 5556 -# protocol: TCP -# livenessProbe: -# httpGet: -# path: /metrics -# port: jmxxp -# initialDelaySeconds: 30 -# periodSeconds: 15 -# timeoutSeconds: 60 -# failureThreshold: 8 -# successThreshold: 1 -# readinessProbe: -# httpGet: -# path: /metrics -# port: jmxxp -# initialDelaySeconds: 30 -# periodSeconds: 15 -# timeoutSeconds: 60 -# failureThreshold: 8 -# successThreshold: 1 -# serviceMonitor: -# interval: 30s -# scrapeTimeout: 30s -# scheme: http - -livenessProbe: - initialDelaySeconds: 60 - periodSeconds: 30 - failureThreshold: 8 - -readinessProbe: - initialDelaySeconds: 60 - periodSeconds: 30 - failureThreshold: 8 - -global: - graph_service_impl: neo4j - datahub_analytics_enabled: true - - elasticsearch: - host: "elasticsearch" - port: "9200" - - kafka: - bootstrap: - server: "broker:9092" - schemaregistry: - url: "http://schema-registry:8081" - - neo4j: - host: "neo4j:7474" - uri: "bolt://neo4j" - username: "neo4j" - password: - secretRef: "neo4j-secrets" - secretKey: "neo4j-password" - - datahub: - mae_consumer: - port: "9091" - - hostAliases: - - ip: "192.168.0.104" - hostnames: - - "broker" - - "mysql" - - "elasticsearch" - - "neo4j" diff --git a/datahub-kubernetes/datahub/charts/datahub-mce-consumer/.helmignore b/datahub-kubernetes/datahub/charts/datahub-mce-consumer/.helmignore deleted file mode 100644 index 0e8a0eb36f4ca2..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-mce-consumer/.helmignore +++ /dev/null @@ -1,23 +0,0 @@ -# Patterns to ignore when building packages. -# This supports shell glob matching, relative path matching, and -# negation (prefixed with !). Only one pattern per line. -.DS_Store -# Common VCS dirs -.git/ -.gitignore -.bzr/ -.bzrignore -.hg/ -.hgignore -.svn/ -# Common backup files -*.swp -*.bak -*.tmp -*.orig -*~ -# Various IDEs -.project -.idea/ -*.tmproj -.vscode/ diff --git a/datahub-kubernetes/datahub/charts/datahub-mce-consumer/Chart.yaml b/datahub-kubernetes/datahub/charts/datahub-mce-consumer/Chart.yaml deleted file mode 100644 index bd00d74aa261ae..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-mce-consumer/Chart.yaml +++ /dev/null @@ -1,21 +0,0 @@ -apiVersion: v2 -name: datahub-mce-consumer -description: A Helm chart for Kubernetes - -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. -type: application - -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -version: 0.2.1 - -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. -appVersion: 0.3.1 diff --git a/datahub-kubernetes/datahub/charts/datahub-mce-consumer/README.md b/datahub-kubernetes/datahub/charts/datahub-mce-consumer/README.md deleted file mode 100644 index 5a7483fbb2b972..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-mce-consumer/README.md +++ /dev/null @@ -1,52 +0,0 @@ -datahub-mce-consumer -==================== -A Helm chart for datahub-mce-consumer - -Current chart version is `0.2.0` - -## Chart Values - -| Key | Type | Default | Description | -|-----|------|---------|-------------| -| affinity | object | `{}` | | -| exporters.jmx.enabled | boolean | false | | -| extraEnvs | Extra [environment variables][] which will be appended to the `env:` definition for the container | `[]` | -| extraVolumes | Templatable string of additional `volumes` to be passed to the `tpl` function | "" | -| extraVolumeMounts | Templatable string of additional `volumeMounts` to be passed to the `tpl` function | "" | -| fullnameOverride | string | `""` | | -| global.kafka.bootstrap.server | string | `"broker:9092"` | | -| global.kafka.schemaregistry.url | string | `"http://schema-registry:8081"` | | -| global.datahub.gms.port | string | `"8080"` | | -| global.hostAliases[0].hostnames[0] | string | `"broker"` | | -| global.hostAliases[0].hostnames[1] | string | `"mysql"` | | -| global.hostAliases[0].hostnames[2] | string | `"elasticsearch"` | | -| global.hostAliases[0].hostnames[3] | string | `"neo4j"` | | -| global.hostAliases[0].ip | string | `"192.168.0.104"` | | -| image.pullPolicy | string | `"IfNotPresent"` | | -| image.repository | string | `"linkedin/datahub-mce-consumer"` | | -| image.tag | string | `"v0.8.5"` | | -| imagePullSecrets | list | `[]` | | -| ingress.annotations | object | `{}` | | -| ingress.enabled | bool | `false` | | -| ingress.hosts[0].host | string | `"chart-example.local"` | | -| ingress.hosts[0].paths | list | `[]` | | -| ingress.tls | list | `[]` | | -| livenessProbe.initialDelaySeconds | int | `60` | | -| livenessProbe.periodSeconds | int | `30` | | -| livenessProbe.failureThreshold | int | `4` | | -| nameOverride | string | `""` | | -| nodeSelector | object | `{}` | | -| podAnnotations | object | `{}` | | -| podSecurityContext | object | `{}` | | -| readinessProbe.initialDelaySeconds | int | `60` | | -| readinessProbe.periodSeconds | int | `30` | | -| readinessProbe.failureThreshold | int | `4` | | -| replicaCount | int | `1` | | -| resources | object | `{}` | | -| securityContext | object | `{}` | | -| service.port | int | `80` | | -| service.type | string | `"ClusterIP"` | | -| serviceAccount.annotations | object | `{}` | | -| serviceAccount.create | bool | `true` | | -| serviceAccount.name | string | `nil` | | -| tolerations | list | `[]` | | diff --git a/datahub-kubernetes/datahub/charts/datahub-mce-consumer/templates/NOTES.txt b/datahub-kubernetes/datahub/charts/datahub-mce-consumer/templates/NOTES.txt deleted file mode 100644 index 8124ab7da3da8d..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-mce-consumer/templates/NOTES.txt +++ /dev/null @@ -1,21 +0,0 @@ -1. Get the application URL by running these commands: -{{- if .Values.ingress.enabled }} -{{- range $host := .Values.ingress.hosts }} - {{- range .paths }} - http{{ if $.Values.ingress.tls }}s{{ end }}://{{ $host.host }}{{ . }} - {{- end }} -{{- end }} -{{- else if contains "NodePort" .Values.service.type }} - export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "datahub-mce-consumer.fullname" . }}) - export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}") - echo http://$NODE_IP:$NODE_PORT -{{- else if contains "LoadBalancer" .Values.service.type }} - NOTE: It may take a few minutes for the LoadBalancer IP to be available. - You can watch the status of by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "datahub-mce-consumer.fullname" . }}' - export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "datahub-mce-consumer.fullname" . }} --template "{{"{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}"}}") - echo http://$SERVICE_IP:{{ .Values.service.port }} -{{- else if contains "ClusterIP" .Values.service.type }} - export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "datahub-mce-consumer.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}") - echo "Visit http://127.0.0.1:8080 to use your application" - kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8080:80 -{{- end }} diff --git a/datahub-kubernetes/datahub/charts/datahub-mce-consumer/templates/_helpers.tpl b/datahub-kubernetes/datahub/charts/datahub-mce-consumer/templates/_helpers.tpl deleted file mode 100644 index f7c47a79c45e34..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-mce-consumer/templates/_helpers.tpl +++ /dev/null @@ -1,63 +0,0 @@ -{{/* vim: set filetype=mustache: */}} -{{/* -Expand the name of the chart. -*/}} -{{- define "datahub-mce-consumer.name" -}} -{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} -{{- end -}} - -{{/* -Create a default fully qualified app name. -We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). -If release name contains chart name it will be used as a full name. -*/}} -{{- define "datahub-mce-consumer.fullname" -}} -{{- if .Values.fullnameOverride -}} -{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} -{{- else -}} -{{- $name := default .Chart.Name .Values.nameOverride -}} -{{- if contains $name .Release.Name -}} -{{- .Release.Name | trunc 63 | trimSuffix "-" -}} -{{- else -}} -{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} -{{- end -}} -{{- end -}} -{{- end -}} - -{{/* -Create chart name and version as used by the chart label. -*/}} -{{- define "datahub-mce-consumer.chart" -}} -{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} -{{- end -}} - -{{/* -Common labels -*/}} -{{- define "datahub-mce-consumer.labels" -}} -helm.sh/chart: {{ include "datahub-mce-consumer.chart" . }} -{{ include "datahub-mce-consumer.selectorLabels" . }} -{{- if .Chart.AppVersion }} -app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} -{{- end }} -app.kubernetes.io/managed-by: {{ .Release.Service }} -{{- end -}} - -{{/* -Selector labels -*/}} -{{- define "datahub-mce-consumer.selectorLabels" -}} -app.kubernetes.io/name: {{ include "datahub-mce-consumer.name" . }} -app.kubernetes.io/instance: {{ .Release.Name }} -{{- end -}} - -{{/* -Create the name of the service account to use -*/}} -{{- define "datahub-mce-consumer.serviceAccountName" -}} -{{- if .Values.serviceAccount.create -}} - {{ default (include "datahub-mce-consumer.fullname" .) .Values.serviceAccount.name }} -{{- else -}} - {{ default "default" .Values.serviceAccount.name }} -{{- end -}} -{{- end -}} diff --git a/datahub-kubernetes/datahub/charts/datahub-mce-consumer/templates/config-jmx-exporter.yaml b/datahub-kubernetes/datahub/charts/datahub-mce-consumer/templates/config-jmx-exporter.yaml deleted file mode 100644 index 7d1f05c346a143..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-mce-consumer/templates/config-jmx-exporter.yaml +++ /dev/null @@ -1,17 +0,0 @@ -{{- if .Values.exporters.jmx.enabled }} -apiVersion: v1 -kind: ConfigMap -metadata: - name: {{ include "datahub-mce-consumer.fullname" . }}-config-jmx-exporter - labels: - {{- include "datahub-mce-consumer.labels" . | nindent 4 }} -data: - config.yml: |- - hostPort: localhost:{{ .Values.env.JMXPORT }} - lowercaseOutputName: {{ .Values.exporters.jmx.config.lowercaseOutputName }} - lowercaseOutputLabelNames: {{ .Values.exporters.jmx.config.lowercaseOutputLabelNames }} - rules: -{{ .Values.exporters.jmx.config.rules | toYaml | indent 6 }} - ssl: false - startDelaySeconds: {{ .Values.exporters.jmx.config.startDelaySeconds }} -{{- end }} diff --git a/datahub-kubernetes/datahub/charts/datahub-mce-consumer/templates/deployment.yaml b/datahub-kubernetes/datahub/charts/datahub-mce-consumer/templates/deployment.yaml deleted file mode 100644 index 650668e6fbfb98..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-mce-consumer/templates/deployment.yaml +++ /dev/null @@ -1,124 +0,0 @@ -apiVersion: apps/v1 -kind: Deployment -metadata: - name: {{ include "datahub-mce-consumer.fullname" . }} - labels: - {{- include "datahub-mce-consumer.labels" . | nindent 4 }} -spec: - replicas: {{ .Values.replicaCount }} - selector: - matchLabels: - {{- include "datahub-mce-consumer.selectorLabels" . | nindent 6 }} - template: - metadata: - {{- with .Values.podAnnotations }} - annotations: - {{- toYaml . | nindent 8 }} - {{- end }} - labels: - {{- include "datahub-mce-consumer.selectorLabels" . | nindent 8 }} - spec: - {{- with .Values.global.hostAliases }} - hostAliases: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.imagePullSecrets }} - imagePullSecrets: - {{- toYaml . | nindent 8 }} - {{- end }} - serviceAccountName: {{ include "datahub-mce-consumer.serviceAccountName" . }} - securityContext: - {{- toYaml .Values.podSecurityContext | nindent 8 }} - volumes: - {{- if .Values.global.credentialsAndCertsSecrets }} - - name: datahub-certs-dir - secret: - defaultMode: 0444 - secretName: {{ .Values.global.credentialsAndCertsSecrets.name }} - {{- end }} - {{- if .Values.exporters.jmx.enabled }} - - name: config-jmx-exporter - configMap: - name: {{ include "datahub-mce-consumer.fullname" . }}-config-jmx-exporter - {{- end }} - {{- with .Values.extraVolumes }} - {{- toYaml . | nindent 8 }} - {{- end }} - initContainers: - {{- with .Values.extraInitContainers }} - {{- toYaml . | nindent 8 }} - {{- end }} - containers: - - name: {{ .Chart.Name }} - securityContext: - {{- toYaml .Values.securityContext | nindent 12 }} - image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" - imagePullPolicy: {{ .Values.image.pullPolicy }} - livenessProbe: - httpGet: - path: /actuator/health - port: 9090 - initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }} - periodSeconds: {{ .Values.livenessProbe.periodSeconds }} - failureThreshold: {{ .Values.livenessProbe.failureThreshold }} - readinessProbe: - httpGet: - path: /actuator/health - port: 9090 - initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }} - periodSeconds: {{ .Values.readinessProbe.periodSeconds }} - failureThreshold: {{ .Values.readinessProbe.failureThreshold }} - env: - - name: MCE_CONSUMER_ENABLED - value: "true" - - name: KAFKA_BOOTSTRAP_SERVER - value: "{{ .Values.global.kafka.bootstrap.server }}" - - name: KAFKA_SCHEMAREGISTRY_URL - value: "{{ .Values.global.kafka.schemaregistry.url }}" - - name: GMS_HOST - value: {{ printf "%s-%s" .Release.Name "datahub-gms" }} - - name: GMS_PORT - value: "{{ .Values.global.datahub.gms.port }}" - {{- if .Values.global.springKafkaConfigurationOverrides }} - {{- range $configName, $configValue := .Values.global.springKafkaConfigurationOverrides }} - - name: SPRING_KAFKA_PROPERTIES_{{ $configName | replace "." "_" | upper }} - value: {{ $configValue }} - {{- end }} - {{- end }} - {{- if .Values.global.credentialsAndCertsSecrets }} - {{- range $envVarName, $envVarValue := .Values.global.credentialsAndCertsSecrets.secureEnv }} - - name: SPRING_KAFKA_PROPERTIES_{{ $envVarName | replace "." "_" | upper }} - valueFrom: - secretKeyRef: - name: {{ $.Values.global.credentialsAndCertsSecrets.name }} - key: {{ $envVarValue }} - {{- end }} - {{- end }} - {{- with .Values.extraEnvs }} - {{- toYaml . | nindent 12 }} - {{- end }} - volumeMounts: - {{- with .Values.global.credentialsAndCertsSecrets }} - - name: datahub-certs-dir - mountPath: {{ .path | default "/mnt/certs" }} - {{- end }} - {{- with .Values.extraVolumeMounts }} - {{- toYaml . | nindent 12 }} - {{- end }} - resources: - {{- toYaml .Values.resources | nindent 12 }} - - {{- include "datahub-jmxexporter.container" . }} - - {{- with .Values.nodeSelector }} - nodeSelector: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.affinity }} - affinity: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.tolerations }} - tolerations: - {{- toYaml . | nindent 8 }} - {{- end }} diff --git a/datahub-kubernetes/datahub/charts/datahub-mce-consumer/templates/serviceaccount.yaml b/datahub-kubernetes/datahub/charts/datahub-mce-consumer/templates/serviceaccount.yaml deleted file mode 100644 index d8570643f3bf61..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-mce-consumer/templates/serviceaccount.yaml +++ /dev/null @@ -1,12 +0,0 @@ -{{- if .Values.serviceAccount.create -}} -apiVersion: v1 -kind: ServiceAccount -metadata: - name: {{ include "datahub-mce-consumer.serviceAccountName" . }} - labels: - {{- include "datahub-mce-consumer.labels" . | nindent 4 }} - {{- with .Values.serviceAccount.annotations }} - annotations: - {{- toYaml . | nindent 4 }} - {{- end }} -{{- end -}} diff --git a/datahub-kubernetes/datahub/charts/datahub-mce-consumer/values.yaml b/datahub-kubernetes/datahub/charts/datahub-mce-consumer/values.yaml deleted file mode 100644 index ef9a75a3da9acb..00000000000000 --- a/datahub-kubernetes/datahub/charts/datahub-mce-consumer/values.yaml +++ /dev/null @@ -1,164 +0,0 @@ -# Default values for datahub-mce-consumer. -# This is a YAML-formatted file. -# Declare variables to be passed into your templates. - -replicaCount: 1 - -image: - repository: linkedin/datahub-mce-consumer - pullPolicy: Always - tag: v0.8.5 - -imagePullSecrets: [] -nameOverride: "" -fullnameOverride: "" - -serviceAccount: - # Specifies whether a service account should be created - create: true - # Annotations to add to the service account - annotations: {} - # The name of the service account to use. - # If not set and create is true, a name is generated using the fullname template - name: - -podAnnotations: {} - # co.elastic.logs/enabled: "true" - -podSecurityContext: {} - # fsGroup: 2000 - -securityContext: {} - # capabilities: - # drop: - # - ALL - # readOnlyRootFilesystem: true - # runAsNonRoot: true - # runAsUser: 1000 - -service: - type: ClusterIP - port: 80 - -ingress: - enabled: false - annotations: {} - # kubernetes.io/ingress.class: nginx - # kubernetes.io/tls-acme: "true" - hosts: - - host: chart-example.local - paths: [] - tls: [] - # - secretName: chart-example-tls - # hosts: - # - chart-example.local - -# Extra environment variables -# This will be appended to the current 'env:' key. You can use any of the kubernetes env -# syntax here -extraEnvs: [] - # - name: MY_ENVIRONMENT_VAR - # value: the_value_goes_here - -extraVolumes: [] - # - name: extras - # emptyDir: {} - -extraVolumeMounts: [] - # - name: extras - # mountPath: /usr/share/extras - # readOnly: true - -extraInitContainers: [] - -resources: {} - # We usually recommend not to specify default resources and to leave this as a conscious - # choice for the user. This also increases chances charts run on environments with little - # resources, such as Minikube. If you do want to specify resources, uncomment the following - # lines, adjust them as necessary, and remove the curly braces after 'resources:'. - # limits: - # cpu: 100m - # memory: 128Mi - # requests: - # cpu: 100m - # memory: 128Mi - -nodeSelector: {} - -tolerations: [] - -affinity: {} - -env: - JMXPORT: 1099 - -exporters: - jmx: - enabled: false -# image: -# repository: bitnami/jmx-exporter -# tag: 0.15.0 -# pullPolicy: IfNotPresent -# config: -# lowercaseOutputName: true -# rules: -# - pattern: ".*" -# startDelaySeconds: 30 -# env: {} -# resources: {} -# path: /metrics -# ports: -# jmxxp: -# containerPort: 5556 -# protocol: TCP -# livenessProbe: -# httpGet: -# path: /metrics -# port: jmxxp -# initialDelaySeconds: 30 -# periodSeconds: 15 -# timeoutSeconds: 60 -# failureThreshold: 8 -# successThreshold: 1 -# readinessProbe: -# httpGet: -# path: /metrics -# port: jmxxp -# initialDelaySeconds: 30 -# periodSeconds: 15 -# timeoutSeconds: 60 -# failureThreshold: 8 -# successThreshold: 1 -# serviceMonitor: -# interval: 30s -# scrapeTimeout: 30s -# scheme: http - -livenessProbe: - initialDelaySeconds: 60 - periodSeconds: 30 - failureThreshold: 4 - -readinessProbe: - initialDelaySeconds: 60 - periodSeconds: 30 - failureThreshold: 4 - -global: - kafka: - bootstrap: - server: "broker:9092" - schemaregistry: - url: "http://schema-registry:8081" - - datahub: - gms: - port: "8080" - - hostAliases: - - ip: "192.168.0.104" - hostnames: - - "broker" - - "mysql" - - "elasticsearch" - - "neo4j" \ No newline at end of file diff --git a/datahub-kubernetes/datahub/quickstart-values-without-neo4j.yaml b/datahub-kubernetes/datahub/quickstart-values-without-neo4j.yaml deleted file mode 100644 index fae60c874a07c1..00000000000000 --- a/datahub-kubernetes/datahub/quickstart-values-without-neo4j.yaml +++ /dev/null @@ -1,78 +0,0 @@ -# Values to start up datahub after starting up the datahub-prerequisites chart with "prerequisites" release name -# Copy this chart and change configuration as needed. -datahub-gms: - enabled: true - image: - repository: linkedin/datahub-gms - tag: "v0.8.5" - -datahub-frontend: - enabled: true - image: - repository: linkedin/datahub-frontend-react - tag: "v0.8.5" - # Set up ingress to expose react front-end - ingress: - enabled: false - -elasticsearchSetupJob: - enabled: true - image: - repository: linkedin/datahub-elasticsearch-setup - tag: "v0.8.5" - -kafkaSetupJob: - enabled: true - image: - repository: linkedin/datahub-kafka-setup - tag: "v0.8.5" - -mysqlSetupJob: - enabled: true - image: - repository: acryldata/datahub-mysql-setup - tag: "v0.8.5" - -datahubUpgrade: - enabled: true - image: - repository: acryldata/datahub-upgrade - tag: "v0.8.5" - -datahub-ingestion-cron: - enabled: false - -global: - graph_service_impl: elasticsearch - - elasticsearch: - host: "elasticsearch-master" - port: "9200" - indexPrefix: demo - - kafka: - bootstrap: - server: "prerequisites-kafka:9092" - zookeeper: - server: "prerequisites-zookeeper:2181" - schemaregistry: - url: "http://prerequisites-cp-schema-registry:8081" - - sql: - datasource: - host: "prerequisites-mysql:3306" - hostForMysqlClient: "prerequisites-mysql" - port: "3306" - url: "jdbc:mysql://prerequisites-mysql:3306/datahub?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8&enabledTLSProtocols=TLSv1.2" - driver: "com.mysql.jdbc.Driver" - username: "root" - password: - secretRef: mysql-secrets - secretKey: mysql-root-password - - datahub: - gms: - port: "8080" - mae_consumer: - port: "9091" - appVersion: "1.0" diff --git a/datahub-kubernetes/datahub/quickstart-values.yaml b/datahub-kubernetes/datahub/quickstart-values.yaml deleted file mode 100644 index 6d062db1fb51b3..00000000000000 --- a/datahub-kubernetes/datahub/quickstart-values.yaml +++ /dev/null @@ -1,86 +0,0 @@ -# Values to start up datahub after starting up the datahub-prerequisites chart with "prerequisites" release name -# Copy this chart and change configuration as needed. -datahub-gms: - enabled: true - image: - repository: linkedin/datahub-gms - tag: "v0.8.5" - -datahub-frontend: - enabled: true - image: - repository: linkedin/datahub-frontend-react - tag: "v0.8.5" - # Set up ingress to expose react front-end - ingress: - enabled: false - -elasticsearchSetupJob: - enabled: true - image: - repository: linkedin/datahub-elasticsearch-setup - tag: "v0.8.5" - -kafkaSetupJob: - enabled: true - image: - repository: linkedin/datahub-kafka-setup - tag: "v0.8.5" - -mysqlSetupJob: - enabled: true - image: - repository: acryldata/datahub-mysql-setup - tag: "v0.8.5" - -datahubUpgrade: - enabled: true - image: - repository: acryldata/datahub-upgrade - tag: "v0.8.5" - -datahub-ingestion-cron: - enabled: false - -global: - graph_service_impl: neo4j - - elasticsearch: - host: "elasticsearch-master" - port: "9200" - indexPrefix: demo - - kafka: - bootstrap: - server: "prerequisites-kafka:9092" - zookeeper: - server: "prerequisites-zookeeper:2181" - schemaregistry: - url: "http://prerequisites-cp-schema-registry:8081" - - neo4j: - host: "prerequisites-neo4j-community:7474" - uri: "bolt://prerequisites-neo4j-community" - username: "neo4j" - password: - secretRef: neo4j-secrets - secretKey: neo4j-password - - sql: - datasource: - host: "prerequisites-mysql:3306" - hostForMysqlClient: "prerequisites-mysql" - port: "3306" - url: "jdbc:mysql://prerequisites-mysql:3306/datahub?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8&enabledTLSProtocols=TLSv1.2" - driver: "com.mysql.jdbc.Driver" - username: "root" - password: - secretRef: mysql-secrets - secretKey: mysql-root-password - - datahub: - gms: - port: "8080" - mae_consumer: - port: "9091" - appVersion: "1.0" diff --git a/datahub-kubernetes/datahub/templates/_helpers.tpl b/datahub-kubernetes/datahub/templates/_helpers.tpl deleted file mode 100644 index 586c1ad54f1057..00000000000000 --- a/datahub-kubernetes/datahub/templates/_helpers.tpl +++ /dev/null @@ -1,63 +0,0 @@ -{{/* vim: set filetype=mustache: */}} -{{/* -Expand the name of the chart. -*/}} -{{- define "datahub.name" -}} -{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} -{{- end -}} - -{{/* -Create a default fully qualified app name. -We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). -If release name contains chart name it will be used as a full name. -*/}} -{{- define "datahub.fullname" -}} -{{- if .Values.fullnameOverride -}} -{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} -{{- else -}} -{{- $name := default .Chart.Name .Values.nameOverride -}} -{{- if contains $name .Release.Name -}} -{{- .Release.Name | trunc 63 | trimSuffix "-" -}} -{{- else -}} -{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} -{{- end -}} -{{- end -}} -{{- end -}} - -{{/* -Create chart name and version as used by the chart label. -*/}} -{{- define "datahub.chart" -}} -{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} -{{- end -}} - -{{/* -Common labels -*/}} -{{- define "datahub.labels" -}} -helm.sh/chart: {{ include "datahub.chart" . }} -{{ include "datahub.selectorLabels" . }} -{{- if .Chart.AppVersion }} -app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} -{{- end }} -app.kubernetes.io/managed-by: {{ .Release.Service }} -{{- end -}} - -{{/* -Selector labels -*/}} -{{- define "datahub.selectorLabels" -}} -app.kubernetes.io/name: {{ include "datahub.name" . }} -app.kubernetes.io/instance: {{ .Release.Name }} -{{- end -}} - -{{/* -Create the name of the service account to use -*/}} -{{- define "datahub.serviceAccountName" -}} -{{- if .Values.serviceAccount.create -}} - {{ default (include "datahub.fullname" .) .Values.serviceAccount.name }} -{{- else -}} - {{ default "default" .Values.serviceAccount.name }} -{{- end -}} -{{- end -}} diff --git a/datahub-kubernetes/datahub/templates/datahub-upgrade/datahub-cleanup-job-template.yml b/datahub-kubernetes/datahub/templates/datahub-upgrade/datahub-cleanup-job-template.yml deleted file mode 100644 index 7969bad8fb4a26..00000000000000 --- a/datahub-kubernetes/datahub/templates/datahub-upgrade/datahub-cleanup-job-template.yml +++ /dev/null @@ -1,117 +0,0 @@ -{{- if .Values.datahubUpgrade.enabled -}} -# Job template for running datahub cleanup -# Creates a suspended cronJob that you can use to create an adhoc job when ready to run clean up. -# Run the following command to do so -# kubectl create job --from=cronjob/<>-datahub-cleanup-job-template datahub-cleanup-job -apiVersion: batch/v1beta1 -kind: CronJob -metadata: - name: {{ .Release.Name }}-datahub-cleanup-job-template - labels: - app.kubernetes.io/managed-by: {{ .Release.Service | quote }} - app.kubernetes.io/instance: {{ .Release.Name | quote }} - app.kubernetes.io/version: {{ .Chart.AppVersion }} - helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" -spec: - schedule: "* * * * *" - suspend: true - jobTemplate: - spec: - template: - spec: - {{- with .Values.global.hostAliases }} - hostAliases: - {{- toYaml . | nindent 12 }} - {{- end }} - {{- with .Values.datahubUpgrade.serviceAccount }} - serviceAccountName: {{ . }} - {{- end }} - {{- with .Values.imagePullSecrets }} - imagePullSecrets: - {{- toYaml . | nindent 12 }} - {{- end }} - volumes: - {{- with .Values.datahubUpgrade.extraVolumes }} - {{- toYaml . | nindent 12 }} - {{- end }} - restartPolicy: Never - securityContext: - runAsUser: 1000 - fsGroup: 1000 - containers: - - name: datahub-upgrade-job - image: "{{ .Values.datahubUpgrade.image.repository }}:{{ .Values.datahubUpgrade.image.tag }}" - args: [ "-u", "NoCodeDataMigrationCleanup" ] - env: - - name: DATAHUB_GMS_HOST - value: {{ printf "%s-%s" .Release.Name "datahub-gms" }} - - name: DATAHUB_GMS_PORT - value: "{{ .Values.global.datahub.gms.port }}" - - name: DATAHUB_MAE_CONSUMER_HOST - value: {{ printf "%s-%s" .Release.Name "datahub-mae-consumer" }} - - name: DATAHUB_MAE_CONSUMER_PORT - value: "{{ .Values.global.datahub.mae_consumer.port }}" - - name: EBEAN_DATASOURCE_USERNAME - value: "{{ .Values.global.sql.datasource.username }}" - - name: EBEAN_DATASOURCE_PASSWORD - valueFrom: - secretKeyRef: - name: "{{ .Values.global.sql.datasource.password.secretRef }}" - key: "{{ .Values.global.sql.datasource.password.secretKey }}" - - name: EBEAN_DATASOURCE_HOST - value: "{{ .Values.global.sql.datasource.host }}" - - name: EBEAN_DATASOURCE_URL - value: "{{ .Values.global.sql.datasource.url }}" - - name: EBEAN_DATASOURCE_DRIVER - value: "{{ .Values.global.sql.datasource.driver }}" - - name: KAFKA_BOOTSTRAP_SERVER - value: "{{ .Values.global.kafka.bootstrap.server }}" - - name: KAFKA_SCHEMAREGISTRY_URL - value: "{{ .Values.global.kafka.schemaregistry.url }}" - - name: ELASTICSEARCH_HOST - value: {{ .Values.global.elasticsearch.host | quote }} - - name: ELASTICSEARCH_PORT - value: {{ .Values.global.elasticsearch.port | quote }} - {{- with .Values.global.elasticsearch.useSSL }} - - name: ELASTICSEARCH_USE_SSL - value: {{ . | quote }} - {{- end }} - {{- with .Values.global.elasticsearch.auth }} - - name: ELASTICSEARCH_USERNAME - value: {{ .username }} - - name: ELASTICSEARCH_PASSWORD - valueFrom: - secretKeyRef: - name: "{{ .password.secretRef }}" - key: "{{ .password.secretKey }}" - {{- end }} - - name: GRAPH_SERVICE_IMPL - value: {{ .Values.global.graph_service_impl }} - {{- if eq .Values.global.graph_service_impl "neo4j" }} - - name: NEO4J_HOST - value: "{{ .Values.global.neo4j.host }}" - - name: NEO4J_URI - value: "{{ .Values.global.neo4j.uri }}" - - name: NEO4J_USERNAME - value: "{{ .Values.global.neo4j.username }}" - - name: NEO4J_PASSWORD - valueFrom: - secretKeyRef: - name: "{{ .Values.global.neo4j.password.secretRef }}" - key: "{{ .Values.global.neo4j.password.secretKey }}" - {{- end }} - {{- with .Values.datahubUpgrade.extraEnvs }} - {{- toYaml . | nindent 16 }} - {{- end }} - volumeMounts: - {{- with .Values.datahubUpgrade.extraVolumeMounts }} - {{- toYaml . | nindent 16 }} - {{- end }} - resources: - limits: - cpu: 500m - memory: 512Mi - requests: - cpu: 300m - memory: 256Mi -{{- end -}} \ No newline at end of file diff --git a/datahub-kubernetes/datahub/templates/datahub-upgrade/datahub-restore-indices-job-template.yml b/datahub-kubernetes/datahub/templates/datahub-upgrade/datahub-restore-indices-job-template.yml deleted file mode 100644 index a98128677cb777..00000000000000 --- a/datahub-kubernetes/datahub/templates/datahub-upgrade/datahub-restore-indices-job-template.yml +++ /dev/null @@ -1,119 +0,0 @@ -{{- if .Values.datahubUpgrade.enabled -}} -# Job template for restoring indices by sending MAE corresponding to all entities in the local db -# Creates a suspended cronJob that you can use to create an adhoc job when ready to run clean up. -# Run the following command to do so -# kubectl create job --from=cronjob/<>-datahub-restore-indices-job-template datahub-restore-indices-job -apiVersion: batch/v1beta1 -kind: CronJob -metadata: - name: {{ .Release.Name }}-datahub-restore-indices-job-template - labels: - app.kubernetes.io/managed-by: {{ .Release.Service | quote }} - app.kubernetes.io/instance: {{ .Release.Name | quote }} - app.kubernetes.io/version: {{ .Chart.AppVersion }} - helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" -spec: - schedule: "* * * * *" - suspend: true - jobTemplate: - spec: - template: - spec: - {{- with .Values.global.hostAliases }} - hostAliases: - {{- toYaml . | nindent 12 }} - {{- end }} - {{- with .Values.datahubUpgrade.serviceAccount }} - serviceAccountName: {{ . }} - {{- end }} - {{- with .Values.imagePullSecrets }} - imagePullSecrets: - {{- toYaml . | nindent 12 }} - {{- end }} - volumes: - {{- with .Values.datahubUpgrade.extraVolumes }} - {{- toYaml . | nindent 12 }} - {{- end }} - restartPolicy: Never - securityContext: - runAsUser: 1000 - fsGroup: 1000 - containers: - - name: datahub-upgrade-job - image: "{{ .Values.datahubUpgrade.image.repository }}:{{ .Values.datahubUpgrade.image.tag }}" - args: - - "-u" - - "RestoreIndices" - env: - - name: DATAHUB_GMS_HOST - value: {{ printf "%s-%s" .Release.Name "datahub-gms" }} - - name: DATAHUB_GMS_PORT - value: "{{ .Values.global.datahub.gms.port }}" - - name: DATAHUB_MAE_CONSUMER_HOST - value: {{ printf "%s-%s" .Release.Name "datahub-mae-consumer" }} - - name: DATAHUB_MAE_CONSUMER_PORT - value: "{{ .Values.global.datahub.mae_consumer.port }}" - - name: EBEAN_DATASOURCE_USERNAME - value: "{{ .Values.global.sql.datasource.username }}" - - name: EBEAN_DATASOURCE_PASSWORD - valueFrom: - secretKeyRef: - name: "{{ .Values.global.sql.datasource.password.secretRef }}" - key: "{{ .Values.global.sql.datasource.password.secretKey }}" - - name: EBEAN_DATASOURCE_HOST - value: "{{ .Values.global.sql.datasource.host }}" - - name: EBEAN_DATASOURCE_URL - value: "{{ .Values.global.sql.datasource.url }}" - - name: EBEAN_DATASOURCE_DRIVER - value: "{{ .Values.global.sql.datasource.driver }}" - - name: KAFKA_BOOTSTRAP_SERVER - value: "{{ .Values.global.kafka.bootstrap.server }}" - - name: KAFKA_SCHEMAREGISTRY_URL - value: "{{ .Values.global.kafka.schemaregistry.url }}" - - name: ELASTICSEARCH_HOST - value: {{ .Values.global.elasticsearch.host | quote }} - - name: ELASTICSEARCH_PORT - value: {{ .Values.global.elasticsearch.port | quote }} - {{- with .Values.global.elasticsearch.useSSL }} - - name: ELASTICSEARCH_USE_SSL - value: {{ . | quote }} - {{- end }} - {{- with .Values.global.elasticsearch.auth }} - - name: ELASTICSEARCH_USERNAME - value: {{ .username }} - - name: ELASTICSEARCH_PASSWORD - valueFrom: - secretKeyRef: - name: "{{ .password.secretRef }}" - key: "{{ .password.secretKey }}" - {{- end }} - - name: GRAPH_SERVICE_IMPL - value: {{ .Values.global.graph_service_impl }} - {{- if eq .Values.global.graph_service_impl "neo4j" }} - - name: NEO4J_HOST - value: "{{ .Values.global.neo4j.host }}" - - name: NEO4J_URI - value: "{{ .Values.global.neo4j.uri }}" - - name: NEO4J_USERNAME - value: "{{ .Values.global.neo4j.username }}" - - name: NEO4J_PASSWORD - valueFrom: - secretKeyRef: - name: "{{ .Values.global.neo4j.password.secretRef }}" - key: "{{ .Values.global.neo4j.password.secretKey }}" - {{- end }} - {{- with .Values.datahubUpgrade.extraEnvs }} - {{- toYaml . | nindent 16 }} - {{- end }} - volumeMounts: - {{- with .Values.datahubUpgrade.extraVolumeMounts }} - {{- toYaml . | nindent 16 }} - {{- end }} - resources: - limits: - cpu: 500m - memory: 512Mi - requests: - cpu: 300m - memory: 256Mi -{{- end -}} \ No newline at end of file diff --git a/datahub-kubernetes/datahub/templates/datahub-upgrade/datahub-upgrade-job.yml b/datahub-kubernetes/datahub/templates/datahub-upgrade/datahub-upgrade-job.yml deleted file mode 100644 index ee03347426d657..00000000000000 --- a/datahub-kubernetes/datahub/templates/datahub-upgrade/datahub-upgrade-job.yml +++ /dev/null @@ -1,123 +0,0 @@ -{{- if .Values.datahubUpgrade.enabled -}} -apiVersion: batch/v1 -kind: Job -metadata: - name: {{ .Release.Name }}-datahub-upgrade-job - labels: - app.kubernetes.io/managed-by: {{ .Release.Service | quote }} - app.kubernetes.io/instance: {{ .Release.Name | quote }} - app.kubernetes.io/version: {{ .Chart.AppVersion }} - helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" - annotations: - # This is what defines this resource as a hook. Without this line, the - # job is considered part of the release. - "helm.sh/hook": post-install,post-upgrade - "helm.sh/hook-weight": "-5" - "helm.sh/hook-delete-policy": before-hook-creation -spec: - template: - spec: - {{- with .Values.global.hostAliases }} - hostAliases: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.datahubUpgrade.serviceAccount }} - serviceAccountName: {{ . }} - {{- end }} - {{- with .Values.imagePullSecrets }} - imagePullSecrets: - {{- toYaml . | nindent 8 }} - {{- end }} - volumes: - {{- with .Values.datahubUpgrade.extraVolumes }} - {{- toYaml . | nindent 8}} - {{- end }} - restartPolicy: Never - securityContext: - runAsUser: 1000 - fsGroup: 1000 - containers: - - name: datahub-upgrade-job - image: "{{ .Values.datahubUpgrade.image.repository }}:{{ .Values.datahubUpgrade.image.tag }}" - args: - - "-u" - - "NoCodeDataMigration" - - "-a" - - "batchSize=1000" - - "-a" - - "batchDelayMs=100" - - "-a" - - "dbType={{ .Values.datahubUpgrade.noCodeDataMigration.sqlDbType }}" - env: - - name: DATAHUB_GMS_HOST - value: {{ printf "%s-%s" .Release.Name "datahub-gms" }} - - name: DATAHUB_GMS_PORT - value: "{{ .Values.global.datahub.gms.port }}" - - name: DATAHUB_MAE_CONSUMER_HOST - value: {{ printf "%s-%s" .Release.Name "datahub-mae-consumer" }} - - name: DATAHUB_MAE_CONSUMER_PORT - value: "{{ .Values.global.datahub.mae_consumer.port }}" - - name: EBEAN_DATASOURCE_USERNAME - value: "{{ .Values.global.sql.datasource.username }}" - - name: EBEAN_DATASOURCE_PASSWORD - valueFrom: - secretKeyRef: - name: "{{ .Values.global.sql.datasource.password.secretRef }}" - key: "{{ .Values.global.sql.datasource.password.secretKey }}" - - name: EBEAN_DATASOURCE_HOST - value: "{{ .Values.global.sql.datasource.host }}" - - name: EBEAN_DATASOURCE_URL - value: "{{ .Values.global.sql.datasource.url }}" - - name: EBEAN_DATASOURCE_DRIVER - value: "{{ .Values.global.sql.datasource.driver }}" - - name: KAFKA_BOOTSTRAP_SERVER - value: "{{ .Values.global.kafka.bootstrap.server }}" - - name: KAFKA_SCHEMAREGISTRY_URL - value: "{{ .Values.global.kafka.schemaregistry.url }}" - - name: ELASTICSEARCH_HOST - value: {{ .Values.global.elasticsearch.host | quote }} - - name: ELASTICSEARCH_PORT - value: {{ .Values.global.elasticsearch.port | quote }} - {{- with .Values.global.elasticsearch.useSSL }} - - name: ELASTICSEARCH_USE_SSL - value: {{ . | quote }} - {{- end }} - {{- with .Values.global.elasticsearch.auth }} - - name: ELASTICSEARCH_USERNAME - value: {{ .username }} - - name: ELASTICSEARCH_PASSWORD - valueFrom: - secretKeyRef: - name: "{{ .password.secretRef }}" - key: "{{ .password.secretKey }}" - {{- end }} - - name: GRAPH_SERVICE_IMPL - value: {{ .Values.global.graph_service_impl }} - {{- if eq .Values.global.graph_service_impl "neo4j" }} - - name: NEO4J_HOST - value: "{{ .Values.global.neo4j.host }}" - - name: NEO4J_URI - value: "{{ .Values.global.neo4j.uri }}" - - name: NEO4J_USERNAME - value: "{{ .Values.global.neo4j.username }}" - - name: NEO4J_PASSWORD - valueFrom: - secretKeyRef: - name: "{{ .Values.global.neo4j.password.secretRef }}" - key: "{{ .Values.global.neo4j.password.secretKey }}" - {{- end }} - {{- with .Values.datahubUpgrade.extraEnvs }} - {{- toYaml . | nindent 12 }} - {{- end }} - volumeMounts: - {{- with .Values.datahubUpgrade.extraVolumeMounts }} - {{- toYaml . | nindent 12 }} - {{- end }} - resources: - limits: - cpu: 500m - memory: 512Mi - requests: - cpu: 300m - memory: 256Mi -{{- end -}} \ No newline at end of file diff --git a/datahub-kubernetes/datahub/templates/elasticsearch-setup-job.yml b/datahub-kubernetes/datahub/templates/elasticsearch-setup-job.yml deleted file mode 100644 index 1aaed30ee7c1bc..00000000000000 --- a/datahub-kubernetes/datahub/templates/elasticsearch-setup-job.yml +++ /dev/null @@ -1,76 +0,0 @@ -{{- if .Values.elasticsearchSetupJob.enabled -}} -apiVersion: batch/v1 -kind: Job -metadata: - name: {{ .Release.Name }}-elasticsearch-setup-job - labels: - app.kubernetes.io/managed-by: {{ .Release.Service | quote }} - app.kubernetes.io/instance: {{ .Release.Name | quote }} - app.kubernetes.io/version: {{ .Chart.AppVersion }} - helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" - annotations: - # This is what defines this resource as a hook. Without this line, the - # job is considered part of the release. - "helm.sh/hook": pre-install,pre-upgrade - "helm.sh/hook-weight": "-5" - "helm.sh/hook-delete-policy": before-hook-creation -spec: - template: - spec: - {{- with .Values.global.hostAliases }} - hostAliases: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.elasticsearchSetupJob.serviceAccount }} - serviceAccountName: {{ . }} - {{- end }} - {{- with .Values.imagePullSecrets }} - imagePullSecrets: - {{- toYaml . | nindent 8 }} - {{- end }} - volumes: - {{- with .Values.elasticsearchSetupJob.extraVolumes }} - {{- toYaml . | nindent 8}} - {{- end }} - restartPolicy: Never - securityContext: - runAsUser: 1000 - fsGroup: 1000 - containers: - - name: elasticsearch-setup-job - image: "{{ .Values.elasticsearchSetupJob.image.repository }}:{{ .Values.elasticsearchSetupJob.image.tag }}" - env: - - name: ELASTICSEARCH_HOST - value: {{ .Values.global.elasticsearch.host | quote }} - - name: ELASTICSEARCH_PORT - value: {{ .Values.global.elasticsearch.port | quote }} - {{- with .Values.global.elasticsearch.useSSL }} - - name: ELASTICSEARCH_USE_SSL - value: {{ . | quote }} - {{- end }} - {{- with .Values.global.elasticsearch.auth }} - - name: ELASTICSEARCH_USERNAME - value: {{ .username }} - - name: ELASTICSEARCH_PASSWORD - valueFrom: - secretKeyRef: - name: "{{ .password.secretRef }}" - key: "{{ .password.secretKey }}" - {{- end }} - - name: DATAHUB_ANALYTICS_ENABLED - value: {{ .Values.global.datahub_analytics_enabled | quote }} - {{- with .Values.elasticsearchSetupJob.extraEnvs }} - {{- toYaml . | nindent 12 }} - {{- end }} - volumeMounts: - {{- with .Values.elasticsearchSetupJob.extraVolumeMounts }} - {{- toYaml . | nindent 12 }} - {{- end }} - resources: - limits: - cpu: 500m - memory: 512Mi - requests: - cpu: 300m - memory: 256Mi -{{- end -}} \ No newline at end of file diff --git a/datahub-kubernetes/datahub/templates/kafka-setup-job.yml b/datahub-kubernetes/datahub/templates/kafka-setup-job.yml deleted file mode 100644 index 5776398e3670c9..00000000000000 --- a/datahub-kubernetes/datahub/templates/kafka-setup-job.yml +++ /dev/null @@ -1,86 +0,0 @@ -{{- if .Values.kafkaSetupJob.enabled -}} -apiVersion: batch/v1 -kind: Job -metadata: - name: {{ .Release.Name }}-kafka-setup-job - labels: - app.kubernetes.io/managed-by: {{ .Release.Service | quote }} - app.kubernetes.io/instance: {{ .Release.Name | quote }} - app.kubernetes.io/version: {{ .Chart.AppVersion }} - helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" - annotations: - # This is what defines this resource as a hook. Without this line, the - # job is considered part of the release. - "helm.sh/hook": pre-install,pre-upgrade - "helm.sh/hook-weight": "-5" - "helm.sh/hook-delete-policy": before-hook-creation -spec: - template: - spec: - {{- with .Values.global.hostAliases }} - hostAliases: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.kafkaSetupJob.serviceAccount }} - serviceAccountName: {{ . }} - {{- end }} - restartPolicy: Never - securityContext: - runAsUser: 1000 - fsGroup: 1000 - {{- with .Values.imagePullSecrets }} - imagePullSecrets: - {{- toYaml . | nindent 8 }} - {{- end }} - volumes: - {{- with .Values.global.credentialsAndCertsSecrets }} - - name: datahub-certs-dir - secret: - defaultMode: 0444 - secretName: {{ .name }} - {{- end }} - {{- with .Values.kafkaSetupJob.extraVolumes }} - {{- toYaml . | nindent 8 }} - {{- end }} - containers: - - name: kafka-setup-job - image: "{{ .Values.kafkaSetupJob.image.repository }}:{{ .Values.kafkaSetupJob.image.tag }}" - env: - - name: KAFKA_ZOOKEEPER_CONNECT - value: {{ .Values.global.kafka.zookeeper.server | quote }} - - name: KAFKA_BOOTSTRAP_SERVER - value: {{ .Values.global.kafka.bootstrap.server | quote }} - {{- if .Values.global.springKafkaConfigurationOverrides }} - {{- range $configName, $configValue := .Values.global.springKafkaConfigurationOverrides }} - - name: KAFKA_PROPERTIES_{{ $configName | replace "." "_" | upper }} - value: {{ $configValue }} - {{- end }} - {{- end }} - {{- if .Values.global.credentialsAndCertsSecrets }} - {{- range $envVarName, $envVarValue := .Values.global.credentialsAndCertsSecrets.secureEnv }} - - name: KAFKA_PROPERTIES_{{ $envVarName | replace "." "_" | upper }} - valueFrom: - secretKeyRef: - name: {{ $.Values.global.credentialsAndCertsSecrets.name }} - key: {{ $envVarValue }} - {{- end }} - {{- end }} - {{- with .Values.kafkaSetupJob.extraEnvs }} - {{- toYaml . | nindent 12 }} - {{- end }} - volumeMounts: - {{- if .Values.global.credentialsAndCertsSecrets }} - - name: datahub-certs-dir - mountPath: {{ .Values.global.credentialsAndCertsSecrets.path | default "/mnt/certs" }} - {{- end }} - {{- with .Values.kafkaSetupJob.extraVolumeMounts }} - {{- toYaml . | nindent 12 }} - {{- end }} - resources: - limits: - cpu: 500m - memory: 512Mi - requests: - cpu: 300m - memory: 256Mi -{{- end -}} diff --git a/datahub-kubernetes/datahub/templates/mysql-setup-job.yml b/datahub-kubernetes/datahub/templates/mysql-setup-job.yml deleted file mode 100644 index 52a1e3b8fe931e..00000000000000 --- a/datahub-kubernetes/datahub/templates/mysql-setup-job.yml +++ /dev/null @@ -1,68 +0,0 @@ -{{- if .Values.mysqlSetupJob.enabled -}} -apiVersion: batch/v1 -kind: Job -metadata: - name: {{ .Release.Name }}-mysql-setup-job - labels: - app.kubernetes.io/managed-by: {{ .Release.Service | quote }} - app.kubernetes.io/instance: {{ .Release.Name | quote }} - app.kubernetes.io/version: {{ .Chart.AppVersion }} - helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" - annotations: - # This is what defines this resource as a hook. Without this line, the - # job is considered part of the release. - "helm.sh/hook": pre-install,pre-upgrade - "helm.sh/hook-weight": "-5" - "helm.sh/hook-delete-policy": before-hook-creation -spec: - template: - spec: - {{- with .Values.global.hostAliases }} - hostAliases: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.mysqlSetupJob.serviceAccount }} - serviceAccountName: {{ . }} - {{- end }} - {{- with .Values.imagePullSecrets }} - imagePullSecrets: - {{- toYaml . | nindent 8 }} - {{- end }} - volumes: - {{- with .Values.mysqlSetupJob.extraVolumes }} - {{- toYaml . | nindent 8}} - {{- end }} - restartPolicy: Never - securityContext: - runAsUser: 1000 - fsGroup: 1000 - containers: - - name: mysql-setup-job - image: "{{ .Values.mysqlSetupJob.image.repository }}:{{ .Values.mysqlSetupJob.image.tag }}" - env: - - name: MYSQL_USERNAME - value: {{ .Values.global.sql.datasource.username | quote }} - - name: MYSQL_PASSWORD - valueFrom: - secretKeyRef: - name: "{{ .Values.global.sql.datasource.password.secretRef }}" - key: "{{ .Values.global.sql.datasource.password.secretKey }}" - - name: MYSQL_HOST - value: {{ .Values.global.sql.datasource.hostForMysqlClient | quote }} - - name: MYSQL_PORT - value: {{ .Values.global.sql.datasource.port | quote }} - {{- with .Values.mysqlSetupJob.extraEnvs }} - {{- toYaml . | nindent 12 }} - {{- end }} - volumeMounts: - {{- with .Values.mysqlSetupJob.extraVolumeMounts }} - {{- toYaml . | nindent 12 }} - {{- end }} - resources: - limits: - cpu: 500m - memory: 512Mi - requests: - cpu: 300m - memory: 256Mi -{{- end -}} \ No newline at end of file diff --git a/datahub-kubernetes/datahub/values.yaml b/datahub-kubernetes/datahub/values.yaml deleted file mode 100644 index 64ab73562b907e..00000000000000 --- a/datahub-kubernetes/datahub/values.yaml +++ /dev/null @@ -1,133 +0,0 @@ -# Values to start up datahub after starting up the datahub-prerequisites chart with "prerequisites" release name -# Copy this chart and change configuration as needed. -datahub-gms: - enabled: true - image: - repository: linkedin/datahub-gms - tag: "v0.8.5" - -datahub-frontend: - enabled: true - image: - repository: linkedin/datahub-frontend-react - tag: "v0.8.5" - # Set up ingress to expose react front-end - ingress: - enabled: false - -datahub-mae-consumer: - enabled: true - image: - repository: linkedin/datahub-mae-consumer - tag: "v0.8.5" - -datahub-mce-consumer: - enabled: true - image: - repository: linkedin/datahub-mce-consumer - tag: "v0.8.5" - -datahub-ingestion-cron: - enabled: false - image: - repository: linkedin/datahub-ingestion - tag: "v0.8.5" - -elasticsearchSetupJob: - enabled: true - image: - repository: linkedin/datahub-elasticsearch-setup - tag: "v0.8.5" - -kafkaSetupJob: - enabled: true - image: - repository: linkedin/datahub-kafka-setup - tag: "v0.8.5" - -mysqlSetupJob: - enabled: true - image: - repository: acryldata/datahub-mysql-setup - tag: "v0.8.5" - -datahubUpgrade: - enabled: true - image: - repository: acryldata/datahub-upgrade - tag: "v0.8.5" - noCodeDataMigration: - sqlDbType: "MYSQL" - -global: - graph_service_impl: neo4j - datahub_analytics_enabled: true - datahub_standalone_consumers_enabled: false - - elasticsearch: - host: "elasticsearch-master" - port: "9200" - - kafka: - bootstrap: - server: "prerequisites-kafka:9092" - zookeeper: - server: "prerequisites-zookeeper:2181" - schemaregistry: - url: "http://prerequisites-cp-schema-registry:8081" - - neo4j: - host: "prerequisites-neo4j-community:7474" - uri: "bolt://prerequisites-neo4j-community" - username: "neo4j" - password: - secretRef: neo4j-secrets - secretKey: neo4j-password - - sql: - datasource: - host: "prerequisites-mysql:3306" - hostForMysqlClient: "prerequisites-mysql" - port: "3306" - url: "jdbc:mysql://prerequisites-mysql:3306/datahub?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8&enabledTLSProtocols=TLSv1.2" - driver: "com.mysql.jdbc.Driver" - username: "root" - password: - secretRef: mysql-secrets - secretKey: mysql-root-password - - datahub: - gms: - port: "8080" - mae_consumer: - port: "9091" - appVersion: "1.0" - -# hostAliases: -# - ip: "192.168.0.104" -# hostnames: -# - "broker" -# - "mysql" -# - "elasticsearch" -# - "neo4j" - -## Add below to enable SSL for kafka -# credentialsAndCertsSecrets: -# name: datahub-certs -# path: /mnt/datahub/certs -# secureEnv: -# ssl.key.password: datahub.linkedin.com.KeyPass -# ssl.keystore.password: datahub.linkedin.com.KeyStorePass -# ssl.truststore.password: datahub.linkedin.com.TrustStorePass -# kafkastore.ssl.truststore.password: datahub.linkedin.com.TrustStorePass -# -# springKafkaConfigurationOverrides: -# ssl.keystore.location: /mnt/datahub/certs/datahub.linkedin.com.keystore.jks -# ssl.truststore.location: /mnt/datahub/certs/datahub.linkedin.com.truststore.jks -# kafkastore.ssl.truststore.location: /mnt/datahub/certs/datahub.linkedin.com.truststore.jks -# security.protocol: SSL -# kafkastore.security.protocol: SSL -# ssl.keystore.type: JKS -# ssl.truststore.type: JKS -# ssl.protocol: TLS -# ssl.endpoint.identification.algorithm: diff --git a/datahub-kubernetes/prerequisites/.gitignore b/datahub-kubernetes/prerequisites/.gitignore deleted file mode 100644 index 6eb125ea453a63..00000000000000 --- a/datahub-kubernetes/prerequisites/.gitignore +++ /dev/null @@ -1,2 +0,0 @@ -*.tgz -*.lock \ No newline at end of file diff --git a/datahub-kubernetes/prerequisites/Chart.yaml b/datahub-kubernetes/prerequisites/Chart.yaml deleted file mode 100644 index 8bd0309b841a42..00000000000000 --- a/datahub-kubernetes/prerequisites/Chart.yaml +++ /dev/null @@ -1,37 +0,0 @@ -apiVersion: v2 -name: datahub-prerequisites -description: A Helm chart for packages that Datahub depends on -type: application -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -version: 0.0.1 -dependencies: - - name: elasticsearch - version: 7.9.3 - repository: https://helm.elastic.co - condition: elasticsearch.enabled - # This chart deploys an enterprise version of neo4j that requires commercial license - - name: neo4j - version: 4.2.2-1 - repository: https://neo4j-contrib.github.io/neo4j-helm/ - condition: neo4j.enabled - # This chart deploys a community version of neo4j - - name: neo4j-community - version: 1.2.4 - repository: https://equinor.github.io/helm-charts/charts/ - condition: neo4j-community.enabled - - name: mysql - version: 8.5.4 - repository: https://charts.bitnami.com/bitnami - condition: mysql.enabled - # This chart deploys an enterprise version of kafka that requires commercial license - # Note, Schema registry and kafka rest proxy do not require the commercial license - - name: cp-helm-charts - version: 0.6.0 - repository: https://confluentinc.github.io/cp-helm-charts/ - condition: cp-helm-charts.enabled - # This chart deploys a community version of kafka - - name: kafka - version: 12.17.4 - repository: https://charts.bitnami.com/bitnami - condition: kafka.enabled \ No newline at end of file diff --git a/datahub-kubernetes/prerequisites/values.yaml b/datahub-kubernetes/prerequisites/values.yaml deleted file mode 100644 index 1581d583249d36..00000000000000 --- a/datahub-kubernetes/prerequisites/values.yaml +++ /dev/null @@ -1,87 +0,0 @@ -# Default configuration for pre-requisites to get you started -# Copy this file and update to the configuration of choice -elasticsearch: - enabled: true # set this to false, if you want to provide your own ES instance. - cluster: - env: - MINIMUM_MASTER_NODES: 1 - EXPECTED_MASTER_NODES: 1 - RECOVER_AFTER_MASTER_NODES: 1 - master: - replicas: 1 - data: - replicas: 1 - client: - replicas: 1 - - # Uncomment if running on minikube - Reference https://github.com/elastic/helm-charts/blob/master/elasticsearch/examples/minikube/values.yaml - # # Permit co-located instances for solitary minikube virtual machines. - # antiAffinity: "soft" - - # # Shrink default JVM heap. - # esJavaOpts: "-Xmx128m -Xms128m" - - # # Allocate smaller chunks of memory per pod. - # resources: - # requests: - # cpu: "100m" - # memory: "512M" - # limits: - # cpu: "1000m" - # memory: "512M" - - # # Request smaller persistent volumes. - # volumeClaimTemplate: - # accessModes: [ "ReadWriteOnce" ] - # storageClassName: "standard" - # resources: - # requests: - # storage: 100M - -# Official neo4j chart uses the Neo4j Enterprise Edition which requires a license -neo4j: - enabled: false # set this to true, if you have a license for the enterprise edition - acceptLicenseAgreement: "yes" - defaultDatabase: "graph.db" - neo4jPassword: "datahub" - # For better security, add password to neo4j-secrets k8s secret and uncomment below - # existingPasswordSecret: neo4j-secrets - core: - standalone: true - -# Deploys neo4j community version. Only supports single node -neo4j-community: - enabled: true # set this to false, if you have a license for the enterprise edition - acceptLicenseAgreement: "yes" - defaultDatabase: "graph.db" - # For better security, add password to neo4j-secrets k8s secret and uncomment below - existingPasswordSecret: neo4j-secrets - -mysql: - enabled: true - auth: - # For better security, add mysql-secrets k8s secret with mysql-root-password, mysql-replication-password and mysql-password - existingSecret: mysql-secrets - -cp-helm-charts: - # Schema registry is under the community license - cp-schema-registry: - enabled: true - kafka: - bootstrapServers: "prerequisites-kafka:9092" ## <>-kafka:9092 - cp-kafka: - enabled: false - cp-zookeeper: - enabled: false - cp-kafka-rest: - enabled: false - cp-kafka-connect: - enabled: false - cp-ksql-server: - enabled: false - cp-control-center: - enabled: false - -# Bitnami version of Kafka that deploys open source Kafka https://artifacthub.io/packages/helm/bitnami/kafka -kafka: - enabled: true diff --git a/docs-website/generateDocsDir.ts b/docs-website/generateDocsDir.ts index e620b72f84bd79..1b142f6b625cc2 100644 --- a/docs-website/generateDocsDir.ts +++ b/docs-website/generateDocsDir.ts @@ -71,7 +71,7 @@ function list_markdown_files(): string[] { // Don't want hosted docs for these. /^contrib\//, // Keep main docs for kubernetes, but skip the inner docs - /^datahub-kubernetes\/datahub\//, + /^datahub-kubernetes\//, /^datahub-web\//, /^metadata-ingestion-examples\//, /^docs\/rfc\/templates\/000-template\.md$/, diff --git a/docs-website/sidebars.js b/docs-website/sidebars.js index ab0cb14f916b34..42881b1f65460d 100644 --- a/docs-website/sidebars.js +++ b/docs-website/sidebars.js @@ -119,7 +119,7 @@ module.exports = { Deployment: [ "docs/how/kafka-config", "docker/README", - "datahub-kubernetes/README", + "docs/deploy/kubernetes", "docker/datahub-upgrade/README", "docs/deploy/aws", "docs/deploy/gcp", diff --git a/docs/deploy/aws.md b/docs/deploy/aws.md index 26f8849b2afc2d..7897f6c2dbd11f 100644 --- a/docs/deploy/aws.md +++ b/docs/deploy/aws.md @@ -56,7 +56,7 @@ ip-192-168-8-126.us-west-2.compute.internal Ready 3h v1.18.9-ek ## Setup DataHub using Helm Once the kubernetes cluster has been set up, you can deploy DataHub and it’s prerequisites using helm. Please follow the -steps in this [guide](../../datahub-kubernetes/README.md) +steps in this [guide](kubernetes.md) ## Expose endpoints using a load balancer diff --git a/docs/deploy/gcp.md b/docs/deploy/gcp.md index f1d8cd862f501d..3713d69f90636c 100644 --- a/docs/deploy/gcp.md +++ b/docs/deploy/gcp.md @@ -55,7 +55,7 @@ gke-datahub-default-pool-e5be7c4f-rksj Ready 34h v1.19.10-gke.16 ## Setup DataHub using Helm Once the kubernetes cluster has been set up, you can deploy DataHub and it’s prerequisites using helm. Please follow the -steps in this [guide](../../datahub-kubernetes/README.md) +steps in this [guide](kubernetes.md) ## Expose endpoints using GKE ingress controller diff --git a/docs/deploy/kubernetes.md b/docs/deploy/kubernetes.md new file mode 100644 index 00000000000000..a413a2fd80ae3f --- /dev/null +++ b/docs/deploy/kubernetes.md @@ -0,0 +1,154 @@ +--- +title: "Deploying with Kubernetes" +--- + +# Deploying DataHub with Kubernetes + +## Introduction + +Helm charts for deploying DataHub on a kubernetes cluster is located in +this [repository](https://github.com/acryldata/datahub-helm). We provide charts for +deploying [Datahub](https://github.com/acryldata/datahub-helm/tree/master/charts/datahub) and +it's [dependencies](https://github.com/acryldata/datahub-helm/tree/master/charts/prerequisites) +(Elasticsearch, optionally Neo4j, MySQL, and Kafka) on a Kubernetes cluster. + +This doc is a guide to deploy an instance of DataHub on a kubernetes cluster using the above charts from scratch. + +## Setup + +1. Set up a kubernetes cluster + - In a cloud platform of choice like [Amazon EKS](https://aws.amazon.com/eks), + [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine), + and [Azure Kubernetes Service](https://azure.microsoft.com/en-us/services/kubernetes-service/) OR + - In local environment using [Minikube](https://minikube.sigs.k8s.io/docs/). Note, more than 7GB of RAM is required + to run Datahub and it's dependencies +2. Install the following tools: + - [kubectl](https://kubernetes.io/docs/tasks/tools/) to manage kubernetes resources + - [helm](https://helm.sh/docs/intro/install/) to deploy the resources based on helm charts. Note, we only support + Helm 3. + +## Components + +Datahub consists of 4 main components: [GMS](https://datahubproject.io/docs/gms), +[MAE Consumer](https://datahubproject.io/docs/metadata-jobs/mae-consumer-job) (optional), +[MCE Consumer](https://datahubproject.io/docs/metadata-jobs/mce-consumer-job) (optional), and +[Frontend](https://datahubproject.io/docs/datahub-frontend). Kubernetes deployment for each of the components are +defined as subcharts under the main +[Datahub](https://github.com/acryldata/datahub-helm/tree/master/charts/datahub) +helm chart. + +The main components are powered by 4 external dependencies: + +- Kafka +- Local DB (MySQL, Postgres, MariaDB) +- Search Index (Elasticsearch) +- Graph Index (Supports either Neo4j or Elasticsearch) + +The dependencies must be deployed before deploying Datahub. We created a separate +[chart](https://github.com/acryldata/datahub-helm/tree/master/charts/prerequisites) +for deploying the dependencies with example configuration. They could also be deployed separately on-prem or leveraged +as managed services. To remove your dependency on Neo4j, set enabled to false in +the [values.yaml](https://github.com/acryldata/datahub-helm/blob/master/charts/prerequisites/values.yaml#L54) for +prerequisites. Then, override the `graph_service_impl` field in +the [values.yaml](https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml#L63) of datahub +instead of `neo4j`. + +## Quickstart + +Assuming kubectl context points to the correct kubernetes cluster, first create kubernetes secrets that contain MySQL +and Neo4j passwords. + +```(shell) +kubectl create secret generic mysql-secrets --from-literal=mysql-root-password=datahub +kubectl create secret generic neo4j-secrets --from-literal=neo4j-password=datahub +``` + +The above commands sets the passwords to "datahub" as an example. Change to any password of choice. + +Add datahub helm repo by running the following + +```(shell) +helm repo add datahub https://helm.datahubproject.io/ +``` + +Then, deploy the dependencies by running the following + +```(shell) +helm install prerequisites datahub/datahub-prerequisites +``` + +Note, the above uses the default configuration +defined [here](https://github.com/acryldata/datahub-helm/blob/master/charts/prerequisites/values.yaml). You can change +any of the configuration and deploy by running the following command. + +```(shell) +helm install prerequisites datahub/datahub-prerequisites --values <> +``` + +Run `kubectl get pods` to check whether all the pods for the dependencies are running. You should get a result similar +to below. + +``` +NAME READY STATUS RESTARTS AGE +elasticsearch-master-0 1/1 Running 0 62m +elasticsearch-master-1 1/1 Running 0 62m +elasticsearch-master-2 1/1 Running 0 62m +prerequisites-cp-schema-registry-cf79bfccf-kvjtv 2/2 Running 1 63m +prerequisites-kafka-0 1/1 Running 2 62m +prerequisites-mysql-0 1/1 Running 1 62m +prerequisites-neo4j-community-0 1/1 Running 0 52m +prerequisites-zookeeper-0 1/1 Running 0 62m +``` + +deploy Datahub by running the following + +```(shell) +helm install datahub datahub/datahub +``` + +Values in [values.yaml](https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml) +have been preset to point to the dependencies deployed using +the [prerequisites](https://github.com/acryldata/datahub-helm/tree/master/charts/prerequisites) +chart with release name "prerequisites". If you deployed the helm chart using a different release name, update the +quickstart-values.yaml file accordingly before installing. + +Run `kubectl get pods` to check whether all the datahub pods are running. You should get a result similar to below. + +``` +NAME READY STATUS RESTARTS AGE +datahub-datahub-frontend-84c58df9f7-5bgwx 1/1 Running 0 4m2s +datahub-datahub-gms-58b676f77c-c6pfx 1/1 Running 0 4m2s +datahub-datahub-mae-consumer-7b98bf65d-tjbwx 1/1 Running 0 4m3s +datahub-datahub-mce-consumer-8c57d8587-vjv9m 1/1 Running 0 4m2s +datahub-elasticsearch-setup-job-8dz6b 0/1 Completed 0 4m50s +datahub-kafka-setup-job-6blcj 0/1 Completed 0 4m40s +datahub-mysql-setup-job-b57kc 0/1 Completed 0 4m7s +elasticsearch-master-0 1/1 Running 0 97m +elasticsearch-master-1 1/1 Running 0 97m +elasticsearch-master-2 1/1 Running 0 97m +prerequisites-cp-schema-registry-cf79bfccf-kvjtv 2/2 Running 1 99m +prerequisites-kafka-0 1/1 Running 2 97m +prerequisites-mysql-0 1/1 Running 1 97m +prerequisites-neo4j-community-0 1/1 Running 0 88m +prerequisites-zookeeper-0 1/1 Running 0 97m +``` + +You can run the following to expose the frontend locally. Note, you can find the pod name using the command above. In +this case, the datahub-frontend pod name was `datahub-datahub-frontend-84c58df9f7-5bgwx`. + +```(shell) +kubectl port-forward 9002:9002 +``` + +You should be able to access the frontend via http://localhost:9002. + +Once you confirm that the pods are running well, you can set up ingress for datahub-frontend to expose the 9002 port to +the public. + +## Other useful commands + +| Command | Description | +|-----|------| +| helm uninstall datahub | Remove DataHub | +| helm ls | List of Helm charts | +| helm history | Fetch a release history | diff --git a/docs/how/migrating-graph-service-implementation.md b/docs/how/migrating-graph-service-implementation.md index f9636d464cbae9..024740b2ce61f0 100644 --- a/docs/how/migrating-graph-service-implementation.md +++ b/docs/how/migrating-graph-service-implementation.md @@ -36,13 +36,13 @@ This prevents your DataHub instance from coming up in neo4j mode in the future. First, adjust your helm variables to turn off neo4j and set your graph_service_impl to elasticsearch. To turn off neo4j in your prerequisites file, set `neo4j-community`'s `enabled` property to `false` -in [datahub-kubernetes/prerequisites/values.yaml](https://github.com/linkedin/datahub/blob/69dc2682ec36d68d2ac2d88b86cbd9c84d766b0b/datahub-kubernetes/prerequisites/values.yaml#L54). +in this [values.yaml](https://github.com/acryldata/datahub-helm/blob/master/charts/prerequisites/values.yaml#L54). -Then, set `graph_service_impl` to `elasticsearch` in -[datahub-kubernetes/datahub/values.yaml](https://github.com/linkedin/datahub/blob/eb60da975dce1f855883e628541cdbcef89717af/datahub-kubernetes/datahub/values.yaml#L63). +Then, set `graph_service_impl` to `elasticsearch` in the +[values.yaml](https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml#L63) of datahub. -See the [deployment helm guide](../../datahub-kubernetes/README.md#components) for more details on how to +See the [deployment helm guide](../deploy/kubernetes.md#components) for more details on how to set up your helm deployment. Finally, follow the [restore-indices helm guide](./restore-indices.md) to re-build