chore: add network policy (#875)

* chore: add limits and requests to pods Signed-off-by: SdgJlbl <[email protected]> * chore: add network policy Signed-off-by: SdgJlbl <[email protected]> * revert: "chore: add limits and requests to pods" This reverts commit 8d277d6. * feat: deny all traffic Signed-off-by: Guilhem Barthés <[email protected]> * feat: allow connection to redis Signed-off-by: Guilhem Barthés <[email protected]> * feat: allow connection to database Signed-off-by: Guilhem Barthés <[email protected]> * feat: allow connection to docker-registry Signed-off-by: Guilhem Barthés <[email protected]> * feat: allow connection to storage Signed-off-by: Guilhem Barthés <[email protected]> * feat: allow connection to orc Signed-off-by: Guilhem Barthés <[email protected]> * feat: allow connection to internet Signed-off-by: Guilhem Barthés <[email protected]> * feat: add communication with server Signed-off-by: Guilhem Barthés <[email protected]> * feat: add communication with k8s api server Signed-off-by: Guilhem Barthés <[email protected]> * feat: add internet communication for builder Signed-off-by: Guilhem Barthés <[email protected]> * fix: add http requests between backends in same cluster Signed-off-by: Guilhem Barthés <[email protected]> * fix: add `skaffold-local-ingress` network policies Signed-off-by: Guilhem Barthés <[email protected]> * chore: test with weaker policy for api k8s Signed-off-by: Guilhem Barthés <[email protected]> * wip: test with weaker deny all Signed-off-by: Guilhem Barthés <[email protected]> * chore: remove commented code Signed-off-by: Guilhem Barthés <[email protected]> * chore: remove commented code as it is parsed by helm but not k8s Signed-off-by: Guilhem Barthés <[email protected]> * fix: `-api-server-egress` was not an array Signed-off-by: Guilhem Barthés <[email protected]> * fix: `-api-server-egress` `ipBlock` indentation Signed-off-by: Guilhem Barthés <[email protected]> * fix: add more private IP range to target kubernetes apiserver Signed-off-by: Guilhem Barthés <[email protected]> * feat: add `server.allowLocalRequests` parameter Signed-off-by: Guilhem Barthés <[email protected]> * chore: remove duplicate network policies (replaced by `allowLocalRequests`) Signed-off-by: Guilhem Barthés <[email protected]> * fix: add `role-api-k8s-client: 'true'` to builder when `privateCa.enabled` is enabled Signed-off-by: Guilhem Barthés <[email protected]> * chore: remove unused `deploy.kubectl` Signed-off-by: Guilhem Barthés <[email protected]> * fix: allow communication between backends in the same cluster Signed-off-by: Guilhem Barthés <[email protected]> * fix: allow communication between local backends with `allowLocalRequests` Signed-off-by: Guilhem Barthés <[email protected]> * fix: readd policy to prevent compute-task to communicate with internet Signed-off-by: Guilhem Barthés <[email protected]> * doc: update chart version and changelog Signed-off-by: Guilhem Barthés <[email protected]> * chore: add network policies for Harbor (#906) * chore: add network policies for Harbor * chore: add newline Signed-off-by: Guilhem Barthés <[email protected]> --------- Signed-off-by: Guilhem Barthés <[email protected]> Co-authored-by: Guilhem Barthés <[email protected]> * fix: remove wrong value for profile `three-org` Signed-off-by: Guilhem Barthés <[email protected]> --------- Signed-off-by: SdgJlbl <[email protected]> Signed-off-by: Guilhem Barthés <[email protected]> Co-authored-by: Guilhem Barthés <[email protected]>
Substra · May 23, 2024 · c1d1fde · c1d1fde
1 parent ae20351
commit c1d1fde
Show file tree

Hide file tree

Showing 30 changed files with 569 additions and 74 deletions.
diff --git a/backend/builder/image_builder/image_builder.py b/backend/builder/image_builder/image_builder.py
@@ -197,6 +197,8 @@ def _build_pod(dockerfile_mount_path: str, image_tag: str) -> kubernetes.client.
                 Label.PodName: pod_name,
                 Label.PodType: "image-build",
                 Label.Component: Label.Component_Compute,
+                Label.RoleRegistryClient: "true",
+                Label.RoleInternetEgress: "true",
             },
         ),
         spec=pod_spec,

diff --git a/backend/substrapp/compute_tasks/compute_pod.py b/backend/substrapp/compute_tasks/compute_pod.py
@@ -22,7 +22,8 @@ class Label:
     ComputePlanKey = "substra.ai/compute-plan-key"
     FunctionKey = "substra.ai/function-key"
     RandomToken = "substra.ai/random-token"
-
+    RoleRegistryClient = "role-registry-client"
+    RoleInternetEgress = "role-internet-egress"
     # Values
     PodType_ComputeTask = "compute-task"
     Component_Compute = "substra-compute"

diff --git a/charts/substra-backend/CHANGELOG.md b/charts/substra-backend/CHANGELOG.md
@@ -2,12 +2,18 @@
 
 <!-- towncrier release notes start -->
 
+## [26.6.0] - 2024-05-22
+
+### Added
+
+- Network policies limit connection to pods (#875)
+
 ## [26.5.1] - 2024-05-21
 
 ### Changed
 
 - Add `resources-dev.yaml` file to overwrite resources requirement to a minimal value in a dev setup (#907)
-
+  
 ## [26.5.0] - 2024-04-29
 
 ### Added

diff --git a/charts/substra-backend/Chart.yaml b/charts/substra-backend/Chart.yaml
@@ -1,7 +1,7 @@
 apiVersion: v2
 name: substra-backend
 home: https://github.com/Substra
-version: 26.5.1
+version: 26.6.0
 appVersion: 0.45.0
 kubeVersion: ">= 1.19.0-0"
 description: Main package for Substra

diff --git a/charts/substra-backend/README.md b/charts/substra-backend/README.md
diff --git a/charts/substra-backend/templates/_helpers.tpl b/charts/substra-backend/templates/_helpers.tpl
@@ -327,6 +327,28 @@ Define service URL based on MinIO or LocalStack enablement
     {{- end -}}
 {{- end -}}
 
+{{/*
+Define service port based on MinIO or LocalStack enablement
+*/}}
+{{- define "substra-backend.objectStore.port" -}}
+    {{- if .Values.minio.enabled -}}
+        {{- print "9000" -}}
+    {{- else if .Values.localstack.enabled -}}
+        {{- print "4566" -}}
+    {{- end -}}
+{{- end -}}
+
+{{/*
+Define service port based on MinIO or LocalStack enablement
+*/}}
+{{- define "substra-backend.objectStore.labels" -}}
+    {{- if .Values.minio.enabled -}}
+        {{- print "app.kubernetes.io/name: minio" -}}
+    {{- else if .Values.localstack.enabled -}}
+        {{- print "app.kubernetes.io/name: localstack" -}}
+    {{- end -}}
+{{- end -}}
+
 
 {{/*
 Define objectstore access key based on MinIO or LocalStack enablement

diff --git a/charts/substra-backend/templates/deployment-api-events.yaml b/charts/substra-backend/templates/deployment-api-events.yaml
@@ -16,6 +16,9 @@ spec:
     metadata:
       labels:
         app.kubernetes.io/name: {{ template "substra.name" . }}-api-events
+        role-redis-client: 'true'
+        role-database-client: 'true'
+        role-orchestrator-client: 'true'
         {{ include "substra.labels" . | nindent 8 }}
       annotations:
         # This will cause the pod to restart if the content of the ConfigMap is updated through Helm

diff --git a/charts/substra-backend/templates/deployment-registry-prepopulate.yaml b/charts/substra-backend/templates/deployment-registry-prepopulate.yaml
@@ -19,9 +19,11 @@ spec:
       labels:
         {{ include "substra.labels" $ | nindent 8 }}
         app.kubernetes.io/name: {{ template "substra.name" $ }}-registry-prepopulate-{{ $index }}
+        role-registry-client: 'true'
+        role-internet-egress: 'true'
     spec:
       initContainers:
-      - name: wait-registry
+      - name: wait-registry 
         image: jwilder/dockerize:0.6.1
         resources:
           {{- toYaml $.Values.registryPrepopulate.waitRegistry | nindent 12 }}

diff --git a/charts/substra-backend/templates/deployment-scheduler-worker.yaml b/charts/substra-backend/templates/deployment-scheduler-worker.yaml
@@ -16,6 +16,10 @@ spec:
     metadata:
       labels:
         app.kubernetes.io/name: {{ template "substra.name" . }}-scheduler-worker
+        role-database-client: 'true'
+        role-orchestrator-client: 'true'
+        role-redis-client: 'true'
+        role-registry-client: 'true'
         {{ include "substra.labels" . | nindent 8 }}
       annotations:
         # This will cause the pod to restart if the content of the ConfigMap is updated through Helm

diff --git a/charts/substra-backend/templates/deployment-scheduler.yaml b/charts/substra-backend/templates/deployment-scheduler.yaml
@@ -16,6 +16,10 @@ spec:
     metadata:
       labels:
         app.kubernetes.io/name: {{ template "substra.name" . }}-scheduler
+        role-database-client: 'true'
+        role-orchestrator-client: 'true'
+        role-redis-client: 'true'
+        role-registry-client: 'true'
         {{ include "substra.labels" . | nindent 8 }}
       annotations:
         # This will cause the pod to restart if the content of the ConfigMap is updated through Helm

diff --git a/charts/substra-backend/templates/deployment-server.yaml b/charts/substra-backend/templates/deployment-server.yaml
@@ -16,6 +16,15 @@ spec:
     metadata:
       labels:
         app.kubernetes.io/name: {{ template "substra.name" . }}-server
+        role-redis-client: 'true'
+        role-database-client: 'true'
+        role-internet-ingress: 'true'
+        role-internet-egress: 'true'
+        role-orchestrator-client: 'true'
+        {{- if .Values.server.allowLocalRequests }}
+        role-server-client: 'true'
+        {{- end }}
+        role-storage-client: 'true'
         {{ include "substra.labels" . | nindent 8 }}
       annotations:
         # This will cause the pod to restart if the content of the ConfigMap is updated through Helm

diff --git a/charts/substra-backend/templates/deployment-worker-events.yaml b/charts/substra-backend/templates/deployment-worker-events.yaml
@@ -16,6 +16,10 @@ spec:
     metadata:
       labels:
         app.kubernetes.io/name: {{ template "substra.name" . }}-worker-events
+        role-api-k8s-client: 'true'
+        role-redis-client: 'true'
+        role-database-client: 'true'
+        role-orchestrator-client: 'true'
         {{ include "substra.labels" . | nindent 8 }}
       annotations:
         # This will cause the pod to restart if the content of the ConfigMap is updated through Helm

diff --git a/charts/substra-backend/templates/job-migrations.yaml b/charts/substra-backend/templates/job-migrations.yaml
@@ -12,6 +12,9 @@ spec:
   template:
     metadata:
       name: {{ template "substra.fullname" . }}-migrations
+      labels:
+          {{ include "substra.labels" . | nindent 10 }}
+          role-database-client: 'true'
     spec:
       restartPolicy: OnFailure
       automountServiceAccountToken: false

diff --git a/charts/substra-backend/templates/network-task-deny-all.yaml b/charts/substra-backend/templates/network-task-deny-all.yaml
@@ -0,0 +1,14 @@
+# Deny ALL networking in launched substra ml task
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: {{ template "substra.fullname". }}-deny-ingress
+  labels:
+    {{ include "substra.labels" . | nindent 4 }}
+spec:
+  podSelector:
+    matchLabels:
+      substra.ai/pod-type: compute-task
+  policyTypes:
+  - Ingress
+  - Egress
diff --git a/charts/substra-backend/templates/networkpolicy-api-server.yaml b/charts/substra-backend/templates/networkpolicy-api-server.yaml
@@ -0,0 +1,17 @@
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: {{ template "substra.name". }}-api-server-egress
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/part-of: {{ template "substra.name" . }}
+      role-api-k8s-client: 'true'
+  policyTypes:
+    - Egress
+  egress:
+    - to:
+      - ipBlock:
+          cidr: 10.0.0.0/8
+      - ipBlock:
+          cidr: 172.16.0.0/12
diff --git a/charts/substra-backend/templates/networkpolicy-database.yaml b/charts/substra-backend/templates/networkpolicy-database.yaml
@@ -0,0 +1,42 @@
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: {{ template "substra.name". }}-database-ingress
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/component: primary
+      app.kubernetes.io/name: postgresql
+  policyTypes:
+  - Ingress
+  ingress:
+  - from:
+    - podSelector:
+        matchLabels:
+          app.kubernetes.io/part-of: {{ template "substra.name" . }}
+          role-database-client: 'true'
+    ports:
+    - protocol: TCP
+      port: {{ .Values.database.port }}
+  egress: []
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: {{ template "substra.name". }}-database-egress
+spec:
+  podSelector:
+    matchLabels:
+          app.kubernetes.io/part-of: {{ template "substra.name" . }}
+          role-database-client: 'true'
+  policyTypes:
+  - Egress
+  egress:
+  - to:
+    - podSelector:
+        matchLabels:
+          app.kubernetes.io/component: primary
+          app.kubernetes.io/name: postgresql
+    ports:
+    - protocol: TCP
+      port: {{ .Values.database.port }}
diff --git a/charts/substra-backend/templates/networkpolicy-deny-all.yaml b/charts/substra-backend/templates/networkpolicy-deny-all.yaml
@@ -0,0 +1,23 @@
+kind: NetworkPolicy
+apiVersion: networking.k8s.io/v1
+metadata:
+  name: {{ template "substra.name". }}-deny-all-backend
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/part-of: substra-backend
+  ingress: []
+  egress:
+    - to:
+      # Allow internal DNS reslution
+      - namespaceSelector:
+          matchLabels:
+            kubernetes.io/metadata.name: kube-system
+        podSelector:
+          matchLabels:
+            k8s-app: kube-dns
+      ports:
+      - protocol: TCP
+        port: 53
+      - protocol: UDP
+        port: 53
diff --git a/charts/substra-backend/templates/networkpolicy-internet.yaml b/charts/substra-backend/templates/networkpolicy-internet.yaml
@@ -0,0 +1,61 @@
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: {{ template "substra.name". }}-internet-ingress
+spec:
+  ingress:
+  - from:
+    - ipBlock:
+        cidr: 0.0.0.0/0
+        {{- if not .Values.server.allowLocalRequests }}
+        except:
+        - 10.0.0.0/8
+        - 192.168.0.0/16
+        - 172.16.0.0/20
+        {{- end }}
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/part-of: {{ template "substra.name" . }}
+      role-internet-ingress: 'true'
+  policyTypes:
+  - Ingress
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: {{ template "substra.name". }}-internet-egress
+spec:
+  egress:
+  - to:
+    - ipBlock:
+        cidr: 0.0.0.0/0
+        except:
+        - 10.0.0.0/8
+        - 192.168.0.0/16
+        - 172.16.0.0/20
+  podSelector:
+    matchLabels:
+      # Not filtering on substra.name as kaniko builder needs access too
+      role-internet-egress: 'true'
+  policyTypes:
+  - Egress
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: {{ template "substra.name". }}-internet-builder-egress
+spec:
+  egress:
+  - to:
+    ports:
+      - protocol: TCP
+        port: 53
+      - protocol: UDP
+        port: 53
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/component: substra-compute
+      # Not filtering on substra.name as kaniko builder needs access too
+      role-internet-egress: 'true'
+  policyTypes:
+  - Egress
diff --git a/charts/substra-backend/templates/networkpolicy-orchestrator-client.yaml b/charts/substra-backend/templates/networkpolicy-orchestrator-client.yaml
@@ -0,0 +1,21 @@
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: {{ include "substra.name" . }}-orchestrator-client-egress
+spec:
+  podSelector:
+    matchLabels:
+      role-orchestrator-client: 'true'
+  policyTypes:
+  - Egress
+  egress:
+  - to:
+    - namespaceSelector: {}
+      podSelector:
+        matchLabels:
+          app.kubernetes.io/name: orchestrator
+          app.kubernetes.io/component: "server"
+    ports:
+    - protocol: TCP
+      port: {{ .Values.orchestrator.port }}
+
diff --git a/charts/substra-backend/templates/networkpolicy-redis.yaml b/charts/substra-backend/templates/networkpolicy-redis.yaml
@@ -0,0 +1,43 @@
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: {{ template "substra.name". }}-redis-ingress
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/component: master
+      app.kubernetes.io/name: redis
+  policyTypes:
+  - Ingress
+  ingress:
+  - from:
+    - podSelector:
+        matchLabels:
+          app.kubernetes.io/part-of: {{ template "substra.name" . }}
+          role-redis-client: 'true'
+    ports:
+    - protocol: TCP
+      port: {{ .Values.redis.master.service.ports.redis }}
+  egress: []
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: {{ template "substra.name". }}-redis-egress
+spec:
+  podSelector:
+    matchLabels:
+          app.kubernetes.io/part-of: {{ template "substra.name" . }}
+          role-redis-client: 'true'
+
+  policyTypes:
+  - Egress
+  egress:
+  - to:
+    - podSelector:
+        matchLabels:
+          app.kubernetes.io/component: master
+          app.kubernetes.io/name: redis
+    ports:
+    - protocol: TCP
+      port: {{ .Values.redis.master.service.ports.redis }}