Issue 61: Support external connectivity

[adrianmo]

Change log description

This PR adds support for external connectivity (i.e. connectivity from outside the Kubernetes cluster). Below is a detailed description of the changes made and the reason why they were made.

Create a headless service for the Segmentstore set

As with the Bookies, Segmentstores are managed by a StatefulSet. Headless services provide DNS records for the Segmentstore set that can be used for intra-cluster communication, i.e., segmentstore pods can be access using the <statefulset_name>-<ordinal>.<headless_service_name> hostname (e.g. pravega-segmentstore-1.pravega-segmentstore-headless would resolve to the IP of second pod in the set).

Create an external service per Segmentstore pod

Pravega clients need to be able to directly reach every Segmentstore instance. We found multiple ways to achieve this:

• Using NodePort services. NodePort services allocate a free port (within the 30000-32767 range) in each Kubernetes node and forwards the traffic to the backend pod. However, this requires extra configuration on the external networking layer to allocate IP addresses, and allow and forward external traffic to the service. In addition, NodePort services are not yet supported on PKS w/ vSphere and NSX-T, which is our main platform.
• Using the Ingress resource. Ingress configures an HTTP server with routing rules to allow external communication with a service. However, most Ingress implementations do not support non-HTTP traffic, making it not an option for our HTTP2-gRPC traffic.
• Using LoadBalancer services. A LoadBalancer service configures an external load balancer (e.g. NSX-T in our platform) to allow external communication and provides an external IP once it's allocated by the external load balancer. It can handle any TCP connections or UDP flows, so it works for our use case.

This PR implements support for LoadBalancer and NodePort service types. Configured in externalAccess.type. The operator creates one external service per Segment Store instance.

In addition, these services are created with the externalTrafficPolicy option enabled. By default, LoadBalancer services route traffic to any backend pod in the cluster, however, with this option, services route traffic only to node-local pods.

Create ClusterRole permissions

When external access is enabled and configured with NodePort, the pod needs to find the node IP using the Kubernetes API. To achieve this, the service account needs to have ClusterNode read permissions on the node resource.

Segmentstore instance to advertise external address

By default, a Segmentstore instance uses it's local address when it registers with the Controller. However, the local pod address is only reachable from within the cluster (or namespace, depending on the environment).

We need to initialize each Segmentstore instance with the external address allocated by the corresponding LoadBalancer service. To achieve this, the Pravega Docker image entrypoint.sh needs to modified to obtain the external service IP and port and configure it as system properties before starting the segmentstore process.

The Segmentstore container needs to know the service name and the namespace to introspect and find its external IP. By naming convention, the LoadBalancing service name is the same as the pod name. So this is information can be easily made available to the pod by leveraging the Downward API. Two environment variables (POD_NAME and POD_NAMESPACE) are passed to the pod to allow the pods to find their information via the Kubernetes API. How the container uses this information is being addressed in the Pravega project PR pravega/pravega#3062.

Ability to configure external access

Exposing the Pravega cluster to the outside is a possibility for users, but it's not enforced. For security reasons, by default a Pravega cluster will still be accessible only within the Kubernetes cluster. To enable external access, users need to explicitly create the externalAccess structure and set the enabled field to true, otherwise the cluster will not be exposed. E.g.:

apiVersion: "pravega.pravega.io/v1alpha1"
kind: "PravegaCluster"
metadata:
  name: "pravega1"
spec:
  zookeeperUri: 10.59.240.7:2181
  externalAccess:
    enabled: true
    type: LoadBalancer
...

Purpose of the change

• Fix Support external connectivity #61

How to test the change

Assuming you already have access to a Kubernetes environment (GKE, PKS, Minikube...), follow the instructions in the README file to deploy ZooKeeper, the NFS server provisioner, and the Pravega Tier 2 PVC. Skip the deployment of the Pravega operator.

Check out this branch.

git checkout issue-61-external-traffic

Update the Pravega operator image to use an image built with the PR changes. Open the deploy/operator.yaml file and replace the container image from pravega/pravega-operator:latest to adrianmo/pravega-operator:v0.1.0-23-dirty.

Deploy the operator.

kubectl create -f deploy/operator.yaml

Create a pravega.yaml that will use the updated images that contain the changes made to obtain and advertise the external service IP (ref: pravega/pravega#3062) and the new externalAccess field.

apiVersion: "pravega.pravega.io/v1alpha1"                                                                                                                       
kind: "PravegaCluster"                                                                                                                                          
metadata:                                                                                                                                                       
  name: "pravega"                                                                                                                                               
spec:                                                                                                                                                           
  zookeeperUri: zk-client:2181                                                                                                                                  

  externalAccess:
    enabled: true
    type: LoadBalancer
                                                                                                                                                                
  bookkeeper:                                                                                                                                                                                                                                                                        
    image:
      repository: adrianmo/bookkeeper
      tag: 0.5.0-2023.5353487-SNAPSHOT
      pullPolicy: IfNotPresent

    replicas: 3

    storage:
      ledgerVolumeClaimTemplate:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "standard"
        resources:
          requests:
            storage: 10Gi

      journalVolumeClaimTemplate:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "standard"
        resources:
          requests:
            storage: 10Gi

    autoRecovery: true

  pravega:
    controllerReplicas: 1
    segmentStoreReplicas: 3

    cacheVolumeClaimTemplate:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "standard"
      resources:
        requests:
          storage: 20Gi

    image:
      repository: adrianmo/pravega
      tag: 0.5.0-2023.5353487-SNAPSHOT
      pullPolicy: IfNotPresent

    tier2:
      filesystem:
        persistentVolumeClaim:
          claimName: pravega-tier2

Copy the contents above to a pravega.yaml file and deploy the Pravega cluster:

kubectl create -f pravega.yaml

Watch the Pravega cluster being created...

watch kubectl get all -l pravega_cluster=pravega

Until the Controller and all Segmentstore services have been allocated an external address in the EXTERNAL-IP column.

NAME                                              READY     STATUS    RESTARTS   AGE
pod/pravega-bookie-0                              1/1       Running   0          4m
pod/pravega-bookie-1                              1/1       Running   0          4m
pod/pravega-bookie-2                              1/1       Running   0          4m
pod/pravega-pravega-controller-76f6d74c55-kkwgb   1/1       Running   0          4m
pod/pravega-pravega-segmentstore-0                1/1       Running   0          4m
pod/pravega-pravega-segmentstore-1                1/1       Running   0          4m
pod/pravega-pravega-segmentstore-2                1/1       Running   0          4m

NAME                                            TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                          AGE
service/pravega-bookie-headless                 ClusterIP      None            <none>           3181/TCP                         4m
service/pravega-pravega-controller              LoadBalancer   10.59.243.23    35.240.18.191    10080:31356/TCP,9090:32287/TCP   4m
service/pravega-pravega-segmentstore-0          LoadBalancer   10.59.248.137   35.241.234.206   12345:31849/TCP                  4m
service/pravega-pravega-segmentstore-1          LoadBalancer   10.59.243.10    35.189.250.112   12345:30177/TCP                  4m
service/pravega-pravega-segmentstore-2          LoadBalancer   10.59.252.97    35.240.48.116    12345:32460/TCP                  4m
service/pravega-pravega-segmentstore-headless   ClusterIP      None            <none>           12345/TCP                        4m

NAME                                               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.extensions/pravega-pravega-controller   1         1         1            1           4m

NAME                                                          DESIRED   CURRENT   READY     AGE
replicaset.extensions/pravega-pravega-controller-76f6d74c55   1         1         1         4m

NAME                                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/pravega-pravega-controller   1         1         1            1           4m

NAME                                                    DESIRED   CURRENT   READY     AGE
replicaset.apps/pravega-pravega-controller-76f6d74c55   1         1         1         4m

NAME                                            DESIRED   CURRENT   AGE
statefulset.apps/pravega-bookie                 3         3         4m
statefulset.apps/pravega-pravega-segmentstore   3         3         4m

Then, copy the Controller External IP and connect to it with the Pravega client. You can use the samples in the Pravega samples repositories.

---

Signed-off-by: Adrian Moreno [email protected]

[adrianmo]

@maddisondavid @EronWright @arvindkandhare can you please review the PR? Thanks!

[EronWright]

It would be great to see the output of kubectl get -n nautilus-pravega all, attached to this PR, thanks!

[adrianmo]

@EronWright thanks for the review. I'll make some changes based on your comments.

[adrianmo]

I've made the changes suggested by @EronWright and updated the PR description.

[adrianmo]

@EronWright @maddisondavid can you please review the PR again after the changes?

outdated review.

[maddisondavid]

LGTM, my only concern is that IF external access is enabled then the LoadBalancer IP will be given to any clients inside the cluster if connecting to Pravega.

[fpj]

+1