Storing TLS objects as Kubernetes secrets and Distributing them to Pravega pods as files

[ravisharda]

Background:
Kubernetes supports an option to store and distribute sensitive data to containers, via the Secrets object. You first create a secret object (using kubectcl, yaml, etc.), and then pass the entries to the containers in the form of either environment variables or files in a volume. See an example in action, here.

When deploying a Pravega cluster with TLS enabled for Controllers and Segment Store pods, admins need to be able to distribute externally created TLS material to the containers/pods. Such material includes:

• X.509 certificates
• Cryptographic keys
• File containing the password of the key file
• Keystore (jks)
• Truststore (jks)
• PasswordAuthHandler input file

Kubernetes Secret object fits the bill rather nicely, both for keeping the TLS material safe and for distributing it to the pods. Currently, Pravega operator does not support using Secrets (It supports ConfigMap though, which is quite similar to the Secret object).

Requirement:

The ask is to build support for creating Kubernetes Secret objects and distributing them to Pravega containers through volumes. The admin shall provide definitions for one or more secret objects in the Pravega cluster definition file. Pravega operator, which consumes the definition file, shall automatically create those secret objects and expose the entries in the objects as files on the volume with specified mount-paths. This example in Kubernetes documentation illustrates its working.

More Details:
Here's roughly how things should roughly work:

1. The admin creates Secret object(s) using commands like these (or alternatively, through yaml spec).

# Create a secret object called "controller-pki" with entries tlsCertFile, cacert, etc. 
# The values (such as ./controller01.pem) are path on the local file system.
$ kubectl create secret generic controller-pki \
        --from-file=tlsCertFile=./controller01.pem \
        --from-file=cacert=./ca-cert \
        --from-file=tlsKeyFile=./controller01.key.pem \
        --from-file=tlsKeyStoreFile=./controller01.jks \
        --from-file=passwordfile=./password 

# Create another one for segment stores.
$ kubectl create secret generic segmentstore-pki \
        --from-file=tlsCertFile=./segmentstore01.pem \
	--from-file=cacert=./ca-cert \
	--from-file=tlsKeyFile=./segmentstore01.key.pem

2. In the Pravega cluster definition file (pravega.yaml for example), the admin specifies the name and entries of the secret objects created in step 1. They also specify the mount-path of the secret object (such as /etc/secret-volume). Alternatively, we can fix the name of the objects and entries, so this step will not be needed. That'll limit the flexibility in introducing new secrets but still meet the current requirements.

The admin uses TLS file paths relative to the mount-path in the pravega cluster definition file to specify JVM system properties for the Pravega pod as shown below:

pravega:
   ...
   options:
       options:
           controller.auth.tlsEnabled: "true"
           controller.auth.tlsCertFile: "/etc/secret-volume/controller01.pem"
           controller.auth.tlsKeyFile: "/etc/secret-volume/controller01.key.pem"
           ...

3. The admin executes the command to initiate the deployment. The Pravega operator system creates a volume at the specified mount path for each secret object, and makes the entries available as files in that volume. A corresponding pod definition snippet is shown below.

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: ...
      image: ...
      volumeMounts:
          - name: secret-volume
            mountPath: /etc/secret-volume
  # The secret data is exposed to Containers in the Pod through a Volume.
  volumes:
    - name: secret-volume
      secret:
        secretName: test-secret

The JVM system properties passed to the container will now automatically resolve file paths in volumes created for the secret (for example: ... -Dcontroller.auth.tlsCertFile: "/etc/secret-volume/controller01.pem" -Dcontroller.auth.tlsKeyFile: "/etc/secret-volume/controller01.key.pem ...").

In effect, the admin is able to inject externally created TLS files into Pravega pods in a safe and secure manner.

[EronWright]

See related: pravega/pravega#3652 which calls for compatibility with the secrets generated by cert-manager.