trust_system_cert_pool CloudNativePG PostgreSQL ?

[albundy83]

Hello,

using CloudNativePG operator for my PostgreSQL databases in Kubernetes, I want to connect to PG using teleport.
I have created a simple database in namespace my-db-namespace with the following configuration:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
spec:
  affinity:
    podAntiAffinityType: preferred
  backup:
    ...
  bootstrap:
    initdb:
      dataChecksums: true
      database: my-db
      encoding: UTF8
      localeCType: C
      localeCollate: C
      owner: my-db
  enableSuperuserAccess: false
  imageName: ghcr.io/cloudnative-pg/postgresql:16.6
  instances: 1
  managed:
    roles: # Needed to connect to PG see https://mirakl.tech/mastering-kubernetes-and-database-administration-with-teleport-and-cloudnative-pg-a-step-by-step-f768f1c614de
      - comment: Teleport user with read-only access
        connectionLimit: 20
        ensure: present
        inherit: true
        login: true
        name: teleport_user_read
      - comment: Teleport user with read/write access
        connectionLimit: 20
        ensure: present
        inRoles:
          - my-db
        inherit: true
        login: true
        name: teleport_user_write
  monitoring:
    ...
  postgresGID: 26
  postgresUID: 26
  postgresql:
    parameters:
      ...
    pg_hba:
      - hostssl all /^teleport_user_.*$ 10.42.0.0/16 cert # Needed to allow cert authentication inside Kubernetes / RKE2
  primaryUpdateStrategy: unsupervised
  resources:
    ...
  storage:
    ....

On teleport server, according to doc Use a custom CA, I have created db-client.cas file using command:

tctl auth export --type=db-client --auth-server=teleport.youhou.fr:443 > db-client.cas

Then, I have appended db-client.cas content to Secret my-db-ca inside key ca.crt generated by CloudNativePG during Cluster creation.

Finally, I have deployed teleport kube agent inside the same namespace my-db-namespace with the following configuration:

helm upgrade --install teleport-kube-agent teleport-kube-agent \
  --repo https://charts.releases.teleport.dev \
  --namespace my-db-namespace \
  --version 17.1.2 \
  --set authToken=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
  --set proxyAddr=teleport.youhou.fr:443 \
  --set roles=db \
  --set labels.env=prod \
  --set labels.project=my-db-prod \
  --set labels.teleport_version=17.1.2 \
  --set databases[0].name=my-db \
  --set databases[0].protocol=postgres \
  --set databases[0].uri=my-db-rw:5432 \ # Service name inside Kubernetes
  --set databases[0].tls.mode=verify-full \
  --set databases[0].tls.ca_cert_file=/etc/teleport-tls-db/cnpg/ca.crt \
  --set databases[0].tls.trust_system_cert_pool=false \
  --set extraVolumes[0].name=cnpg \
  --set extraVolumes[0].secret.secretName=my-db-ca \ # With CloudNativePG, Certificate Authority is generated in Secret name <cluster_name>-ca
  --set extraVolumeMounts[0].name=cnpg \
  --set extraVolumeMounts[0].mountPath=/etc/teleport-tls-db/cnpg \
  --set extraVolumeMounts[0].readOnly=true \
  --set rbac.create=false # I already have a teleport kube agent inside this Kubernetes

And it works perfectly:

gbellongervais@localhost:~/work/my-db> tsh db connect my-db --db-user=teleport_user_write --db-name=my-db
psql (17.2, server 16.6 (Debian 16.6-1.pgdg110+1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256, compression: off, ALPN: none)
Type "help" for help.

Now, the part I think I did not understand correctly, I hope I could use the feature trust_system_cert_pool.

According to this message, if I update the local system store with the content of Secret my-db-ca on my teleport server, I don't need to mount and use ca_cert_file parameter.

So I collect content of Secret my-db-ca, I add it to local Ubuntu 24.04 system store (in short file /etc/ssl/certs/ca-certificates.crt) where my teleport server (teleport.youhou.fr) is installed and I restart it.
I deploy again my teleport kube agent with the updated parameters:

helm upgrade --install teleport-kube-agent teleport-kube-agent \
  --repo https://charts.releases.teleport.dev \
  --namespace my-db-namespace \
  --version 17.1.2 \
  --set authToken=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
  --set proxyAddr=teleport.youhou.fr:443 \
  --set roles=db \
  --set labels.env=prod \
  --set labels.project=my-db-prod \
  --set labels.teleport_version=17.1.2 \
  --set databases[0].name=my-db \
  --set databases[0].protocol=postgres \
  --set databases[0].uri=my-db-rw:5432 \ # Service name inside Kubernetes
  --set databases[0].tls.mode=verify-full \
  --set databases[0].tls.trust_system_cert_pool=true \ # Now we set it to true
  --set rbac.create=false # I already have a teleport kube agent inside this Kubernetes

But it does not work, I have the following error:

gbellongervais@localhost:~/work/my-db> tsh db connect my-db --db-user=teleport_user_write --db-name=my-db
psql: error: connection to server at "localhost" (::1), port 43415 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?
connection to server at "localhost" (127.0.0.1), port 43415 failed: Database service cannot validate database's certificate: failed to connect to `host=my-db-rw user=teleport_user_write database=my-db`: failed to write startup message (tls: failed to verify certificate: x509: certificate signed by unknown authority). Please verify if the correct CA bundle is used in the database config.
ERROR: exit status 2

Any idea ? Did I understand correctly the documentation ?
Thanks for reading me :)

[webvictim]

My understanding is that your teleport-kube-agent would need the my-db-ca added to the system cert pool, as that's where the Teleport db_service is running - inside the Kubernetes pod.

[albundy83]

Ah that's what I was afraid.
So, I think in Kubernetes, this feature is not very useful no ?
I hope I could load the database certificate directly in my teleport server, not in my teleport kube agent.