Acme parsing error - Multiple certificationResolvers

[petrleocompel]

Classification

• Serious bug

Reproducibility

• Always

Docker information

Client:
 Version:    24.0.5
 Context:    default
 Debug Mode: false

Server:
 Containers: 14
  Running: 14
  Paused: 0
  Stopped: 0
 Images: 26
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: journald
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: /usr/libexec/docker/docker-init
 containerd version:
 runc version:
 init version:
 Security Options:
  seccomp
   Profile: builtin
  selinux
  cgroupns
 Kernel Version: 6.5.11-300.fc39.x86_64
 Operating System: Fedora CoreOS 39.20231119.3.0
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.6GiB
 Name: cac
 ID: 63493c27-f150-4b61-a3db-f6880880998b
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: petrleocompel
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: true

docker images mailserver2/mailserver --digests --filter "dangling=false"
REPOSITORY               TAG       DIGEST                                                                    IMAGE ID       CREATED        SIZE
mailserver2/mailserver   1.1.15    sha256:c85fc055d805333a18210fa6d8fc7227a2f0d3dff519b2cfa805bc0410b61c63   d7b64bc841d8   6 months ago   421MB

docker images traefik --digests --filter "dangling=false"
REPOSITORY   TAG       DIGEST                                                                    IMAGE ID       CREATED        SIZE
traefik      2.10      sha256:c5181ddf303f1ccfd4bd6d1d9c4867b0500efb6089a0f9ccb16612438f6e934f   64586c703ab1   5 weeks ago    153MB

Description

Wildcard letsencrypt certificate cannot throws error in parsing.

Steps to reproduce

1. Wildcard domain configuration with traefik:2.10

Expected results

Parsing correctly PEM

Actual results

Acme certificate is present in JSON but cannot be parsed..

Debugging information

[INFO] MariaDB/PostgreSQL hostname not found in /etc/hosts
[INFO] Container IP found, adding a new record in /etc/hosts
[INFO] Redis hostname not found in /etc/hosts
[INFO] Container IP found, adding a new record in /etc/hosts
[INFO] Search for SSL certificates generated by Traefik
[INFO] acme.json found with Traefik v2 format, dumping into pem files
[ERROR] The certificate for mail.xxx.xx or the private key was not found !
[INFO] Don't forget to add a new traefik frontend rule to generate a certificate for mail.xxx.xx subdomain
[INFO] Look /mnt/docker/traefik/acme/dump.log and 'docker logs traefik' for more information
[INFO] Starting services

dump.log

[INFO] acme.json found with Traefik v2 format, dumping into pem files
Could not read private key from <stdin>
40E79918CB7F0000:error:1E08010C:DECODER routines:OSSL_DECODER_from_bio:unsupported:../crypto/encode_decode/decoder_lib.c:101:No supported data to decode. Input type: PEM

Configuration (docker-compose.yml, traefik.toml...etc)

docker-compose.yml

Mailserver:
    ....
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=http_network"
      - "traefik.http.routers.spam.entrypoints=websecure"
      - "traefik.http.routers.spam.rule=Host(`spam.${MAILSERVER_DOMAIN}`)"
      - "traefik.http.routers.spam.service=spam"
      - "traefik.http.routers.spam.tls=true"
      - "traefik.http.routers.spam.tls.certresolver=letsencrypt"
      - "traefik.http.routers.spam.tls.domains[0].main=xxx.xx"
      - "traefik.http.routers.spam.tls.domains[0].sans=*.xxx.xx"

[AndrewSav]

Thank you for reporting. How can I reproduce it?

[petrleocompel]

I actually just updated traefik to version 2.10. After renewing of the certs this happened. I validated that I can extract them with traefik-certs-dumper (works).

I have no clue where to start how to simulate that. If you would lead me where this output is coming from "Could not read private key from" and where it is executed I will try to get more info why it is happening.

[AndrewSav]

Most likely it's this line:

mailserver/rootfs/usr/local/bin/dumpcerts.traefik.v2.sh

Line 155 in e10532a

| openssl rsa -inform pem -out "${pdir}/letsencrypt.key" >/dev/null

[petrleocompel]

Thanks I will try in few days. I will investigate and post results.

[ksylvan]

@petrleocompel Do you have any updates?

[petrleocompel]

I will have to try tomorrow. Also certs should be renewing these days. Lets see if DNS challenge got back to normal and can be parsed.

[ksylvan]

@petrleocompel Any updates on this?

[petrleocompel]

Sorry. Yes. So It few days ago it was looking like it still has issues. Currently I seems to be working fine. No clue why this happened. (There was no internet/power/system outage) Just started working and since then I have not done anything.
I will try to investigate in anycase I will reopen this issue. But for now I have no info to provide sadly. Hopefully it was just signle time problem.

[petrleocompel]

Finally !! Found some more details.

1. There was a Selinux file permission problem.

All my mounts have z and Z labels. For acme it is z since it is shared.

Since I always tested in docker exec as root I had permissions.
Today I had weird problem with "manual cert update" and found out permission was different for postfix user.

[root@srv mail]# docker exec -it -u 102 mail-mailserver-1 bash
postfix@mail:/$ cd /etc/letsencrypt/acme/
bash: cd: /etc/letsencrypt/acme/: Permission denied

[root@srv traefik]# ls -la
drw-------.  4 root root 4096 Dec 11 14:52 acme
[root@srv traefik]# ls -la acme/
-rw-------. 1 root  root      65427 Jan 11 23:37 acme.json

So after fixing this ...

2. Still happens

I tested under postfix user. Not it can "read" the file acme.json. (I tested cat and permissions i can see file contents under postfix user)

[INFO] acme.json found with Traefik v2 format, dumping into pem files
[ERROR] The certificate for mail.xxx.xx or the private key was not found !

postfix@mail:/$ /usr/local/bin/dumpcerts.traefik.v2.sh /etc/letsencrypt/acme/acme.json /tmp/
Could not read private key from <stdin>
40179710417F0000:error:1E08010C:DECODER routines:OSSL_DECODER_from_bio:unsupported:../crypto/encode_decode/decoder_lib.c:101:No supported data to decode. Input type: PEM

So investigated more...

Finally I stumbled across the jq command which is input of private file to openssl.
Mine returned 2 entries for path .[].Account.PrivateKey

postfix@mail:/$ jq  -e -r '.[].Account.PrivateKey'  /etc/letsencrypt/acme/acme.json
xxx==
yyy=

click happened two base64 returns.

So the problem is ... Multiple certificationResolvers. For me one is httpChallange second is dnsChallange.
And so there are 2 private keys.

traefik-certs-dumper understands this. But the dumpcerts.traefik.v2.sh does not.

It took me a year to figure this out....

[AndrewSav]

@petrleocompel since you have the test case you are probably in the best postition to fix this. Would you like to submit a PR?

[petrleocompel]

@AndrewSav Probably I will.

But is there a reason why to use custom script and not to use something like -> https://github.com/ldez/traefik-certs-dumper
Those guys had to solve it long time ago.

Currently in the script I do not know how would I fix that.
Only solution i can think of is Iterate by the Account and do it in loop.

[AndrewSav]

@petrleocompel

But is there a reason why to use custom script and not to use something like

I have to guess here, but I think the reasons could be that the output of that tool is not what mailserver wants, and that it's not a bash script which mailserver seems to use for scripting.

Currently in the script I do not know how would I fix that.

What is that you do not know? May be I can help, what is the problem?

Only solution i can think of is Iterate by the Account and do it in loop.

It's very hard for me to tell because I have no idea of what input and output are. Can you post your file please? You can substitute all values for "redacted", we just need the structure.

[petrleocompel]

@AndrewSav

I have to guess here, but I think the resasons could be that the output of that tool is not what mailserver wants, and that it's not a bash script which mailserver seems to use for scripting.

I was using it as fallback ssl export. Just exported the certs, renamed and moved to "fallback ssl folder" and mailserver found them.

./traefik-certs-dumper file --domain-subdir=true --source ./acme.json --version v2
cd dump/xx.xx/
mv certificate.crt cert.pem
mv privatekey.key privkey.pem
cd ..
cd mail.xx.xx/
mv privatekey.key privkey.pem
mv certificate.crt cert.pem

# example only
mv dump .../xxx/ssl-mount 

The whole script could be replaced by pre-bundled tool.

What is that you do not know? May be I can help, what is the problem?

As explained jq retrurns 2 outputs. 2 base64 lines separated by \n

mailserver/rootfs/usr/local/bin/dumpcerts.traefik.v2.sh

Line 107 in 8cb6ee2

priv=$(${jq} -e -r '.[].Account.PrivateKey' "${acmefile}") || bad_acme

And openssl later does not expect this and crashes.

mailserver/rootfs/usr/local/bin/dumpcerts.traefik.v2.sh

Line 154 in 8cb6ee2

echo -e "-----BEGIN RSA PRIVATE KEY-----\n${priv}\n-----END RSA PRIVATE KEY-----" \

Not sure why do we even need to extract the letsencrypt.key ??? Is it needed ? Cannot find any other mentions ?
Than just "export" of it to file.

structure

{
  "http-acme": {
    "Account": {
      "Email": "redacted",
      "Registration": {
        "body": {
          "status": "valid",
          "contact": [
            "mailto:redacted"
          ]
        },
        "uri": "redacted"
      },
      "PrivateKey": "redacted",
      "KeyType": "4096"
    },
    "Certificates": [
      {
        "domain": {
          "main": "mail.redacted"
        },
        "certificate": "redacted",
        "key": "redacted",
        "Store": "default"
      }
    ]
  },
  "dns-acme": {
    "Account": {
      "Email": "redacted",
      "Registration": {
        "body": {
          "status": "valid",
          "contact": [
            "mailto:redacted"
          ]
        },
        "uri": "redacted"
      },
      "PrivateKey": "redacted",
      "KeyType": "4096"
    },
    "Certificates": [
      {
        "domain": {
          "main": "redacted",
          "sans": [
            "*.redacted"
          ]
        },
        "certificate": "redacted",
        "key": "redacted",
        "Store": "default"
      },
      {
        "domain": {
          "main": "redacted"
        },
        "certificate": "redacted",
        "key": "redacted",
        "Store": "default"
      }
    ]
  }
}

[petrleocompel]

I think just by "not exporting" the letsencrypt.key everything will work.
I really do not know why is it exported.

[petrleocompel]

@AndrewSav I commented out the line

mailserver/rootfs/usr/local/bin/dumpcerts.traefik.v2.sh

Lines 154 to 155 in 8cb6ee2

	echo -e "-----BEGIN RSA PRIVATE KEY-----\n${priv}\n-----END RSA PRIVATE KEY-----" \
	| openssl rsa -inform pem -out "${pdir}/letsencrypt.key" >/dev/null

restarted docker container and it works.
It is dirty fix because it did not extract mail.domain.xx from the HTTP challenge but since there is a DNS challenge for the *.domain.xx... So it is overwritten by selecting "upper domain"

select (.domain.main == $domain )

mailserver/rootfs/usr/local/bin/dumpcerts.traefik.v2.sh

Line 163 in 8cb6ee2

select (.domain.main == $domain )| .certificate' ${acmefile}) || bad_acme

❗ but the "extraction" of letsencrypt.key is unnecessary and can crash in multiple certificationResolvers scenario.

[AndrewSav]

I've create mailserver2/debian-mail-overlay#28 to add traefik-cert-dumper into the based image. Let's see how it goes, we also will need to build a new version of mailserver based on that image if it gets merged. After that we can switch to using that instead of the script.