What is the "appropriate location" for the root ca?

[GRRLjay]

Under the heading "Migrating from Self-Signed Certificates" on the "Certificates" page, it says "If the certs are signed, ensure you also include the root ca in the appropriate location as well." What is the appropriate location? This is an Ubuntu 22.04 install using the Ansible playbook. It is not clear to me whether we're talking about a location on the host or someplace in a container.

Please make this more clear.

[aarz-snl]

certificates are stored in the lme_certs volume

You can find its location by using sudo -i podman volume inspect lme_certs

you will see storage locations here for the ca, elasticsearch, kibana, etc.

Replace certs in this volume and they will be replicated out to the all containers using them. Ensure permissions for these files match what they were before.

Dont forget on your new certs that they contain the ip address/hostname of your choice -- and add the ca to your trusted stores. This is how you get rid of the 'untrusted cert' error when browsing

https://learn.microsoft.com/en-us/skype-sdk/sdn/articles/installing-the-trusted-root-certificate
https://ubuntu.com/server/docs/install-a-root-ca-certificate-in-the-trust-store

[GRRLjay]

I understand this is not supported. All I really want is for our web interface to use our cert so browsers won't alarm.

I have not been able to replace the self-signed certs with our own corp signed certs. My process is this:

1. Generate .csr files from the pre-existing .key files.
2. Sign the .csr files with our CA.
3. Replace the self-signed certs.
4. Replace the ca.crt with our ca certficate.
5. Generate the elasticsearch.chain.pem file.
6. Fix up ownership and permissions.
7. Start lme.service

For whatever reason Kibana can never fully get started. I tried replacing the kibana cert only, but for some reason that causes kibana to start and restart repeatedly. While started, it seems to work until it goes to restart.

[cbaxley]

Check your permissions on the files. Often servers will not start up if they aren't secure enough. Just do a ls -l on the existing ones and check users and permissions. Thanks, Clint Baxley

…

On Wed, Nov 27, 2024 at 4:39 PM GRRLjay ***@***.***> wrote: I understand this is not supported. All I really want is for our web interface to use our cert so browsers won't alarm. I have not been able to replace the self-signed certs with our own corp signed certs. My process is this: 1. Generate .csr files from the pre-existing .key files. 2. Sign the .csr files with our CA. 3. Replace the self-signed certs. 4. Replace the ca.crt with our ca certficate. 5. Generate the elasticsearch.chain.pem file. 6. Fix up ownership and permissions. 7. Start lme.service For whatever reason Kibana can never fully get started. I tried replacing the kibana cert only, but for some reason that causes kibana to start and restart repeatedly. While started, it seems to work until it goes to restart. — Reply to this email directly, view it on GitHub <#519 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQY33SAC56E7XZWCYEEAUD2CY3ZHAVCNFSM6AAAAABSJ4ZE5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBUHAYDQNJYGU> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[aarz-snl]

Do you have to use your corp self signed certs?

If you HAVE to use your own corp signed certs we will have to look into what his process would look like to retroactively replace certs. But he should be something as described above and as cbaxley said may be a permissions issue after you're replacing the certs in the volume. Your container must be able to read those certs the same as it did before. You must also include all the private docker domain names before as well or the containers will not be able to speak to each other See the instances.yml file to see what those would look like

If you dont have to and are at a place where you can do a full re-install (its not that big of a deal if you aren't collecting data yet and just testing things out) but you could just utilize the instances.yml file. If you're still just using self-signed certs.

https://github.com/cisagov/LME/blob/main/config/setup/instances.yml

This file will be located in LME/config/setup after you download LME and by default handles the localhost ip address and the private docker network domain name. This means these certs really only work via the private docker network communication or if you're actually ON the server going to https://localhost. But you can add anything you want to get OUTSIDE of that private network as well. For instance the actual ip of your server, or even a domain name. Then those certs will also work for that domain name / ip address. You would just add that to the kibana section under dns and ip.

Then you would ensure that your ca is loaded in the trusted store of all the devices that would access it. For instance in instances.yml

  - name: "kibana"
    dns:
      - "lme-kibana" (This is the private docker network domain name, and only works when talking from container to container)
      - "localhost"
      - "your.private.domain.name.if.you.have.one"
    ip:
      - "127.0.0.1"
      - "another ip address" (ie. 192.168.0.23) or whatever ip you use to get to it from an endpoint device.

Then run the install process as you did before. Add the ca to the trusted stores of devices that will access kibana, then when you go to your.domain.name or https://ipaddress you should no longer get the browser error for untrusted certs.

[aarz-snl]

Also, I do not think replacing just ONE cert would work. As the containers talk to each other via their docker dns name and use the ca to do that. You would need to do a self signed cert from the same ca for each application. I haven't verified that with testing, but that would be my assumption.

For instance when you review the quadlet for kibana:

https://github.com/cisagov/LME/blob/main/quadlet/lme-kibana.container

You'll see it has env variables to talk to elasticsearch via the private docker dns name at https://lme-elasticsearch:9200 -- and points to the ca.crt in order to do that

[GRRLjay]

I am specifically trying to use all corp-signed certificates. We don't add self-signed certs or CAs to our trust stores. I did maintain the DNS and IP listed in instances.yml when I was signing it. I just added our lan IP and intranet fqdn. I did check and fix the ownership and perms of the signed certificates after replacing the self-signed ones (step 6 above).

I'm happy to do whatever troubleshooting is needed or provide more detail about the steps I took. Just know that the reason we turned to LME is we don't know enough about the ELK stack to figure it out on our own.

[aarz-snl]

So you may be able to do this by placing your ca in a zip and placing it

~/LME/config/setup/ca.zip

What happens during install this location is completely cloned to /opt/lme as you see here in the ansible script:

- name: Copy config files /opt/lme/config
  copy:
    src: "{{ clone_directory }}/config/"
    dest: /opt/lme/config/
    owner: "{{ install_user }}"
    group: "{{ install_user }}"
    mode: '0644'
  become: yes

Then during install the ca is checked for its existence and if it doesn't exist one is created. See the init-setup.sh script

https://github.com/cisagov/LME/blob/main/config/setup/init-setup.sh

It then uses this ca and the instances.yml to generate certs. My thought here is that instead of creating a new ca it will just use your ca to then generate each individual cert for elasticsearch, kibana etc.

What I think you would need to do in this scenario:

1. After you download LME place your custom ca.crt and ca.key in ~/LME/config/setup/ - zip them up to also have a file called ca.zip included in the same location. (This will force the script to skip generating a new ca.). So, 3 files, ca.crt, ca.key, and ca.zip
2. configure your instances yml to include domain names and ip addresses of your choice.
3. Ensure permissions for these files are set as they currently are for your install before running the install process. See the ansible snippet above -- I think it should do this automatically during the clone process so we should be good there.
4. uninstall and reinstall lme. The cert generation process should now use your ca to generate the certs.

I hope that makes sense. The init-setup.sh could use some cleaning up to integrate this better, but we will need time to test/implement.

[aarz-snl]

This entire idea may need some quick testing - should be able to do this in a few minutes as proof of concept.

The path may actually be this: ~/LME/config/setup/certs/

[aarz-snl]

That doesn't appear to be the solution. As the certs are mounted to volume lme_certs not the path as previously thought This will take some more time to look at

[GRRLjay]

Here's the steps I've just tried. It did not result in a corp-signed set of certificates.

Create my own ca.key and ca.crt signed by our corporate root ca.
Placed them in ~/LME/config/setup/certs/.
sudo systemctl stop lme.service
sudo -i podman volume rm lme_certs (per https://github.com/cisagov/LME/blob/main/docs/markdown/maintenance/certificates.md)
sudo systemctl start lme.service

When I connect with a browser, it's not using the cert's signed by us.

I have tried the procedure with several different directories and none worked:
~/LME/config/setup/certs/
~/LME/config/setup/certs/ca
~/LME/config/setup/

It's easy enough to try other directories if you can tell me where. The init-setup.sh says /usr/share/elasticsearch/config/certs, but that elasticsearch tree doesn't exist so I assume it's some scripting magic happening somewhere else I'm not aware of.

[aarz-snl]

So that path is in the container not on your host. When you review a quadlet you'll see this:

https://github.com/cisagov/LME/blob/main/quadlet/lme-setup-certs.container

Volume=lme_certs:/usr/share/elasticsearch/config/certs

This is saying mount lme_certs volume on the host machine to that location within the container. To find the location on your host machine you would run:

sudo -i podman volume inspect lme_certs

This should give you a location like : /var/lib/containers/storage/volumes/lme_certs/_data

You can then do a sudo su and then cd /var/lib/containers/storage/volumes/lme_certs/_data

From here you will see ALL your certs used across all services. ie. ca, elasticsearch, fleet-server, etc. In theory you could just replace each on here in the volume and it will replicate to each of your services. -- but I'm not sure what that might break as well if we missing something that happens during install -- hence trying to get to work during install.

As a workaround I have created this script that you will replace the init-setup.sh script with before running the install ansible yaml:

First uninstall using uninstall steps.

sudo -i -u root 
systemctl stop lme* && systemctl reset-failed && podman volume rm -a &&  podman secret rm -a && rm -rf /opt/lme && rm -rf /etc/lme && rm -rf /etc/containers/systemd

Replace the contents of init-setup.sh with this code instead(located at ~/LME/config/setup/init-setup.sh)

#!/bin/bash
set -euo pipefail

if [[ -z "${ELASTIC_PASSWORD:-}" || -z "${KIBANA_PASSWORD:-}" ]]; then
  echo "ERROR: ELASTIC_PASSWORD and/or KIBANA_PASSWORD are missing."
  exit 1
fi

CONFIG_DIR="/usr/share/elasticsearch/config"
CERTS_DIR="${CONFIG_DIR}/certs"
DATA_DIR="/usr/share/elasticsearch/data"
INSTANCES_PATH="${CONFIG_DIR}/setup/instances.yml"

# Check for either custom CA or existing CA
if [ -f "/usr/share/elasticsearch/config/setup/ca.crt" ] && [ -f "/usr/share/elasticsearch/config/setup/ca.key" ]; then
    echo "Using provided custom CA..."
    mkdir -p "${CERTS_DIR}/ca"
    cp "/usr/share/elasticsearch/config/setup/ca.crt" "${CERTS_DIR}/ca/ca.crt"
    cp "/usr/share/elasticsearch/config/setup/ca.key" "${CERTS_DIR}/ca/ca.key"
elif [ ! -f "${CERTS_DIR}/ca/ca.crt" ] || [ ! -f "${CERTS_DIR}/ca/ca.key" ]; then
    echo "No CA found, creating new CA..."
    elasticsearch-certutil ca --silent --pem --out "${CERTS_DIR}/ca.zip"
    unzip -o "${CERTS_DIR}/ca.zip" -d "${CERTS_DIR}"
fi

if [ ! -f "${CERTS_DIR}/elasticsearch/elasticsearch.crt" ]; then
    echo "Creating certificates..."
    elasticsearch-certutil cert --silent --pem --in "${INSTANCES_PATH}" --out "${CERTS_DIR}/certs.zip" --ca-cert "${CERTS_DIR}/ca/ca.crt" --ca-key "${CERTS_DIR}/ca/ca.key"
    unzip -o "${CERTS_DIR}/certs.zip" -d "${CERTS_DIR}"
    cat "${CERTS_DIR}/elasticsearch/elasticsearch.crt" "${CERTS_DIR}/ca/ca.crt" > "${CERTS_DIR}/elasticsearch/elasticsearch.chain.pem"
fi

echo "Setting file permissions... certs"
chown -R elasticsearch:elasticsearch "${CERTS_DIR}"
find "${CERTS_DIR}" -type d -exec chmod 755 {} \;
find "${CERTS_DIR}" -type f -exec chmod 644 {} \;

echo "Setting file permissions... data"
chown -R elasticsearch:elasticsearch "${DATA_DIR}"

add your ca.crt and ca.key to this location ~/LME/config/setup The script will now use your ca to generate certificates instead.

NOW go through the install steps of setting up the .env file, running the ansible playbook.

[aarz-snl]

If this works for you it could guide in a direction to implementing this forever into the install. For me i was able to get all my services up and running using a random ca i generated using openssl

See here just checking one of my custom certs which shows a city i added during generation instead of "randomly generated by elasticsearch"

[GRRLjay]

So to be clear, this can't be done post-install? The corporate ca must be provisioned prior to installation?

I have already tried the approach of replacing all the certs in /var/lib/containers/storage/volumes/lme_certs/_data and that did not work as stated earlier.

I will give this script a try.

[aarz-snl]

My assumption is that it SHOULD work... but that just hasn't been done or tested. Id have to make sure your creation of a ca and subsequent creation of every other cert is in line with what we are currently doing with elastic cert util

It may take actually installing elast cert util on the ubuntu instance and then running CLI commands using the custom ca.crt and ca.key. THEN replacing all the certs in /var/lib/containers/storage/volumes/lme_certs/_data

[aarz-snl]

This is how you can manually generate certs with elastic cert util

make a directory

mkdir ~/LME/config/setup/generated_certs

run the command (youll need to adjust the path to your user)

podman run -it --rm \
  -v /home/lme-user/LME/config/setup/instances.yml:/usr/share/elasticsearch/config/instances.yml:Z \
  -v /home/lme-user/LME/config/setup/ca.crt:/usr/share/elasticsearch/config/ca.crt:Z \
  -v /home/lme-user/LME/config/setup/ca.key:/usr/share/elasticsearch/config/ca.key:Z \
  -v /home/lme-user/LME/config/setup/generated_certs:/usr/share/elasticsearch/config/certs:Z \
  localhost/elasticsearch:LME_LATEST \
  /bin/bash -c "bin/elasticsearch-certutil cert --silent --pem \
    --ca-cert config/ca.crt \
    --ca-key config/ca.key \
    --in config/instances.yml \
    --out config/certs/certificates.zip \
    && unzip -o config/certs/certificates.zip -d config/certs \
    && cat config/certs/elasticsearch/elasticsearch.crt config/ca.crt > config/certs/elasticsearch/elasticsearch.chain.pem"

this assumes you have your ca.crt and ca.key and updated instances.yml in ~/LME/config/setup

it will generate your certs into ~/LME/config/setup/generated_certs

You should be able to manually move these into the volume location. Dont forget to move your ca.crt and ca.key as well

I ended up having to also adjust permissions;

sudo chown -R 165536:165536 /var/lib/containers/storage/volumes/lme_certs/_data

sudo chmod -R 755 /var/lib/containers/storage/volumes/lme_certs/_data

sudo systemctl restart lme.service

This APPEARS to be working for me -- replacing the certs AFTER install

[GRRLjay]

At first I got the following result. This was because the self-signed certificate was not trusted by the server OS.

Trying to pull localhost/elasticsearch:LME_LATEST...
Error: initializing source docker://localhost/elasticsearch:LME_LATEST: pinging container registry localhost: Get "https://localhost/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority

After adding the self-signed ca.crt to the servers ca store, the error is now:

Trying to pull localhost/elasticsearch:LME_LATEST...
Error: initializing image from source docker://localhost/elasticsearch:LME_LATEST: invalid character '<' looking for beginning of value

podman run -it --rm
-v /home/lme-user/LME/config/setup/instances.yml:/usr/share/elasticsearch/config/instances.yml:Z
-v /home/lme-user/LME/config/setup/ca.crt:/usr/share/elasticsearch/config/ca.crt:Z
-v /home/lme-user/LME/config/setup/ca.key:/usr/share/elasticsearch/config/ca.key:Z
-v /home/lme-user/LME/config/setup/generated_certs:/usr/share/elasticsearch/config/certs:Z
localhost/elasticsearch:LME_LATEST
/bin/bash -c "bin/elasticsearch-certutil cert --silent --pem
--ca-cert config/ca.crt
--ca-key config/ca.key
--in config/instances.yml
--out config/certs/certificates.zip
&& unzip -o config/certs/certificates.zip -d config/certs
&& cat config/certs/elasticsearch/elasticsearch.crt config/ca.crt > config/certs/elasticsearch/elasticsearch.chain.pem"

[aarz-snl]

try just pulling the public elasticsearch image instead:

podman run -it --rm \
  -v /home/lme-user/LME/config/setup/instances.yml:/usr/share/elasticsearch/config/instances.yml:Z \
  -v /home/lme-user/LME/config/setup/ca.crt:/usr/share/elasticsearch/config/ca.crt:Z \
  -v /home/lme-user/LME/config/setup/ca.key:/usr/share/elasticsearch/config/ca.key:Z \
  -v /home/lme-user/LME/config/setup/generated_certs:/usr/share/elasticsearch/config/certs:Z \
  docker.elastic.co/elasticsearch/elasticsearch:8.15.3 \
  /bin/bash -c "bin/elasticsearch-certutil cert --silent --pem \
    --ca-cert config/ca.crt \
    --ca-key config/ca.key \
    --in config/instances.yml \
    --out config/certs/certificates.zip \
    && unzip -o config/certs/certificates.zip -d config/certs \
    && cat config/certs/elasticsearch/elasticsearch.crt config/ca.crt > config/certs/elasticsearch/elasticsearch.chain.pem"

This ca doesn't have anything to do with the pulling of the image. So it shouldn't have an impact on it. It's using podman. Does your org use a proxy with TLS inspection? Because that can complicate things

[GRRLjay]

For testing purposes, I removed any TLS inpsection for this machine and the result is the same. I am an absolute novice with podman, but the original command given seems to be pulling from a service internal to this machine. However, if the service is using an untrusted certificate, it makes sense that there would be an issue.

This last modification executed without error. The browser trusts the Kibana web interface. However, I'm back where I've been before with "Kibana server is not ready yet."

$ sudo systemctl status lme-kibana

provides the following error:

Dec 05 15:00:24 sv-slme01 lme-kibana[44428]: [2024-12-05T15:00:24.634+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. unable to get issuer certificate

[aarz-snl]

Yes, it should be pulling internal -- that image should exist in the local repo on the machine. I'm not exactly sure why you would have TLS issues to a local repo as I dont get this error on my ubuntu machine.

The other error is because the cert isn't talking correctly with elasticsearch.

Did you cp all certs generated in generated certs path and replace in the volume? And adjust the permissions?

You can also run the command

sudo -i podman logs lme-elasticsearch

And sudo -i podman logs lme-kibana

And paste the logs in here

[aarz-snl]

I'm going to run through all my steps again on an install and verify

[aarz-snl]

verify ca.crt and ca.crt are in config/setup

run podman command

should now have these crts in generated_certs: ( you wont have all these -- some like 'keycloak' are for my local testing)

move the ca and the certs to replace them in the volume - adjust to match your home directory

sudo su

cp -r /home/lme-user/LME/config/setup/generated_certs/* /var/lib/containers/storage/volumes/lme_certs/_data/

cp /home/lme-user/LME/config/setup/ca.* /var/lib/containers/storage/volumes/lme_certs/_data/ca/

adjust permissions

chown -R 165536:165536 /var/lib/containers/storage/volumes/lme_certs/_data
chmod -R 755 /var/lib/containers/storage/volumes/lme_certs/_data

exit sudo with exit

restart lme.service

sudo systemctl restart lme.service

I dont get the same error you're seeing:

From browser im seeing my custom cert details

[GRRLjay]

I followed your steps with one adjustment. I had to include our corp-signed root and intermediate cert in the elasticsearch.chain.pem. If I don't, elasticsearch won't even start.

Requested logs are attached.

lme-kibana.log
lme-elasticsearch.log

[aarz-snl]

Does your ca.crt thats located in:

/var/lib/containers/storage/volumes/lme_certs/_data/ca/ca.crt

Also contain the intermediate and root ca details?

[GRRLjay]

Does your ca.crt thats located in:

/var/lib/containers/storage/volumes/lme_certs/_data/ca/ca.crt

Also contain the intermediate and root ca details?

It is signed by our intermediate, but it is not a chain file.

[GRRLjay]

My theory is kibana or the container itself doesn't trust our root. I wouldn't know how to achieve that.

[aarz-snl]

I would attempt to do this by included all the crts into the .crt to make it a chain.

For some reason kibana cant find the issuer -- and thats the only thing I can think of.

The order of your elasticsearch chain would be like:

elasticsearch.crt
intermediate.crt
root-ca.crt

And your custom ca would be

ca.crt
intermediate.crt
root-ca.crt

Granted I'm not really an expert of certs in an organization like this -- but its unable to determine the issuer and thats the only thing I can think of. You can can just copy the contents of those certs into the ca.crt

The same thing is done in ubuntu when trusting crts:

/etc/ssl/certs/ca-certificates.crt is just a chain of a bunch of crts

[GRRLjay]

There it is. I can log in now. My commands:

sudo su -
cd /var/lib/containers/storage/volumes/lme_certs/_data/ca
cp ca.crt ca.crt.bak
cat ca.crt.bak Enterprise.crt ROOT.crt > ca.crt

[aarz-snl]

Great -- now when you install elastic agents we typically have users add --insecure to the end of the install command for development / testing (which keeps it tls but skips verification)

if your endpoints already have this ca trusted -- you should be able to not add --insecure and it will verify successfully using your orgs certs that are already installed.

[GRRLjay]

Alright, here is the whole thing in one procedure for posterity:

After ansible-playbook -K ~/LME/ansible/install_lme_local.yml:

1. Create a key and csr with CA extensions on the LME machine.

2. Sign the csr with the organizations root or intermediate CA.

3. Bring the crt back to the LME machine along with the root.crt and intermediate.crt (if applicable).

4. Rename the .key file to ca.key

5. Rename the .crt file to ca.crt

6. Copy the ca files to the setup folder
   ~$ cp ca.* ~/LME/config/setup

7. Create ca.zip file

~$ mkdir ca
~$ cp ca.* ca/
~$ zip -r ca.zip ca/

8. Update instances.yml to add organization specific DNS and IPs
   ~$ vi ~/LME/config/setup/instances.yml

9. Create the folder for the elasticsearch signed certs and update permissions so it's writable for the next command.

~$ mkdir ~/LME/config/setup/generated_certs
~$ chmod 777 ~/LME/config/setup/generated_certs

10. Run the podman command to generate certificates signed by our ca.

~$ podman run -it --rm \
  -v /home/lme-user/LME/config/setup/instances.yml:/usr/share/elasticsearch/config/instances.yml:Z \
  -v /home/lme-user/LME/config/setup/ca.crt:/usr/share/elasticsearch/config/ca.crt:Z \
  -v /home/lme-user/LME/config/setup/ca.key:/usr/share/elasticsearch/config/ca.key:Z \
  -v /home/lme-user/LME/config/setup/generated_certs:/usr/share/elasticsearch/config/certs:Z \
  docker.elastic.co/elasticsearch/elasticsearch:8.15.3 \
  /bin/bash -c "bin/elasticsearch-certutil cert --silent --pem \
    --ca-cert config/ca.crt \
    --ca-key config/ca.key \
    --in config/instances.yml \
    --out config/certs/certs.zip \
    && unzip -o config/certs/certs.zip -d config/certs \
    && cat config/certs/elasticsearch/elasticsearch.crt config/ca.crt  > config/certs/elasticsearch/elasticsearch.chain.pem"

11. Copy the certificate zip files to the lme_certs volume.

~$ sudo cp ~/ca.zip /var/lib/containers/storage/volumes/lme_certs/_data
~$ sudo cp ~/LME/config/setup/generated_certs/certs.zip /var/lib/containers/storage/volumes/lme_certs/_data

12. Unzip the certificates into their target directories.

~$ sudo unzip /var/lib/containers/storage/volumes/lme_certs/_data/ca.zip -d  /var/lib/containers/storage/volumes/lme_certs/_data/
~$ sudo unzip /var/lib/containers/storage/volumes/lme_certs/_data/certs.zip -d  /var/lib/containers/storage/volumes/lme_certs/_data/

13. Backup the ca.crt file before we turn it into a chain.
    ~$ sudo cp /var/lib/containers/storage/volumes/lme_certs/_data/ca/ca.crt /var/lib/containers/storage/volumes/lme_certs/_data/ca/lmeca.crt

14. Chain the ca.crt file with our corporate intermediate and root certificate.

~$ cat ~/corp-intermediate.crt | sudo tee --append  /var/lib/containers/storage/volumes/lme_certs/_data/ca/ca.crt > /dev/null
~$ cat ~/corp-root.crt | sudo tee --append  /var/lib/containers/storage/volumes/lme_certs/_data/ca/ca.crt > /dev/null

15. Replace the elasticsearch.chain.pem with one containing our corporate intermediate and root certificate.
~$ sudo cat /var/lib/containers/storage/volumes/lme_certs/_data/elasticsearch/elasticsearch.crt | sudo tee /var/lib/containers/storage/volumes/lme_certs/_data/elasticsearch/elasticsearch.chain.pem > /dev/null
~$ sudo cat /var/lib/containers/storage/volumes/lme_certs/_data/ca/ca.crt | sudo tee --append  /var/lib/containers/storage/volumes/lme_certs/_data/elasticsearch/elasticsearch.chain.pem > /dev/null

16. Update ownership and permissions.

~$ sudo chown -R 165536:165536 /var/lib/containers/storage/volumes/lme_certs/_data
~$ sudo chmod -R 755 /var/lib/containers/storage/volumes/lme_certs/_data

17. Restart lme.service.
    ~$ sudo -i systemctl restart lme.service

[GRRLjay]

This covers post-install. I'm not sure about the pre-install method above. I have not been able to install from scratch for several days due to some error in the process.

[aarz-snl]

for the pre-install method the script would have to be updated to also include copying of root-ca's and intermediate ca's into the final ca.crt

This should be fine going forward for you as any updates to your crts you'd probably want to do post install without an uninstall anyway

[GRRLjay]

I think it's important to note that elasticsearch-certutil wants the ca.crt file to be a single certificate. It threw an error when I tried to give it a chain. That's why my process assembles the ca chain afterward.

[aarz-snl]

That's pretty typical when dealing with generating certs using a ca and its associated key. Its normal to see chains combined afterwards.

I'm going to add to some docs on this to capture this process for other users in the same boat