Let's Encrypt Integration

[jerrinot]

An integration with Let's Encrypt for easy to use & secure HTTPS could be a killer feature. Configuring TLS is PITA in most application containers.

For inspiration see Caddy

Also: https://github.com/shred/acme4j

[matthiasblaesing]

Not sure whether this is appropriate to hijack this - but it is related, as it covers the TLS integration. At work I have a gateway system for which access is protected by a custom CA which is configured to regularly create a CRL. Big problem: I have to restart the gateway system to reload the CRL.

If I'm not mistaken this is also true for changes to the trust- and keystores. So to sum it up it would be nice if the TLS implementation would dynamicly adapt to changes to the trust-/keystores and CRLs.

[smillidge]

How would you see this working from a user perspective. What steps would you see them doing?

@matthiasblaesing dynamically reacting to keystore changes is doable but is an enhancement request can you raise an issue for that?

[jerrinot]

@smillidge: See https://caddyserver.com/docs/automatic-https

Also:

:-)

For Payara Micro one could use whatever Payara Micro is using for configuration.

Having this in Payara Micro would be really a killer feature. "The simplest way to run your Java application in production." (in your face Spring Boot! :)))

[smillidge]

Cool picture. As long as people realise it probably won't work seamlessly behind a proxy, it would be a great feature to have.

[mikecroft]

Internal issue id PAYARA-1061

[Davio]

If you would use and enable this, you could maybe also enable secure admin by default?
And perhaps this would require the hostname to be set to some real value, i.e. not localhost.

[ratcashdev]

"The simplest way to run your Java application in production."
I like the idea, although IMHO it would take more just an integration with letsencrypt. The wireframe should also include options to:

• adding the header: Strict-Transport-Security:max-age with a sane default value
• force Perfect Forward secrecy by pre-selecting (or disabling) the necessary cipher suites

While we're at it, if on-the-fly certificate reload and change would be possible, I would actually prefer doing it through asadmin CLI in a scriptable fashion (sample script being part of the payara package).

[ghost]

Hi all,

I'd like to contribute to the LetsEncrypt integration if I can. What I have so far is a servlet filter that responds to the LetsEncrypt-challenge (installed via default-web.xml) and a post processing Java application that updates the GF keystore when it gets triggered by the acme renew process. It just sends an e-mail when the SSL certificates have been renewed and I have to manually restart the server:

I do have an issue with the web admin console after the keystore update, because it breaks the login. What I do to solve it is basically disabling secure admin, restart the server, enable secure admin again, restart the server.

It would be good to have a mBean that does whatever gets done during this process, i.e. reloading the SSL certificates from the keystore file and fixing the login keys if secure admin is enabled. My post processing Java program could then simply call that mBean to finish the job.

I'm happy to provide the mBean (empty body for the server side to get finished by someone who knows GF) and the full source of my LE code. I think this is a viable approach, because it requires minimal effort on the Glassfish code.

Cheers,
Andy

[mikecroft]

Hey @a-genius that's great! Community contributions are very gratefully received. I can't see that we have any signed CLA from you. If you haven't already signed and emailed on back to us, could you do that so that we can accept your PR when it's ready? If you'd like to contribute unfinished code, then I guess probably @arjantijms or @smillidge would need to review it and see if we can schedule in some time to work on it ourselves (we're pretty busy lately!) Thanks again

[ggam]

@a-genius while the idea is great, I think Servlet Filters might bee executed too late (JASPIC is loaded before any filters). I think you will probably need to implement it in a lower level tier (Grizzly filters, Tomcat valves or similar).

[ratcashdev]

My vote goes for a Custom SOTERIA - JSR375 AuthenticationMechanism that would handle the callbacks. A Valve is also good, as well as a ServerauthenticationModule of JASPIC - JSR196, but then again, those are very low level. Why not using the facilities that are available in a JAVA EE 7/8 app server?

A couple of factors to consider before you opt for any approach:

1. Payara must be reachable from the internet for LetsEncrypt to work. Most installations on home laptops/computers are behind NAT and thus callbacks won't work, unless there is a port mapping on the NAT router.
2. Cloud-based DEV instances of app-servers are usually pure, out-of-the box installations directly connected to the internet. I think this is the group of users who would benefit the most from letsencrypt integration.
3. production instances of app servers on the other hand tend to be behind load-balancing reverse proxies (think of nginx, HAProxy). @mikecroft correct me if I am wrong - you know the best how is payara usually deployed in production among your clients. Anyway, if my assumption is correct, we need to consider that:
   1. proxies may act as SSL terminators, and thus there would be little reason to use Let's encrypt, Some companies (mostly banks) tend to configure even internal channels as SSL, but then again, they won't use Let's Encrypt for internal certificates.
   2. we need to know the external (exposed) URL of our app-server
   3. proxies may not let through the callbacks, or may not direct it to the correct node.

Ideally, the solution chosen would be usable with little to no modifications in all scenarios (where it makes sense, that is). For this it is imperative that the (externally visible) callback URL is configurable.
If we wanted to really do the extra mile for the (production) customers, we'd:

1. check if we have an external IP
2. try to do some UPNP hole punching (if the user agrees)
3. include detailed explanation and instructions with ready-made templates for NGINX and HAProxy that could be copy-pasted so that the callback is forwarded to the correct node.

[ghost]

I'm sure that there are many ways to integrate GF with LetsEncrypt. My idea was to interfere with the application server as little as possible. That is why I didn't consider a library like acme4j for this job - the application server should not implement the acme protocol itself. This would be another burden that needed maintenance.
The ACME protocol (and the certbot client) support different sorts of verification, e.g. web-root verification or DNS verification. This is the reason why I split the code into two artefacts that can be installed independently:

The acme-challenge filter
The acme post processor

The acme-challenge filter can be installed if one has a web-application at the root-context of each (virtual) domain set up at your server. If someone can't use that filter, they may need to use DNS validation, or someone implements a different mechanism to achieve the goal, e.g. for JASPIC ( I never used it so I can't help here).

The post processor is a Java stand-alone application. It gets started by the certbot tool and just copies the received certificate (which is a pem-file that contains the cert chain and another one that contains the private key) into the GF keystore. It doesn't need any connection to the outside world, so it will run happily on localhost as well.

The only part that is missing is getting the server reading in the new certificate. If it had an mBean API that triggers the re-load, the post-processor could just call it once its finished. That way no server restart would be required and the integration would be complete.

If the mBean is the only change, users who don't need that integration don't have to handle all that stuff. It'd still be a good feature, for instance one could trigger a re-load manually using JVisualVM after an update of the keystore.

I think this is a good place to discuss different approaches.

[smillidge]

There a couple of requirements together here. The first is Let's Encrypt integration which would require the challenge filter.

The second is a simple way to add a certificate to the keystore and configure it without a restart. I would like to add the ability to load any certificate into the keystore as an asadmin command. In that way the certs could be loaded into the keystore, the configuration refreshed and the keystore replicated across all running instances in the domain.
There is an example asadmin command here if someone fancies having a go https://github.com/payara/Payara/blob/master/nucleus/payara-modules/service-exemplar/src/main/java/fish/payara/service/example/config/admin/SetExampleServiceMessage.java.

Keystore manager is here https://github.com/payara/Payara/blob/0ff8de27c0083a6fd8118f6a7caf7a53e27ba04c/nucleus/admin/server-mgmt/src/main/java/com/sun/enterprise/admin/servermgmt/KeystoreManager.java

[ratcashdev]

I have glanced through the preferred ways of integrating letsencrypt. They mention two basic scenarios: with shell access or without.
I assume we want to go with the shell route (so that we don't need to integrate ACME diretly). In this scenario all we need is:

• asadmin command for reloading/changing keys and a suitable keystore manager (as suggested above) - a plugin for certbot (instead of the acme-challenge filter being part of the app-server) so that it uploads a .war and uses the correct context-path.

Another possibility is embedding a minimalistic web-server into asadmin (see @AdamBien 's https://github.com/AdamBien/nano web server). Chances are with this approach we would not need to write a certbot plugin and could do with the existing webroot (https://certbot.eff.org/docs/using.html#webroot) plugin for certbot.

Once the cert is ready, we would use asadmin in both cases to upload it to payara. What do you think?

[ratcashdev]

I'll try to do the asadmin command (unless someone will be faster).
@smillidge how does key management work in GF? Specifically:

1. if I execute asadmin command changing the keystore against a remote instance that happens to be a DAS instance, will it distribute the changed keystore/truststore automatically after the change to all nodes?
2. is there already a mechanism to reload/change keys and aliases on the fly without restarting GF or will we have to do this as well?
   Update: Just tested it. GF is not able to use keys/aliases added to the keystore while GF is running. Switching existing aliases becomes effective when the corresponding Network listener is restarted.

[ratcashdev]

Tried to implement an AdminCommand just like in your example, but its not recognized by asadmin - not until i change it to LocalDomainCommand. What am I missing?

[ghost]

I like the idea to have an asadmin command for reloading the keys. As expected, some others have the same problem, maybe we can learn something from their ideas, e. g. Jetty:
jetty/jetty.project#918
or Tomcat:
https://serverfault.com/questions/328533/can-tomcat-reload-its-ssl-certificate-without-being-restarted

[ratcashdev]

Never mind, I think I got it. This helped a lot: https://www.youtube.com/watch?v=gByztPUkqyY

[smillidge]

@ratcashdev if you put a branch out on GitHub somewhere I can help if you have issues

[ratcashdev]

That's very welcome. I hope it's ok if I fork my branch from 4.1.2.174. I have just tried the master, but asadmin throws exceptions there. Trying to go through the path of least resistance. I have just managed to build/add the example service and run it. Should not take long to make it import keys.

[ratcashdev]

@smillidge Question: Is there a way to call KeystoreManager.KeystoreExecutor inside server-mgmt from the 'service-exemplar.jar' (the payara AdminCommand example deployment fragment)? I could only make it work by copying server-mgmt.jar into glassfish/modules but it seems wrong because normally it's inside lib/asadmin/

[ghost]

Phew, a month already since the last post. I finally managed to put the code for the filter and the post-processor on github: https://github.com/a-genius/schnerpfel
Feel free to give it a try.
@ratcashdev how are things going with the asadmin command?

[ratcashdev]

Part of it is done, but I don't really like how it is done and I am waiting for @smillidge to answer my former question.

[ratcashdev]

I am back working on this. Prliminary results here: https://github.com/ratcashdev/Payara/commit/94af56eb341558e48042234d7783fa522414363e
At this point I need some answers from @smillidge or the team to the following questions:

1. Am I free to extend the server-mgmt.jar or need to do the entire work in a separate module? If the latter, I would need to duplicate a good chunk of code around KeyToolExecutor (currently protected) and seems of little value to me. Currently, the only code in a separate module is the new asadmin command add-pkcs8 But even this seems to be overkill. I would prefer doing everything inside server-mgmt.jar
2. Do I have to use the java keytool at all? Other parts of the code is using it, but since I can write to the keystore directly from within JAVA, I was wondering, what's the value of going through keytool? Ability to replicate it through Cluster instances, perhaps?
3. I presume I don't have to deal with the actual orchestration of [load keypar, reconfigure listener, restart listener] as part of a higher-order asadmin command because that can be achieved using an external script. Correct?
4. Currently supporting only unencrypted PKCS#8 private keys, but that's exactly what Letsencrypt delivers. I hope this is fine.

[ghost]

@ratcashdev I don't really understand why you need to use keytool or the Java equivalent code. After the LetsEncrypt post-process run (see https://github.com/a-genius/schnerpfel/tree/master/gfTools) the new certificate will be in the Payara/Glassfish keystore already. All the new command should do is to reload the cert and to reconfigure the listener.

Unless you want to reimplement everything...

[ratcashdev]

quoting @smillidge from a couple of posts above:

I would like to add the ability to load any certificate into the keystore as an asadmin command. In that way the certs could be loaded into the keystore, the configuration refreshed and the keystore replicated across all running instances in the domain.

That's exactly what I did: ability to load PEM and .KEY files into the keystore as an asadmin command. The configuration and listener restart can already be done with the existing asadmin commands. The seamless SSL configuration reload will be done in a separate PR, if at all possible.

[smillidge]

yeah we actually have a task for this sprint to simplify SSL handling so the as admin command will be create as I envisage an administration console section for keystore management.

On the specific question of using keytool directly from the source. There's no need to do that in fact I was thinking of refactoring all that to use the apis. I suspect that is really old code from the days when there was no java api to modify the keystore.

[smillidge]

On the question of modifying server-mgmt. go for it that is the best place to put the code.

[ghost]

"I would like to add the ability to load any certificate into the keystore as an asadmin command. In that way the certs could be loaded into the keystore, the configuration refreshed and the keystore replicated across all running instances in the domain."

These are two different tasks and there should be two asadmin-commands for them in my opinion:

1. Load any certificate into the keystore
2. Refresh the http listener to use the new certificate

"The configuration and listener restart can already be done with the existing asadmin commands."

Which command is that?

[ratcashdev]

@smillidge hi, should the new API that is not using keytool allow empty keystore or private key passwords (specifically char[] {}) or actively refuse it? It seems keytool does not allow such passwords, while the JAVA API does, with the exception of null.

[smillidge]

I would refuse it. Also in Payara the private key password must be the same as the keystore password.

[ratcashdev]

@a-genius

Which command is that?

1. asadmin set configs.config.server-config.network-config.protocols.protocol.http-listener-2.ssl.cert-nickname=newLeAlias
2. asadmin restart-domain <yourdomain>

The following commands do not (YET, unfortunately) reload the keystore:

2. asadmin set server.network-config.network-listeners.network-listener.http-listener-2.enabled=false
3. asadmin set server.network-config.network-listeners.network-listener.http-listener-2.enabled=true

[ratcashdev]

@smillidge submitted a PR for the keypair import: #2599 Please have someone have a look on it. Thanks.

[ratcashdev]

another PR: #2635 . With this change the above commands (setting listener to disabled and then enabled) would reload the keystores and activate the new key/alias/ssl setting without restarting the whole app-server.

[ratcashdev]

#2635 got merged. @a-genius also did quite a bit of work.
Basically what we have currently is:

1. leChallengeFilter by @a-genius (not yet integrated)
2. ability to import keys via asadmin (merged)
3. ability to reload keys on listener restart (merged)

@smillidge what would be the next step for Let's Encrypt integration? How would you suggest to integrate the filter?

[smillidge]

Is the filter a servlet filter? One option is to add it to the admin console web application however that would mean exposing that to the internet for this to work automatically. How do you envisage this working with Let's Encrypt, would a user expose the admin console to the Internet?

[ratcashdev]

@a-genius only published the complied jars, not the sources, so can't tell, if its a servlet filter or not.
@smillidge how about a listener/server setting, that would (temporarily?) add such a filter to the ROOT? i.e. deploy a ROOT.war or something like a tomcat VALVE, maybe? In those cases the exposure to the internet would already have been solved.

[ratcashdev]

@smillidge I have re-read the ACME certbot documentation and have an idea.

1. Bundle a minimalistic WAR inside ASADMIN (or generate it on the fly). It may contain a single folder + file: .well-known/acme-challenge but that's not necessary at all. It can be completely empty. Call it le.war
2. deploy this war using: asadmin deploy --contextroot "/" le.war
3. use the certbot to retrieve the keypair with the webroot plugin like this:
   sudo certbot certonly --webroot -w [domains]/domain1/applications/le -d example.com -d www.example.com
4. any change to that folder or file(s) will immediately be visible and the http challenge can be completed.
5. Once the keypair is downloaded, use asadmin add-pkcs8 and afterwards configure and disable/enable the relevant listener (thanks to Bugfix: listener not using the correct keystore #2635 )
6. undeploy le.war, or leave it. No harm in having it there.

This is nicely scriptable and repeatable every 90 days using CRON and the only missing element is the actual (empty) WAR, and/or the script calling asadmin and the certbot. Obviously, it assumes the user already has an account with LetsEncrypt, has his keypair available to be used by certbot, etc. But Certbot helps with all that.
Optional: ASADMIN could play the role of the script and ASADMIN would call certbot. But I like the external script more - more transparent, easier to tweak.

Another possibility would be to use ACME4J inside asadmin and not rely on the certbot at all. A bit more programming, but nothing terribly serious. This would add ~400Kb of extra dependency to ASADMIN (acme4j - 100kb + jose4j - 252kB and slf4j - 40Kb if not already available) and the related risks (outdated acme4j, need to update the distribution just to make LE integration working, etc.) I really like the former approach much much more.

Edit: Certbot is only supported on Linux. If we wanted to be cross-platform, we would need to go the acme4j route.

[ghost]

"a-genius only published the complied jars, not the sources, so can't tell, if its a servlet filter or not."

The leChallengeFilter is a servlet filter that should be added to the default-web.xml. This requires a web application at the root context of every virtual host that shall be verified by Letsencrypt. All documentation and source code is on my github repository. I mentioned it in the accompanying readme.md, but it may not be obvious enough: "The zip-files contain source, binaries and property-files."

It may not work for every possible setup, as a few people mentioned (see further up this thread). I got my cert updated two times since I installed it and it worked fine; I just have to restart the server manually. It would be much better just to trigger a reload of the cert without restarting the connector.

[ratcashdev]

Simplifying further.
With in-place deployments a .war is not necessary, just a directory (named e.g. le_war) containing a single other directory WEB-INF. This can be deployed with asadmin as follows:
asadmin deploy --name le --contextroot "/" /home/glassfish/le_war

Certbot would then be used like this:
sudo certbot certonly --webroot -w /home/glassfish/le_war -d example.com -d www.example.com

Afterwards you can undeploy the war:
asadmin undeploy le

and upload the keys, etc.

I think it won't get simpler than that. No coding necessary to achieve full integration and automation.

[ratcashdev]

Here's a PoC script using python (as it is cross-platform and certbot anyway requires python): https://gist.github.com/ratcashdev/1b09877d37e02ef5170bf9e60c377f34
The idea is to put this script in the same directory as asadmin

Usage:
le.py -h

Example:
sudo le.py -a myalias -l http-listener-2 -d domain1.com -d www.domain1.com

First run must be manual (account creation, etc). The subsequent calls could be done without user interaction.

[ratcashdev]

@smillidge have a look on this please, if it's worthy for a PR.

[smillidge]

I'm not too familiar with Python so can't comment on the script. Feel free to make the PR and I can get it reviewed

[ratcashdev]

submitted as: #2727

[ratcashdev]

@lprimak with the last PR merged, I think this issue/feature request can also be merged. As long as @jerrinot and the others agree.

[lprimak]

This issue is already closed :). I hope that that’s what you mean

[ratcashdev]

lol

[thirion]

For anyone finding this issue please note that the current script uses different arguments than listed above. (for instance -d is domain dir and not the domain you want to create a certificate for)

It also has a few issues that need to be tweaked before it will work correctly:
First, change /tmp/payara_le_war to /tmp/payara_le.war, else it won't deploy in Payara.
You also need to add args.listener when calling configure_listener_alias or that part will fail.

Once you've manually ran it once to create and insert your certificate you should add "-n" to the certbot_call_args, else it will ask you what to do if your cert isn't due to be renewed yet. I'm assuming you eventually want to run it from cron with unattended mode enabled.
Something like:
0 0 */3 * * cd /your_payara_install/bin && ./letsencrypt.py -c your_domain.com -n domain1

Here is my file that seems to work well:
letsencrypt.zip

[ratcashdev]

@thirion I am happy that you found this thread and the script. At least someone found it useful (even though it seems it's not really working trouble free).

Back when I was writing this (especially the in-place deployoment: #1047 (comment)) it worked for me to deploy the dir named as it was in the script. If it does not work now, maybe something changed on Payara's side. The LetsEncrypt must have changed a lot too.
To be honest, i moved to other areas so can't really comment if it works today or not. But once again, thanks for posting the update. Maybe you'll want to submit it as a PR, too.

[smillidge]

Please raise a new issue if there are problems with the script as we are shipping and therefore maintaining it.