Docker Registry and Certbot for Let's Encrypt

[anton-kasperovich]

Thanks to #91 and #18 PR's, it's helped a lot to make this PR possible.

Added:

• Official Docker Registry v2.5.1 image.
• Certbot script which would used to generate free SSL certificates via Let's Encrypt service, used by default to make Docker Registry secure.

Usage example:

export TARGET_HOST=52.123.254.123
./adop certbot gen-letsencrypt-certs registry.${TARGET_HOST}.nip.io registry
./adop compose -i ${TARGET_HOST} init
./adop certbot export-letsencrypt-certs registry.${TARGET_HOST}.nip.io registry

Notes:

• Certbot script could be also used to make secure whole ADOP stack, but NGINX configuration changes would be required for it, not sure, do we want to add documentation for it in this PR or (imo) it's better to add it separately.
• Creating this PR now to collect a feedback, how we could make it as part of QuickStart script by default.
• Fixed EOF for Docker Registry adop-nginx#19 - consider this PR as dependency for current one.
• https://github.com/Accenture/letsencrypt - this repository should be replaced on my (or as Pull Request) https://github.com/anton-kasperovich/certbot

[ismarslomic]

Have you considered Docker Trusted Registry, which includes security, monitoring and UI in addition to private Docker Registry?

[anton-kasperovich]

@ismarslomic not really, at least for now, as it's the paid solution "Starting at $150 monthly"

[ismarslomic]

If you install it on your own you dont have any costs as I understood it.

Edit: Sorry, you are right. DTR requires UCP, which is a payable service.

[nickdgriffin]

@ismarslomic We only use free and open source software in the community edition of ADOP, so we can't use DTR. It also requires UCP it seems, amongst other things: https://docs.docker.com/docker-trusted-registry/install/license/

[ismarslomic]

Sorry, my bad. Licensing for both, UCP and DTR was hidden very nicely on Docker docs.

[anton-kasperovich]

After all discussions and my quick fixes, rebase required. Will do it soon.

[anton-kasperovich]

Rebase - done. @nickdgriffin @SachinKSingh28 sorry for a delay, please review and merge if it looks fine, thanks!

[nickdgriffin]

Test cases:

• docker login registry.IP.nip.io (with admin credentials from provisioning)
• Tag an ADOP image: docker tag accenture/adop-nginx:0.3.0 registry.IP.nip.io/adop-nginx:0.3.0
• Push the tagged image: docker push registry.IP.nip.io/adop-nginx:0.3.0
• Remove the tag: docker rmi registry.IP.nip.io/adop-nginx:0.3.0
• Pull the tagged image: docker pull registry.IP.nip.io/adop-nginx:0.3.0

Tested with:

• From Centos 7 using quickstart on AWS

To be tested:

• From Windows 7 in Git Bash using quickstart on AWS
• From Windows 10 in Git Bash using quickstart on AWS

[anton-kasperovich]

Thanks @nickdgriffin , he suggested to make a prompt, in case if we have already up&running ADOP with NGINX container it's allocating 80 and 443 ports and because of this, we can't issuing new certificates for other services as Let's Encrypt requires these ports to be open. Now, we can use the same "./adop gen-certs" command multiple times (for example if i want to generate new certificate for new platform extension etc.), as i added a check if we have running Proxy (NGINX) container with user prompt confirmation, if he agree to stop this container and issuing new certificates or not.

46a0051#diff-006650ed73143f2ce2864bfcf2e172d2R77

@nickdgriffin could you please test it on Windows again, i can cover only Linux & Mac testing. Thank you in advance!

[nickdgriffin]

@anton-kasperovich , has this been run for "./quickstart.sh -t local"? As I'm not sure certbot will work at the moment like that:

INFO: SSL certificates will be issued for registry.192.168.99.100.nip.io.
Unable to find image 'accenture/certbot:0.0.1' locally
0.0.1: Pulling from accenture/certbot
117f30b7ae3d: Pulling fs layer
20b1656c04c6: Pulling fs layer
67b558020018: Pulling fs layer
117f30b7ae3d: Verifying Checksum
117f30b7ae3d: Download complete
117f30b7ae3d: Pull complete
20b1656c04c6: Verifying Checksum
20b1656c04c6: Download complete
67b558020018: Verifying Checksum
67b558020018: Download complete
20b1656c04c6: Pull complete
67b558020018: Pull complete
Digest: sha256:adecb80b3afeeec12b1621b2703c88bcf4d1a3c84a8b694f7cce67a33872ba98
Status: Downloaded newer image for accenture/certbot:0.0.1
2016-10-08 09:11:29,130:WARNING:certbot.client:Registering without email!
/usr/lib/python2.7/site-packages/OpenSSL/rand.py:58: UserWarning: implicit cast from 'char *' to a different pointer type: will                                                      be forbidden in the future (check that the types are as you expect; use an explicit ffi.cast() if they are correct)
  result_code = _lib.RAND_bytes(result_buffer, num_bytes)
Failed authorization procedure. registry.192.168.99.100.nip.io (tls-sni-01): urn:acme:error:unknownHost :: The server could not                                                      resolve a domain name :: No valid IP addresses found for registry.192.168.99.100.nip.io
IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: registry.192.168.99.100.nip.io
   Type:   unknownHost
   Detail: No valid IP addresses found for
   registry.192.168.99.100.nip.io

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address.
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.

I'll try "-t aws" out too when I have time.

[nickdgriffin]

I've tested from Windows 7 with "-t aws" and it works fine, still need to complete the Windows 10 test. I think it'd be good to get an additional sanity test on Windows too (@dsingh07 /@SachinKSingh28 please?).

I think the quickstart script needs to have some logic added that will only allow the registry commands to be run for "-t aws" as we know that will have an accessible public IP for Lets Encrypt to reach to verify the target of the certificate, and we need a caveat mentioned somewhere that the Registry will only function with this type. I don't think it's worth holding up this PR to get something usable out there.

[nickdgriffin]

Another failure case that I think needs handling - I ran quickstart and it failed when it got to "adop compose init" because I had an old secrets file (which is legit), so after fixing that I ran the same command again and got this:

You have an existing certificate that contains exactly the same domains you requested and isn't close to expiry.
(ref: /etc/letsencrypt/renewal/registry.IP.nip.io.conf)

What would you like to do?

I couldn't actually enter anything for the prompt, and I just wanted it to leave the certificates in place and carry on without regenerating them ideally.

@anton-kasperovich , is there a way that can be achieved for the case where the CLI fails for whatever reason and someone just reruns quickstart?

[nickdgriffin]

Works from Windows 10 too with "-t" AWS.

I think the only things that need handling are:

• Only calling certbot in quickstart for "-t aws"
• Handling the case mentioned above where valid certificates already exist

And one final sanity test from someone else and then this is good to go :)

[dsingh07]

Tested successfully on Windows 7 through Git Bash

Test scenario for registry:

• Spin up ADOP using quickstart with flag "-t aws"
• docker-machine ssh <MACHINE_NAME>
• docker login registry.IP.nip.io (with admin credentials from provisioning)
• Tag an ADOP image: docker tag redis:3.0.7 registry.IP.nip.io/redis:3.0.7
• Push the tagged image: registry.IP.nip.io/redis:3.0.7
• Remove the tag: docker rmi registry.IP.nip.io/redis:3.0.7
• Pull the tagged image: docker pull registry.IP.nip.io/redis:3.0.7

Further tested certbot by trying to regenerate unexpired certs which didn't work, as has been pointed out above by @nickdgriffin

export TARGET_HOST=<IP>
./adop certbot gen-certs registry.${TARGET_HOST}.nip.io registry

Got the following output:

You have an existing certificate that contains exactly the same domains you requested and isn't close to expiry.
(ref: /etc/letsencrypt/renewal/registry.<IP>.nip.io.conf)

What would you like to do?

We should handle this case, as we don't know what the valid answers to this question are.

[anton-kasperovich]

Oh, it's took me some time to find more proper way (at least i think so...). So the issue which was raised by @nickdgriffin and @dsingh07 , it's because we can not support Docker TTY in Git Bash (Windows) without massive work around like "winpty" or "babun" which obviously is not the option for us.

So what I've done:

• Just added "non-interactive" flag in Let's Encrypt and fortunately for me, they've actually support everything in their own CLI, there's check like if there's no certificate - generate, if exists but doesn't require renewal process - just skip, if exists and require renewal - then executing renewal process automatically.
• Added checks on: ${MACHINE_TYPE} == "aws", that we officially can support SSL only in case if docker-machine was created on Amazon and have public ip address (public ip - behavior by default), otherwise we will just run ADOP without Let's Encrypt as it was before.

Example after i update on non-interactive usage with use case which was mention above (tested on Win10 and Git Bash which was installed from Docker Toolbox):

INFO: SSL certificates will be issued for registry.******.nip.io.
We found running Proxy (NGINX) container. To issue new SSL certificate we have to stop & start Proxy container, which will be cause of ADOP short outage. Please, confirm restart of Proxy (NGINX) container? [y/n] y
Stopping Proxy (NGINX) container...proxy
-------------------------------------------------------------------------------
Certificate not yet due for renewal; no action taken.
-------------------------------------------------------------------------------
Starting Proxy (NGINX) container...proxy

Please give it a try one more time by someone and hopefully we can merge it now, sorry for not getting into it earlier...

[nickdgriffin]

Carried out the tests in #123 (comment) again for Centos 7 and Windows 7, and it all looks good. Also performed the test after rebasing on master just to be safe and it was fine from Centos 7.

I'll try Windows 10 again when I can.

I also happened to look at the rate limits (because I couldn't do the rebase test from Windows), https://letsencrypt.org/docs/rate-limits/, which implies that only 20 instances a week with a registry will be possible as the domain counts as "nip.io" I think. You can see what's already been registered at https://crt.sh/?q=%25.nip.io. Right now I can't create another one :(

[anton-kasperovich]

Unfortunately, because of rate limits on Let's Encrypt, we can't keep it as solution by default, 'cause it might fail ADOP initialization at all. I'm rewriting this PR with self-signed certificates by default, but at the same time i will keep Certbot solution as additional, that people would still able to generate real SSL certificates issued by Let's Encrypt, if they would like to.

[anton-kasperovich]

@nickdgriffin @dsingh07 i've updated the Certbot bash script, please take a look on the "gen-export-certs" new function, which was developed to generate and export (in needed places) self-signed certificates

[SachinKSingh28]

@anton-kasperovich - Stack was created successfully with registry when I launched quickstart from linux dev box but haven't tried git bash yet. I was getting error message when trying to login to the registry.

$ docker login registry.IP.nip.io
Username:
Password:
Error response from daemon: Get https://registry.IP.nip.io/v1/users/: x509: certificate signed by unknown authority
Will try to add the certificate to trusted store and see if that solves the problem.

[anton-kasperovich]

05.02.17 Updated PR.

Changed strategy of copying certificate on copying it to the /etc/docker/certs.d/{domain_name} folder, as per instruction below:

Instruct every docker daemon to trust that certificate.
This is done by copying the domain.crt file to /etc/docker/certs.d/myregistrydomain.com:5000/ca.crt.

So, at the moment, we should have working Docker Registry whatever a way of deployment we use, i mean quickstart or just ./adop compose init... both should work, as i've added docker-machine cp commands in order to support quickstart as well.

Please, pay attention, that this solution will work only internally i.e. inside the stack. If someone wants to use Docker Registry remotely (like i want to pull image from stack on my own laptop), i have to copy and trust certificate. My next step is to provide documentation, on how to use Self-Signed (evaluation mode) and Let's Encrypt (close to "production" way of usage).

[nickdgriffin]

I think this should be "./adop certbot" here?

[nickdgriffin]

Given this another round of testing on Windows 7 with the latest Docker Toolbox and it worked as expected, and I also rebased on master and tested and it also still worked (and rebased cleanly). I focused on the new self-signed certificate stuff from inside Jenkins but didn't attempt to trust it locally.

I'd like to give it a go on Windows 10 and Ubuntu too just to cover the bases I think.

[nickdgriffin]

Tested on Windows 10 with the latest Docker Toolbox and it worked as expected with the self-signed certificates, also not attempting to trust it locally.

Gave it a shot from Windows 10 with virtualbox too and it behaved as expected - the registry ended up in a restart loop due to lack of certs but that didn't stop me accessing the rest of the tools. I commented out the "if aws" section which resulted in a failure but I don't think that'd be too tricky to resolve in a future PR:
Copying self-signed certificate to the trusted location /etc/docker/certs.d/...
mkdir: cannot create directory ‘/etc/docker’: Permission denied

I think this is probably good to go once the typo in the docs mentioned above has been sorted, but could you also rebase on master please?

[anton-kasperovich]

@nickdgriffin Finally found a time, I've made rebase and fixed the typo

[nickdgriffin]

LGTM :)