HTTP Proxy support is broken after upgrade to Version 17.12.0-ce-mac46 (21698)

[alvarow]

Expected behavior

Docker pull from internal or external registries work through HTTP proxy that does SSL Interception.

Actual behavior

If the HTTP and HTTPS Proxy field are set on Docker Preferences, then the internal registry fails with ERROR: Get https://dtr.cdl.es.ad.xxx.com/v2/: net/http: TLS handshake timeout message, but works for external registries (i.e. AWS ECR or Docker Hub).

If the HTTP Proxy preference is removed (but HTTPS is left) then the internal registry works (its SSL certificate is signed by the same internal CA that signs the SSL Intercepted requests through the proxy), but external registries fail to load with Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Information

• Full output of the diagnostics from "Diagnose & Feedback" in the menu

Diagnostic ID 94B69674-4A18-4FC4-A9E4-2644691FA387

Docker for Mac: version: 17.12.0-ce-mac46 (a61e84b8bca06b1ae6ce058cdd7beab1520ad622)
macOS: version 10.12.6 (build: 16G1036)
logs: /tmp/94B69674-4A18-4FC4-A9E4-2644691FA387/20180109-215355.tar.gz
[OK]     db.git
[OK]     vmnetd
[OK]     dns
[OK]     driver.amd64-linux
[OK]     virtualization VT-X
[OK]     app
[OK]     moby
[OK]     system
[OK]     moby-syslog
[OK]     kubernetes
[OK]     env
[OK]     virtualization kern.hv_support
[OK]     slirp
[OK]     osxfs
[OK]     moby-console
[OK]     logs
[OK]     docker-cli
[OK]     menubar
[OK]     disk

Steps to reproduce the behavior

1. From a network that requires authenticated proxy usage with SSL Interception (i.e. BlueCoat), try to perform docker login or docker pull.

[djs55]

Thanks for your report.

I've not been able to reproduce this myself yet, but I've found a bug which I suspect could be the cause of this-- in the proxy configuration code we conflated the settings for the HTTP and HTTPS proxies, leading to unexpected results in some environments.

I've made a prototype fix. If you would like to try it I've made an experimental build: https://download-stage.docker.com/mac/pr/21758/Docker.dmg -- note this build is suitable for testing only. The build is version

Version 17.12.1-ce-rc1-mac49 (21758)
Channel: pr
f007f92cf0

If it offers an auto-update, don't accept it since it will probably update to another experimental build without the fix.

If this doesn't work, could you let me know a little bit more about your proxies? Do you have different URLs for HTTP and HTTPS?

[alvarow]

Thanks, I will give it a try in a few minutes.

I have the same URL for HTTP and HTTPS, I don't have a whole lot of details, I simply know it is a Bluecoat cluster that requires authentication and performs SSL Interception (DLP checks etc).

This has been working for well over a year, just now, whatever the changes are seems to have affected it.

Thanks for the quick reply.

[alvarow]

No joy :-(

regulya@ROSELCDV0001LHJ:~$ docker login dtr.cdl.es.ad.xxx.com
Username: adpe
Password:
Error response from daemon: Get https://dtr.cdl.es.ad.xxx.com/v2/: net/http: TLS handshake timeout
regulya@ROSELCDV0001LHJ:~$ docker pull redis:alpine
alpine: Pulling from library/redis
605ce1bd3f31: Pull complete
d12d456b03b7: Pull complete
b6e6ee69c5e2: Pull complete
813ea0248ee4: Pull complete
62208b7f07e1: Pull complete
35b7abefa067: Pull complete
Digest: sha256:a5fd9334b28d6afdc8b84aad8a01056d8d1d849cb5a744919caf40d5b57597b2
Status: Downloaded newer image for redis:alpine

The proxy works, I can pull things from the external registry, just not the internal one, seems it won't trust the internal CA even with its certificate being present on Keychain.

docker info however gives me conflicting information about the proxies:

---snip---
HTTP Proxy: docker.for.mac.http.internal:3128
HTTPS Proxy: docker.for.mac.http.internal:3129
---snip---

Which most certainly is completely different that the proxy configured on Preferences.

[alvarow]

DiagnosticID is 94B69674-4A18-4FC4-A9E4-2644691FA387

[djs55]

Thanks for giving it a go. I'll investigate more tomorrow.

The docker.for.mac.http.internal name resolves to an internal TCP proxy which connects to the proxy configured in Preferences. Or at least that is the theory :-)

[rubyisbeautiful]

This also affected me and some colleagues. We use the same proxy URL for HTTP and HTTPS. We use an external proxy, i.e. not one on the local Docker host...it seemed to only affect HTTPS URLs.

What was weird is that it affected not only Docker daemon actions, like docker pull but actions in containers, that have their own proxy configurations and normally wouldn't rely on the docker daemon (such as https_proxy=http://whatev curl -v https://github.com (github being down this morning notwithstanding))

If I can provide any more info I will try but at this time I've rolled back to a previous version. I can re-upgrade cautiously :)

[djs55]

We've fixed a couple of HTTP proxy issues and are hoping to release an update either later today or earlier tomorrow. I'll comment here when the update is available.

Regarding in-container operations that have their own proxy configurations, Docker 17.07 and later have a new CLI feature: docker/cli#93 The file ~/.docker/config.json can contain settings as follows:

{
  "auths" : {

  },
  "orchestrator" : "kubernetes",
  "proxies": {
    "default": {
      "httpProxy": "http://172.16.1.61:3128",
      "httpsProxy": "http://172.16.1.61:3128",
      "noProxy": "",
      "ftpProxy": ""
    }
  }
}

These are automatically inserted by docker run and docker build into the container's environment.

[rubyisbeautiful]

whoa that's big. lots of automation will get shelved now (in a good way) :)
Thanks for fixing this proxy bug quickly @djs55

[djs55]

Sorry I should have emphasised -- we've fixed some proxy bugs for sure, but there might still be one or two left. In particular the original issue reported by @alvarow may not be fixed -- I'm still unable to reproduce it locally (for me using proxies to access a secure registry is working, unfortunately). The update will be worth re-testing, but we might need to release another update soon afterwards.

Thanks for all your patience in the meantime!

[alvarow]

Thanks @djs55, please let me know if there's anything I can do to help. I am happy to demonstrate as well, over a BlueJeans/WebEx/etc session.

[sharenz]

I think I have the same issue. I am behind a proxy with SSL interception. Connecting to external registries (using the proxy) works. But I can not access the internal registry anymore. I think there is a problem with bypassing the proxy for internal domains.

I just downgraded to 17.09.0-ce-mac33 (19543), now it works again.

[danieladams456]

I tried the new 17.12.0-ce-mac47 version and still having several problems. Authentication information is now not passed when set in the proxy settings as http://user:[email protected]:80.

Also proxy exclusion isn't working in the .domainname.net format (preferred) as well as hostname.domainname.net.

I'm rolling back to 17.09 for now to fix.

(no auth headers passed)

[djs55]

@danieladams456 thanks very much for the report and screenshot. I can reproduce these so I'll work on fixing them and adding some regression tests.

[neurostream]

When behind a corporate MITM transparent SSL proxy, and experiencing most of this issue, I found that running a local transparent SSL intercept Proxy for MacOS (which I'll refer to by its name, Charles) "fixed" it - so I just wanted to drop a line here in case the following observations help anyone else.

From the time Docker.app launches to when the status bar Whale menu is clicked, it appears to connect to:

api.wootric.com
api.segment.io
download.docker.com (run "Check for Updates..." from menu)
cloud.docker.com (login to Docker and switch accounts from menu)
registry-1.docker.io
auth.docker.io

Exempting these domains from Charles intercept one-at-a-time (starting and stopping Docker.app to it whittle down), the connections that weren't be intercepted by Charles before leaving MacOS would connect normally - with the exception of cloud.docker.com and download.docker.com. Connections to those two domains would work via Charles, but log several of these events to the MacOS system Console before finally negotiating and connecting:

Jan 12 19:31:16 macbook1 Docker[10744]: CFNetwork SSLHandshake failed (-9824) Jan 12 19:31:16 macbook1 4]: NSURLSession/NSURLConnection HTTP load failed (kCFStreamErrorDomainSSL, -9824) Jan 12 19:31:16 macbook1 Docker[10744]: HttpClient: Failed to send request. An SSL error has occurred and a secure connection to the server cannot be made.

( Also noticed that cloud.docker.com and download.docker.com were both from different *.docker.com wildcard certs with slightly different subject alternative names, including an extra docker.io SAN on the cloud.docker.com cert... not a thing, I guess. Just interesting.)

[djs55]

I've got an experimental build which addresses several issues in particular

• proxy authentication should work
• it was possible for DNS lookups of the internal proxy name to hang, leading to timeouts
• proxy excludes of the form .foo.com should work as well as *.foo.com and 10.0.0.0/16 and 10.0.0.1 (comma-separated)

I've tested the fixes end-to-end with various combinations of insecure registries, hub, DTR (with cert accepted via safari) with and without a proxy and with and without proxy authentication. Note that when I tested authenticated proxies I used the Whale menu -> Preferences -> Proxies -> Manual proxy configuration and typed in Urls of the form "http://username:password@host:port/". Unfortunately we can't currently use the username and password typed into the System Preferences proxy settings.

If you'd like to try it the experimental build is here: https://download-stage.docker.com/mac/bysha1/2dde662345343861125e80c570b12c1a8c211dc8/Docker.dmg

It should report version

Version 17.12.0-ce-mac49 (21932)
Channel: pr
2dde662345

It's a developer build from a PR based on the stable branch-- if it offers an auto-update, don't take it or you'll end up on another branch without the fixes.

If it still doesn't work, please reproduce the problem and then upload a fresh diagnostic from the Whale menu -> Diagnose and Feedback ... -> Diagnose & Upload. I also fixed a bug where some of the networking logs are missing, so I'm keen to see any logs after new failures.

Thanks for all your help so far and sorry for the inconvenience.

[danieladams456]

Works! 🎉

[alvarow]

Works here as well! 🎉 Tested with two associates plus mine.

Just in case you were looking for something in particular, I have uploaded the Diagnostic logs as well. (unique id is: 94B69674-4A18-4FC4-A9E4-2644691FA387)

Thanks for the help!

Now... when can we expect a stable build with this fix? :-)

[sharenz]

The experimental build works, thanks.

[pgayvallet]

@djs55 can i close this ?

[alvarow]

Any official build with this fix?

[rubyisbeautiful]

I haven't been upgrading because essentially, this bricked Docker at "work" for all development teams. I'd like to be confident it's in a regular upgrade channel if possible. Thanks for all your hard work on Docker all.

[rubyisbeautiful]

I can confirm that this is now fixed in latest edge, Version 18.02.0-ce-mac53 (22617) c3ae48f0d5.
I prefer to stay on the stable channel, but I'm not really sure how to track if/when this fix makes it there.

[alvarow]

I'll wait for confirmation of the official build before I tell the gang to move on. We're holding on 17.09

[lenguyenthedat]

just tried but 18.02 does not fix it for us 😞. Holding on 17.09 for now.

dat$ docker --version
Docker version 18.02.0-ce-rc2, build f968a2c

dat$ [---]
Connecting to docker-registry (port [---])...
retrieving latest docker image...
Error response from daemon: Get [---]: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

[djs55]

The fix from the experimental build should be in the stable channel from 17.12.0-ce-mac49 onwards. The current stable version is 17.12.0-ce-mac55 released yesterday (2018-02-27). In the releasenotes it's

Fix using an HTTP proxy with authentication. Fixes #2386

I'll tentatively close this ticket since the fix has been released. @alvarow let me know if it works / doesn't work for you.

@lenguyenthedat I think you must have a different (although related) problem. Could you reproduce with the latest edge build and upload a diagnostics into a fresh ticket? Also it would be really useful if I could reproduce it too locally -- so any reproduction steps you could think of would be great. Please link the new ticket to this one and ping me on it so I see it. Thanks for your help!

[alvarow]

This is confirmed working on release mac55!

Thanks so much @djs55

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked