Unable to get capabilities working in Docker swarm under Windows WSL2

[trajano]

• I have tried with the latest version of my channel (Stable or Edge)
• I have uploaded Diagnostics
• Diagnostics ID:

I am trying to test the capabilities feature specified in #2687

docker stack deploy -c havege.yml havege

> docker service inspect havege_haveged --pretty

ID:             svh3aovntbvm8hmf3bv1l31ls
Name:           havege_haveged
Labels:
 com.docker.stack.image=hortonworks/haveged:1.1.0
 com.docker.stack.namespace=havege
Service Mode:   Replicated
 Replicas:      1
Placement:
UpdateConfig:
 Parallelism:   1
 On failure:    pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Update order:      stop-first
RollbackConfig:
 Parallelism:   1
 On failure:    pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Rollback order:    stop-first
ContainerSpec:
 Image:         hortonworks/haveged:1.1.0
Resources:
Networks: havege_default
Endpoint Mode:  vip


>docker service inspect havege_haveged | grep -i admin

Expected behavior

I expected to see NET_ADMIN or some sort of capability

Actual behavior

No capabilities shown.

Information

• Windows Version:
• Docker Desktop Version:
• Are you running inside a virtualized Windows e.g. on a cloud server or on a mac VM:

Steps to reproduce the behavior

1. My havege.yml file

version: "3.8"
services:
  haveged:
    image: hortonworks/haveged:1.1.0
    cap_add:
      - NET_ADMIN
    # docker run --privileged -d --restart always hortonworks/haveged:1.1.0

2. ...

Client: Docker Engine - Community
 Cloud integration: 1.0.4
 Version:           20.10.0
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        7287ab3
 Built:             Tue Dec  8 18:55:31 2020
 OS/Arch:           windows/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.0
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       eeddea2
  Built:            Tue Dec  8 18:58:04 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Originally in docker/for-win#9853

[thaJeztah]

I'm not able to reproduce this on a Linux machine;

docker service inspect havege_haveged --pretty

ID:		uldc824xefrllsgfzwoy3wfi3
Name:		havege_haveged
Labels:
 com.docker.stack.image=nginx:alpine
 com.docker.stack.namespace=havege
Service Mode:	Replicated
 Replicas:	1
Placement:
UpdateConfig:
 Parallelism:	1
 On failure:	pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Update order:      stop-first
RollbackConfig:
 Parallelism:	1
 On failure:	pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Rollback order:    stop-first
ContainerSpec:
 Image:		nginx:alpine@sha256:c2ce58e024275728b00a554ac25628af25c54782865b3487b11c21cafb7fabda
Capabilities:
 Add: CAP_NET_ADMIN
Resources:
Networks: havege_default
Endpoint Mode:	vip

Wondering if it's because the CLI is running on Windows; are you seeing the same problem if you use a Linux CLI on WSL2 ?

[thaJeztah]

Ah! I see the same issue when using Docker Desktop on Mac;

docker service inspect havege_haveged | grep -i admin

But if I deploy the stack using the "raw" socket (bypassing the API proxy that's used in Docker Desktop), it works;

reproduction steps

Start a container with the docker 20.10.1 CLI in it, and with the raw socket bind-mounted as /var/run/docker.sock

docker run -it --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /var/run/docker.sock.raw:/var/run/docker.sock.raw \
  docker:20.10.1 sh

Inside the container, deploy the stack using the "regular" /var/run/docker.sock that was bind-mounted.
This socket is proxied on Docker Desktop (which is needed to make some adjustments, e.g. to remap host paths);

docker -H unix:///var/run/docker.sock stack deploy -c- through_proxy <<'EOF'
version: "3.9"
services:
  haveged:
    image: nginx:alpine
    cap_add:
      - NET_ADMIN
EOF

Inspecting the service created through the API proxy, the capability is missing

$ docker service inspect --format=pretty through_proxy_haveged

ID:		yvkglw2pwh1gr7wujd2e2vf0y
Name:		through_proxy_haveged
Labels:
 com.docker.stack.image=nginx:alpine
 com.docker.stack.namespace=through_proxy
Service Mode:	Replicated
 Replicas:	1
Placement:
UpdateConfig:
 Parallelism:	1
 On failure:	pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Update order:      stop-first
RollbackConfig:
 Parallelism:	1
 On failure:	pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Rollback order:    stop-first
ContainerSpec:
 Image:		nginx:alpine@sha256:c2ce58e024275728b00a554ac25628af25c54782865b3487b11c21cafb7fabda
Resources:
Networks: through_proxy_default
Endpoint Mode:	vip

Deploying the stack, but this time using the "raw" socket;

docker -H unix:///var/run/docker.sock.raw stack deploy -c- without_proxy <<'EOF'
version: "3.9"
services:
  haveged:
    image: nginx:alpine
    cap_add:
      - NET_ADMIN
EOF

Inspecting the service shows that this time the capabilities are set:

$ docker service inspect --format=pretty without_proxy_haveged

ID:		yg8ve22zmrsheu257uuvjigm3
Name:		without_proxy_haveged
Labels:
 com.docker.stack.image=nginx:alpine
 com.docker.stack.namespace=without_proxy
Service Mode:	Replicated
 Replicas:	1
Placement:
UpdateConfig:
 Parallelism:	1
 On failure:	pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Update order:      stop-first
RollbackConfig:
 Parallelism:	1
 On failure:	pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Rollback order:    stop-first
ContainerSpec:
 Image:		nginx:alpine@sha256:c2ce58e024275728b00a554ac25628af25c54782865b3487b11c21cafb7fabda
Capabilities:
 Add: CAP_NET_ADMIN
Resources:
Networks: without_proxy_default
Endpoint Mode:	vip

So looks like this is a bug in the Docker Desktop API not passing through the new options for API v1.41

[trajano]

Maybe see if moby/moby#32957 (comment) is similar. Since it's a Windows specific issue and does not occur with a Linux client.

[trajano]

But if I deploy the stack using the "raw" socket (bypassing the API proxy that's used in Docker Desktop), it works;

Is there any documentation on what the API proxy does? Why can't we just call the socket directly?

[thaJeztah]

The proxy is used for various things, e.g., when bind-mounting files from the host, the proxy takes care of rewriting paths from the host to their equivalent locations in the Linux VM, as well as setting up monitoring for those paths. I think there's additional things (remapping IP-addresses "inside" vs "outside" of the VM etc.)

So, while the .raw socket can be used for debugging, it's not recommended to use for other reasons.

[davehope]

I'm seeing similar behaviour with Docker 20.10.1, build 831ebea (docker-ce/buster,now 5:20.10.13-0debian-buster amd64).

Capabilities are successfully applied when using docker stack deploy --compose-file ..., or post-deployment with docker service update xxx --cap-add xxx but not via something like Portainer. I assume the root cause is the same as this issue.

[thaJeztah]

Yes, that sounds like an issue with portainer; may be that they didn't update their API integration yet for the latest API changes in Docker 20.10.x

[thaJeztah]

Also short update on the above; the Docker Desktop team informed me that a fix for this was merged, but may not yet have been released due to Christmas and New Year; I'm asking them which release of Docker Desktop the fix will be in 👍

[thaJeztah]

Looks to be fixed (tested on Docker Desktop for Mac 3.2.2), using the reproduction steps from #2893 (comment)

docker stack deploy -c- through_proxy <<'EOF'
version: "3.9"
services:
  haveged:
    image: nginx:alpine
    cap_add:
      - NET_ADMIN
EOF

Check if the capabilities were set on the service:

docker service inspect --format=pretty through_proxy_haveged | grep -1 Capabilities
 Image:		nginx:alpine@sha256:e20c21e530f914fb6a95a755924b1cbf71f039372e94ac5ddcf8c3b386a44615
Capabilities:
 Add: CAP_NET_ADMIN