Mounting to /var/lib/docker results in creation of invalid and useless volumes

[orbelico]

Short description and how to reproduce

If I mount a volume or host path to /var/lib/docker inside a container, using docker-py:
>>> client.containers.run('alpine', volumes=['my_volume:/var/lib/docker']) ,
a very strange behavior occurs:

• My volume (or host path) is mounted as expected to /var/lib/docker (so far, so good)
• However, another docker volume with long cryptic name is created automatically
• This additional volume is mounted to: /var/lib/docke [sic!] - without the trailing "r"!
• This new volume is empty and stays empty, since it is not expected and not used. However, it remains after the container is stopped and removed.

While this does not actually break any functionality, it leads to a lot of useless volumes accumulating on my system. And these are not easy to filter from other, also automatically created volumes, that I actually still need, so cleaning up is very tedious.

This is apparently a problem of docker-py, not of docker itself: when doing the same using docker CLI everything works as expected:
$ docker run -v my_volume:/var/lib/docker alpine

The specific use case where I would like to mount a volume to that path is the docker:dind image from the official docker library. It requires /var/lib/docker to be a volume in its Dockerfile. I want to provide my own volume (or host path) for this, so I can manage and re-use the docker cache.

Logs / Inspects

Inspect container (shortened to relevant part) clearly shows the two mount paths:

  "Mounts": [
    {
      "Type": "volume",
      "Name": "22a311221e43824e3e12e689532b16387fe0c21624f40fe72ad1acd52a931667",
      "Source": "/var/lib/docker/volumes/22a311221e43824e3e12e689532b16387fe0c21624f40fe72ad1acd52a931667/_data",
      "Destination": "/var/lib/docke",
      "Driver": "local",
      "Mode": "",
      "RW": true,
      "Propagation": ""
    },
    {
      "Type": "bind",
      "Source": "my_volume",
      "Destination": "/var/lib/docker",
      "Mode": "",
      "RW": true,
      "Propagation": "rprivate"
    }
  ],

Inspect the invalidly created volume:

{
  "CreatedAt": "2022-03-29T23:03:26+02:00",
  "Driver": "local",
  "Labels": null,
  "Mountpoint": "/var/lib/docker/volumes/22a311221e43824e3e12e689532b16387fe0c21624f40fe72ad1acd52a931667/_data",
  "Name": "22a311221e43824e3e12e689532b16387fe0c21624f40fe72ad1acd52a931667",
  "Options": null,
  "Scope": "local",
  "CreatedTime": 1648587806000,
  "Containers": {
    "d5cffd2e149298d36971f3702b63d42eaddbc7f0080cfdcaff691c2be879120a": {
      "Name": "nostalgic_sanderson",
      "Destination": "/var/lib/docke"
    }
  }
}

Shell into affected container clearly shows that both dirs exist, but only one is in use:

~ # ls -la /var/lib/docker
total 56
drwx--x---   14 root     root          4096 Mar 29 21:03 .
drwxr-xr-x    1 root     root          4096 Mar 29 21:03 ..
drwx--x--x    4 root     root          4096 Mar 29 21:03 buildkit
drwx--x--x    3 root     root          4096 Mar 29 21:03 containerd
drwx--x---    2 root     root          4096 Mar 29 21:03 containers
drwx------    3 root     root          4096 Mar 29 21:03 image
drwxr-x---    3 root     root          4096 Mar 29 21:03 network
drwx--x---    3 root     root          4096 Mar 29 21:03 overlay2
drwx------    4 root     root          4096 Mar 29 21:03 plugins
drwx------    2 root     root          4096 Mar 29 21:03 runtimes
drwx------    2 root     root          4096 Mar 29 21:03 swarm
drwx------    2 root     root          4096 Mar 29 21:03 tmp
drwx------    2 root     root          4096 Mar 29 21:03 trust
drwx-----x    2 root     root          4096 Mar 29 21:03 volumes
~ # ls -la /var/lib/docke
total 8
drwxr-xr-x    2 root     root          4096 Mar 29 21:03 .
drwxr-xr-x    1 root     root          4096 Mar 29 21:03 ..

Version information

$ pip freeze | grep docker && python --version && docker version

docker==5.0.3
Python 3.8.10
Client:
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.8
 Git commit:        20.10.7-0ubuntu5~20.04.2
 Built:             Mon Nov  1 00:34:17 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       20.10.7-0ubuntu5~20.04.2
  Built:            Fri Oct 22 00:45:53 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.5-0ubuntu3~20.04.2
  GitCommit:        
 runc:
  Version:          1.0.1-0ubuntu2~20.04.1
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:      

Host OS:

• ubuntu Linux 20.04
• Kernel 5.4.0-104-generic x86_64

Please let me know if I can provide any more information.

[pevidex]

This seems related to #3073. Try working that around by supplying a volumes dict instead of list. It will bypass this step and not call the problematic function (this line).

[orbelico]

Thx @pevidex, will try that as soon as I find the time.