moby-engine issue in 41.20241122.3.0

[tamberlo]

Describe the bug

After auto upgrade to version 41.20241122.3.0 docker.service and docker.socket fails to start.

Reproduction steps

1. Boot the OS

Expected behavior

Docker starts normally

Actual behavior

[systemd]
Failed Units: 2
  docker.service
  docker.socket

# journalctl -xeu docker.service
dockerd[18501]:         /builddir/build/BUILD/moby-engine-27.3.1-build/moby-27.3.1/volume/drivers/extpoint.go:118 +0x48
dockerd[18501]: github.com/docker/docker/volume/drivers.(*Store).lookup(0xc00098d710, {0xc0004447b0, 0x10}, 0x0)
dockerd[18501]:         /builddir/build/BUILD/moby-engine-27.3.1-build/moby-27.3.1/volume/drivers/extpoint.go:97 +0x645
dockerd[18501]: github.com/docker/docker/volume/drivers.(*Store).GetDriver(0xc00098d710, {0xc0004447b0, 0x10})
dockerd[18501]:         /builddir/build/BUILD/moby-engine-27.3.1-build/moby-27.3.1/volume/drivers/extpoint.go:150 +0x66
dockerd[18501]: github.com/docker/docker/volume/service.lookupVolume({0x5646e6de5c28, 0x5646e825e6a0}, 0xc00098d710, {0xc0004447b0, 0x10}, {0xc0009b>
dockerd[18501]:         /builddir/build/BUILD/moby-engine-27.3.1-build/moby-27.3.1/volume/service/store.go:770 +0xbc
dockerd[18501]: github.com/docker/docker/volume/service.(*VolumeStore).restore.func2({{0xc0009b21e0, 0x17}, {0xc0004447b0, 0x10}, 0xc0009e4300, 0xc0>
dockerd[18501]:         /builddir/build/BUILD/moby-engine-27.3.1-build/moby-27.3.1/volume/service/restore.go:37 +0x1f5
dockerd[18501]: created by github.com/docker/docker/volume/service.(*VolumeStore).restore in goroutine 1
dockerd[18501]:         /builddir/build/BUILD/moby-engine-27.3.1-build/moby-27.3.1/volume/service/restore.go:31 +0x397
systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

System details

VM QEMU Fedora CoreOS 41.20241122.3.0

Additional information

On previous version 41.20241109.3.0 everything was running smooth.
Fedora CoreOS is running on 3 vm in a docker swarm.

[dustymabe]

can you run rpm-ostree status --verbose and report the results here?

[tamberlo]

core@m-2:~$ rpm-ostree status --verbose
State: idle
AutomaticUpdatesDriver: Zincati (zincati.service)
  DriverState: active; periodically polling for updates (last checked Wed 2024-12-18 17:23:57 UTC)
Deployments:
● fedora:fedora/x86_64/coreos/stable (index: 0)
                  Version: 41.20241122.3.0 (2024-12-16T21:04:43Z)
               BaseCommit: c81a94e71f7bbb4a7eaf1aa2162183d1c63fda4e5b7fb8a3054f42ff9607d8a1
                           └─ fedora-coreos-pool (2024-12-16T18:59:38Z)
                   Commit: 7af77fbe9a2fed23295ff5908157c8097bb43cb64125e47170964ed4ccb313a1
                           ├─ fedora (2024-10-24T13:55:59Z)
                           ├─ fedora-cisco-openh264 (2024-03-11T19:22:31Z)
                           ├─ updates (2024-12-17T03:55:59Z)
                           └─ updates-archive (2024-12-17T04:21:09Z)
                   Staged: no
                StateRoot: fedora-coreos
             GPGSignature: 1 signature
                           Signature made Mon Dec 16 21:06:19 2024 using RSA key ID D0622462E99D6AD1
                           Good signature from "Fedora <[email protected]>"
          LayeredPackages: qemu-guest-agent tcpdump

  fedora:fedora/x86_64/coreos/stable (index: 1)
                  Version: 41.20241109.3.0 (2024-11-25T02:09:37Z)
               BaseCommit: 870e2c8fd02e81652b30bc9a33b5da9d47de66c8bc2bae0a3739ecf38f652660
                           └─ fedora-coreos-pool (2024-11-24T22:20:49Z)
                   Commit: 441caf0716d0f4afcde3ff47d3627fb9666ba3ee430d57792ff95ac37c8aee90
                           ├─ fedora (2024-10-25T08:41:19Z)
                           ├─ fedora-cisco-openh264 (2024-03-11T19:22:31Z)
                           ├─ updates (2024-11-25T01:51:23Z)
                           └─ updates-archive (2024-11-25T02:38:28Z)
                StateRoot: fedora-coreos
             GPGSignature: 1 signature
                           Signature made Mon Nov 25 02:11:25 2024 using RSA key ID D0622462E99D6AD1
                           Good signature from "Fedora <[email protected]>"
          LayeredPackages: qemu-guest-agent tcpdump

[dustymabe]

thanks. odd. I have not seen any reports similar to this and there doesn't really seem to be any bug reports on it. From the trace it looks like maybe something related to your use of volumes?

@fifofonix - have you seen anything like this before?

@tamberlo it might be worth searching upstream and/or opening a bug against moby-engine in bugzilla to see if those maintainers have seen anything like it.

[tamberlo]

Thank you for the support... tomorrow I'll investigate a bit more...

[tamberlo]

I cannot reproduce the problem by reinstalling CoreOS 41.20241122.3.0 vanilla on a VM. I feared it might be some Docker drivers used to mount GlusterFS but even after installing them Docker continues to work correctly. However, I take this opportunity to ask you what is the correct way to share Gluster with containers on CoreOS, since the installation is always discouraged to avoid update problems. Thank you! I'll try to investigate a bit more...