Passing through Port 514/UDP eventually stops passing through

[csm10495]

Description

I have a simple server that listens on port 514 via UDP and just prints what it gets. (It's a very sketchy syslog-ish server). After a while (seemingly once a day-ish it just stops listening on port 514.). The app inside the container is still running.

Inside the container, i can see my process is still listening on 514:

root@syslog-server:/home/csmutil# sudo -u csmutil lsof -p 7 | grep \:syslog
python    7 csmutil    6u  IPv4 262532      0t0               UDP *:syslog

In ps auxx:

csmutil      7  3.6  0.4  51912 20844 ?        S    Nov18  43:16 python -m csmutil.syslog_server

Though outside the container in Windows, it seems like port 514 is no longer in use:

PS C:\Windows\System32> Get-Process -Id (Get-NetUDPEndpoint -LocalPort 514)
Get-NetUDPEndpoint : No MSFT_NetUDPEndpoint objects found with property 'LocalPort' equal to '514'.  Verify the value
of the property and retry.
At line:1 char:18
+ Get-Process -Id (Get-NetUDPEndpoint -LocalPort 514)
+                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (514:UInt16) [Get-NetUDPEndpoint], CimJobException
    + FullyQualifiedErrorId : CmdletizationQuery_NotFound_LocalPort,Get-NetUDPEndpoint

Get-Process : Cannot bind argument to parameter 'Id' because it is null.
At line:1 char:17
+ Get-Process -Id (Get-NetUDPEndpoint -LocalPort 514)
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (:) [Get-Process], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.GetProcessC
   ommand

Though when i first started (until the passthrough stopped) it would show up in windows via ^ as well.

Here is a simple repro for csmutil/syslog_server.py:

import socketserver
from logging import getLogger, basicConfig, DEBUG

logger = getLogger(__file__)
basicConfig(level=DEBUG)

class SyslogUDPHandler(socketserver.BaseRequestHandler):
    def handle(self):
        data = self.request[0].strip().decode()
        log_msg = f"{self.client_address[0]}: {data}"
        if "ERR" in data.upper():
            logger.error(log_msg)
        elif "WARN" in data.upper():
            logger.warning(log_msg)
        else:
            logger.debug(log_msg)


if __name__ == "__main__":
    logger.info("Starting SyslogUDP Handler")
    try:
        server = socketserver.UDPServer(("0.0.0.0", 514), SyslogUDPHandler)
        server.serve_forever(poll_interval=0.5)
    except KeyboardInterrupt:
        logger.info("Crtl+C Pressed. Shutting down.")

I'm running against: python:3.10.0-bullseye

If you try to start another instance of the server (inside the same container).. it fails as expected:

OSError: [Errno 98] Address already in use

Furthermore, docker inspect still shows the port as being forwarded (despite it not being according to Windows):

...
            "Ports": {
                "514/udp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "514"
                    }
                ]
            },
...

Reproduce

1. docker run -p 514:514/udp -it --rm python:3.10.0-bullseye bash
2. Inside the container, create a syslog_server.py file with the contents given above.
3. Run the server via python syslog_server.py inside the container
4. Back in Windows run via powershell: Get-Process -Id (Get-NetUDPEndpoint -LocalPort 514) you should get something. Come back in several hours and run this again, and if it reproduces, it'll no longer find the process listening on port 514, though the docker container (and python script) will still be running

Expected behavior

Ports should be forwarded indefinitely as long as the container is running.

docker version

PS C:\Windows\System32> docker version
Client:
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:42:27 2024
 OS/Arch:           windows/amd64
 Context:           desktop-linux

Server: Docker Desktop 4.36.0 (175267)
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978
  Built:            Fri Sep 20 11:41:11 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.21
  GitCommit:        472731909fa34bd7bc9c087e4c27943f9835f111
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

PS C:\Windows\System32> docker info
Client:
 Version:    27.3.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Ask Gordon - Docker Agent (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-ai.exe
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.18.0-desktop.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.30.3-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.37
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.15
    Path:     C:\Program Files\Docker\cli-plugins\docker-desktop.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.15.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 17
  Running: 17
  Paused: 0
  Stopped: 0
 Images: 14
 Server Version: 27.3.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.133.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 5
 Total Memory: 4.807GiB
 Name: docker-desktop
 ID: df8868a6-7187-4971-80e1-8090e022429e
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

Diagnostics ID

C273F745-3836-433C-93C7-2151320BA8E6/20241119221212

Additional Info

I think this started after updating to docker desktop 4.36.0 .. I'm not really sure what version I was on before.

[csm10495]

I rolled back to docker desktop 4.34.3 and still see the same issue.

This time when it happened, I exec'd into a bash session and played a bit more.

root@syslog-server:/home/csmutil# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.18.0.9  netmask 255.255.0.0  broadcast 172.18.255.255
        ether 02:42:ac:12:00:09  txqueuelen 0  (Ethernet)
        RX packets 1213  bytes 593779 (579.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 272  bytes 20951 (20.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 1250  bytes 479950 (468.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1250  bytes 479950 (468.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

I also installed netcat inside it so I could send some UDP packets directly to see if my syslog server gets them.

Trying the ipv4 address:

root@syslog-server:/home/csmutil# echo hello | nc -v -u -w 0 172.18.0.9 514
Connection to 172.18.0.9 514 port [udp/syslog] succeeded!
root@syslog-server:/home/csmutil#

and it gets logged by the syslog server. Similarly trying loopback:

root@syslog-server:/home/csmutil# echo hello | nc -v -u -w 0 127.0.0.1 514
Connection to 127.0.0.1 514 port [udp/syslog] succeeded!

gets logged as well. So the python app is definitely alive and happy inside the container. Just docker is losing the port forward for some reason.

[csm10495]

Here is the output of docker inspect when the container is in this weird state:

C:\Users\csm10495>docker inspect syslog-server
[
    {
        "Id": "3743778c7f266be0c88c2b2b57268f52a3e76084c601f40aeafa7edc733feaf2",
        "Created": "2024-11-21T06:52:41.332021608Z",
        "Path": "python",
        "Args": [
            "-m",
            "csmutil.run",
            "--auto-restart-delay",
            "5",
            "--auto-restart",
            "--process-name",
            "syslog-server",
            "--notify-on-non-zero-exit-code-daily-quota",
            "5",
            "python",
            "-m",
            "csmutil.syslog_server"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 100163,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2024-11-21T17:10:25.374877817Z",
            "FinishedAt": "2024-11-21T17:10:24.914230544Z"
        },
        "Image": "sha256:af2f5d5f83891d4622484cd19c45056fb28761c61a0ebf4252adcc9486c36dbe",
        "ResolvConfPath": "/var/lib/docker/containers/3743778c7f266be0c88c2b2b57268f52a3e76084c601f40aeafa7edc733feaf2/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/3743778c7f266be0c88c2b2b57268f52a3e76084c601f40aeafa7edc733feaf2/hostname",
        "HostsPath": "/var/lib/docker/containers/3743778c7f266be0c88c2b2b57268f52a3e76084c601f40aeafa7edc733feaf2/hosts",
        "LogPath": "/var/lib/docker/containers/3743778c7f266be0c88c2b2b57268f52a3e76084c601f40aeafa7edc733feaf2/3743778c7f266be0c88c2b2b57268f52a3e76084c601f40aeafa7edc733feaf2-json.log",
        "Name": "/syslog-server",
        "RestartCount": 0,
        "Driver": "overlayfs",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": [
            "ef5307c215dbd7a51fc47ed150711720d01c5368ad2d3274109cfaacd664e40a"
        ],
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "serverfmmachalowcom_default",
            "PortBindings": {
                "514/udp": [
                    {
                        "HostIp": "",
                        "HostPort": "514"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                0,
                0
            ],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "host",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": [],
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "volume",
                    "Source": "serverfmmachalowcom_z_drive",
                    "Target": "/mnt/z-drive",
                    "VolumeOptions": {}
                }
            ],
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware",
                "/sys/devices/virtual/powercap"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": null,
            "Name": "overlayfs"
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "serverfmmachalowcom_z_drive",
                "Source": "/var/lib/docker/volumes/serverfmmachalowcom_z_drive/_data",
                "Destination": "/mnt/z-drive",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "syslog-server",
            "Domainname": "",
            "User": "csmutil",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "514/udp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "CSMUTIL_LOG_DIR=/mnt/z-drive/Backup/docker/docker-binds/server.fm.machalow.com/syslog-server/logs",
                "PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/cargo/bin",
                "LANG=C.UTF-8",
                "GPG_KEY=A035C8C19219BA821ECEA86B64E628F8D684696D",
                "PYTHON_VERSION=3.11.10",
                "PYTHON_SHA256=07a4356e912900e61a15cb0949a06c4a05012e213ecd6b4e84d0f67aabbee372",
                "RUSTUP_HOME=/usr/local/rustup",
                "CARGO_HOME=/usr/local/cargo",
                "RUST_VERSION=1.82.0",
                "PYTHONUNBUFFERED=1",
                "PYTHONDONTWRITEBYTECODE=1",
                "TZ=America/Los_Angeles"
            ],
            "Cmd": [
                "python",
                "-m",
                "csmutil.run",
                "--auto-restart-delay",
                "5",
                "--auto-restart",
                "--process-name",
                "syslog-server",
                "--notify-on-non-zero-exit-code-daily-quota",
                "5",
                "python",
                "-m",
                "csmutil.syslog_server"
            ],
            "Image": "container-registry.fm.machalow.com/csmutil-debian",
            "Volumes": null,
            "WorkingDir": "/home/csmutil",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "cfa2f6c17fb1c95ff8db2d1492e78bb9b03922b60fc9b5eae1b401d753445d1b",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "mysql:service_started:false",
                "com.docker.compose.image": "sha256:af2f5d5f83891d4622484cd19c45056fb28761c61a0ebf4252adcc9486c36dbe",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "serverfmmachalowcom",
                "com.docker.compose.project.config_files": "Z:\\Backup\\Repos\\csmutil\\compose\\server.fm.machalow.com\\docker-compose.yml",
                "com.docker.compose.project.working_dir": "Z:\\Backup\\Repos\\csmutil\\compose\\server.fm.machalow.com",
                "com.docker.compose.replace": "ffba29facd9131a0f19adc37b03323dc8c59907379f3463be6f040e05bee6de7",
                "com.docker.compose.service": "syslog-server",
                "com.docker.compose.version": "2.29.1"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "5d26851edd247ee682c17b8a96837ea4f07a1a5b6385efc3ca06a1b8c8e8bea0",
            "SandboxKey": "/var/run/docker/netns/5d26851edd24",
            "Ports": {
                "514/udp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "514"
                    }
                ]
            },
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "serverfmmachalowcom_default": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "syslog-server",
                        "syslog-server"
                    ],
                    "MacAddress": "02:42:ac:12:00:09",
                    "DriverOpts": null,
                    "NetworkID": "8c5ea2b3f977c064972d4ecc3975e285904007cd8db677290c37ae56de3ae9a2",
                    "EndpointID": "0cd1fe5b814cf06e6b3842d95f01eb7f301ceb7453bcfb5c3584d5086580832f",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.9",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "DNSNames": [
                        "syslog-server",
                        "3743778c7f26"
                    ]
                }
            }
        }
    }
]

[csm10495]

As one additional piece of info, while the container is in this state, I went back to windows and successfully started the same syslog server just fine, further showing that the port forwarding stopped (since i could bind to the port in windows, even though the container is supposed to be bound / passed through to that port).