issues with swarm encrypted overlay network and iptables-nft on RHEL 8+

[luca-heitmann]

Description

We see errors in the journal log when creating an encrypted overlay network with RHEL 8.5 and Docker 20.10. I created an test environment with two Rocky Linux 8.5 VMs where the issue is reproducible (see below). As Mirantis does not support RHEL 8.5 for Docker 20.10 yet, I also verified that the issue is reproducible with Rocky 8.4.

could not install mangle rule:  (iptables failed: iptables --wait -t mangle -A OUTPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1048832 -j MARK --set-mark 13681891: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2))

The errors in the log show that some iptables rules cannot be created. The reason seems to be u32 extension was deprecated in RHEL 8: https://bugzilla.redhat.com/show_bug.cgi?id=2061288

The encrypted overlay network seem to use u32:

moby/libnetwork/drivers/overlay/encryption.go

Line 211 in 2b70006

rule = []string{"-p", "udp", "--dport", p, "-m", "u32", "--u32", c, "-j", "MARK", "--set-mark", m}

I am not an expert and don't fully understand the consequences of the missing iptables/nftables rules. In the test environment, I cannot detect any misbehavior of the routing mesh or other components.

Steps to reproduce the issue:

1. Apply the following compose file by running docker stack deploy -c stack.yml test-stack on a system which uses nftables:

version: '3.8'

networks:
  default:
    driver_opts:
      encrypted: "true"
    name: test-net

services:
  test-srv-1:
    deploy:
      placement:
        constraints:
          - node.hostname==node1
    image: nginx
    ports:
    - published: 30001
      target: 80
  test-srv-2:
    deploy:
      placement:
        constraints:
          - node.hostname==node2
    image: nginx
    ports:
    - published: 30002
      target: 80

2. Execute journalctl

Describe the results you received:

Found these errors when executing journalctl:

Mar 16 10:20:23 node1 dockerd[5991]: time="2022-03-16T10:20:23.640857615Z" level=warning msg="could not install mangle rule:  (iptables failed: iptables --wait -t mangle -A OUTPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j MARK --set-mark 13681891: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2))"
Mar 16 10:20:23 node1 dockerd[5991]: time="2022-03-16T10:20:23.640885838Z" level=warning msg=" (iptables failed: iptables --wait -t mangle -A OUTPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j MARK --set-mark 13681891: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2))"
Mar 16 10:20:23 node1 dockerd[5991]: time="2022-03-16T10:20:23.652697451Z" level=error msg="could not add input rule:  (iptables failed: iptables --wait -t filter -A INPUT -m policy --dir in --pol ipsec -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j ACCEPT: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry
iptables -h' or 'iptables --help' for more information.\n (exit status 2)). Please do it manually."
Mar 16 10:20:23 node1 dockerd[5991]: time="2022-03-16T10:20:23.664394572Z" level=error msg="could not add input rule:  (iptables failed: iptables --wait -t filter -A INPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j DROP: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2)). Please do it manually."
Mar 16 10:20:24 node1 dockerd[5991]: time="2022-03-16T10:20:24.041791548Z" level=warning msg="could not install mangle rule:  (iptables failed: iptables --wait -t mangle -A OUTPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j MARK --set-mark 13681891: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2))"
Mar 16 10:20:24 node1 dockerd[5991]: time="2022-03-16T10:20:24.042004393Z" level=warning msg=" (iptables failed: iptables --wait -t mangle -A OUTPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j MARK --set-mark 13681891: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2))"
Mar 16 10:20:24 node1 dockerd[5991]: time="2022-03-16T10:20:24.052837063Z" level=error msg="could not add input rule:  (iptables failed: iptables --wait -t filter -A INPUT -m policy --dir in --pol ipsec -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j ACCEPT: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry
iptables -h' or 'iptables --help' for more information.\n (exit status 2)). Please do it manually."
Mar 16 10:20:24 node1 dockerd[5991]: time="2022-03-16T10:20:24.064097404Z" level=error msg="could not add input rule:  (iptables failed: iptables --wait -t filter -A INPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j DROP:
 iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2)). Please do it manually."
Mar 16 10:26:46 node1 dockerd[5991]: time="2022-03-16T10:26:46.492985295Z" level=warning msg="could not install mangle rule:  (iptables failed: iptables --wait -t mangle -A OUTPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j MARK --set-mark 13681891: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2))"
Mar 16 10:26:46 node1 dockerd[5991]: time="2022-03-16T10:26:46.493022369Z" level=warning msg=" (iptables failed: iptables --wait -t mangle -A OUTPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j MARK --set-mark 13681891: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2))"
Mar 16 10:26:46 node1 dockerd[5991]: time="2022-03-16T10:26:46.508195851Z" level=error msg="could not add input rule:  (iptables failed: iptables --wait -t filter -A INPUT -m policy --dir in --pol ipsec -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j ACCEPT: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry
iptables -h' or 'iptables --help' for more information.\n (exit status 2)). Please do it manually."
Mar 16 10:26:46 node1 dockerd[5991]: time="2022-03-16T10:26:46.521236295Z" level=error msg="could not add input rule:  (iptables failed: iptables --wait -t filter -A INPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j DROP: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2)). Please do it manually."

Describe the results you expected:

not having these errors

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

[root@node1 ~]# docker version
Client: Docker Engine - Community
 Version:           20.10.13
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 10 14:07:38 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.13
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       906f57f
  Built:            Thu Mar 10 14:05:59 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.10
  GitCommit:        2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

[root@node1 ~]# docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.0-docker)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 20.10.13
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: x0ntpiod2v7v86o3m0iu6gnq9
  Is Manager: true
  ClusterID: 6nciqolcj6zogy3sj79d5n10c
  Managers: 1
  Nodes: 2
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 172.16.16.101
  Manager Addresses:
   172.16.16.101:2377
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.18.0-305.19.1.el8_4.x86_64
 Operating System: Rocky Linux 8.4 (Green Obsidian)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 1.775GiB
 Name: node1
 ID: H5J2:TIN2:BZYH:EGX5:3F4K:HX5A:RTVT:WJO7:437D:NK2S:WROP:P55O
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

I use the following Vagrantfile in my test environment:

# -*- mode: ruby -*-
# vi: set ft=ruby :

ENV['VAGRANT_DEFAULT_PROVIDER'] = 'virtualbox'
ENV['VAGRANT_NO_PARALLEL'] = 'yes'

Vagrant.configure(2) do |config|

  NodeCount = 2

  (1..NodeCount).each do |i|

    config.vm.define "node#{i}" do |node|

      node.vm.box               = "bento/rockylinux-8.4"
      node.vm.box_check_update  = false
      node.vm.box_version       = "202110.26.0"
      node.vm.hostname          = "node#{i}"

      node.vm.network "private_network", ip: "172.16.16.10#{i}"

      node.vm.provider :virtualbox do |v|
        v.gui     = true
        v.name    = "node#{i}"
        v.memory  = 2048
        v.cpus    = 2
      end

      node.vm.provision "shell", path: "bootstrap.sh"
      if i == 1
        node.vm.provision "shell", inline: "docker swarm init --advertise-addr 172.16.16.101"
        node.vm.provision "shell", inline: "docker swarm join-token -q worker > /vagrant/token"
      else
        node.vm.provision "shell", inline: "docker swarm join --advertise-addr 172.16.16.10#{i} --listen-addr 172.16.16.10#{i}:2377 --token `cat /vagrant/token` 172.16.16.101:2377"
      end
    end
  end
end

this bootstrap.sh file is used to provision the VMs:

#!/bin/bash

# Enable ssh password authentication
echo "Enable ssh password authentication"
sed -i 's/^PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
sed -i 's/.*PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config
systemctl reload sshd

# Set Root password
echo "Set root password"
echo -e "admin\nadmin" | passwd root >/dev/null 2>&1

# Install Docker
echo "Install Docker"
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
dnf install -y docker-ce-20.10.13-3.el8 docker-ce-cli-20.10.13-3.el8 containerd.io
systemctl enable docker
systemctl start docker

If you have vagrant with VirtualBox set up and want to use this configuration:

1. Create the Vagrantfile and bootstrap.sh in a new directory with the content from above
2. Create the file "stack.yml" with the content from above
3. Execute "vagrant up"
4. Execute "vagrant ssh node1"
5. Execute "sudo docker stack deploy -c /vagrant/stack.yml test-stack"
6. Execute "sudo journalctl | grep iptables"

Edit: I just noticed that Mirantis does not support RHEL 8.5 but 8.4. So I changed the test environment to Rocky 8.4 and verified that the issue is also reproducible there.

[johnbizokk]

@luca-heitmann Hello! After a kernel rolled back (5.17.3-> 4.18.0) during a system update, I had the same issue. It all works like a clockwork after reverting to the "5. X" kernel.

[xinfengliu]

This is because RHEL 8.x and other variants (Rocky, CentOS) do not include xt_u32 kernel module by default. You can install it by sudo dnf install kernel-modules-extra then restart docker daemon.

[MarcelGeo]

@xinfengliu we have the same error messages with iptables after sudo dnf install kernel-modules-extra. RHEL 8

[johnbizokk]

@xinfengliu we have the same error messages with iptables after sudo dnf install kernel-modules-extra. RHEL 8

I would suggest upgrading to the latest kernel.

[MarcelGeo]

Thanks for answer johnbizokk and xinfengliu. Our version is:

4.18.0-372.9.1.el8.x86_64

[corhere]

RedHat decided to deprecate the xt_u32 kernel module in RHEL 8 and remove it in RHEL 9 as a way to nudge users towards migrating to native nftables. This is an arbitrary limitation imposed by RedHat; iptables-nft is compatible with all legacy packet matches which are available for the running kernel. iptables-nft is deprecated as of RHEL 9 but is still available...for now.

[neersighted]

This issue became, and was fixed during the remediation of:

• CVE-2023-28840: Encrypted overlay network may be unauthenticated
• CVE-2023-28841: Encrypted overlay network traffic may be unencrypted
• CVE-2023-28842: Encrypted overlay network with a single endpoint is unauthenticated