Linux masquerading broken on all 0.30.* versions

[bravosierrasierra]

Linux masquerading broken on all 0.30.* versions

After 0.29.4 release inside NETBIRD-RT-NAT iptables chain added network interfaces, who broke -j MASQUERADE: inside POSTROUTING chain source interfaces does not exists and all MASUERADE rules does not works.

On all 0.30.* releases netbird client incorrectly appended interfaces to NETBIRD-RT-NAT rules from iptables-save:

Generated by iptables-save v1.8.4 on Thu Oct 17 11:05:40 2024

*nat
:PREROUTING ACCEPT [2:280]
:INPUT ACCEPT [2:280]
:OUTPUT ACCEPT [51:3869] :POSTROUTING ACCEPT [51:3869]
:DOCKER - [0:0]
:NETBIRD-RT-NAT - [0:0]
-A POSTROUTING -j NETBIRD-RT-NAT
-A NETBIRD-RT-NAT -d 10.239.8.0/26 -i wt0 ! -o lo -j MASQUERADE
-A NETBIRD-RT-NAT -s 10.239.8.0/26 ! -i lo -o wt0 -j MASQUERADE
-A NETBIRD-RT-NAT -d 10.239.5.128/26 -i wt0 ! -o lo -j MASQUERADE
-A NETBIRD-RT-NAT -s 10.239.5.128/26 ! -i lo -o wt0 -j MASQUERADE
-A NETBIRD-RT-NAT -d 10.239.6.0/26 -i wt0 ! -o lo -j MASQUERADE
-A NETBIRD-RT-NAT -s 10.239.6.0/26 ! -i lo -o wt0 -j MASQUERADE

Mangle does not work and packet goes outside with mesh ip address (100.104.127.184 -> 10.239.6.202.8123):
root@netbird-analytic-gw-prod-b-01:~# ./tcpdump -i any -n host 10.239.6.202 and port 8123
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
11:07:22.398211 wt0 In IP 100.104.127.184.54659 > 10.239.6.202.8123: Flags [S], seq 2681453072, win 65535, options [mss 1240,nop,wscale 6,nop,nop,TS val 669360886 ecr 0,sackOK,eol], length 0
11:07:22.398244 eth0 Out IP 100.104.127.184.54659 > 10.239.6.202.8123: Flags [S], seq 2681453072, win 65535, options [mss 1240,nop,wscale 6,nop,nop,TS val 669360886 ecr 0,sackOK,eol], length 0
11:07:23.398321 wt0 In IP 100.104.127.184.54659 > 10.239.6.202.8123: Flags [S], seq 2681453072, win 65535, options [mss 1240,nop,wscale 6,nop,nop,TS val 669361887 ecr 0,sackOK,eol], length 0
11:07:23.398382 eth0 Out IP 100.104.127.184.54659 > 10.239.6.202.8123: Flags [S], seq 2681453072, win 65535, options [mss 1240,nop,wscale 6,nop,nop,TS val 669361887 ecr 0,sackOK,eol], length 0

on a 0.29.4 and before iptables rules from iptables-save look as

*nat
:PREROUTING ACCEPT [11:810]
:INPUT ACCEPT [1:140]
:OUTPUT ACCEPT [123:8994]
:POSTROUTING ACCEPT [24:2413]
:DOCKER - [0:0]
:NETBIRD-RT-NAT - [0:0]
-A POSTROUTING -j NETBIRD-RT-NAT
-A NETBIRD-RT-NAT -o lo -j RETURN
-A NETBIRD-RT-NAT -s 10.239.7.64/26 -j MASQUERADE
-A NETBIRD-RT-NAT -d 10.239.7.64/26 -j MASQUERADE
-A NETBIRD-RT-NAT -s 10.239.6.128/26 -j MASQUERADE
-A NETBIRD-RT-NAT -d 10.239.6.128/26 -j MASQUERADE
-A NETBIRD-RT-NAT -s 10.239.5.128/26 -j MASQUERADE
...

Tcpdump proofs: Mangle worked (100.104.127.184 -> Mangle to eth0 ip 10.239.7.162 -> 10.239.6.202.8123)
root@netbird-analytic-gw-prod-b-01:~# ./tcpdump -i any -n host 10.239.6.202 and port 8123
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
11:12:17.983354 wt0 In IP 100.104.127.184.54729 > 10.239.6.202.8123: Flags [S], seq 2871055947, win 65535, options [mss 1240,nop,wscale 6,nop,nop,TS val 3187012617 ecr 0,sackOK,eol], length 0
11:12:17.983418 eth0 Out IP 10.239.7.162.54729 > 10.239.6.202.8123: Flags [S], seq 2871055947, win 65535, options [mss 1240,nop,wscale 6,nop,nop,TS val 3187012617 ecr 0,sackOK,eol], length 0
11:12:17.985243 eth0 In IP 10.239.6.202.8123 > 10.239.7.162.54729: Flags [S.], seq 899166765, ack 2871055948, win 65160, options [mss 1460,sackOK,TS val 1737359530 ecr 3187012617,nop,wscale 7], length 0
11:12:17.985258 wt0 Out IP 10.239.6.202.8123 > 100.104.127.184.54729: Flags [S.], seq 899166765, ack 2871055948, win 65160, options [mss 1460,sackOK,TS val 1737359530 ecr 3187012617,nop,wscale 7], length 0

[bravosierrasierra]

uname -a

Linux netbird-analytic-gw-prod-b-01 5.4.0-196-generic #216-Ubuntu SMP Thu Aug 29 13:26:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/issue

Ubuntu 20.04.6 LTS \n \l

[teoder]

Same issue here, also on Ubuntu 20.04.6 LTS.

[mlsmaycon]

The older version of ubuntu/kernel doesn't support input interfaces in the postrouting chains. We will work on a fix for Ubuntu 20.04 this week.

In the meantime, you can use the 0.29.4 version.

[connect369]

Same here - Ubuntu 20.04.6 LTS

[davidhan888-code]

I am facing the same issue. I have tried 0.30.2 and 0.30.3 on Rhel8 with the latest kernel. Every time I restart the Net Bird service, it will put the drop rule below at the top of the forward table and then randomly place it if there is one at the top already. If I restart five times, I will get five drop rules. I can make it work by finding the line number of the drop rule and deleting it.

iptables -L FORWARD --line-numbers
Chain FORWARD (policy DROP)
num target prot opt source destination
1 DROP all -- anywhere anywhere
2 ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
3 ACCEPT all -- anywhere anywhere
4 DROP all -- anywhere anywhere

[mgarces]

hey, can you please test if release 0.31.1 fixes this issue?

[teoder]

Looks good to me @mgarces