IP masquerading issue in k8s cluster worker nodes

[MohitBairagi]

Hi All,

I have a k8s metal cluster of 3 control nodes and around 25 worker nodes running on laptops with base OS Ubuntu 22.04.1 LTS in which we are also using Canal as CNI plugin. We added some new nodes/laptops in the cluster as worker nodes. Now on these nodes we are facing the Source Network Address Translation(SNAT) or IP masquerading on these nodes.
I realised about this issue, when I found that pod network communication is working both on the same node and also on different nodes, but internet connectivity is not working. I also ran tcpdump to troubleshoot this and found that on those nodes where internet connectivity is working Pod IP gets converted to Node Ip, but on the nodes with issue, Pod IP is not getting converted to Node Ip. Hence traffic leaves from the node with Pod IP and I am assuming that packets are unable to find the path when coming back.

Expected Behavior

Internet connectivity should work on the issue nodes as working on other nodes

Current Behavior

Internet connectivity is not working

Possible Solution

Not Sure

Steps to Reproduce (for bugs)

Unable to reproduce the issue as not sure what is the root cause of the issue.