ECS CNI Plugin: unable to attach the veth interface to bridge

[mythri-garaga]

Summary

Customers launching tasks in awsvpc mode are experiencing tasks failing to start issue intermittently.

The tasks are failing to start with the following error

evel=error time=2022-06-02T16:00:59Z msg="Unable to configure pause container namespace" task="<task id>" error="add network failed: bridge create veth pair: unable to attach the veth interface veth5f8829e5 to bridge: exchange full"
level=error time=2022-06-02T16:00:59Z msg="Error transitioning container" nextState="RESOURCES_PROVISIONED" error="container resource provisioning: failed to setup network namespace: add network failed: bridge create veth pair: unable to attach the veth interface veth5f8829e5 to bridge: exchange full" task="<task id>" container="~internal~ecs~pause" runtimeID="<runtime id>"
level=warn time=2022-06-02T16:00:59Z msg="Error starting/provisioning container[%s (Runtime ID: %s)];" container="~internal~ecs~pause" runtimeID="<runtime id>" error="container resource provisioning: failed to setup network namespace: add network failed: bridge create veth pair: unable to attach the veth interface veth5f8829e5 to bridge: exchange full"

ecs cni plugin error

2022-06-02T16:00:59Z [INFO] Creating the bridge: ecs-bridge
2022-06-02T16:00:59Z [INFO] Creating veth pair for namespace: /host/proc/14172/ns/net
2022-06-02T16:00:59Z [INFO] Attaching veth pair veth5f8829e5 to bridge
2022-06-02T16:00:59Z [ERROR] Error executing ADD command: bridge create veth pair: unable to attach the veth interface veth5f8829e5 to bridge: exchange full
2022-06-02T16:00:59Z [INFO] IPAM configuration not found, skip DEL for IPAM

AMI Ids: ami-061c10a2cb32f3491, ami-0c5c9bcfb36b772fe
Instance types: m6i.2xlarge, m5.2xlarge

[muthu-vgh]

"Version": "Amazon ECS Agent - v1.61.0 (0573061)" and issue seems to resolve it self after reboot, Linux 4.14.276-211.499.amzn2.x86_64 #1 SMP Wed Apr 27 21:08:48 UTC 2022

[Realmonia]

This should be fixed by #3311. The problem is that ECS CNI plugin doesn't always clean up Veth which causes exchange full and no more Veth interfaces can be attached.

@muthu-vgh I suppose "issue resolves it self after reboot" means reboot the instance? The instance reboot should clean up the Veths.

Closing this, please kindly reopen if you find more occurrences.