ECS Agent not reporting network bindings

[jmetzmeier]

Summary

After upgrading to the latest ECS agent, our containers are no longer registered to the target group.

Description

We upgraded to the latest ECS agent version (v1.15.0) via switching to the latest Amazon Linux ECS AMI, from a previous version of the AMI. After this change, our tasks no longer get registered to the target group for the load balancer. This appears to be caused by the agent not reporting the network bindings to ECS. In the AWS ECS UI, the tasks show as "Network bindings - not configured". No relevant errors appear in the ecs-agent log.

Inspecting /var/lib/ecs/data/ecs_agent_data.json shows that ecs-agent knows about the ports:

                        "KnownPortBindings": [
                            {
                                "BindIp": "0.0.0.0",
                                "ContainerPort": 3000,
                                "HostPort": 32770,
                                "Protocol": "tcp"
                            }
                        ],
                        "KnownStatus": "RUNNING",
                        "portMappings": [
                            {
                                "BindIp": "",
                                "ContainerPort": 3000,
                                "HostPort": 0,
                                "Protocol": "tcp"
                            }
                        ],

All of our tasks use dynamic port mapping.

Expected Behavior

The network bindings properly display via describe-tasks API call as well as show up in the UI. Tasks properly register to ALB target group.

Observed Behavior

Opposite of expected behavior.

Environment Details

{
    "Cluster": "knotweed-staging-BackendCluster-1VI257U11T4R5",
    "ContainerInstanceArn": "arn:aws:ecs:us-east-1:12345:container-instance/d736b12e-941f-4613-8dd3-b9ded76a43d8",
    "Version": "Amazon ECS Agent - v1.15.0 (d2dd240)"
}

[nmeyerhans]

Thanks for reporting this. This was actually an issue in the ECS backend service, so it won't require agent changes. The problem has been resolved at this time, and newly launched tasks should get the proper network configuration. Please let me know if you continue to experience problems.

[jmetzmeier]

@nmeyerhans I redeployed the updated AMI to our staging environment and redeployed the tasks, but the network binding still do not register with ECS.

[jmetzmeier]

For anyone else finding this report in a search and also having this issue, this was the reply we got from AWS support:

We have already identified a bug with the ecs-agent versions 1.14.5 & 1.15.0, where the container fails to register with the load balancer. As a workaround, current recommendation to fix the problem is to rollback to the agent version 1.14.4. Our backend team is actively working to resolve the issue and probably will roll out the fix in a week.

[oanabotezat]

actually I had to wait a bit for the task to get in the RUNNING state. tested on agent version:

2017.09.d | 1.16.0 | 17.06.2-ce | 1.16.0-1
2017.03.f | 1.14.4 | 17.03.2-ce | 1.14.4-1
2017.03.e | 1.14.3 | 17.03.1-ce | 1.14.3-1

i get the networkBindings only after the task is in RUNNING state and not while is in PENDING.

                    "lastStatus": "RUNNING",
                    "name": "ecs-demo",
                    "networkBindings": [
                        {
                            "bindIP": "0.0.0.0",
                            "containerPort": 80,
                            "hostPort": 80
                        }
                    ]

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/container_agent_versions.html