Unable to start Service: failed to pull image amazon/amazon-ecs-pause:0.1.0

[rickalm]

#1837

Summary

Hardcoded/Referenced image is not available on Docker Hub

Description

While 1837 discussed the issue with "pulling" the referenced Pause Container image, without anywhere to pull the image from, the issue blocking deployment of services using PidMode or any other pause container use-case is broken.

It seems that the Amazon Linux has this container baked-into the image, but that is not an acceptable approach, and by the docker image specification the above referenced image is expected to be on docker hub. This would be like baking the ecs-agent image into an Amazon AMI and then telling all customers which use their own AMI to build the agent from scratch

Expected Behavior

If a tool is referencing a "published image" then that image should be published.

If AWS is not going to publish the image, then provide a way to override the image name rather than hardcoding the image name within the code

[rickalm]

Further investigation has identified the issue

ECS_ENABLE_TASK_ENI indirectly controls creation of the pause image

https://github.com/aws/amazon-ecs-agent/blob/master/agent/app/agent.go#L265-L266

https://github.com/aws/amazon-ecs-agent/blob/master/agent/app/agent.go#L265-L266

initializeTaskENIDependencies is the function which creates the image locally, but since its part of the ENI Dependencies in order to get the image created unless all ENI deps can be initialized

[CRITICAL] Unable to initialize Task ENI dependencies: agent is not started with an init system

[rickalm]

To provide some context:

We are using the ECS feature known as PidMode: Task in order to allow a task in container 1 to send a sighup to a process in container 2 to inform it of the change to its configuration files

Which is what is invoking use of the pause container.

https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_TaskDefinition.html

[rickalm]

Our current work-around is

docker pull k8s.gcr.io/pause:3.1; docker tag k8s.gcr.io/pause:3.1 amazon/amazon-ecs-pause:0.1.0

I would be happy to specify an ENV var at launch to change from the ecs pause image to the k8s pause but those 2 items are not exposed through ENV

[petderek]

The pause container is baked into the agent container, not the optimized AMI. If you are pulling agent from Docker Hub or building it yourself with the 'make release' target, you will have the pause container available as a tarball inside the agent.

It looks like we never load it correctly for the case where task networking is disabled, but PID / IPC sharing is enabled.

[rickalm]

@petderek the work-around is not sustainable

[petderek]

Agreed; we'll fix this -- likely by unconditionally loading the pause container into docker. If agent is running, the pause container should be available.

[petderek]

This shipped in 1.35.0