Killed container process leads to "containerd: container not found"

[arnaud-lb]

docker ps shows running containers that are not actually running (I can not see the process on the host). Trying to stop them leads to this error in the logs:

Container a1e0ba168351ca586c069235bdc171366c715d38451cfd17dbdcb3289ce3bd3d failed to exit within 10 seconds of signal 15 - using the force

Then:

container kill failed because of 'container not found' or 'no such process': Cannot kill container a1e0ba168351ca586c069235bdc171366c715d38451cfd17dbdcb3289ce3bd3d: rpc error: code = 2 desc = containerd: container not found

And docker stop returns successfully (exit status 0, no error message).

However, when trying to start the container, I see this in the logs:

Handler for POST /v1.27/containers/xxx/start returned error: Container already started

Observed this repeatedly on a process that happened to be killed by the kernel's OOM killer.

Expected behavior

Non running containers do not appear in docker ps
Non running containers can be killed or started
docker stop returns non-0 status when it fails

Information

• Debian
• Docker CE 17.03.1~ce-0~debian-jessie

I'm unable to find a way to reproduce this easily.

[ionelmc]

Just sending SIGKILL should reproduce this (I just hit this issue as well).

[LeonLiuY]

I met the same problem.

• docker ps shows the container
• docker kill can not kill the container and "container not found" was logged in docker daemon log
• docker rm -f can make it disappear from docker ps

[henry-zhang]

we also have this issue with 17.03

[cpuguy83]

This is fixed on 18.03.
Thanks!

[vrothberg]

This is fixed on 18.03.

@cpuguy83 Do you know which commit{,s} fixed the issue?

[cpuguy83]

@vrothberg I couldn't give you just one commit.
Since 17.03 uses the old containerd codebase.
Since 17.12 we've been using containerd 1.0 which is totally different, and likewise the client integration we use in moby is completely different.

That said moby/moby@e55bead could be at least a partial fix against the older codebase... but I'm not sure as I've not looked at the commit history there or where the bug might have come in.

[vrothberg]

@cpuguy83 Thanks a lot for the quick reply, much appreciated!

[monstermax]

same issue with docker 17.09.1-ce

$ docker -v
Docker version 17.09.1-ce, build 19e2cf6

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a3b75c3e76de debian8-cta3 "/bin/sh -c /opt/b..." 5 months ago Up 4 days cta3

$ docker exec -ti cta3 bash
containerd: container not found

i understand i have to update docker. but it's not easy in production while the server is used 24/24h

[FilipRy]

This is fixed on 18.03.
Thanks!

I am running in the same problems on 18.09, see 38501

[aniknaemmm]

same issue with docker

$ docker -v
Docker version 19.03.1, build 74b1e89

$docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b36988af25aa xxxx "powershell -Command…" 3 months ago Up 3 weeks

Container b36988af25a failed to exit within 5 seconds of signal 15 - using the force
Sending kill signal 9 to container b36988af25aa664c95
container kill failed because of 'container not found' or 'no such process' [action=kill container=b36988af25aa664c95332 error=no such container]

[hron84]

Is there any way to get rid on this problem other than upgrading to a newest Docker version? We faced with this issue on a prod env, and we need a much less invasive solution for this.