Tasks should be removed when a node doesn't matches service constraint

[dongluochen]

In moby/moby#31377 (comment), a task from a global service is stuck at pending state. Reconciliation should clean up such tasks.

root@manager1:~# docker service ps consulagent
ID            NAME                                   IMAGE                        NODE         DESIRED STATE  CURRENT STATE            ERROR  PORTS
nrv6ofiwlsbz  consulagent.dqxztwj3rrycomdvy7y4xkstd  127.0.0.1:5000/consul:x.x.x  manager3     Running        Pending 19 minutes ago
5umgd96jc4np  consulagent.d2swwbt2n2j58dyzwjs2xc8f7  127.0.0.1:5000/consul:x.x.x  node2        Running        Running 19 minutes ago
kgs950z3hvdo  consulagent.vae6tkmhc9mapv9fcck38nu5m  127.0.0.1:5000/consul:x.x.x  node1        Shutdown       Shutdown 18 minutes ago
qzvs3jg10u5m  consulagent.d2swwbt2n2j58dyzwjs2xc8f7  127.0.0.1:5000/consul:x.x.x  node2        Shutdown       Shutdown 19 minutes ago

Signed-off-by: Dong Chen [email protected]

[aaronlehmann]

This makes the global orchestrator's responsibility overlap with constraintenforcer. I think it's better to fix this check in constraintenforcer:

https://github.com/docker/swarmkit/blob/f19ff43e522a1516c171ea801227d28f40509224/manager/orchestrator/constraintenforcer/constraint_enforcer.go#L101

[codecov]

Codecov Report

Merging #2053 into master will increase coverage by 0.08%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master    #2053      +/-   ##
==========================================
+ Coverage   54.22%   54.31%   +0.08%     
==========================================
  Files         111      111              
  Lines       19319    19317       -2     
==========================================
+ Hits        10476    10492      +16     
+ Misses       7594     7569      -25     
- Partials     1249     1256       +7

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f19ff43...c82f358. Read the comment docs.

[aaronlehmann]

I think the problem has been fixed by #1958