Skip to content

Commit

Permalink
container stats: ignore context.Canceled errors
Browse files Browse the repository at this point in the history
we often receive context.Canceled errors when a container exits during
docker stats collection.

there is a benign race condition where if we process the error first
before processing that the context is "Done", then we will log this
canceled error as a warning message here:
https://github.com/aws/amazon-ecs-agent/blob/5be7aa08bed215a557f48c16d8201ad3db59a9be/agent/stats/container.go#L118-L122

this change ignores these context.Canceled errors so that we don't log
them. This will eliminate log messages that look like this when a
container exits:

level=warn time=2020-12-25T07:51:33Z msg="Error encountered processing metrics stream from docker, this may affect cloudwatch metric accuracy: DockerGoClient: Unable to decode stats for container REDACTED: context canceled" module=container.go
  • Loading branch information
sparrc committed Feb 24, 2021
1 parent 8784e86 commit 59b62e0
Showing 1 changed file with 7 additions and 2 deletions.
9 changes: 7 additions & 2 deletions agent/stats/container.go
Original file line number Diff line number Diff line change
Expand Up @@ -118,8 +118,13 @@ func (container *StatsContainer) processStatsStream() error {
case <-container.ctx.Done():
return nil
case err := <-errC:
seelog.Warnf("Error encountered processing metrics stream from docker, this may affect cloudwatch metric accuracy: %s", err)
returnError = true
select {
case <-container.ctx.Done():
// ignore error when container.ctx.Done()
default:
seelog.Warnf("Error encountered processing metrics stream from docker, this may affect cloudwatch metric accuracy: %s", err)
returnError = true
}
case rawStat, ok := <-dockerStats:
if !ok {
if returnError {
Expand Down

0 comments on commit 59b62e0

Please sign in to comment.