A variety of network calls from GoCD seem less reliable

[robrap]

Context:

The symptom we are seeing is that we have many more OpsGenie alerts coming from GoCD for the edxapp pipeline nw than in the past. Additionally, we have OpsGenie configured to only alert the team if the failure happens twice, and most alerts self-close. This typically means the issue was flaky, which is often a networking issue.

We've been adding a bunch of retries recently around failures related to failed downloads, as an example.

• https://github.com/openedx/configuration/pull/6988/files
• openedx-unsupported/configuration@bee7eaa

We have other transient failures ticketed:

• prod_deploy_ami_job fails transiently #403
• create_merge_pr_job fails transiently openedx-unsupported/tubular#687

And still others that have yet to be ticketed.

See GoCD alerts graph:

Question:

• Is it possible that the NAT has reached capacity for outbound requests, or some other infrastructure limitation that has been reached?

[github-actions]

This is now https://2u-internal.atlassian.net/browse/PSRE-2723

[robrap]

[inform] These were the tags I used for OpsGenie Analytics:

[robrap]

We've had enough success with retries at the moment that we've decided not to pursue determining if there is an underlying issue. I'm closing this ticket.