Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: add client side retry to prevent 'transport is closing' errors #6402

Merged
merged 1 commit into from
Jun 4, 2021

Conversation

alexmt
Copy link
Collaborator

@alexmt alexmt commented Jun 4, 2021

Signed-off-by: Alexander Matyushentsev [email protected]

We are periodically seeing 'transport is closing' errors from argocd CLI in production. The reason is that Argo CD API server runs behind load balancer but uses default ( infinite ) idle connection timeout. Load balancer periodically closes idle connections which causes the error. PR adds retry to client side as well as changes max idle connection timeout to 5 mins (configurable via env variable).

@alexmt alexmt requested a review from mayzhang2000 June 4, 2021 17:17
@alexmt alexmt force-pushed the transport-closing-error branch 2 times, most recently from bd9b324 to da94121 Compare June 4, 2021 17:25
@codecov
Copy link

codecov bot commented Jun 4, 2021

Codecov Report

Merging #6402 (313ad5d) into master (c3b1220) will increase coverage by 0.01%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #6402      +/-   ##
==========================================
+ Coverage   41.00%   41.01%   +0.01%     
==========================================
  Files         152      152              
  Lines       20081    20088       +7     
==========================================
+ Hits         8234     8239       +5     
- Misses      10715    10716       +1     
- Partials     1132     1133       +1     
Impacted Files Coverage Δ
reposerver/repository/repository.go 60.85% <0.00%> (+0.09%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c3b1220...313ad5d. Read the comment docs.

@alexmt alexmt force-pushed the transport-closing-error branch from da94121 to 313ad5d Compare June 4, 2021 17:52
@alexmt alexmt changed the title fix: change default idle timeout of API server and add client side retry to prevent 'transport is closing' errors fix: add client side retry to prevent 'transport is closing' errors Jun 4, 2021
@alexmt alexmt merged commit 2eb6b3b into argoproj:master Jun 4, 2021
@alexmt alexmt deleted the transport-closing-error branch June 4, 2021 18:57
alexmt pushed a commit that referenced this pull request Jun 4, 2021
mcanevet added a commit to mcanevet/devops-stack that referenced this pull request Jun 23, 2021
Note: thanks to argoproj/argo-cd#6402, it's not
required anymore to retry.
@jannfis jannfis added the needs-verification PR requires pre-release verification label Jun 25, 2021
mcanevet added a commit to mcanevet/devops-stack that referenced this pull request Aug 11, 2021
Note: thanks to argoproj/argo-cd#6402, it's not
required anymore to retry.
@alexmt alexmt removed the needs-verification PR requires pre-release verification label Aug 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants