OCPBUGS-48320,SDN-4930: Increase probe timeouts on UDN pod #29458
Conversation
openshift-ci
bot
added
the
do-not-merge/work-in-progress
|
/payload-job periodic-ci-openshift-release-master-nightly-4.19-e2e-vsphere-ovn-techpreview |
|
/test e2e-gcp-ovn-techpreview |
|
@tssurya: trigger 4 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/422b3d50-d809-11ef-9095-911f8eab4ccc-0 |
|
/payload-job periodic-ci-openshift-release-master-nightly-4.19-e2e-vsphere-ovn-techpreview |
|
@tssurya: trigger 4 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/9f9edb10-d80c-11ef-9b1f-9851ec981cd1-0 |
|
/payload-job periodic-ci-openshift-release-master-nightly-4.19-e2e-vsphere-ovn-techpreview |
|
@tssurya: trigger 4 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/d44a8ca0-d830-11ef-9237-b435d7189f1c-0 |
tssurya
force-pushed
the
increase-probe-timeouts
branch
2 times, most recently
from
3097cee to
2d3339e
Compare
|
/payload-job periodic-ci-openshift-release-master-nightly-4.19-e2e-vsphere-ovn-techpreview |
1 similar comment
|
/payload-job periodic-ci-openshift-release-master-nightly-4.19-e2e-vsphere-ovn-techpreview |
|
@tssurya: trigger 4 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/f7edf6b0-d8c1-11ef-9154-d5ab7bd9ab9c-0 |
|
/payload-job periodic-ci-openshift-release-master-nightly-4.19-e2e-vsphere-ovn-techpreview |
|
@tssurya: trigger 4 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/fa0f01f0-d8c1-11ef-906b-6dc76be18a87-0 |
|
@tssurya: trigger 4 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/0017a7a0-d8c2-11ef-9257-d12072dbf52d-0 |
tssurya
changed the title
openshift-ci
bot
removed
the
do-not-merge/work-in-progress
|
/label acknowledge-critical-fixes-only |
openshift-ci
bot
added
the
acknowledge-critical-fixes-only
|
I have vetted 12 runs on this PR all looking good with restart issue |
|
Job Failure Risk Analysis for sha: 2d3339e
|
tssurya
force-pushed
the
increase-probe-timeouts
branch
2 times, most recently
from
5bdbd13 to
c8ba555
Compare
Signed-off-by: Surya Seetharaman <[email protected]>
tssurya
force-pushed
the
increase-probe-timeouts
branch
from
c8ba555 to
0bc15f9
Compare
tssurya
changed the title
openshift-ci-robot
added
the
jira/valid-reference
|
@tssurya: This pull request references SDN-4930 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target either version "4.19." or "openshift-4.19.", but it targets "openshift-4.18" instead. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/test e2e-gcp-ovn-techpreview |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: trozet, tssurya The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
/override ci/prow/e2e-aws-ovn-serial |
|
@knobunc: Overrode contexts on behalf of knobunc: ci/prow/e2e-aws-ovn-serial In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
1 similar comment
|
/override ci/prow/e2e-aws-ovn-serial |
|
@knobunc: Overrode contexts on behalf of knobunc: ci/prow/e2e-aws-ovn-serial In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
tssurya
changed the title
openshift-ci-robot
added
the
jira/severity-moderate
|
@tssurya: This pull request references Jira Issue OCPBUGS-48320, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. This pull request references SDN-4930 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target either version "4.19." or "openshift-4.19.", but it targets "openshift-4.18" instead. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
added
the
jira/invalid-bug
|
/jira refresh |
openshift-ci-robot
added
the
jira/valid-bug
|
@tssurya: This pull request references Jira Issue OCPBUGS-48320, which is valid. 3 validation(s) were run on this bug
Requesting review from QA contact: This pull request references SDN-4930 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target either version "4.19." or "openshift-4.19.", but it targets "openshift-4.18" instead. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
removed
the
jira/invalid-bug
|
/tide refresh |
|
though failed did not fail for reasons related to UDN, all |
|
@tssurya: Jira Issue OCPBUGS-48320: Some pull requests linked via external trackers have merged: The following pull requests linked via external trackers have not merged:
These pull request must merge or be unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with Jira Issue OCPBUGS-48320 has not been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@tssurya: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
This PR increases the liveness and readiness probe failure threshold to 3 and also increases each probe's timeout to 3seconds instead of the default value 1 for both which is pretty aggressive.
We have seen failures of the following pattern:
where randomly after 15 or 30 seconds the liveness probe timeones out waiting for headers -> so at least we know the TCP conn was established and kubelet unfortunately didn't receive the headers within the 1sec? Hard to tell why that's the case. But this PR when increasing it to 3 seconds so far in the CI has not hit this flake even once.
We think increasing the failure threshold to 3 is safer on ocp just like we have for startup probes.