Skip to content
This repository has been archived by the owner on Sep 5, 2019. It is now read-only.

Resync every 10h instead of every 30s #543

Closed
wants to merge 1 commit into from

Conversation

imjasonh
Copy link
Member

Fixes #541

Proposed Changes

  • Resync our custom resources every 10 hours, and rely on build/pod watch updates to trigger syncs in most cases

Resyncing every 30s seems to lead us to hit rate limits that we never recover from when there are too many builds in the system at a given time. Serving also switched to resyncing every 10h for similar scaling reasons.

Release Note

NONE

@knative-prow-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ImJasonH

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copy link

@knative-prow-robot knative-prow-robot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@imjasonh: 0 warnings.

In response to this:

Fixes #541

Proposed Changes

  • Resync our custom resources every 10 hours, and rely on build/pod watch updates to trigger syncs in most cases

Resyncing every 30s seems to lead us to hit rate limits that we never recover from when there are too many builds in the system at a given time. Serving also switched to resyncing every 10h for similar scaling reasons.

Release Note

NONE

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@imjasonh
Copy link
Member Author

$ kubectl get build sleep-500 -o=custom-columns-file=./test/columns.txt
NAME        TYPE        STATUS   CREATED                START                  END
sleep-500   Succeeded   True     2019-01-25T21:34:34Z   2019-01-25T21:39:58Z   <none>

(End time is not populated for any of the builds created in the load test, likely another bug)

@imjasonh
Copy link
Member Author

/test pull-knative-build-integration-tests

1 similar comment
@imjasonh
Copy link
Member Author

/test pull-knative-build-integration-tests

@knative-prow-robot
Copy link

@imjasonh: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
pull-knative-build-integration-tests 174a55a link /test pull-knative-build-integration-tests

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@sebgoa
Copy link
Contributor

sebgoa commented Feb 4, 2019

We ran into this at ~200 builds. Happy to test a fix.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants