Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do conntrack deletions in the background. #1498

Merged
merged 2 commits into from
Jul 12, 2017

Conversation

fasaxc
Copy link
Member

@fasaxc fasaxc commented Jul 7, 2017

Block on pending deletions if IP is reused. Improves the slow performance observed in https://github.com/projectcalico/felix/issues/1501

Todos

  • Unit tests (full coverage)
  • Redo manual tests to verify improvement. (Completed by @frnkdny and @doublek)

Release Note

Improve performance when the conntrack table contains many entries by doing conntrack deletions in the background.

Block on pending deletions if IP is reused.
@fasaxc fasaxc requested a review from nelljerram July 10, 2017 12:35
@fasaxc
Copy link
Member Author

fasaxc commented Jul 10, 2017

@neiljerram, @doublek tested this and found it to be a strict improvement over what we have now in his high-connection-load testing. Please can you review?

Copy link
Member

@nelljerram nelljerram left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice change - but a couple of thoughts to consider.

@@ -347,6 +355,45 @@ func (r *RouteTable) syncRoutesForLink(ifaceName string) error {
return nil
}

// startConntrackDeletion starts the deletion of conntrack entries for the given CIDR in the background. Pending
// deletions are tracked in the pendingConntrackCleanups map so we can block waiting for them later.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a bit more context somewhere to record why we're making this change to move conntrack deletions into the background. I'd be happy with any of:

  • adding to the comment here
  • adding to the commit message
  • adding more to the PR comments - either inline explanation or a link to an issue where it's explained.

At the moment - unless I missed it - I don't think we have any of those.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added an issue and updated the comment.

})
start := time.Now()
rt.Apply()
Expect(time.Since(start)).To(BeNumerically("<", 10*time.Millisecond))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks fragile if we're running on a really slow test VM. One option would be a factor of 10, as 0.3s would still be nearly unnoticeable. Is there anything better we can do?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't think of a better way to interlock the test without quite a bit of plumbing. Let's just up the timer.

Copy link
Member

@nelljerram nelljerram left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

@fasaxc
Copy link
Member Author

fasaxc commented Jul 11, 2017

retest this please

@fasaxc fasaxc assigned fasaxc and unassigned nelljerram Jul 11, 2017
@caseydavenport caseydavenport added this to the Calico v2.4.0 milestone Jul 11, 2017
@fasaxc fasaxc merged commit d231298 into projectcalico:master Jul 12, 2017
@fasaxc fasaxc deleted the bg-conntrack branch July 12, 2017 09:05
song-jiang pushed a commit to song-jiang/felix that referenced this pull request Aug 30, 2017
Do conntrack deletions in the background.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants