NodeLifecycleController treats node lease renewal as a heartbeat signal #69241

wangzhen127 · 2018-09-28T23:28:26Z

What this PR does / why we need it:
This PR makes NodeLifecycleController treat node lease renewal as a heartbeat signal, in addition to NodeStatus update. This is part of KEP-0009, feature #589 and issue #14733.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Release note:

NodeLifecycleController: Now node lease renewal is treated as the heartbeat signal from the node, in addition to NodeStatus Update.

wangzhen127 · 2018-09-28T23:29:39Z

Still work in progress. Need to add tests. Will notify once it's done. For visibility:
/cc @wojtek-t @wasylkowski-a

k8s-ci-robot · 2018-09-28T23:29:40Z

@wangzhen127: GitHub didn't allow me to request PR reviews from the following users: wasylkowski-a.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

Still work in progress. Need to add tests. Will notify once it's done. For visibility:
/cc @wojtek-t @wasylkowski-a

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

neolit123 · 2018-09-30T05:15:46Z

/kind feature

wojtek-t · 2018-10-01T08:50:54Z

/assign

wojtek-t

This looks reasonable, but it requires tests.
Also, as I mentioned in one comment, if you could move renames to a separate PR, this PR would be so much easier to review (and renames are no-controversial at all and doesn't require tests, so we could merge them immediately).

wojtek-t · 2018-10-01T12:37:52Z

pkg/controller/nodelifecycle/node_lifecycle_controller.go

@@ -132,10 +137,11 @@ const (
 	retrySleepTime = 20 * time.Millisecond
 )

-type nodeStatusData struct {
+type nodeHealthData struct {
 	probeTimestamp           metav1.Time
 	readyTransitionTimestamp metav1.Time
 	status                   v1.NodeStatus


Since you're touching this code anyway - can we change this to pointer? [potentially in a separate PR if that's easier - you can do all the renames there too]

After reading this PR more, if you could move just renames to a separate PR, that would make this PR so much easier.
Thanks!

wojtek-t · 2018-10-01T12:38:18Z

pkg/controller/nodelifecycle/node_lifecycle_controller.go

-	nodeStatusMap map[string]nodeStatusData
+	// Per Node map storing last observed healthy data, including Status and Lease,
+	// together with a local time when it was observed.
+	nodeHealthMap map[string]nodeHealthData


Since you're touching it anyway - this should probably be map[string]*nodeHealthData.

wangzhen127

Please see #69305 for the PR doing the renames. I will rebase and add tests after that PR gets merged.

jennybuckley · 2018-10-01T20:07:52Z

/cc @cheftako

wangzhen127 · 2018-10-01T20:38:26Z

/sig node
/cc @yujuhong

wojtek-t · 2018-10-04T13:40:08Z

@wangzhen127 - this is ready for rebase I think.

wangzhen127 · 2018-10-04T15:48:34Z

Yes. I am adding tests. Will push another version soon.

wangzhen127 · 2018-10-09T16:38:30Z

Ping @gmarek :)

wojtek-t · 2018-10-09T19:16:11Z

Ping @gmarek :)

I talked offline with @gmarek. He won't have time to look into that in the next 2 weeks or so. But he is fine with merging without his review and he will take a look some time towards end of October and if needed we will apply his comments then.

@yujuhong - will you be able to take a look in that case (in addition to my review)?

yujuhong

Looks good overall. I have some questions about the initialization of the nodeHealth map and various conditions, but most probably because I'm not that familiar with the code

yujuhong · 2018-10-09T21:04:31Z

pkg/controller/nodelifecycle/node_lifecycle_controller.go

+	if utilfeature.DefaultFeatureGate.Enabled(features.NodeLease) {
+		// Always update the probe time if node lease is renewed.
+		observedLease, _ = nc.leaseLister.Leases(v1.NamespaceNodeLease).Get(node.Name)
+		if observedLease != nil && (!found || savedLease == nil || savedLease.Spec.RenewTime.Before(observedLease.Spec.RenewTime)) {


nit: Is it necessary to check found? It was set 50 lines ago and savedLease == nil seems to be sufficient.

You are right. We do not need to check found. Updated.

yujuhong · 2018-10-09T21:24:23Z

pkg/controller/nodelifecycle/node_lifecycle_controller.go

-			probeTimestamp:           node.CreationTimestamp,
-			readyTransitionTimestamp: node.CreationTimestamp,
-		}
+		if _, found := nc.nodeHealthMap[node.Name]; found {


When do we expect the node to be in the nodeHealthMap but does not have the ready condition set?

Another question is whether it's possible that nc.nodeHealthMap[node.Name] is set to nil?

It's not obvious to me that when setting the value in line 929 below nc.nodeHealthMap[node.Name] = savedNodeHealth, the savedNodeHealth will always be non-nil...

When do we expect the node to be in the nodeHealthMap but does not have the ready condition set?

Look at this example:

time t0: Node is created.

time t1: Node renew lease, but NodeStatus does not have ready condition. And found is false. We create and save a fake ready condition (unknown) here. And we also save the lease below.

time t2: Still, NodeStatus does not have ready condition. So currentReadyCondition is still nil. We saved the node at t1 with unknown ready condition. So found is true.

Another question is whether it's possible that nc.nodeHealthMap[node.Name] is set to nil?

savedNodeHealth is always not nil at line 929. So once this function runs at least once, nc.nodeHealthMap[node.Name] is always non-nil.

Consider 2 cases:

If currentReadyCondition is nil, we either use the existing one or create a fake one. Then savedNodeHealth on line 855 is not nil.

If currentReadyCondition is not nil, and if at this point we do not have any saved copy, i.e., nc.nodeHealthMap[node.Name] is nil for now. Then at line 855, we check it again, we will have found to be false, so we create a savedNodeHealth at line 879, and then assign it to nc.nodeHealthMap[node.Name] at line 929.

Thanks for explaining. I think the code here is not very easy to follow (not this PR's fault).

One case that seems a bit strange to me is that even if the node doesn't have status, as long as the it continues renewing the lease, the controller will take no action. I think this is by design, but still feels strange :\

In terms of heartbeat, I think renewing the lease satisfy the need of making sure node is alive.

@wojtek-t, WDYT?

So the situation where a node does not have status, but keeps renewing lease depends on the value of node-status-update-period.

So what we have been thinking is to switch:

set lease refresh period (however we call it) to how frequently we update node status now

decrease frequency of node status updates to something like 1 minute

but still compute node statuses with the same frequency (equal of lease refresh period)

So if we make leases and node-statuses independent, then yes - there may be small period when there is no node status (this can pretty much happen only after node registration though, because after that it should always be there).
But if we would make them somehow correlated (I'm not saying we should do that - I'm saying it's theoretically possible), then it would only happen on races.

But yeah - I agree the latter probably doesn't make much sense.

@yujuhong, does this small period that there is no node status sound ok?

I was more interested in the pathological case where kubelet could not post the status ever, while it could still sending heartbeats. I think this theoretically could happen if we decouple heartbeat and status updates in kubelet (which is probably the right thing to do to not add more complexity and dependency). I don't think we need to deal with the pathological case now, but in case we want to put a safeguard in the future, let's settle on leaving a comment saying something like If kubelet never posted the node status, but continues renewing the heartbeat leases, the node controller will assume the node is healthy and take no action.

Done. Added the comments at line 923 now. PTAL

pkg/controller/nodelifecycle/node_lifecycle_controller_test.go

wangzhen127

Updated. PTAL

wangzhen127 · 2018-10-09T22:59:40Z

pkg/controller/nodelifecycle/node_lifecycle_controller.go

-			probeTimestamp:           node.CreationTimestamp,
-			readyTransitionTimestamp: node.CreationTimestamp,
-		}
+		if _, found := nc.nodeHealthMap[node.Name]; found {


When do we expect the node to be in the nodeHealthMap but does not have the ready condition set?

Look at this example:

time t0: Node is created.

time t1: Node renew lease, but NodeStatus does not have ready condition. And found is false. We create and save a fake ready condition (unknown) here. And we also save the lease below.

time t2: Still, NodeStatus does not have ready condition. So currentReadyCondition is still nil. We saved the node at t1 with unknown ready condition. So found is true.

Another question is whether it's possible that nc.nodeHealthMap[node.Name] is set to nil?

savedNodeHealth is always not nil at line 929. So once this function runs at least once, nc.nodeHealthMap[node.Name] is always non-nil.

Consider 2 cases:

If currentReadyCondition is nil, we either use the existing one or create a fake one. Then savedNodeHealth on line 855 is not nil.

If currentReadyCondition is not nil, and if at this point we do not have any saved copy, i.e., nc.nodeHealthMap[node.Name] is nil for now. Then at line 855, we check it again, we will have found to be false, so we create a savedNodeHealth at line 879, and then assign it to nc.nodeHealthMap[node.Name] at line 929.

wangzhen127 · 2018-10-09T23:02:22Z

pkg/controller/nodelifecycle/node_lifecycle_controller.go

+	if utilfeature.DefaultFeatureGate.Enabled(features.NodeLease) {
+		// Always update the probe time if node lease is renewed.
+		observedLease, _ = nc.leaseLister.Leases(v1.NamespaceNodeLease).Get(node.Name)
+		if observedLease != nil && (!found || savedLease == nil || savedLease.Spec.RenewTime.Before(observedLease.Spec.RenewTime)) {


You are right. We do not need to check found. Updated.

pkg/controller/nodelifecycle/node_lifecycle_controller_test.go

yujuhong · 2018-10-11T21:10:26Z

/lgtm

wangzhen127 · 2018-10-11T21:40:51Z

Thanks @yujuhong!

I just noticed and fixed the formatting issue. @wojtek-t, can you approve?

wojtek-t · 2018-10-12T06:16:12Z

I took one more final look and it also looks good to me now.

/lgtm
/retest

wojtek-t · 2018-10-12T08:48:48Z

/approve

k8s-ci-robot · 2018-10-12T08:48:56Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wangzhen127, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kube-controller-manager/OWNERS~~ [wojtek-t]
~~pkg/controller/nodelifecycle/OWNERS~~ [wojtek-t]
~~test/integration/scheduler/OWNERS~~ [wojtek-t]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

gmarek · 2018-10-24T10:46:04Z

@wangzhen127 @yujuhong @wojtek-t I took a look at this PR and IMO the change is technically correct. What I'm wondering about is the decision to not set the NodeReady condition to true when NC observes a lease. Can someone explain me the reasoning behind this decision? I feel that users might depend on NodeReady=True as a signal to detect when Nodes are up (e.g. kubeUp depended on that at some point).

wojtek-t · 2018-10-24T10:49:50Z

What I'm wondering about is the decision to not set the NodeReady condition to true when NC observes a lease. Can someone explain me the reasoning behind this decision? I feel that users might depend on NodeReady=True as a signal to detect when Nodes are up (e.g. kubeUp depended on that at some point).

I think that's a missing thing - if NodeReady != True and NC observed a Lease, I think we should set NodeReady = True. Would that solve all your concerns?

gmarek · 2018-10-24T10:56:21Z

Yup.

wojtek-t · 2018-10-24T10:59:43Z

@wangzhen127 - will you be able to add this? ^^

wangzhen127 · 2018-10-24T21:41:16Z

Yes, will create another PR for it.

gmarek · 2018-10-25T10:05:42Z

@wangzhen127 - thanks!

wangzhen127 · 2018-10-26T18:09:27Z

I thought more on this and also talked to @yujuhong offline. I think we should not let NC change NodeReady status to True if it observed a lease renewal. Because node lease is just a heartbeat signal. It says nothing about whether the node is ready. Let's look at all 4 cases below.

Case 1: When node just starts, there was no status:
To NC, it means: "I see you. I know you are reachable. So you are not unknown. But I don't know if you are ready. I will let you decide later".

Case 2, When node already has status as NodeReady == False
NC just know the node is reachable. NC does not know when node will become ready because kubelet could just keep deciding NodeReady == False if there is some error on the node.

Case 3, When node already has status as NodeReady == Unknown
In this case, if NC starts to get node lease, it may want to change the status. The problem is that NC does not know whether it should change to NodeReady == True or NodeReady == False. On the other hand, every time kubelet checks node status, ReadyCondition setter will try to update the status to either NodeReady == True or NodeReady == False, but not unknown. That will trigger a node status change, and kubelet then should report new NodeStatus to NC immediately.

Case 4, When node already has status as NodeReady == True
There is nothing NC needs to do.

Make periodic NodeStatus updates cheaper cherry-pick master, create lease api, kubernetes#64246 master, node lifecycle controller, kubernetes#69241 kubelet, kubernetes#66257 See merge request !184866

k8s-ci-robot requested review from aveshagarwal and brendandburns September 28, 2018 23:28

k8s-ci-robot requested a review from wojtek-t September 28, 2018 23:29

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Sep 30, 2018

k8s-ci-robot assigned wojtek-t Oct 1, 2018

wojtek-t reviewed Oct 1, 2018

View reviewed changes

wangzhen127 mentioned this pull request Oct 1, 2018

Rename node status to node health in NodeLifecycleController #69305

Merged

wangzhen127 commented Oct 1, 2018

View reviewed changes

k8s-ci-robot requested a review from cheftako October 1, 2018 20:07

k8s-ci-robot requested a review from yujuhong October 1, 2018 20:38

k8s-ci-robot added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Oct 1, 2018

wangzhen127 force-pushed the heartbeat branch from 56be9dc to 1e13ae1 Compare October 4, 2018 21:13

k8s-ci-robot removed the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 4, 2018

yujuhong reviewed Oct 9, 2018

View reviewed changes

wangzhen127 force-pushed the heartbeat branch from ae04ce7 to 2bf8515 Compare October 9, 2018 23:21

wangzhen127 commented Oct 9, 2018

View reviewed changes

wangzhen127 force-pushed the heartbeat branch from 2bf8515 to 1934437 Compare October 11, 2018 20:01

k8s-ci-robot assigned yujuhong Oct 11, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 11, 2018

wangzhen127 force-pushed the heartbeat branch from 1934437 to ec0cda7 Compare October 11, 2018 21:38

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 11, 2018

NodeLifecycleController treats node lease renewal as a heartbeat signal

e35d808

wangzhen127 force-pushed the heartbeat branch from ec0cda7 to e35d808 Compare October 11, 2018 23:24

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 12, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 12, 2018

k8s-ci-robot merged commit 23a8477 into kubernetes:master Oct 12, 2018

wangzhen127 deleted the heartbeat branch October 12, 2018 16:48

yujuhong mentioned this pull request Oct 16, 2018

[sig-cluster-lifecycle] Reboot [Disruptive] [Feature:Reboot] each node by ordering unclean reboot and ensure they function upon restartChanges #69786

Closed

NodeLifecycleController treats node lease renewal as a heartbeat signal #69241

NodeLifecycleController treats node lease renewal as a heartbeat signal #69241

Conversation

wangzhen127 commented Sep 28, 2018 • edited Loading

wangzhen127 commented Sep 28, 2018

k8s-ci-robot commented Sep 28, 2018

neolit123 commented Sep 30, 2018

wojtek-t commented Oct 1, 2018

wojtek-t left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangzhen127 left a comment

Choose a reason for hiding this comment

jennybuckley commented Oct 1, 2018

wangzhen127 commented Oct 1, 2018

wojtek-t commented Oct 4, 2018

wangzhen127 commented Oct 4, 2018

wangzhen127 commented Oct 9, 2018

wojtek-t commented Oct 9, 2018

yujuhong left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangzhen127 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yujuhong commented Oct 11, 2018

wangzhen127 commented Oct 11, 2018

wojtek-t commented Oct 12, 2018

wojtek-t commented Oct 12, 2018

k8s-ci-robot commented Oct 12, 2018

gmarek commented Oct 24, 2018

wojtek-t commented Oct 24, 2018

gmarek commented Oct 24, 2018

wojtek-t commented Oct 24, 2018

wangzhen127 commented Oct 24, 2018

gmarek commented Oct 25, 2018

wangzhen127 commented Oct 26, 2018

wangzhen127 commented Sep 28, 2018 •

edited

Loading