kubernetes: Received event spamming? #449

jonaz · 2016-06-09T09:38:20Z

I get this every second. I dont think traefik should listen to this event or even subscribe to it since its the internal leader election when running multiple masters.

time="2016-06-09T09:35:38Z" level=debug msg="Received event from kubernetes map[object:map[metadata:map[resourceVersion:4596497 creationTimestamp:2016-06-07T10:40:19Z annotations:map[control-plane.alpha.kubernetes.io/leader:{\"holderIdentity\":\"utv-kube01\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2016-06-07T10:40:19Z\",\"renewTime\":\"2016-06-09T09:35:38Z\",\"leaderTransitions\":0}] name:kube-controller-manager namespace:kube-system selfLink:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager uid:3b0f4e50-2c9c-11e6-acdb-005056885071] subsets:[] kind:Endpoints apiVersion:v1] type:MODIFIED]"

This might be a kubernetes issue

When i tried manually watching http://localhost:8001/api/v1/endpoints?watch=true i get:

{"type":"MODIFIED","object":{"kind":"Endpoints","apiVersion":"v1","metadata":{"name":"kube-scheduler","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/endpoints/kube-scheduler","uid":"3a9db9fc-2c9c-11e6-acdb-005056885071","resourceVersion":"4597683","creationTimestamp":"2016-06-07T10:40:19Z","annotations":{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"utv-kube01\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2016-06-07T10:40:19Z\",\"renewTime\":\"2016-06-09T09:46:21Z\",\"leaderTransitions\":0}"}},"subsets":[]}}
{"type":"MODIFIED","object":{"kind":"Endpoints","apiVersion":"v1","metadata":{"name":"kube-controller-manager","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/endpoints/kube-controller-manager","uid":"3b0f4e50-2c9c-11e6-acdb-005056885071","resourceVersion":"4597685","creationTimestamp":"2016-06-07T10:40:19Z","annotations":{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"utv-kube01\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2016-06-07T10:40:19Z\",\"renewTime\":\"2016-06-09T09:46:21Z\",\"leaderTransitions\":0}"}},"subsets":[]}}
{"type":"MODIFIED","object":{"kind":"Endpoints","apiVersion":"v1","metadata":{"name":"kube-scheduler","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/endpoints/kube-scheduler","uid":"3a9db9fc-2c9c-11e6-acdb-005056885071","resourceVersion":"4597687","creationTimestamp":"2016-06-07T10:40:19Z","annotations":{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"utv-kube01\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2016-06-07T10:40:19Z\",\"renewTime\":\"2016-06-09T09:46:23Z\",\"leaderTransitions\":0}"}},"subsets":[]}}
{"type":"MODIFIED","object":{"kind":"Endpoints","apiVersion":"v1","metadata":{"name":"kube-controller-manager","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/endpoints/kube-controller-manager","uid":"3b0f4e50-2c9c-11e6-acdb-005056885071","resourceVersion":"4597688","creationTimestamp":"2016-06-07T10:40:19Z","annotations":{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"utv-kube01\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2016-06-07T10:40:19Z\",\"renewTime\":\"2016-06-09T09:46:23Z\",\"leaderTransitions\":0}"}},"subsets":[]}}
{"type":"MODIFIED","object":{"kind":"Endpoints","apiVersion":"v1","metadata":{"name":"kube-scheduler","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/endpoints/kube-scheduler","uid":"3a9db9fc-2c9c-11e6-acdb-005056885071","resourceVersion":"4597689","creationTimestamp":"2016-06-07T10:40:19Z","annotations":{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"utv-kube01\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2016-06-07T10:40:19Z\",\"renewTime\":\"2016-06-09T09:46:25Z\",\"leaderTransitions\":0}"}},"subsets":[]}}
{"type":"MODIFIED","object":{"kind":"Endpoints","apiVersion":"v1","metadata":{"name":"kube-controller-manager","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/endpoints/kube-controller-manager","uid":"3b0f4e50-2c9c-11e6-acdb-005056885071","resourceVersion":"4597690","creationTimestamp":"2016-06-07T10:40:19Z","annotations":{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"utv-kube01\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2016-06-07T10:40:19Z\",\"renewTime\":\"2016-06-09T09:46:25Z\",\"leaderTransitions\":0}"}},"subsets":[]}}
{"type":"MODIFIED","object":{"kind":"Endpoints","apiVersion":"v1","metadata":{"name":"kube-scheduler","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/endpoints/kube-scheduler","uid":"3a9db9fc-2c9c-11e6-acdb-005056885071","resourceVersion":"4597692","creationTimestamp":"2016-06-07T10:40:19Z","annotations":{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"utv-kube01\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2016-06-07T10:40:19Z\",\"renewTime\":\"2016-06-09T09:46:27Z\",\"leaderTransitions\":0}"}},"subsets":[]}}
{"type":"MODIFIED","object":{"kind":"Endpoints","apiVersion":"v1","metadata":{"name":"kube-controller-manager","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/endpoints/kube-controller-manager","uid":"3b0f4e50-2c9c-11e6-acdb-005056885071","resourceVersion":"4597693","creationTimestamp":"2016-06-07T10:40:19Z","annotations":{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"utv-kube01\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2016-06-07T10:40:19Z\",\"renewTime\":\"2016-06-09T09:46:27Z\",\"leaderTransitions\":0}"}},"subsets":[]}}

The only thing changed is renewTime.
So i guess traefik could filter out those messages.

I found a issue upstream: kubernetes/kubernetes#23812

The text was updated successfully, but these errors were encountered:

emilevauge · 2016-06-15T21:25:27Z

@errm: did you get something similar on your platform ?

olavmrk · 2016-06-16T08:55:52Z

I got the same problem here.

It appears that the kubernetes provider sends a new configuration for every event, even though the event doesn't actually change the configuration. Since these events arrive very frequently (several times each second), the result is that a new configuration is never applied, since it wants a two second pause in the configuration updates:

time="2016-06-16T08:54:31Z" level=debug msg="Last kubernetes config received less than 2s, waiting..."

mikespokefire · 2016-06-16T11:24:11Z

Also encountering this issue myself. Maybe instead of waiting for a 2 second pause from kubernetes, is traefik aware of the last time it updated its own config and we can use that instead?

jonaz · 2016-06-16T12:33:22Z

Or just ignore events with annotation control-plane.alpha.kubernetes.io/leader

errm · 2016-06-16T15:41:34Z

@jonaz @mikespokefire which version of kubernetes are you currently running?

errm · 2016-06-16T15:49:04Z

I am not seeing these, I think they are related to the --leader-elect flag on kube-scheduler or kube-controller-manager/ it seems that using endpoints like this for leader-election also causes issues for kube-proxy so it might be fixed upstream.

Currently we don't check what is in a watched event, rather just using it as a trigger to rebuild the whole config. I guess the best thing would be to filter these events to just look for changes to an IP or port...

jonaz · 2016-06-16T16:48:44Z

@errm We're running latest stable which is v1.2.4

And yes we have --leader-elect=true following best practice guide from coreos: https://coreos.com/kubernetes/docs/latest/deploy-master.html

mikespokefire · 2016-06-16T21:50:17Z

We are using v1.2.3. Looking further into the problem, it seems to be the
controller manager with --leader-elect for us too.
On 16 Jun 2016 5:49 p.m., "jonaz" [email protected] wrote:

@errm https://github.com/errm We're running latest stable which is
v1.2.4

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#449 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAT9ITdzQnXcFgIxhJKqjSzVm72J2wRJks5qMX5xgaJpZM4IxytR
.

nlf · 2016-06-17T20:56:35Z

I get this spam with leader election enabled also

errm · 2016-06-20T10:23:57Z

Unless you are running a HA master on multiple nodes you should be able to safely remove --leader-elect so that may be a work around for some.

I have a few ideas for fixing this...

Cache templateObjects in the kubernetes provider so we could test if the config has changed before it is submitted to traefik.
Deserialise events as we receive them, and try to do something smart to work out if we need to do an update.

emilevauge · 2016-06-20T11:29:35Z

See my comment #448 (comment)

jonaz mentioned this issue Jun 9, 2016

kubernetes: backends not updated when i scale replication controller? #448

Closed

emilevauge added the investigation-needed label Jun 15, 2016

whs mentioned this issue Jun 18, 2016

etcd tries to connect with TLS even with --etcd.tls=false #456

Closed

emilevauge mentioned this issue Jun 22, 2016

Fix spamming events in listenProviders #477

Merged

vdemeester closed this as completed in #477 Jun 23, 2016

emilevauge mentioned this issue Nov 14, 2016

Still a memory leak with k8s - 1.1 RC4 #844

Closed

ldez removed the investigation-needed label May 12, 2017

traefik locked and limited conversation to collaborators Sep 1, 2019

traefiker added the status/5-frozen-due-to-age label Sep 1, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubernetes: Received event spamming? #449

kubernetes: Received event spamming? #449

jonaz commented Jun 9, 2016 •

edited

Loading

emilevauge commented Jun 15, 2016

olavmrk commented Jun 16, 2016

mikespokefire commented Jun 16, 2016

jonaz commented Jun 16, 2016

errm commented Jun 16, 2016

errm commented Jun 16, 2016

jonaz commented Jun 16, 2016 •

edited

Loading

mikespokefire commented Jun 16, 2016

nlf commented Jun 17, 2016

errm commented Jun 20, 2016

emilevauge commented Jun 20, 2016

kubernetes: Received event spamming? #449

kubernetes: Received event spamming? #449

Comments

jonaz commented Jun 9, 2016 • edited Loading

emilevauge commented Jun 15, 2016

olavmrk commented Jun 16, 2016

mikespokefire commented Jun 16, 2016

jonaz commented Jun 16, 2016

errm commented Jun 16, 2016

errm commented Jun 16, 2016

jonaz commented Jun 16, 2016 • edited Loading

mikespokefire commented Jun 16, 2016

nlf commented Jun 17, 2016

errm commented Jun 20, 2016

emilevauge commented Jun 20, 2016

jonaz commented Jun 9, 2016 •

edited

Loading

jonaz commented Jun 16, 2016 •

edited

Loading