Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

destination: Frequent redundant discovery updates #8677

Closed
olix0r opened this issue Jun 16, 2022 · 4 comments
Closed

destination: Frequent redundant discovery updates #8677

olix0r opened this issue Jun 16, 2022 · 4 comments
Labels
area/controller priority/triage wontfix

Comments

@olix0r
Copy link
Member

olix0r commented Jun 16, 2022

#8666 includes logs that show the destination controller serving redundant
discovery updates roughly every 10 seconds (though sometimes up to 60s apart):

[   390.340324s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   400.338156s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   410.359043s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   420.344174s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   430.338443s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   440.343410s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   470.336826s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   480.343436s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   490.339913s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   500.345823s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   510.341853s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   520.337896s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   550.335535s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   560.345701s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   610.338643s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   620.343356s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   630.338879s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   640.338325s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   670.336862s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   690.339669s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   730.335908s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   750.339219s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   770.336963s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   790.335917s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })
[   850.341237s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.7.14:50170}:proxy{addr=10.0.115.236:5432}:tcp: linkerd_proxy_discover::from_resolve: Changed change=Insert(10.244.2.2:5432, Endpoint { addr: Remote(ServerAddr(10.244.2.2:5432)), tls: Some(ClientTls { server_id: ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local")), alpn: Some(AlpnProtocols("transport.l5d.io/v1")) }), metadata: Metadata { labels: {"control_plane_ns": "linkerd", "namespace": "multi", "pod": "postgres-instance1-5f7t-0", "service": "postgres-ha", "serviceaccount": "postgres-instance", "statefulset": "postgres-instance1-5f7t"}, protocol_hint: Http2, opaque_transport_port: Some(4143), identity: Some(ServerId(Name("postgres-instance.multi.serviceaccount.identity.linkerd.cluster.local"))), authority_override: None }, logical_addr: Some(LogicalAddr(postgres-ha.multi.svc.cluster.local:5432)), protocol: (), opaque_protocol: false })

While these updates should not cause the logic bug described in the issue, they
do seem likely to cause unnececessary work: the balancer endpoint will be
replaced each time an update is processed, causing new connections to be
created. In large clusters, this is probably taxing on the destination
controller as it is forced to perform unnecessary I/O with all of its clients.

Why are these redundant updates being sent? Are they preventable?
@olix0r olix0r added this to the stable-2.12.0 milestone Jun 16, 2022
@olix0r olix0r added the priority/P1 Planned for Release label Jun 16, 2022
@olix0r
Copy link
Member Author

olix0r commented Jun 16, 2022

FWIW, I'm not seeing this behavior on my local k3d cluster with emojivoto...

@olix0r olix0r mentioned this issue Jun 16, 2022
Closed
@johnswarbrick
Copy link

johnswarbrick commented Jun 16, 2022 •
edited
Loading

Hi @olix0r, there is very little information in logs -fn linkerd deploy/linkerd-destination destination

Defaulted container "linkerd-proxy" out of: linkerd-proxy, destination, sp-validator, policy, linkerd-init (init)
time="2022-06-13T20:20:59Z" level=info msg="Found pre-existing key: /var/run/linkerd/identity/end-entity/key.p8"
time="2022-06-13T20:20:59Z" level=info msg="Found pre-existing CSR: /var/run/linkerd/identity/end-entity/csr.der"
[     0.001206s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.001593s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.001603s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.001605s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.001606s]  INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
[     0.001607s]  INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
[     0.001609s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.001610s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via localhost:8086
[     0.010845s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local

@olix0r
Copy link
Member Author

olix0r commented Jun 16, 2022

@johnswarbrick Sorry, I may have given you a slightly incorrect command. These logs are from the proxy, but I'm curious to see the logs from the destination controller. I think this should work:

kubectl logs -fn linkerd deploy/linkerd-destination --container=destination

hawkw added a commit to linkerd/linkerd2-proxy that referenced this issue Jun 17, 2022
@hawkw
update Tower to 0.4.13 to fix load balancer panic
c201375 
Tower [v0.4.13] includes a fix for a bug in the `tower::ready_cache`
module, tower-rs/tower#415. The `ready_cache` module is used internally
in Tower's load balancer. This bug resulted in panics in the proxy
(linkerd/linkerd2#8666, linkerd/linkerd2#6086) in cases where the
Destination service sends a very large number of service discovery
updates (see linkerd/linkerd2#8677).

This commit updates the proxy's dependency on `tower` to 0.4.13, to
ensure that this bugfix is picked up.

Fixes linkerd/linkerd2#8666
Fixes linkerd/linkerd2#6086

[v0.4.13]: https://github.com/tower-rs/tower/releases/tag/tower-0.4.13
@hawkw hawkw mentioned this issue Jun 17, 2022
Merged
olix0r pushed a commit to linkerd/linkerd2-proxy that referenced this issue Jun 17, 2022
@hawkw
update Tower to 0.4.13 to fix load balancer panic (#1758)
16ec721 
Tower [v0.4.13] includes a fix for a bug in the `tower::ready_cache`
module, tower-rs/tower#415. The `ready_cache` module is used internally
in Tower's load balancer. This bug resulted in panics in the proxy
(linkerd/linkerd2#8666, linkerd/linkerd2#6086) in cases where the
Destination service sends a very large number of service discovery
updates (see linkerd/linkerd2#8677).

This commit updates the proxy's dependency on `tower` to 0.4.13, to
ensure that this bugfix is picked up.

Fixes linkerd/linkerd2#8666
Fixes linkerd/linkerd2#6086

[v0.4.13]: https://github.com/tower-rs/tower/releases/tag/tower-0.4.13
olix0r added a commit to linkerd/linkerd2-proxy that referenced this issue Jun 18, 2022
@olix0r
Dedupe discovery updates
5984690 
The proxy can receive redundant discovery updates. When this occurs, it
causes the balancer to churn, replacing an endpoint stack (and
therefore needlessly dropping a connection).

This change updates the discovery module to keep clones of the
discovered endpoint metadata, so that updated values can be compared to
eliminate duplicate updates.

Relates to linkerd/linkerd2#8677

Signed-off-by: Oliver Gould <[email protected]>
@olix0r olix0r mentioned this issue Jun 18, 2022
Merged
olix0r added a commit to linkerd/linkerd2-proxy that referenced this issue Jun 18, 2022
@olix0r
Dedupe discovery updates
9fa15b7 
The proxy can receive redundant discovery updates. When this occurs, it
causes the balancer to churn, replacing an endpoint stack (and
therefore needlessly dropping a connection).

This change updates the discovery module to keep clones of the
discovered endpoint metadata, so that updated values can be compared to
eliminate duplicate updates.

Relates to linkerd/linkerd2#8677

Signed-off-by: Oliver Gould <[email protected]>
olix0r added a commit to linkerd/linkerd2-proxy that referenced this issue Jun 20, 2022
@olix0r
Dedupe discovery updates (#1759)
ff47f8d 
The proxy can receive redundant discovery updates. When this occurs, it
causes the balancer to churn, replacing an endpoint stack (and
therefore needlessly dropping a connection).

This change updates the discovery module to keep clones of the
discovered endpoint metadata, so that updated values can be compared to
eliminate duplicate updates.

Relates to linkerd/linkerd2#8677
@adleong adleong removed this from the stable-2.12.0 milestone Jun 30, 2022
@adleong adleong added priority/triage and removed priority/P1 Planned for Release labels Jun 30, 2022
hawkw added a commit to linkerd/linkerd2-proxy that referenced this issue Jun 30, 2022
@hawkw
update Tower to 0.4.13 to fix load balancer panic (#1758)
c4efeee 
Tower [v0.4.13] includes a fix for a bug in the `tower::ready_cache`
module, tower-rs/tower#415. The `ready_cache` module is used internally
in Tower's load balancer. This bug resulted in panics in the proxy
(linkerd/linkerd2#8666, linkerd/linkerd2#6086) in cases where the
Destination service sends a very large number of service discovery
updates (see linkerd/linkerd2#8677).

This commit updates the proxy's dependency on `tower` to 0.4.13, to
ensure that this bugfix is picked up.

Fixes linkerd/linkerd2#8666
Fixes linkerd/linkerd2#6086

[v0.4.13]: https://github.com/tower-rs/tower/releases/tag/tower-0.4.13
@hawkw hawkw mentioned this issue Jun 30, 2022
Closed
hawkw added a commit to linkerd/linkerd2-proxy that referenced this issue Jun 30, 2022
@hawkw
update Tower to 0.4.13 to fix load balancer panic (#1758)
17db1cf 
Tower [v0.4.13] includes a fix for a bug in the `tower::ready_cache`
module, tower-rs/tower#415. The `ready_cache` module is used internally
in Tower's load balancer. This bug resulted in panics in the proxy
(linkerd/linkerd2#8666, linkerd/linkerd2#6086) in cases where the
Destination service sends a very large number of service discovery
updates (see linkerd/linkerd2#8677).

This commit updates the proxy's dependency on `tower` to 0.4.13, to
ensure that this bugfix is picked up.

Fixes linkerd/linkerd2#8666
Fixes linkerd/linkerd2#6086

[v0.4.13]: https://github.com/tower-rs/tower/releases/tag/tower-0.4.13
@hawkw hawkw mentioned this issue Jun 30, 2022
Merged
hawkw added a commit to linkerd/linkerd2-proxy that referenced this issue Jun 30, 2022
@hawkw
update Tower to 0.4.13 to fix load balancer panic (#1758)
b239e8e 
Tower [v0.4.13] includes a fix for a bug in the `tower::ready_cache`
module, tower-rs/tower#415. The `ready_cache` module is used internally
in Tower's load balancer. This bug resulted in panics in the proxy
(linkerd/linkerd2#8666, linkerd/linkerd2#6086) in cases where the
Destination service sends a very large number of service discovery
updates (see linkerd/linkerd2#8677).

This commit updates the proxy's dependency on `tower` to 0.4.13, to
ensure that this bugfix is picked up.

Fixes linkerd/linkerd2#8666
Fixes linkerd/linkerd2#6086

[v0.4.13]: https://github.com/tower-rs/tower/releases/tag/tower-0.4.13
hawkw pushed a commit to linkerd/linkerd2-proxy that referenced this issue Jun 30, 2022
@olix0r @hawkw
Dedupe discovery updates (#1759)
1f03230 
The proxy can receive redundant discovery updates. When this occurs, it
causes the balancer to churn, replacing an endpoint stack (and
therefore needlessly dropping a connection).

This change updates the discovery module to keep clones of the
discovered endpoint metadata, so that updated values can be compared to
eliminate duplicate updates.

Relates to linkerd/linkerd2#8677
@hawkw hawkw mentioned this issue Jun 30, 2022
Merged
hawkw added a commit to linkerd/linkerd2-proxy that referenced this issue Jun 30, 2022
@hawkw
update Tower to 0.4.13 to fix load balancer panic (#1758)
1817b3f 
Tower [v0.4.13] includes a fix for a bug in the `tower::ready_cache`
module, tower-rs/tower#415. The `ready_cache` module is used internally
in Tower's load balancer. This bug resulted in panics in the proxy
(linkerd/linkerd2#8666, linkerd/linkerd2#6086) in cases where the
Destination service sends a very large number of service discovery
updates (see linkerd/linkerd2#8677).

This commit updates the proxy's dependency on `tower` to 0.4.13, to
ensure that this bugfix is picked up.

Fixes linkerd/linkerd2#8666
Fixes linkerd/linkerd2#6086

[v0.4.13]: https://github.com/tower-rs/tower/releases/tag/tower-0.4.13
hawkw pushed a commit to linkerd/linkerd2-proxy that referenced this issue Jun 30, 2022
@olix0r @hawkw
Dedupe discovery updates (#1759)
ebcf7cb 
The proxy can receive redundant discovery updates. When this occurs, it
causes the balancer to churn, replacing an endpoint stack (and
therefore needlessly dropping a connection).

This change updates the discovery module to keep clones of the
discovered endpoint metadata, so that updated values can be compared to
eliminate duplicate updates.

Relates to linkerd/linkerd2#8677
@stale
Copy link

stale bot commented Sep 28, 2022

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Sep 28, 2022
@stale stale bot closed this as completed Oct 12, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 12, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area/controller priority/triage wontfix
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@olix0r @adleong @johnswarbrick