consumer: unexpected re-connection attempts #177
Comments
|
@imjoshholloway no, that isn't intended behavior, are you running the latest release of |
mreiferson
changed the title
|
@imjoshholloway Can you clarify exactly what was used to initialize this consumer, i.e. (a variation of |
|
Sorry for the delay in replying @jehiah and @mreiferson. As Yes... We're using a fork for extra logging. (https://github.com/hailocab/go-nsq). It's about 93 commits behind this repo but from having a quick look at the diff there doesn't seem to be anything that'd affect this behaviour. - That said, I'll pull the upstream changes and retry :-) In the instance I'm describing above we're using |
|
Can confirm even when using this repo directly we still get the same problem: |
|
Oh, when you use |
|
@mreiferson Just to confirm we definitely do call |
|
Right, but you're also using Basically, you shouldn't use both and we might need to make that more explicit. |
|
For the most recent test (with this repo) had you configured both |
|
... unless in your test it isn't calling |
|
Sorry @mreiferson, we're using both This is the code that's being triggered when a host is removed from our config. Normally this happens when an instance fails some healthchecks and is marked as unhealthy.: if hosts := diffHosts(s.subHosts, subHosts); len(hosts) > 0 {
log.Infof("Disconnecting from NSQ hosts: %v", hosts)
for _, host := range hosts {
err := s.consumer.DisconnectFromNSQD(host)
if err != nil && err != nsqlib.ErrNotConnected {
log.Warnf("Error disconnecting from NSQ host %s: %v", host, err)
}
}
}source: https://github.com/hailocab/service-layer/blob/master/nsq/subscriber.go#L175-L194 It's worth noting that this issue doesn't appear if the host is healthy and is simply removed from the config. Logs when node is healthy: Logs when node is unhealthy: Hopefully this helps :-) |
|
@imjoshholloway that was actually very helpful. What you are seeing is that on connection close, there is a loop started here that only exits when a successful connection is made. Calling DisconnectFromNSQD doesn't break that loop, but should. |
That loop should only execute if you don't have any @imjoshholloway can you see if |
|
|
do any of the |
|
No, neither |
|
@imjoshholloway can you reproduce this with vanilla |
|
@mreiferson sure, I'll try and do this over the weekend :-) |
|
@imjoshholloway appreciate all your help here, I haven't seen this behavior before so I'm trying to reduce the test case down to its most basic, so we can figure out what's going on. 😍 |
|
@mreiferson I've not been able to replicate this and it turns out that it's for good reason. There was another part of the codebase that was connecting directly to the NSQds directly and not updating when the config was changed 👎 Apologies for wasting your time with this. |
|
@imjoshholloway no worries, thanks for following up |
I'm currently investigating an issue where even after calling either
DisconnectFromNSQLookupdorDisconnectFromNSQDon a host that is already disconnected, re-connection attempts still occur.In our scenario we are consuming from 3 hosts: 10.0.0.1/10.0.0.2/10.0.0.3 and the host
10.0.0.1becomes unhealthy (service stopped) we remove it from the list of nodes and callDisconnectFromNSQLookupdorDisconnectFromNSQD.As you can see from the following log output, the connection to host
10.0.0.1was lost and re-connection attempts are made. The host is then removed from our list andDisconnectFromNSQLookupdis called. However, the re-connection attempts still occur:Is this intended/expected behaviour?
The text was updated successfully, but these errors were encountered: