client has run out of available brokers to talk to (Is your cluster reachable?)

[boyunzheyue2]

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.
Sarama Version: 46cf3e2
Kafka Version: kafka:2.12-1.0.1
Go Version: 1.9.2

Configuration

conf.Producer.RequiredAcks = sarama.WaitForAll
conf.Producer.Retry.Max = 3
conf.Producer.Return.Successes = true
conf.Producer.Timeout = time.Second * 5

Logs

[sarama] 2018/08/06 08:28:12 Initializing new client
[sarama] 2018/08/06 08:28:12 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama] 2018/08/06 08:28:12 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama] 2018/08/06 08:28:12 client/metadata fetching metadata for all topics from broker kafka-default-2.kafka-default.component.svc.cluster.local:9092
[sarama] 2018/08/06 08:28:12 Connected to broker at kafka-default-2.kafka-default.component.svc.cluster.local:9092 (unregistered)
[sarama] 2018/08/06 08:28:12 client/brokers registered new broker #2 at 172.20.25.94:9092
[sarama] 2018/08/06 08:28:12 client/brokers registered new broker #1 at 172.20.25.92:9092
[sarama] 2018/08/06 08:28:12 client/brokers registered new broker #0 at 172.20.25.93:9092
[sarama] 2018/08/06 08:28:12 Successfully initialized new client
[sarama] 2018/08/06 08:28:12 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama] 2018/08/06 08:28:12 Initializing new client
[sarama] 2018/08/06 08:28:12 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama] 2018/08/06 08:28:12 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama] 2018/08/06 08:28:12 client/metadata fetching metadata for all topics from broker 172.20.2.85:32000
[sarama] 2018/08/06 08:28:42 Failed to connect to broker 172.20.2.85:32000: dial tcp 172.20.2.85:32000: i/o timeout
[sarama] 2018/08/06 08:28:42 client/metadata got error from broker while fetching metadata: dial tcp 172.20.2.85:32000: i/o timeout
[sarama] 2018/08/06 08:28:42 client/metadata no available broker to send metadata request to
[sarama] 2018/08/06 08:28:42 client/brokers resurrecting 1 dead seed brokers
[sarama] 2018/08/06 08:28:42 client/metadata retrying after 250ms... (3 attempts remaining)
[sarama] 2018/08/06 08:28:42 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama] 2018/08/06 08:28:42 client/metadata fetching metadata for all topics from broker 172.20.2.85:32000
[sarama] 2018/08/06 08:29:12 Failed to connect to broker 172.20.2.85:32000: dial tcp 172.20.2.85:32000: i/o timeout
[sarama] 2018/08/06 08:29:12 client/metadata got error from broker while fetching metadata: dial tcp 172.20.2.85:32000: i/o timeout
[sarama] 2018/08/06 08:29:12 client/metadata no available broker to send metadata request to
[sarama] 2018/08/06 08:29:12 client/brokers resurrecting 1 dead seed brokers
[sarama] 2018/08/06 08:29:12 client/metadata retrying after 250ms... (2 attempts remaining)
[sarama] 2018/08/06 08:29:12 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama] 2018/08/06 08:29:12 client/metadata fetching metadata for all topics from broker 172.20.2.85:32000
^[[sarama] 2018/08/06 08:29:42 Failed to connect to broker 172.20.2.85:32000: dial tcp 172.20.2.85:32000: i/o timeout
[sarama] 2018/08/06 08:29:42 client/metadata got error from broker while fetching metadata: dial tcp 172.20.2.85:32000: i/o timeout
[sarama] 2018/08/06 08:29:42 client/metadata no available broker to send metadata request to
[sarama] 2018/08/06 08:29:42 client/brokers resurrecting 1 dead seed brokers
[sarama] 2018/08/06 08:29:42 client/metadata retrying after 250ms... (1 attempts remaining)
[sarama] 2018/08/06 08:29:42 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama] 2018/08/06 08:29:42 client/metadata fetching metadata for all topics from broker 172.20.2.85:32000
[sarama] 2018/08/06 08:30:12 Failed to connect to broker 172.20.2.85:32000: dial tcp 172.20.2.85:32000: i/o timeout
[sarama] 2018/08/06 08:30:12 client/metadata got error from broker while fetching metadata: dial tcp 172.20.2.85:32000: i/o timeout
[sarama] 2018/08/06 08:30:12 client/metadata no available broker to send metadata request to
[sarama] 2018/08/06 08:30:12 client/brokers resurrecting 1 dead seed brokers
[sarama] 2018/08/06 08:30:12 Closing Client

Problem Description

hi:
there are three instance in my service, two of them can connect broker 172.20.2.85:32000 properly,but one of them,only one of them failed to connect,always timeout.

[eapache]

If two can connect and one cannot, then the problem is not with Sarama but probably with the network configuration of the one that can't connect.