Failed to connect to rabbitmq many times during openstack deployemnt

[wenchma]

After delopy openstack successfully, open neutron server log
Expected result: In neutron server.log, it should to have such log to tell user you have to connect to AMQP server. " Connected to AMQP server on 127.0.0.1:5671". It should not have errors.
Actual result: In neutron server.log, it try many times to conenct to AMQP server, 10 mintues later, it connect to AMQP server finally.
Notes: Not only neutron have such issue, same as nova.etc.

/var/log/neutron/server.log

2014-08-15 01:36:33.112 19576 WARNING neutron.agent.securitygroups_rpc [-] Driver configuration doesn't match with enable_security_group
2014-08-15 01:36:33.187 19576 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 127.0.0.1:5671
2014-08-15 01:36:33.198 19576 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 127.0.0.1:5671 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2014-08-15 01:36:34.200 19576 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2014-08-15 01:36:35.202 19576 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 127.0.0.1:5671
2014-08-15 01:36:35.214 19576 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 127.0.0.1:5671 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.
2014-08-15 01:36:38.214 19576 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2014-08-15 01:36:39.222 19576 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 127.0.0.1:5671
2014-08-15 01:36:39.228 19576 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 127.0.0.1:5671 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.
2014-08-15 01:36:44.232 19576 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2014-08-15 01:36:45.234 19576 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 127.0.0.1:5671
2014-08-15 01:36:45.240 19576 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 127.0.0.1:5671 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.

[wenchma]

It is due to delay to restart rabbitmq service, should notify the service to restart immediately to fix this issue.
Related line:
https://github.com/kennonkwok/rabbitmq/blob/master/recipes/default.rb#L157

[michaelklishin]

While it may be desirable to notify with :immediate in OpenStack installations, you don't want to do it in every case. That said, RabbitMQ restarts are quick (unless you have broken message indices and millions of messages in durable queues), so it may be fine to do this for config changes.

@wenchma why don't you submit a pull request?

[wenchma]

@michaelklishin I have sumitted a pull request, #154 ,
can you have a review or make others also review ? thanks!

[jjasghar]

These issues have been merged.

[zhang-jianwei]

2018-04-18 07:28:58.156 3125 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 172.19.14.6:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.
2018-04-18 07:28:58.231 3142 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 172.19.14.6:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.
2018-04-18 07:29:02.382 2834 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 172.19.14.8:5673
2018-04-18 07:29:02.415 3143 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 172.19.14.8:5673
2018-04-18 07:29:02.477 2834 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 172.19.14.7:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.

rabbitmq install as cluster, when restart the cluster, the openstack service will reconnect it, but neutron metadata service sometimes unreachable error, when try connection every node many times, it connected. this issue lead to neutron service instability.

[michaelklishin]

This is not a support forum. "ECONNREFUSED" is pretty unambiguous: the client failed to open a TCP connection. Make sure that the broker is running on the target node, then follow the methodology recommended by RabbitMQ docs.