Fix a race condition in a call to GD conn manager #1750
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
arcusfelis
reviewed
Mar 2, 2018
| Error -> | ||
| ?INFO_MSG("event=outgoing_conn_start_progress,server='~s'," | ||
| "state='failed',error='~p'", [Server, Error]), | ||
| Error | ||
| end. | ||
|
|
||
| -spec get_connection(Server :: jid:lserver()) -> pid(). | ||
| get_connection(Server) -> |
There was a problem hiding this comment.
Can be still relevant
https://github.com/esl/MongooseIM/pull/1660/files#r165371814
I am feeling that we would find more bugs in this part of GD.
There was a problem hiding this comment.
Since it was mentioned in another PR, maybe it should rather be resolved there? But +1 for adding some comment about the behaviour.
arcusfelis
reviewed
Mar 2, 2018
| get_connection(Server) -> | ||
| mod_global_distrib_server_mgr:get_connection(Server). | ||
| case catch mod_global_distrib_server_mgr:get_connection(Server) of |
There was a problem hiding this comment.
I would love to see a test for this fix.
There was a problem hiding this comment.
I'll add a TODO comment here because we need to come up with a uniform method of simulating race conditions. Without it, I would be forced to use meck+RPC which is evil and failure-prone.
arcusfelis
reviewed
Mar 2, 2018
| ?ERROR_MSG("event=cannot_acquire_outgoing_connection,server='~s'", [Server]), | ||
| throw({error, {cannot_acquire_outgoing_connection, Server}}); | ||
| get_connection(Server, RetriesLeft) -> | ||
| case mod_global_distrib_server_sup:get_connection(Server) of |
There was a problem hiding this comment.
Still not covered with tests.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses an issue observed in load tests, where sometimes a call to
get_connectioncrashed withnoprocexception (that applies to connection manager process).It may happen if a process starts outgoing connections supervisor and second one tries to access the pool in a state, when supervisor is already running but manager (child of the supervisor) is not (yet).
Also please check the comments in the code.
Load tests confirm that this branch fixes the issue.