Fix race condition in Global Distribution packet retransmission

[fenek]

This PR fixes #1683 and obsoletes #1684

The race condition was provoked by test_in_order_messages_on_multiple_connections_with_bounce test case. It manifested under following conditions:

1. Due to lack of recipient mapping, e.g. 4 messages were stored in bounce storage
2. Mapping appeared
3. 5th message got sent and it triggered known_recipient hook, which in turn triggered retransmission of stored items
4. For every retransmitted item, full routing logic is executed, so each time the recipient mapping is retrieved
5. Test process may remove (as a part of test case) recipient mapping in the middle of retransmission, so only some of the items reach recipient.
6. Code returns from hook execution to a point, where the mapping is already retrieved and the message 5 gets delivered anyway
7. Result: messages delivered out of order

I've decided not to create new test case nor make the existing one more predictable, as with 100 messages sent it reproduces the issue fairly frequently and there is very little chance it won't test what it is supposed to. I mean an edge case when e.g. all 100 messages hit a window where recipient mapping is available due to test-c2s concurrency.

Note: This version was built on Travis about 6 times with full set of jobs and test_in_order_messages_on_multiple_connections_with_bounce was never a reason for crash.

[arcusfelis]

pamparam, don't like any concurrency in this quantum world^W^W^W any tests.

[arcusfelis]

Can we have it on a separate line?

TargetHost = get_metadata(Acc, target_host_override, undefined),
case lookup_recipients_host(TargetHost, To, LocalHost, GlobalHost) of

It still will be two lines long :D

[arcusfelis]

target_host_override should be documented, I have no idea what is for just by looking at the code.

[arcusfelis]

and why it can be undefined should be described too.

[arcusfelis]

comments!

[arcusfelis]

good