Fatal node sync call stops subsequent calls

[prd-fox]

When the PartyInfo poller makes requests to other nodes there are two classes of exceptions that can occur, general connection issues and then other, more serious issues.

A simple connection issue gets logged and passed over, as this is expected to happen from time to time, and all other exception types were presumed to be ephemeral and eventually resolve. We throw this error back up the call stack to handle later.

This means that when this error occurs, all other parties we wanted to sync with after the current do not happen (as the exception stops the loop). Sometimes, the error will never resolve, which effectively stops the node fully syncing with the network.

Any error, whether it is a general connection issue or something more serious, should not stop the processing of other calls.

[prd-fox]

One example when this can occur is inside a Docker network, where the target does not exist.

Generally these scenarios are very rare, and workarounds such as removing the fault peer entry from the config will enable at least a one time sync to the network.