TLS error on send crashes proactor

[anschweitzer]

TLS recently turned for ATN and scada; Just got this ssl.SSLEOFError: EOF occurred in violation of protocol on both sides, though at slightly different times.

This crashes the main thread so proactor restarts, rather than just closing and re-opening the paho connection.

Issues:

1. proactor should not restart; it should catch this error and restart the client.
2. I don't know why this happened. I've seen it twice on ATN.

The ATN exception:

Traceback (most recent call last):
  File "/Users/Andrew/git/gridworks/venv/lib/python3.11/site-packages/gwproactor/proactor_implementation.py", line 227, in process_messages
...
  File "/Users/Andrew/git/gridworks/venv/lib/python3.11/site-packages/gwproactor/proactor_implementation.py", line 434, in _process_mqtt_message
    self._links.send_ack(
...
  File "/Users/Andrew/git/gridworks/venv/lib/python3.11/site-packages/gwproactor/links/mqtt.py", line 143, in publish
    return self._client.publish(topic, payload, qos)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/Andrew/git/gridworks/venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 1257, in publish
    rc = self._send_publish(
...
  File "/Users/Andrew/git/gridworks/venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 649, in _sock_send
    return self._sock.send(buf)
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/Andrew/.pyenv/versions/3.11.0/lib/python3.11/ssl.py", line 1210, in send
    return self._sslobj.write(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2393)

The scada exception:

Traceback (most recent call last):
  File "/home/pi/.local/lib/python3.10/site-packages/gwproactor/proactor_implementation.py", line 226, in process_messages
    await self.process_message(message)
...
  File "/home/pi/.local/lib/python3.10/site-packages/gwproactor/links/link_manager.py", line 394, in _recv_activated
    self.upload_pending_events()
...
  File "/home/pi/.local/lib/python3.10/site-packages/gwproactor/links/mqtt.py", line 143, in publish
    return self._client.publish(topic, payload, qos)
  File "/home/pi/.local/lib/python3.10/site-packages/paho/mqtt/client.py", line 1257, in publish
    rc = self._send_publish(
...
  File "/home/pi/.local/lib/python3.10/site-packages/paho/mqtt/client.py", line 2464, in _packet_write
    write_length = self._sock_send(
  File "/home/pi/.local/lib/python3.10/site-packages/paho/mqtt/client.py", line 649, in _sock_send
    return self._sock.send(buf)
  File "/usr/local/lib/python3.10/ssl.py", line 1205, in send
    return self._sslobj.write(data)
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2396

[anschweitzer]

This might be fixed in paho 2.0. See this paho issue.

Updating to 2.0 will require some changes (and probably some testing)