Pyzeebe will never time out a connection to a gateway with problems

[kbakk]

Describe the bug
If the gateway fails in a certain way, while a connection has been opened to it, Pyzeebe will not time out.

The issue applies both to the client, and the worker.

For the worker, the issue is noticed by it hanging, if it experience loss of connectivity to gateway.

To Reproduce

Start the gateway and brokes (use https://github.com/camunda-community-hub/zeebe-docker-compose/blob/0.26.0/standalone-gateway/docker-compose.yml) – don't run any workers against this cluster.

docker-compose up -d

In a second terminal, run the following code:

import logging
from pyzeebe import ZeebeClient
logging.basicConfig(level=logging.DEBUG, format="%(asctime)s %(levelname)s [%(thread)d|%(threadName)s] %(filename)s:%(lineno)d: %(message)s")


zeebe_client = ZeebeClient(hostname="localhost", port=26500)
zeebe_client.deploy_workflow("tests/integration/test.bpmn")
try:
    workflow_instance_key, workflow_result = zeebe_client.run_workflow_with_result(
        bpmn_process_id="test",
        timeout=10000
    )
    logging.info("Result: %s", workflow_result)
except Exception as e:
    logging.exception("Run failed")

After a few seconds, pause the broker, using

docker-compose pause gateway

What you'll see is that the client waits forever (actually 2 hours) since the broker has gone down:

2021-06-25 14:11:20,716 DEBUG [123145476907008|Thread-2] zeebe_adapter_base.py:51: Grpc channel connectivity changed to: ChannelConnectivity.IDLE
2021-06-25 14:11:20,716 DEBUG [123145476907008|Thread-2] zeebe_adapter_base.py:55: Connected to localhost:26500
2021-06-25 14:11:20,716 DEBUG [123145476907008|Thread-3] zeebe_adapter_base.py:51: Grpc channel connectivity changed to: ChannelConnectivity.CONNECTING
2021-06-25 14:11:20,716 DEBUG [123145476907008|Thread-3] zeebe_adapter_base.py:61: Connecting to localhost:26500.
2021-06-25 14:11:20,718 DEBUG [123145476907008|Thread-4] zeebe_adapter_base.py:51: Grpc channel connectivity changed to: ChannelConnectivity.READY
2021-06-25 14:11:20,718 DEBUG [123145476907008|Thread-4] zeebe_adapter_base.py:55: Connected to localhost:26500

The reason being that the timeout argument is for the server side, and not the client-side. ​

Expected behavior

If connection to the gateway is lost, Pyzeebe should detect this and throw an error (or retry connection).

Version

• Pyzeebe: 0.26.4

• Zeebe: 0.26.0
  ​
  Additional context

• https://github.com/grpc/grpc/blob/0e20a5fce8e53510255fa1a5199544a86ebfcd8e/src/core/ext/transport/chttp2/transport/chttp2_transport.cc#L65-L70

  • By default, the gRPC client will wait for 2 hours (DEFAULT_SERVER_KEEPALIVE_TIME_MS) before trying to check if the server is still up

This has been solved in the official clients:

• Enable gRPC keepalive camunda/camunda#3368
• Support keep alive pings camunda/camunda#3487

node.js:

• https://github.com/camunda-community-hub/zeebe-client-node-js/blob/2b1312dfed027d942a04c8ec6b0eaac3591aee57/src/lib/GrpcClient.ts#L227-L291

C#:

• https://github.com/camunda-community-hub/zeebe-client-csharp/blob/af5ed6bfb20c353c5e27a362e8778302c7b22e4c/Client/ZeebeClient.cs#L75-L92

[JonatanMartens]

After some experimenting it seems that this:

    def _create_channel(connection_uri: str, credentials: BaseCredentials = None,
                        secure_connection: bool = False) -> grpc.Channel:
        options = (
                ("grpc.keepalive_time_ms", 45000), 
                ("grpc.keepalive_timeout_ms", 120000),
                ("grpc.http2.min_time_between_pings_ms", 60000)
        )
        if credentials:
            return grpc.secure_channel(connection_uri, credentials.grpc_credentials, options=options)
        elif secure_connection:
            return grpc.secure_channel(connection_uri, grpc.ssl_channel_credentials(), options=options)
        else:
            return grpc.insecure_channel(connection_uri, options=options)

would fix it.

Do we want these to be changed? Both the java and go clients allow this, but the node client does not.

[kbakk]

What about allowing to monkeypatch it if the user wants to change it?

We could define GRPC_CHANNEL_OPTIONS as a dict (that we convert to tuples before using), so the user can do

from pyzeebe.grpc import GRPC_CHANNEL_OPTIONS
GRPC_CHANNEL_OPTIONS.update("grpc.keepalive_time_ms": 5000)

With regards to grpc.keepalive_timeout_ms, grpc.http2.min_time_between_pings_ms – I think we should avoid specifying them, and following the convention in go/java to let the underlying C implementation set the defaults.

[kbakk]

I can add this in a PR, though not sure how to test this (except testing that default values, monkeypatching works).

[kbakk]

@JonatanMartens See PR. Let me know if you would like this ported to 3.0 branch as well - I see that I can do it quite similarly.

grpc_channel_utils.py

def create_channel(
    connection_uri: str,
    credentials: Optional[BaseCredentials] = None,
    secure_connection: bool = False
) -> grpc.aio.Channel:
    options = get_channel_options()
    if credentials:
        return grpc.aio.secure_channel(connection_uri, credentials.grpc_credentials, options=options)
    if secure_connection:
        return grpc.aio.secure_channel(connection_uri, grpc.ssl_channel_credentials(), options=options)
    return grpc.aio.insecure_channel(connection_uri, options=options)

get_channel_options could perhaps be extended to accept a dict, that allows the user to override variables (using dict merge).

[JonatanMartens]

@JonatanMartens See PR. Let me know if you would like this ported to 3.0 branch as well - I see that I can do it quite similarly.

That would be appreciated.

get_channel_options could perhaps be extended to accept a dict, that allows the user to override variables (using dict merge).

Good idea!

[JonatanMartens]

I'll keep this issue open until we also fix the 3.0.0 branch.