Unable to connect to GRPC server

[BSchilperoort]

I am unable to connect to the GRPC server, even when starting the GRPC server inside the same python instance.

Setup:

• WSL2, Ubuntu 20.04
• Python 3.8.10
• grpc4bmi 0.2.x (latest)

The code I use to start the server, and connect to the model:

>>> import grpc
>>> from grpc4bmi.bmi_client_subproc import BmiClientSubProcess
>>> model = BmiClientSubProcess("pywflow.wflow.WflowBMI")
Starting GRPC server for <grpc4bmi.bmi_grpc_server.BmiServer object at 0x7fef00b7edf0> at port 46137

>>> import grpc
>>> from grpc4bmi.bmi_grpc_client import BmiClient
>>> mymodel = BmiClient(grpc.insecure_channel("localhost:46137"))

Upon calling get_model_name, an error occurs. AttributeError: 'BmiClientSubProcess' object has no attribute 'pipe'.
It does not matter if I connect to the server in the same python instance, or in a different one (running concurrently).

>>> mymodel.get_component_name()
Exception ignored in: <function BmiClientSubProcess.__del__ at 0x7fea851b8280>
Traceback (most recent call last):
  File "/home/bart/grpc4bmi-0.2.x/grpc4bmi/bmi_client_subproc.py", line 29, in __del__
    self.pipe.terminate()
AttributeError: 'BmiClientSubProcess' object has no attribute 'pipe'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/bart/grpc4bmi-0.2.x/grpc4bmi/bmi_grpc_client.py", line 77, in get_component_name
    return str(self.stub.getComponentName(bmi_pb2.Empty()).name)
  File "/home/bart/venv/grpc/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/bart/venv/grpc/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "Socket closed"
        debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B::1%5D:46137 {created_time:"2022-11-14T09:09:04.782000456+01:00", grpc_status:14, grpc_message:"Socket closed"}"

[sverhoeven]

I am able to somewhat replicate on Ubuntu 20.04, WSL2, Python 3.8.10

heat model and grpc4bmi 0.2.x

Commit c5ed76f

from grpc4bmi.bmi_client_subproc import BmiClientSubProcess
model = BmiClientSubProcess("heat.BmiHeat")
model.get_component_name()
'The 2D Heat Equation'

In background I see run-bmi-server --name heat.BmiHeat --port 46353 running, to which I can connect a second client with

>>> from grpc4bmi.bmi_grpc_client import BmiClient
>>> import grpc
>>> mymodel = BmiClient(grpc.insecure_channel("localhost:46353"))
>>> mymodel.get_component_name()
'The 2D Heat Equation'

pywflow main branch

git clone [email protected]:eWaterCycle/pywflow.git
cd pywflow/
python3 -m venv .venv
 . .venv/bin/activate
pip install -U wheel pip
pip install -e .
pip install grpc4bmi==0.2.15

from pywflow import WflowBMI
...
juliacall.JuliaError: ArgumentError: Package Wflow not found in current path
...

Followed instructions and install wflow package in Julia

.venv/julia_env/pyjuliapkg/install/bin/julia
import Pkg; Pkg.add("Wflow")

from pywflow import WflowBMI
model = WflowBMI()
model.get_component_name()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "..../pywflow/pywflow/wflow.py", line 76, in get_component_name
    return jl.Wflow.BMI.get_component_name(self.model)
AttributeError: 'WflowBMI' object has no attribute 'model'

Seems self.model is initialized during model.initialize() so installed example dataset (needed to correct dir_input=data).

from pywflow import WflowBMI
model = WflowBMI()
model.initialize('sbm_config.toml')
model.get_component_name()
'sbm'

Now able to run wflow in Python BMI.

from grpc4bmi.bmi_client_subproc import BmiClientSubProcess
model = BmiClientSubProcess("pywflow.wflow.WflowBMI")
model.initialize('sbm_config.toml')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/stefanv/git/eWaterCycle/pywflow/.venv/lib/python3.8/site-packages/grpc4bmi/bmi_grpc_client.py", line 62, in initialize
    self.stub.initialize(bmi_pb2.InitializeRequest(config_file=fname))
  File "/home/stefanv/git/eWaterCycle/pywflow/.venv/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/stefanv/git/eWaterCycle/pywflow/.venv/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "Socket closed"
        debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B::1%5D:44233 {grpc_message:"Socket closed", grpc_status:14, created_time:"2022-11-15T14:24:46.083914685+01:00"}"

The same primary exception, but the model.pipe prop is available to me.

The setting up of Julia taking a long time (>1s), causing the client to make a connection with a server which is not yet there.
Similar to #16 . The Docker and Singularity classes have a delay argument which can be used to wait for container to startup, before connecting to it. The BmiClientSubProcess has a hardcoded delay of 1 second (

grpc4bmi/grpc4bmi/bmi_client_subproc.py

Line 25 in c5ed76f

time.sleep(1)

), which is not enough time.

When I expose the delay (subprocess-delay branch) and set it to 30s the model works.

from grpc4bmi.bmi_client_subproc import BmiClientSubProcess
model = BmiClientSubProcess("pywflow.wflow.WflowBMI", delay=30)

I see run-bmi-server --name pywflow.wflow.WflowBMI --port 34083 running in the background.

Calling a model function causes the run-bmi-server process to die.

model.initialize('sbm_config.toml')

>>> eo = model.pipe.communicate()
>>> eo
(None, None)
>>> model.pipe.returncode
-11

Also tried using stdout=PIPE,stderr=PIPE to popen, but output was empty string.

Try to run grpc server in python repl

from pywflow import WflowBMI
model = WflowBMI()
from grpc4bmi.run_server import serve
serve(model, 34523)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/stefanv/git/eWaterCycle/grpc4bmi/grpc4bmi/run_server.py", line 81, in serve
    bmi_pb2_grpc.add_BmiServiceServicer_to_server(model, server)
  File "/home/stefanv/git/eWaterCycle/grpc4bmi/grpc4bmi/bmi_pb2_grpc.py", line 433, in add_BmiServiceServicer_to_server
    servicer.updateUntil,
AttributeError: 'WflowBMI' object has no attribute 'updateUntil'

I expect when using BmiClientSubProcess the run-bmi-server also crashes on this error.

When using grpc4bmi 0.2.x the basic_modeling_interface.Bmi should be used while pywflow uses bmipy.Bmi.

So I think there are 3 problems

1. Model takes longer than 1 second to startup causing client/server handshake problem -> workaround custom delay
2. The stdout/stderr (run-bmi-server stays quiet #121) of BmiClientSubProcess are empty. Making it hard to diagnose -> workaround use grpc4bmi.run_server.serve(model, port), to bypass a subprocess
3. Model uses different BMI version then grpc4bmi expects

[BSchilperoort]

Wow thanks for the extensive debugging session!

It seems that supporting the newer BMI implementation would then be important to solve first, as the old version is hard to find (and archived).

[BSchilperoort]

I re-tried to start a grpc server, using the code below.
I used the master branch grpc4bmi (for bmi 2.0), with the delay patch added in manually.
After installing the latest grpc4bmi release, the segmentation fault occurs there as well.

Start the server:

>>> from pywflow.wflow import WflowBMI
>>> from grpc4bmi.bmi_grpc_server import BmiServer
>>> from grpc4bmi.run_server import serve
>>> mymodel = BmiServer(WflowBMI())
>>> serve(mymodel, 55555)

Try to connect:

>>> from grpc4bmi.bmi_grpc_client import BmiClient
>>> import grpc
>>> mymodel = BmiClient(grpc.insecure_channel("localhost:55555"))
>>> mymodel.initialize("~/grpc4bmi_wflow/pywflow/tests/hbv_config.toml")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/bart/grpc4bmi_wflow/grpc4bmi/grpc4bmi/bmi_grpc_client.py", line 100, in initialize
    handle_error(e)
  File "/home/bart/grpc4bmi_wflow/grpc4bmi/grpc4bmi/bmi_grpc_client.py", line 98, in initialize
    return self.stub.initialize(bmi_pb2.InitializeRequest(config_file=fname))
  File "/home/bart/grpc4bmi_wflow/wfvenv/lib/python3.8/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/bart/grpc4bmi_wflow/wfvenv/lib/python3.8/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "Socket closed"
        debug_error_string = "{"created":"@1669127625.136234664","description":"Error received from peer ipv6:[::1]:55555","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket closed","grpc_status":14}"

At this point, the terminal in which I started the server returned:

Segmentation fault

Which also closed the python terminal.

---

The following code runs just fine:

>>> from pywflow.wflow import WflowBMI
>>> model = WflowBMI()
>>> model.initialize("pywflow/tests/hbv_config.toml")
[ Info: Cyclic parameters are provided by `pywflow/tests/staticmaps-lahn.nc`.
[ Info: Forcing parameters are provided by `pywflow/tests/forcing-lahn.nc`.
[etc...]

Also, when I replace the model with "BmiHeat" (2.0), it works just fine. And the BmiClient's get_component_name returns 'The D Heat Equation"