Latest protobuf #126

sverhoeven · 2023-01-23T12:34:30Z

Adds Dockerfiles for heat model in combinations of languages, and versions of bmi, protobuf, grpc4bmi.

To test

Build images with cd test/heat-images && docker compose build
Edit image var in test/heat-images/heat_tester.py and run it

After merge actions

In test/heat-images/python-bmi02-legacy/Dockerfile, test/heat-images/python-bmi20/Dockerfile, test/heat-images/python-bmi20-pb4/Dockerfile, change RUN pip install https://github.com/eWaterCycle/grpc4bmi/archive/refs/heads/bmi2.zip to RUN pip install grpc4bmi==<new version>

sverhoeven · 2023-01-23T12:39:08Z

@goord I am trying to create a grpc4bmi Docker image around https://github.com/csdms/bmi-example-cxx , but am having link errors. error message

Do you have any quick suggestions?

Having linked to both grc++_unsecure.so and grpc++.so caused while loop and 100% cpu, just having one works

sverhoeven · 2023-01-30T15:42:09Z

In https://github.com/eWaterCycle/grpc4bmi/tree/latest-protobuf/test/heat-images/c-bmi20-pb3 the update and inialize methods work, but get_component_name core dumps.

Debug

Outside Docker do echo core | sudo tee /proc/sys/kernel/core_pattern + make ulimit -c does not return 0.

docker run -ti  --entrypoint bash heat:c-bmi20-pb3
apt-get update
apt-get install gdb
/usr/local/bin/run_bmi_server
# Call get_component_name in python shell
gdb /usr/local/bin/run_bmi_server /core.7
...
bt
#0  __strncpy_ssse3 () at ../sysdeps/x86_64/multiarch/strcpy-ssse3.S:98
#1  0x00007ff5bdd7b2a8 in Get_component_name () from /usr/local/lib/libbmiheatc.so
#2  0x00007ff5bde4cd59 in BmiCWrapper::GetComponentName[abi:cxx11]() () from /usr/local/lib/libgrpc4bmi.so
#3  0x00007ff5bde4dd4d in BmiGRPCService::getComponentName(grpc::ServerContext*, bmi::Empty const*, bmi::GetComponentNameResponse*) ()
...

import grpc
from grpc4bmi.bmi_grpc_client import BmiClient
model = BmiClient(grpc.insecure_channel("172.17.0.2:50051"))
model.get_component_name()
# raises
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "Socket closed"
	debug_error_string = "UNKNOWN:Error received from peer ipv4:172.17.0.2:50051 {grpc_message:"Socket closed", grpc_status:14, created_time:"2023-01-30T16:15:57.894638216+01:00"}"

Co-authored-by: Johan Hidding <[email protected]>

sverhoeven · 2023-02-21T11:43:11Z

Docker files are now hidden away in some sub dir (https://github.com/eWaterCycle/grpc4bmi/tree/latest-protobuf/test/heat-images). We should put them in better location, for example

What to you think @Peter9192 ?

Peter9192 · 2023-02-21T12:44:02Z

What to you think @Peter9192 ?

I thought we were going to retire grpc4bmi-examples in favour of ewatercycle-models. But perhaps we can instead repurpose it:

https://github.com/eWaterCycle/grpc4bmi-examples: all these grpc4bmi-heat containers
https://github.com/eWaterCycle/ewatercycle-models: ewatercycle model containers (or plugins?)

Does that make sense?

Peter9192

Overall, very nice test suite! Got a few issues while testing. Also had some questions about the c++ changes, but I suppose those are not the most relevant, more to trigger myself to see if I could understand what was going on.

cpp/CMakeLists.txt

cpp/bmi_c_wrapper.h

docs/server/Cpp.rst

grpc4bmi/bmi_pb2.py

test/heat-images/python-bmi20-pb4/Dockerfile

Peter9192 · 2023-03-15T16:24:01Z

test/heat-images/heat_tester.py

+    image, config_body = (
+        # Comment out line you want to run
+    'heat:py-0.2',py_config
+    # 'heat:py-0.2-legacy',py_config


with grpc4bmi install from branch bmi2 this gives

(ewatercycle) peter@DESKTOP-S380217:~/ewatercycle/grpc4bmi/test/heat-images$ python heat_tester.py heat:py-0.2-legacy The 2D Heat Equation ('plate_surface__temperature',) [6 8] Traceback (most recent call last): File "/home/peter/ewatercycle/grpc4bmi/test/heat-images/heat_tester.py", line 59, in <module> test_heat(tmp_path=Path('/tmp')) File "/home/peter/ewatercycle/grpc4bmi/test/heat-images/heat_tester.py", line 53, in test_heat values = model.get_value(var_name) File "/home/peter/mambaforge/envs/ewatercycle/lib/python3.10/site-packages/grpc4bmi/bmi_optionaldest.py", line 105, in get_value dest = reserve_values(self.origin, name) File "/home/peter/mambaforge/envs/ewatercycle/lib/python3.10/site-packages/grpc4bmi/reserve.py", line 12, in reserve_values size = total_size // item_size ZeroDivisionError: integer division or modulo by zero

test/heat-images/docker-compose.yaml

Peter9192 · 2023-03-15T16:38:06Z

test/heat-images/heat_tester.py

+
+    image, config_body = (
+        # Comment out line you want to run
+    'heat:py-0.2',py_config


Suggested change

'heat:py-0.2',py_config

'heat:python-bmi02',py_config

With bmi2 branch insides my existing ewatercycle env gives zerodivisionerror:

heat:python-bmi02 The 2D Heat Equation ('plate_surface__temperature',) [6 8] Traceback (most recent call last): File "/home/peter/ewatercycle/grpc4bmi/test/heat-images/heat_tester.py", line 59, in <module> test_heat(tmp_path=Path('/tmp')) File "/home/peter/ewatercycle/grpc4bmi/test/heat-images/heat_tester.py", line 53, in test_heat values = model.get_value(var_name) File "/home/peter/mambaforge/envs/ewatercycle/lib/python3.10/site-packages/grpc4bmi/bmi_optionaldest.py", line 105, in get_value dest = reserve_values(self.origin, name) File "/home/peter/mambaforge/envs/ewatercycle/lib/python3.10/site-packages/grpc4bmi/reserve.py", line 12, in reserve_values size = total_size // item_size ZeroDivisionError: integer division or modulo by zero

With clean mamba env and grpc4bmi installed from bmi2 branch, get

Traceback (most recent call last): File "/home/peter/ewatercycle/grpc4bmi/test/heat-images/heat_tester.py", line 4, in <module> from grpc4bmi.bmi_client_docker import BmiClientDocker File "/home/peter/mambaforge/envs/testheat/lib/python3.10/site-packages/grpc4bmi/bmi_client_docker.py", line 8, in <module> from typeguard import check_argument_types, qualified_name ImportError: cannot import name 'check_argument_types' from 'typeguard' (/home/peter/mambaforge/envs/testheat/lib/python3.10/site-packages/typeguard/__init__.py)

downgrading typeguard from v3.0.0 to v2.13.3 solved this issue.

Fixed in #129

I will have a look at the division by zero as the workaround in bmi_grpc_client.py::get_var_itemsize should fix it

That division by zero workaround is only in branch of this PR at https://github.com/eWaterCycle/grpc4bmi/blob/latest-protobuf/grpc4bmi/bmi_grpc_client.py#L199 so that makes sense then when running the bmi2 branch it fails.

pyproject.toml

Typeguard3 compat

sverhoeven

Thanks for reviewing

cpp/CMakeLists.txt

cpp/bmi_c_wrapper.h

pyproject.toml

test/heat-images/c-bmi20/Dockerfile

test/heat-images/cxx-bmi20/Dockerfile

sverhoeven · 2023-03-16T10:41:48Z

test/heat-images/heat_tester.py

+
+    image, config_body = (
+        # Comment out line you want to run
+    'heat:py-0.2',py_config


Fixed in #129

sverhoeven · 2023-03-16T10:44:42Z

test/heat-images/heat_tester.py

+
+    image, config_body = (
+        # Comment out line you want to run
+    'heat:py-0.2',py_config


I will have a look at the division by zero as the workaround in bmi_grpc_client.py::get_var_itemsize should fix it

test/heat-images/python-bmi02-legacy/Dockerfile

test/heat-images/python-bmi20-pb4/Dockerfile

Co-authored-by: Peter Kalverla <[email protected]>

sverhoeven

Thanks for reviewing

test/heat-images/docker-compose.yaml

sverhoeven · 2023-03-16T15:22:45Z

test/heat-images/heat_tester.py

+
+    image, config_body = (
+        # Comment out line you want to run
+    'heat:py-0.2',py_config


That division by zero workaround is only in branch of this PR at https://github.com/eWaterCycle/grpc4bmi/blob/latest-protobuf/grpc4bmi/bmi_grpc_client.py#L199 so that makes sense then when running the bmi2 branch it fails.

sonarqubecloud · 2023-03-16T15:30:22Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

99.0% Coverage
3.3% Duplication

sverhoeven and others added 7 commits January 20, 2023 12:37

Generate bindings with protobuf v4

49a38ef

Add Dockerfiles for heat model using different versions

56a1fc3

Use latest @actions/cache

939dff0

Working on wrapping heatcxx in a grpc4bmi server container image

75856ca

Move bmi-c submodule to v2.1 tag

99e0a47

Stuck on linker

8a4024b

Get grpc4bmi from gh instead of local mount

f0e0e14

sverhoeven added 4 commits January 26, 2023 14:55

Fix compile

ede0a47

const

117dc24

Dont install

3bb6070

More debug

a573637

sverhoeven changed the base branch from master to bmi2 January 26, 2023 14:59

sverhoeven and others added 10 commits January 26, 2023 16:05

More prints

19247c4

Add reflection

8c5849d

Ignore sif files for git and docker

3adfe23

Prefer install of bmi-c + bmi-cxx over git submodule

84fb60d

Got working server from bmi-example-cxx

727f3bc

Having linked to both grc++_unsecure.so and grpc++.so caused while loop and 100% cpu, just having one works

More cleanup

0926af9

Remove deps of deps

80dd5b9

Make sure bmi.hxx is imported

f0fce18

Wrap bmiheatc with grpc

2241fe8

Install bmi-c and bmi-cxx in CI

2e937b3

sverhoeven and others added 6 commits January 31, 2023 16:56

Allocate strings

4fb5946

Co-authored-by: Johan Hidding <[email protected]>

Drop cache buster

f4d5155

Old python heat model is missing get_var_itemsize, added workaround

c606b5a

Add image with run-bmi-server --bmi-version 0.2 ...

f3323b9

Add build command + simplify non-python dirs

e023b4a

Correct GetGridRank wrapper c function was calling get_grid_size

732f5fd

Structure readme

3798bbc

sverhoeven marked this pull request as ready for review February 21, 2023 11:43

sverhoeven requested a review from Peter9192 March 13, 2023 08:48

Peter9192 requested changes Mar 15, 2023

View reviewed changes

sverhoeven and others added 6 commits March 16, 2023 08:04

Make typeguard==3 compatible

5a129ad

Add check back as str is iterable which passes typechecked

6ab8927

Fix thing I broke with prev commit

b12e5f3

One more test to fix

11c7965

Pin protobuf to v4

4850b2c

Merge pull request #129 from eWaterCycle/typeguard3-compat

c209b22

Typeguard3 compat

sverhoeven commented Mar 16, 2023

View reviewed changes

Apply suggestions from code review

570a1b6

Co-authored-by: Peter Kalverla <[email protected]>

sverhoeven commented Mar 16, 2023

View reviewed changes

Made py image names consistent

cf9928f

Peter9192 approved these changes Mar 16, 2023

View reviewed changes

sverhoeven merged commit eb91cfc into bmi2 Mar 20, 2023

sverhoeven deleted the latest-protobuf branch March 20, 2023 09:48

This was referenced Mar 20, 2023

Replace bad chars in singularity version + fix heat model tests #90

Closed

Upgrade heat model broke our tests #91

Closed

Upgrade to protobuf v4 #115

Closed

Dependency conflict with protobuf and grpcio-reflection #120

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Latest protobuf #126

Latest protobuf #126

sverhoeven commented Jan 23, 2023 •

edited

Loading

sverhoeven commented Jan 23, 2023

sverhoeven commented Jan 30, 2023

sverhoeven commented Feb 21, 2023

Peter9192 commented Feb 21, 2023

Peter9192 left a comment

Peter9192 Mar 15, 2023

Peter9192 Mar 15, 2023

Peter9192 Mar 15, 2023

sverhoeven Mar 16, 2023

sverhoeven Mar 16, 2023

sverhoeven Mar 16, 2023

sverhoeven left a comment

sverhoeven Mar 16, 2023

sverhoeven Mar 16, 2023

sverhoeven left a comment

sverhoeven Mar 16, 2023

sonarqubecloud bot commented Mar 16, 2023

Latest protobuf #126

Latest protobuf #126

Conversation

sverhoeven commented Jan 23, 2023 • edited Loading

sverhoeven commented Jan 23, 2023

sverhoeven commented Jan 30, 2023

sverhoeven commented Feb 21, 2023

Peter9192 commented Feb 21, 2023

Peter9192 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sverhoeven left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sverhoeven left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Mar 16, 2023

sverhoeven commented Jan 23, 2023 •

edited

Loading