Combine 0.2.x branch with master

.github/workflows/ci.yml

@@ -5,39 +5,48 @@ name: CI
 
 on:
   push:
+    branches: [main]
   pull_request:
+    types: [opened, synchronize, reopened]
 
 jobs:
   python:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
     - name: Set up Python
-      uses: actions/setup-python@v2
+      uses: actions/setup-python@v4
       with:
-        python-version: 3.6
+        python-version: '3.10'
     - name: Install dependencies
       run: |
-        python -m pip install --upgrade pip
-        pip install -r requirements.txt
-        python setup.py install
-    - name: Setup Singularity
-      uses: eWaterCycle/setup-singularity@v6
+        python -m pip install --upgrade pip wheel
+        pip install -e .[dev,R]
+    - name: Setup Apptainer
+      uses: eWaterCycle/setup-apptainer@v2
       with:
-        singularity-version: 3.6.4
+        apptainer-version: 1.1.3
     - name: Pull Docker image
       run: docker pull ewatercycle/walrus-grpc4bmi:v0.3.1
-    - name: Cache Singularity image
-      id: cache-singularity-image
+    - name: Cache Apptainer image
+      id: cache-apptainer-image
       uses: actions/cache@v2
       with:
-        path: ~/.singularity
-        key: singularity-image
-    - name: Pull Singularity image
-      if: steps.cache-singularity-image.outputs.cache-hit != 'true'
-      run: singularity pull docker://ewatercycle/walrus-grpc4bmi:v0.3.1
+        path: ~/.apptainer
+        key: apptainer-image
+    - name: Pull Apptainer image
+      if: steps.cache-apptainer-image.outputs.cache-hit != 'true'
+      run: apptainer pull docker://ewatercycle/walrus-grpc4bmi:v0.3.1
+    - uses: r-lib/actions/setup-r@v2
+      with:
+        install-r: false
+    - name: Install R dependencies
+      run: |
+        Rscript -e "install.packages('remotes')"
+        Rscript -e "install.packages('R6')"
     - name: Test with pytest
-      run: pytest --cov=grpc4bmi --cov-report xml
+      run: |
+        pytest -vv --cov=grpc4bmi --cov-report xml
     - name: Correct coverage paths
       run: sed -i "s+$PWD/++g" coverage.xml
     - name: SonarCloud analysis
@@ -46,19 +55,22 @@ jobs:
         GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
   cpp:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
         with:
           submodules: true
       - name: Install grpc
         uses: ewatercycle/setup-grpc@v4
         with:
-          grpc-version: 1.27.2
+          grpc-version: 1.48.2
       - name: Copy grpc to system default
         run: |
           sudo rsync -a $GRPC_ROOT/ /usr/local
           sudo ldconfig
+      - name: Build & install bmi-c
+        run: mkdir -p build && cd build && cmake .. && sudo make install
+        working-directory: cpp/bmi-c
       - name: Build dir
         run: mkdir -p cpp/build
       - name: configure
@@ -70,3 +82,4 @@ jobs:
       - name: ctest
         run: ctest -V
         working-directory: cpp/build
+      # TODO use https://github.com/marketplace/actions/sonarcloud-scan-for-c-and-c

.readthedocs.yaml

@@ -0,0 +1,16 @@
+version: 2
+
+build:
+  os: "ubuntu-22.04"
+  tools:
+    python: "3.10"
+
+sphinx:
+  configuration: docs/conf.py
+
+python:
+  install:
+    - method: pip
+      path: .
+      extra_requirements:
+        - docs

README.md

@@ -8,7 +8,7 @@
 
 ## Purpose
 
-This software allows you to wrap your Basic Model Interface (BMI) implementation ([https://github.com/csdms/bmi](https://github.com/csdms/bmi)) in a server process and communicate with it via the included Python client. The communication is serialized to protocol buffers by gRPC ([https://grpc.io/](https://grpc.io/)) and occurs over network ports.
+This software allows you to wrap your [Basic Model Interface (BMI)](https://github.com/csdms/bmi) implementation in a server process and communicate with it via the included Python client. The communication is serialized to protocol buffers by [GRPC](https://grpc.io/) and occurs over network ports. Can run models in isolated containers using Docker or Apptainer.
 
 ## Installation
 
@@ -43,13 +43,17 @@ in the cpp folder.
 
 ### Model written in Python
 
-For inspiration look at the example in the test directory. To start a server process that allows calls to your BMI implementation, type
+A model should be a subclass of the `Bmi` class from the [bmipy](https://pypi.org/project/bmipy/2.0/) package.
+
+For inspiration look at the [example](test/fake_models.py) in the test directory. 
+
+To start a server process that allows calls to your BMI implementation, type
 
 ```bash
 run-bmi-server --name <PACKAGE>.<MODULE>.<CLASS> --port <PORT> --path <PATH>
 ```
 
-where ```<PACKAGE>, <MODULE>``` are the Python package and module containing your implementation, ```<CLASS>``` is your
+where ```<PACKAGE>, <MODULE>``` are the python package and module containing your implementation, ```<CLASS>``` is your
 bmi model class name, ```<PORT>``` is any available port on the host system, and optionally ```<PATH>``` denotes an
 additional path that should be added to the system path to make your implementation work. The name option above is
 optional, and if not provided the script will look at the environment variables ```BMI_PACKAGE```, ```BMI_MODULE``` and
@@ -99,9 +103,8 @@ mymodel.initialize(<FILEPATH>)
 ...further BMI calls...
 ```
 
-The package contains also client implementation that own the server process, either as a Python subprocess or a docker
-image or a singularity image running the ```run-bmi-server``` script. For instance
-
+The package contains also client implementation that own the server process, either as a Python subprocess or a Docker
+container or a Singularity container or a Apptainer container running the ```run-bmi-server``` script. For instance
 ```python
 from grpc4bmi.bmi_client_subproc import BmiClientSubProcess
 mymodel = BmiClientSubProcess(<PACKAGE>.<MODULE>.<CLASS>)
@@ -110,18 +113,25 @@ mymodel = BmiClientSubProcess(<PACKAGE>.<MODULE>.<CLASS>)
 will automatically launch the server in a sub-process and
 
 ```python
-from grpc4bmi.bmi_client_subproc import BmiClientDocker
-mymodel = BmiClientDocker(<IMAGE>,<PORT>)
+from grpc4bmi.bmi_client_docker import BmiClientDocker
+mymodel = BmiClientDocker(<IMAGE>, <WORK DIR TO MOUNT>, input_dirs=[<INPUT DIRECTORIES TO MOUNT>])
 ```
-
-will launch a docker container, assuming that a gRPC BMI server will start and exposes the port ```<PORT>```.
+will launch a Docker container based on supplied Docker image
+and will mount supplied directories to share files between the container and host.
 
 ```python
 from grpc4bmi.bmi_client_singularity import BmiClientSingularity
-mymodel = BmiClientSingularity(<IMAGE>,<PORT>)
+mymodel = BmiClientSingularity(<IMAGE>, <WORK DIR TO MOUNT>, input_dirs=[<INPUT DIRECTORIES TO MOUNT>])
 ```
+will launch a singularity container on based supplied Singularity image
+and will mount supplied directories to share files between the container and host.
 
-will launch a singularity container, assuming that a gRPC BMI server will start and exposes the port ```<PORT>```.
+```python
+from grpc4bmi.bmi_client_apptainer import BmiClientApptainer
+mymodel = BmiClientApptainer(<IMAGE>, <WORK DIR TO MOUNT>, input_dirs=[<INPUT DIRECTORIES TO MOUNT>])
+```
+will launch a Apptainer container on based supplied Apptainer image
+and will mount supplied directories to share files between the container and host.
 
 For more documentation see [https://grpc4bmi.readthedocs.io/](https://grpc4bmi.readthedocs.io/).
 
@@ -130,10 +140,16 @@ For more documentation see [https://grpc4bmi.readthedocs.io/](https://grpc4bmi.r
 When developers change the proto-file, it is necessary to install gRPC tools Python packages in your Python environment:
 
 ```bash
-pip install -r requirements.txt
-pip install -e .
+# Create virtual env
+python3 -m venv .venv
+. venv/bin/activate
+# Make sure latest pip and wheel are install
+pip install -U pip wheel
+pip install -e .[dev]
 # For R integration also install the R extras with
 pip install -e .[R]
+# For building docs (cd docs && make html) also install the docs extras with
+pip install -e .[docs]
 ```
 
 and install the C++ runtime and `protoc` command as described in <https://github.com/google/protobuf/blob/master/src/README.md>.

docs/_static/design-overview.xml

@@ -1 +1 @@
-<mxfile modified="2019-03-29T15:45:23.435Z" host="www.draw.io" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/8.8.0 Chrome/61.0.3163.100 Electron/2.0.2 Safari/537.36" etag="E8JU1hGCv8wWEk0ACa1m" version="10.5.9" type="device"><diagram id="g6TwU9qutF0pdJ5EpEK3" name="Page-1">5VrbcpswEP0az6QPzYC42DzGJGk704vHnl6mbwoooEZGrpBv/fpKIDAgnJDW2JPYL4iVWFnnaFd7sAeWP9+8Y3ARf6IhIgNghJuBdT0AwDRdIC7Sss0tQ8PODRHDoRq0M8zwH6SMhrIucYjS2kBOKeF4UTcGNElQwGs2yBhd14fdU1KfdQEjpBlmASS69TsOeaysrmPvOt4jHMW8XLGX98xhMVotJY1hSNcVk3UzsHxGKc9b842PiESvACZ/7nZPb/nNGEp4lwem38Y/vIV//TNY//qJPqZ/3OmPt2oZK0iWasWTLY9pImwX6DK6FNfPlKM7Sh/eqGXwbQEOo8skRNK9MbDG6xhzNFvAQPauxXYQtpjPibgzRXOFGMcC2CuCo0TYOJUDUs7oQwmtwGSsvpAYjTZ7V2qW+Imdh+gccbYVQ9QDwFIcqE1nFxSsKxSOlC2usuep7QrVtolK3ztgRUNh+wycHQ3nAXCJmHYc4pVoRrI5/vShsIpJKh1PIF9BuY2Ee0yITwll2bPWffaRdprwWzjHRML0cRngEIpZfJqkVM5zECbsOhOOCXQm3BYmLKMnIkYtRDTARUl4JVOHuAsITFMc1DHuhhsKtczyJGoVVJwWUAobQwRyvKq7b0NKzTChWExckmLbRp2UYQPslC5ZgNRT1YzScDQCTzjikEWIa44y4spl/zuXxSFRITOaTnyZsxBfU/YgWhNGOQ0oKQLrjhVBBWRym6LfS5RybQ+ILc/rrOeZqoijhCaoEVrKBFV+CwTZiLUkvjkOw2yftMVqPbCPEKGO0+DQasmV3jEj1DQ1Vq9p8CCwBLcznERLAhnm2xwIDnEie45xMoUwjTOf8qFIZgbl/wA0jJox2UJDCXmVBmD0xgPYF10zxFYSdOMiLxb+szKoI22e9sQaDRtEFBVsNR6GLUTYTl88WBoPWaFwDui7AFw63fDvLx/pNXKhc3xx3RExADI4pq+aD3fYlY/+4qFTLf01zTJUztCrrqltp2NN7fQWIe6+k8InGGUVY8dc9XJQd06P+rAlDs5MymgKZGTXXXSVMpomajrqW8routTK9cmCbDVaz1edeHbb+dOmT0BfcQf0ekwjKBJILZqSQd5MIBe4J5kFGNIalCImH7UXNfWKE94VkxjPRdM0m+EChhqYx1YZnobma1B7j+ycR3PlkdEv9sNjGk/ciTWPX1ZZ+yz8N3UFoYLDOrXwA/qLECX8zpUS7/RqsEj2j6hBxU4uB/38cksZZ/CFafUDMGZapxeMoE0wnlmhbDVOfrcZIJ3f+ZsNR17D0Z5CWaALt5VhCzkgfcYX9mq/bopG7vGgVTjQhawpq/CZWBSvHIrNnxLsfJCoJPaP0bbc+RTxzS3jjPSM0Lb3+yvhden8ZfpCCZL96n8TwD4MYaYxtOuMgTbZZR+GMnG7+xNEHsm7/5JYN38B</diagram></mxfile>
+<mxfile modified="2022-11-15T11:35:34.226Z" host="app.diagrams.net" agent="5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36" etag="v_puD-7TxF7KD0VfFUmk" version="20.4.0" type="device"><diagram id="g6TwU9qutF0pdJ5EpEK3" name="Page-1">5Vpbb5swFP41SNvDKjCXhMeGrtukXaJE26q9ueCCVwcz49z262eDSQCTlm2hWZeqEubY2Pj7zjk+XxLDDhabNwxmyQcaIWIAM9oY9pUBgOUAYMh/M9qWltHYKw0xw5EatDfM8U+kjKayLnGE8sZATinhOGsaQ5qmKOQNG2SMrpvD7ihprprBGGmGeQiJbv2KI54oq+c6+463CMdJtbTl+WXPAlaj1VbyBEZ0XTPZrw07YJTysrXYBIhI9CpgyueuD/Tu3oyhlPd5YPZlcuNnwdW3cP39G3qf//RmN6/UNlaQLNWOp1ue0FTYXqCL+EJcP1KObim9f6m2wbcVOIwu0wjJ6U3DnqwTzNE8g6HsXQt3ELaEL4i4s0RzhRjHAthLguNU2DiVA3LO6P0OWoHJRN+W2qmcAG1qJrXNN4guEGdbMUT1AltxoJzOqShY1ygcK1tSZ89X7gqV28S7uffAiobC9jdwdjWcDeARsewkwivRjGVz8uFdZRWL1DoeQb6GchcJd5iQgBLKimftu+JP2mnKr+ECEwnT+2WIIyhWCWiaU7nOUZhwmky4FtCZ8DqYsM2BiBh3ENECF6XRpUwd4i4kMM9x2MT4j3FDUSPZ6KjVUHE7QKlsDBHI8aqZorqQUitMKRZvsiPFccwmKaMW2DldshCpp+oZpTXRGDwyEYcsRlybqCBut+0/57I6JGpkxrNpIHMW4mvK7kVryiinISVVYN2yKqiATG4z9GOJcq75gPBv3mS9zFRVHKU0Ra3QUiao8lso2EesI/EtcBQVftIVq83AfoIIdd0Wh3ZHrvSfMkItS2P1iob3AktwfZllHOJUtiUMVfspzqUI5kkxp3wolnlBzX8EEsbtiOwgYQd4nQRgDsYCOBRbc8RWBQEvylLhL+uCJtLWac+r8ahFhNdxXo06iHDcoXiwNR6KMuEc0PcAuHD74T9cNtIr5ErlBOK6J8IAMjhm/zUf3qgvH8PFQ69K+nOujgjJ0H9dUTtuz4raHSxCvEMnRUAwKurFnrnq+aDunh71UUccnJmQ0fTH2GlO0VfIaIqoPdHQQkZXpXapTjKy1Wg9X23iO13nT5c6AUPFHdDrMY2gWCCVtSWDvJlCLnBPCwswpTXciZhyVH/Udp95wttqXfNBNC2rHS5gpIH51CrD19B8ZlrvYSd53NVPh33lDQ8pPHEn9jz5h4vav8V/09QPKjTsU8s+oH8IomTfuVLin14LVqn+AS2o2CnFYFBerinjDP7LSn0Yxiz79HIRdMnFMyuT7da577UDpPfn/VZrIr810YEyWaALt7VhmRyQ/8YL+41vNmVtUMx41Boc6DLWkjX4XGyK1w7F9tcITjlIVBKHx2gudz4lfNtl3LGeEbp8f7gCXhfOn2bPlCDZr34zAZzjEGaZI6fJGOgSXc5xKBO3+x9AlJG8/x2J/foX</diagram></mxfile>

docs/conf.py

@@ -30,7 +30,11 @@
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode', 'sphinx.ext.napoleon', 'sphinx.ext.intersphinx', 'sphinxarg.ext']
+extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode',
+              'sphinx.ext.napoleon', 'sphinx.ext.intersphinx',
+              'sphinxarg.ext', 'sphinxcontrib.apidoc',
+              'sphinx_copybutton',
+              ]
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
@@ -63,7 +67,7 @@
 #
 # This is also used if you do content translation via gettext catalogs.
 # Usually you set "language" from the command line for these cases.
-language = None
+language = 'en'
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
@@ -76,6 +80,11 @@
 # If true, `todo` and `todoList` produce output, else they produce nothing.
 todo_include_todos = False
 
+apidoc_module_dir = "../grpc4bmi"
+apidoc_separate_modules = True
+apidoc_toc_file = False
+# Ignore generated files
+apidoc_excluded_paths = ["bmi_pb2_grpc.py", "bmi_pb2.py"]
 
 # -- Options for HTML output ----------------------------------------------
 

docs/container/usage.rst

@@ -10,15 +10,14 @@ Grpc4bmi can run containers with `Docker engine`_.
 
 Use the :class:`grpc4bmi.bmi_client_docker.BmiClientDocker` class to start a Docker container and get a client to interact with the model running inside the container.
 
-
-
 For example the PCR-GLOBWB model can be started in a Docker container with
 
 .. code-block:: python
 
+    from grpc4bmi.bmi_client_docker import BmiClientDocker
+
     model = BmiClientDocker(image='ewatercycle/pcrg-grpc4bmi:latest', image_port=55555,
-                            input_dir="./input",
-                            output_dir="./output")
+                            work_dir="./input")
     # Interact with model
     model.initialize('config.cfg')
 
@@ -27,6 +26,8 @@ For example the PCR-GLOBWB model can be started in a Docker container with
 
 .. _Docker engine: https://docs.docker.com/
 
+.. _Singularity-section:
+
 Singularity
 -----------
 
@@ -48,15 +49,56 @@ Use the :class:`grpc4bmi.bmi_client_singularity.BmiClientSingularity` class to s
 .. code-block:: python
 
     from grpc4bmi.bmi_client_singularity import BmiClientSingularity
+
     image = '<docker image name of grpc4bmi server of a bmi model>'
-    client = BmiClientSingularity(image, input_dir='<directory with models input data files>')
+    input_dir = '<directory with models input data files>'
+    work_dir = '<directory where model can write output files>'
+    client = BmiClientSingularity(image, work_dir, input_dirs=[input_dir])
+
+     # Interact with model
+    client.initialize('<absolute path to config file in the input directory>')
+
+    # Stop container
+    del client
 
 For example for the wflow Docker image the commands would be the following
 
 .. code-block:: python
 
     from grpc4bmi.bmi_client_singularity import BmiClientSingularity
+
     image = 'docker://ewatercycle/wflow-grpc4bmi:latest'
-    client = BmiClientSingularity(image, input_dir='wflow_rhine_sbm', output_dir='wflow_output')
+    work_dir = '/tmp/run1'
+    client = BmiClientSingularity(image, work_dir, input_dirs=['/scratch/input/wflow_rhine_sbm'])
+
+     # Interact with model
+    client.initialize('/scratch/input/wflow_rhine_sbm/wflow_sbm_bmi.ini')
+
+    # Stop container
+    del client
+
 
 .. _Singularity: https://www.sylabs.io/guides/latest/user-guide/
+
+Apptainer
+---------
+
+Grpc4bmi can run containers on `Apptainer`_.
+
+Apptainer is an open source fork of singularity.
+
+Apptainer behaves very similar to singularity so everything described in :ref:`Singularity-section` chapter also goes for Apptainer.
+You will need to replace the `singularity` command with `apptainer` and :class:`grpc4bmi.bmi_client_singularity.BmiClientSingularity` class with
+:class:`grpc4bmi.bmi_client_apptainer.BmiClientApptainer` class.
+
+.. _Apptainer: https://apptainer.org/docs/user/main/
+
+Sharing files between host computer and container
+-------------------------------------------------
+
+Containers run in an isolated file system and by default can not read/write any files on host computer.
+To get a directory on your host computer inside a container you have mount them with `input_dirs` and
+`work_dir` arguments of :py:class:`grpc4bmi.bmi_client_docker.BmiClientDocker` and
+:py:class:`grpc4bmi.bmi_client_singularity.BmiClientSingularity`.
+
+See :py:class:`grpc4bmi.bmi_client_singularity.BmiClientSingularity` for examples using `input_dirs` and `work_dir`.

docs/introduction.rst

@@ -3,6 +3,6 @@ Introduction
 
 The Basic Modeling Interface (BMI, see https://github.com/csdms/bmi) is a multi-language library interface tailored to earth system models. This software allows you to wrap a BMI implementation in a server process and communicate with it via the included python client. The communication is serialized to protocol buffers by GRPC (https://grpc.io/) and occurs over network ports. On the server side, we support BMI implementations in :ref:`python <pythonservice>`, R or C/C++. Fortran models need to be linked against the C-version of the BMI. On the client side, we expose the standard python BMI (https://github.com/csdms/bmi-python/).
 
-This setup enables you to wrap your BMI-enabled model in a Docker (https://www.docker.com/) or Singularity (https://singularity.lbl.gov/) container  and communicate with it from a python process on the host machine.
+This setup enables you to wrap your BMI-enabled model in a Docker (https://www.docker.com/) or Singularity (https://singularity.lbl.gov/) or Apptainer (https://apptainer.org) container  and communicate with it from a python process on the host machine.
 
 .. image:: _static/design-overview.svg

docs/python_api.rst

@@ -4,4 +4,4 @@ Python API
 .. toctree::
    :maxdepth: 4
 
-   grpc4bmi
+   api/grpc4bmi

docs/server/python.rst

@@ -49,7 +49,7 @@ and inside the ``mymodule.py`` the bmi implementation
 
 .. code-block:: python
 
-    from bmi import Bmi
+    from bmipy import Bmi
 
     class MyBmi(Bmi):
         def __init__(self):
@@ -63,4 +63,19 @@ Then we launch this toy model as a service by executing
 
     $ run-bmi-server --name mypackage.mymodule.MyBmi
 
-This will report the chosen port number in the standard output stream. It can be used to connect to the service via the BMI :ref:`grpc python client <python-grpc4bmi-client>`.
+This will report the chosen port number in the standard output stream. It can be used to connect to the service via the BMI :ref:`grpc python client <python-grpc4bmi-client>`.
+
+Legacy version
+--------------
+
+The grpc4bnmi by default uses the `v2.0 version of the Basic Model Interface (BMI) <https://pypi.org/project/bmipy/2.0/>`_ on the client and server side..
+Grpc4bmi has support for `legacy version 0.2 of BMI <https://pypi.org/project/basic-modeling-interface/0.2/>`_ on the server side. 
+
+A legacy model can be launched with
+
+.. code-block:: sh
+
+    $ run-bmi-server --bmi-version 0.2 --name mypackage.mymodule.MyLegacyBmi
+
+The functions only present in v0.2 will not be callable.
+The functions only present in v2.0 will throw not implemented errors.