Walberla: Morre than one block / rank

[RudolfWeeber]

For better cache locallity.

1. clarify support for for several blocks per mpi rank to the LatticeWalberla class

   • src/walberla_bridge/include/walberla_bridge/LatticeWalberla.hpp and src/walberla_bdirdge/src/LatticeWalberla.cpp
   • we currently just use the node_grid parameter as arguments for the number of blocks per direcoint in the construciont of the waLBerla block forest
   • So, here, only the variable node_grid needs to be renamed to block_grid to it celar what the argument actually contains

2. Support blocks_per_mpi_rank=(nx,ny,nz) keyword argument with default = (1,1,1) in the LatticeWalbelra class (which, in the core holds the waLBerla block forest)

   • src/python/espressomd/detail/walberla.py the LatticeWalberla class, required_params(), valid_params() and default_params()
   • script interface bindings for the class in src/script_interface/walberla/LatticeWalberla.hpp
     • add the new parameter in the add_parameters() call in the constructor. Use Utils::Vector3i as data type on the c++ side
     • in do_construct() calculate block_grid by multiplying the mpi node grid componentwise by the locks_per_mpi_rankparameter. then pass that block_grid as 2nd arg to the conststurcotr of the LatticeWalbelra core class in the maek_shared call.

3. Modify src/python/espressomd/lb.py LBFluidWalberla.validate_parasm() and friends to allow for the blocks_per_mpi_rnak parameter and apss it to the construciont of the LatticeWalberla class

4. Support in src/walberla_bridge/src/lattice_boltzmann/LBWalberlaImpl.hpp
   There might be places in that file where we implied that there is only one block per node. I did not find any at a quick glance, so all our Lb related tests from the test stuie need to be run with more than one block per mpi rnak to check that it works

[RudolfWeeber]

@hidekb

[hidekb]

Following above description, LatticeWalberla core class are modified.
Since the 7th to 9th argument for createUniformBlockGrid(calling createBlockForest) used in src/walberla_bdirdge/src/LatticeWalberla.cpp must be same as the node_grid, we follow it.

After modification, the calculation (maintainer/benchmarks/lb.py) is worked for only blocks_per_mpi_rank=(1,1,1). In other cases, calculations are terminated by walberla_agrid_sanity_checks in src/core/integrate.cpp with the message:
Traceback (most recent call last):
File "../maintainer/benchmarks/lb.py", line 144, in
system.lb = lbf
File "script_interface.pyx", line 468, in espressomd.script_interface.ScriptInterfaceHelper.setattr
File "./src/python/espressomd/system.py", line 279, in lb
lb.call_method("activate")
File "script_interface.pyx", line 172, in espressomd.script_interface.PScriptInterface.call_method
RuntimeError: waLBerla and ESPResSo disagree about domain decomposition.

It may imply that the interface between ESPResSo and walBerla in current code is implemented on the assumption that 1 Block allocate 1 CPU.
We are currently investigating the extent of the impact of this assumption, looking at other codes involved.

[RudolfWeeber]

The implementation of LatticeWalbera::get_local_domain() assumes one block per mpi rank. There is actually an assertion in there. While working on this, it is probably useful to remove -DNDEBUG from CXX flags in CMake or buidl with build tyep DEBUG, so the assertions fire.

So the implementation of that function needs to be replced. I'm pretty sure, the UniformBlcokForest class in waLBerla let's us query the boundaries for which the local MPI rank is responsible.