Cluster specific settings for flusi

Let us collect cluster specific tips and tricks here to make live easier for eveybody

IRENE (skylake)

Updated: 30/jan/2025, TE

FFTW:

./configure FC=mpif90 F77=mpif90 --prefix=$PWD --enable-sse2 --enable-openmp --enable-mpi

make -j12

P3DFFT:

./configure --prefix=$PWD --enable-intel --enable-fftw --with-fftw=${FFT_ROOT} --enable-measure --enable-stride1 LDFLAGS="-lm" FC=mpif90 CC=mpicc

make-j1 ; make install

JOBFILE:

#MSUB -r FLUSI 	# Request name
#MSUB -n 96		# Number of tasks to use. On skylake, this would be 2x24 = 48 cores per node. 
#MSUB -T 86400 		# Elapsed time limit in seconds
#MSUB -o JOB.SKY.FLUSI.%I.out 	# Standard output. %I is the job id
#MSUB -e JOB.SKY.FLUSI.%I.err	# Error output. %I is the job id
#MSUB -q skylake	# partition: skylake
#MSUB -A gen14152	# Project ID
#MSUB -m scratch		# which dataspaces this job can use
#MSUB -@ [email protected]:begin,end

set -x
cd ${BRIDGE_MSUB_PWD}

# command to run the code
RUN="ccc_mprun"
# parameter file
INIFILE="PARAMS.ini"

${RUN} ./flusi ${INIFILE}

Mesocentre (v1, not the current cluster as of 2025)

Mesocentre is located in Marseille (nord) Before doing anything, load ifort compiler + MPI environment:

module load intel/12.1 mpich2/intel/1.4.1

• FFTW: no special remarks, works as specified in the install guide

    export FFT_ROOT="${HOME}/src/fftw-3.3.3"
  

• P3DFFT: content of makeinc (in $P3DFFT_ROOT/build):

    DFLAGS = -DFFTW -DESTIMATE -DSTRIDE1
    CC = mpicc
    FF = mpif90 -c -I${FFT_ROOT}/include
    FFLAGS = -O3 
    AR = ar
    ARFLAGS = -v -r -u
    LDFLAGS= -O3 -L${FFT_ROOT}/lib -lfftw3 -lm 
    INCL = -I${FFT_ROOT}/include
  

  then just type make don't forget the include path:

    export P3DFFT_ROOT="${HOME}/src/p3dfft"
  

  To install p3dfft-2.6.1, follow the instructions for Turing. By default, it is compiled with MPICH2. When compiling with OpenMPI, before configuring, type

    grep -n 'lmpichf90' configure.ac
    grep -n 'lmpichf90' configure
  

  and then modify these two files to remove all instances of -lmpichf90.

• HDF5: is preinstalled, just type

    export HDF_ROOT="/LOGINSTO/softs/hdf5/intel/1.8.8/"  
  

• FLUSI: set compiler flags:

    export FFLAGS="-O3"
  

• Use ssh with git:

    git remote set-url origin [email protected]:pseudospectators/FLUSI.git
  

• Use LAPACK:

    Do Nothing! Just leave line 85 of makefile ("LDFLAGS += -llapack") as it is
  

• SUMMARY: After installation, in my .bash_profile, the following is appended (for simpler copy/paste):

  For use with MPICH

    module load intel/12.1 mpich2/intel/1.4.1       
    export FFT_ROOT="${HOME}/src/fftw-3.3.3"
    export P3DFFT_ROOT="${HOME}/src/p3dfft"
    export HDF_ROOT="/LOGINSTO/softs/hdf5/intel/1.8.8/" 
    export FFLAGS="-O3"
  

  For use with OPENMPI

    module load intel/13.1.2 openmpi/intel/1.6.3 hdf5/intel/1.8.8
    export FFT_ROOT="${HOME}/src/fftw-3.3.3"
    export P3DFFT_ROOT="${HOME}/src/p3dfft-2.6.1-openmpi"
    export HDF_ROOT="/softs/cemracs/selalib/intel"
    export FFLAGS="-O3"
    export PATH="${PATH}:/home/dkolom/mpi2vis"
  

Job-file

a sample OAR job file for the OARSUB system running on the mesocentre:

For use with MPICH

#!/bin/bash
#OAR -n FLUSI_A
#OAR -p cluster
#OAR -l nodes=1/core=8, walltime=24:00:00
#OAR -O FLUSI.%jobid%.output
#OAR -E FLUSI.%jobid%.ERROR
#OAR --project 13b016
#cat $OAR_NODEFILE
module load mpich2/intel/1.4.1 intel/12.1
#MPI_ARCH_PATH=/softs/openmpi/intel/1.4.5
#/softs/openmpi/intel/1.4.5/bin/mpirun -machinefile $OAR_NODEFILE -n 12 ./main PARAMS_test.ini

/softs/mpich2/intel/1.4.1/bin/mpirun -n 8 ./flusi PARAMS_test_fsi.ini

#------------------------------------
# run: oarsub -S ./job.oar
# note job.oar must be executable
#------------------------------------

For use with OPENMPI

#!/bin/bash
#OAR -n FLUSI_A
#OAR -p cluster
#OAR -q medium
##OAR -l core=32, walltime=48:00:00
#OAR -l nodes=1, walltime=48:00:00
#OAR -O FLUSI.%jobid%.output
#OAR -E FLUSI.%jobid%.ERROR
#OAR --project 14b028
module load intel/13.1.2 hdf5/intel/1.8.8 openmpi/intel/1.6.3
echo running
ulimit -s unlimited

CPUS=$(wc -l ${OAR_NODEFILE} | awk '{print $1}')
echo ${CPUS}
#mpirun -n "${CPUS}" -machinefile "${OAR_NODEFILE}" --bind-to-core -bysocket ./exeflusi.sh
#mpirun -n "${CPUS}" -machinefile "${OAR_NODEFILE}" --bind-to-core -bynode ./exeflusi.sh
mpirun -n "${CPUS}" -machinefile "${OAR_NODEFILE}" --bind-to-core ./exeflusi.sh

#------------------------------------
# run: oarsub -S ./batch.oar
# note job.oar must be executable
#------------------------------------

The content of exeflusi.sh is:

#!/bin/bash
ulimit -s unlimited
./flusi PARAMS.ini

Note that the project name changes every year. Read more about options -bysocket and -bynode on the Mesocentre website.

Unit tests

When OPENMPI is used, the tests should be submitted using OAR. Unit test directory /test and executable files flusi and mhd should be copied to /scratch disk space. Here is a sample OAR script (note that it must be executable):

#!/bin/bash
#OAR -n unit_test
#OAR -p cluster
#OAR -l nodes=1, walltime=10:00:00
#OAR -O FLUSI.%jobid%.output
#OAR -E FLUSI.%jobid%.ERROR
#OAR --project 14b028
#cat $OAR_NODEFILE
module load intel/13.1.2 hdf5/intel/1.8.8 openmpi/intel/1.6.3
echo running
ulimit -s unlimited

export nprocs=4
export mpi_command="mpirun -n ${nprocs} -machinefile "${OAR_NODEFILE}" "
export mpi_serial="mpirun -n 1 -machinefile "${OAR_NODEFILE}" "

./unittest.sh

#------------------------------------
# run: oarsub -S ./batch.oar
# note job.oar must be executable
#------------------------------------

Duke

Is M2P2's home cluster

Sugiton

This is a machine with Fedora 18 and the gfortran compiler.

• HDF5 I've simply installed

    sudo yum install hdf5-mpich hdf5-mpich-devel
  

  There is then no need to set HDF_ROOT.

• P3DFFT: the contents of the makeinc are:

    CC = mpicc
    FF = f95 -c -I/home/tommy/fftw/include -I/usr/include/mpich2-x86_64
    FFLAGS = -O3 
    AR = ar
    ARFLAGS = -v -r -u
    LDFLAGS= -O3 -L/home/tommy/fftw/lib -lfftw3 -lm 
    INCL = -I/home/tommy/fftw/include