-
Notifications
You must be signed in to change notification settings - Fork 20
Building the dependencies
Compiling all of the libraries needed to link ORAC is something of a rite of passage for the development team: a notoriously fiddly, inconsistent, and long-winded process. Different computer infrastructures cause issues in unpredictable and unreproducible ways, which change with new versions of each library. As such, the below should be seen as a travelogue through this quagmire rather than a definitive guide. This process will not be easy, it will not be quick, and you will find no sense of satisfaction at the end. Should you get through it, please add your own commentary to this page so the next soul has an easier journey.
A.C. Povey
7 Dec 2021
This document describes how to compile and install the libraries required by ORAC.
- All code is in bash, unless otherwise stated, and has largely been pasted into terminals rather than run in a script (i.e. symbols are escaped to prevent the terminal parsing them, which would not be necessary in a script).
- Tarballs for all of the libraries discussed below are stored on EODG's Oxford server at
/network/group/aopp/eodg/shared/orac/libraries/tarballs/. Links point to the website from which the source can be downloaded at the time of writing. - The argument
-j8is added tomakefor all well-structured libraries to reduce compile time. Reduce this number if your system has fewer than eight cores and remove the argument completely if compilation fails, then rerun, to get error messages in a linear manner. This argument is not recommended for ifort compilation as you run out of licenses. - These instructions use
cmakewherever possible as that is the only way to buildeccodesand it can be impossibly difficult to convince that to link against a version of netCDF/HDF that wasn't built withcmakeor isn't in the default location. If you can't use cmake, the below provides best-effort advice.
The following environment variables are used in code examples:
-
$TAR_DIRis where the tarballs are located. -
$ORAC_DIRis a copy of the ORAC git repository, which is largely the source of patch files. -
$SCRATCHis a temporary or permanent folder where the compilation will be done. This is ideally a local folder on the machine doing the compilation to reduce network delays. -
$FC,$CC,$CXXare the executables for the Fortran, C, and C++ compilers, respectively. These should be exported. -
$LIB_DIRis an exported variable pointing to the folder where you want the final installation to go.
The Adaptive Entropy Coding library is an open-source replacement for SZip, which is used by the HDF libraries. It isn't technically required by ORAC but is needed for the compression level we recommend. It is installed via,
git clone https://gitlab.dkrz.de/k202009/libaec.git
cd libaec
git checkout v1.0.6
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX="$LIB_DIR" \
-DCMAKE_BUILD_TYPE="Release" \
-G "Unix Makefiles" -Wno-dev ..
make install -j8
Different version numbers can be selected with the checkout statement.
The ENVISAT Product Reader is a Brockman Consult C package for opening and manipulating the data products from the now-defunct ENVISAT. It is installed via,
git clone git://github.com/bcdev/epr-api
cd epr-api
patch -p1 -i $ORAC_DIR/feedstocks/epr_api/atsr2.patch
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX="${LIB_DIR}" \
-DCMAKE_BUILD_TYPE="Release" \
-DCMAKE_C_FLAGS="-fPIC -w" \
-DBUILD_TESTS="yes" \
-G "Unix Makefiles" -Wno-dev ..
make -j8
make test
make install
The patch applied extends the functionality of the library to ATSR-2, which uses an almost identical format. A second patch (makefile.patch) is used when compiling within a virtual environment (such as conda), for which the linker is defined by $LD.
Hierarchical Data Format 4 is an HDF Group library for opening manipulating scientific data. It can be installed via your package manager but that tends to be horribly out-of-date and doesn't necessarily install the Fortran hooks. It is installed via,
tar -xzf $TAR_DIR/CMake-hdf-4.2.15.tar.gz
cd CMake-hdf-4.2.15
patch -p0 -i $ORAC_DIR/feedstocks/hdf4/max_files.patch
mv HDF4options.cmake HDF4options.cmake.example
cp $ORAC_DIR/feedstocks/hdf4/HDF4options.cmake HDF4options.cmake
ctest -S HDF4config.cmake,BUILD_GENERATOR=Unix,FORTRAN_LIBRARIES=ON,LIB_DIR="$LIB_DIR" -C Release -VV -O hdf4.log
cd build
make install
# HDF removed the header file we use; pull request 76 removes this dependency but this line generates the old file
sed ../hdf-4.2.15/hdf/src/hdf.inc -e "s:^[Cc]:\!:" > $LIB_DIR/include/hdf.f90
# Prevent clash with netcdf
mv $LIB_DIR/bin/ncdump $LIB_DIR/bin/h4_ncdump
mv $LIB_DIR/bin/ncgen $LIB_DIR/bin/h4_ncgen
The patch increases the maximum number of files that can be opened, which is necessary when running multiple copies of ORAC simultaneously. This causes the test f_hdftest1 to fail. The patch $ORAC_DIR/feedstocks/hdf4/gnu11.patch is also necessary when compiling with gfortran >= 10.0 and/or using locally installed copies of SZip or JPEG.
An example of building with the configure version of the library is in the feedstocks,
autoreconf -vfi
# Build static library the shared isn't compatible with Fortran and I haven't
# been able to build both.
./configure --prefix="${PREFIX}" \
--enable-linux-lfs \
--enable-silent-rules \
--with-ssl \
--with-szlib="${PREFIX}" \
--with-zlib="${PREFIX}" \
--with-jpeg="${PREFIX}" \
--disable-netcdf \
--enable-fortran \
--disable-shared
make -j ${CPU_COUNT}
make check
make install -j ${CPU_COUNT}
make installcheck
We build our own version of this because you can't build the shared libraries at the same time as the Fortran hooks and conda-forge care about the former while we want the latter. There will be some way to fix this problem, but it's probably easier to just write our HDF4 code in C.
It is possible to use the Git repository version of this code, for example,
mkdir hdf
cd hdf
git clone [email protected]:HDFGroup/hdf4
mv hdf4 hdf-4.2.15
pushd hdf-4.2.15
git checkout hdf-4_2_15
popd
ln -s hdf-4.2.15/config/cmake/scripts/CTestScript.cmake CTestScript.cmake
ln -s hdf-4.2.15/config/cmake/scripts/HDF4config.cmake HDF4config.cmake
cp $ORAC_DIR/feedstocks/hdf4/HDF4options.cmake HDF4options.cmake
ctest -S HDF4config.cmake,BUILD_GENERATOR=Unix,FORTRAN_LIBRARIES=ON,LIB_DIR="$LIB_DIR" \
-C "Release" -VV -O hdf4.log
cd build
make install
The Earth Observing System for HDF4 is maintained by Raytheon. To build v2.20,
tar -xzf $TAR_DIR/HDF-EOS2.20v1.00.tar.Z
cd hdfeos
patch -p1 -i $ORAC_DIR/feedstocks/hdfeos2/SWapi.patch
patch -p1 -i $ORAC_DIR/feedstocks/hdfeos2/string.patch
patch -p1 -i $ORAC_DIR/feedstocks/hdfeos2/cmake.patch
mkdir build
cd build
cmake -C $ORAC_DIR/feedstocks/hdfeos2/cacheinit.hdfeos.cmake -Wno-dev -DHDF4_DIR=$LIB_DIR/share/cmake -DCMAKE_INSTALL_PREFIX=$LIB_DIR -G "Unix Makefiles" ..
make -j8
make install
The test drivers are disabled in this build as they didn't write the linking scripts in cmake for several tests (which probably helped them decide to drop it). Should you be using a gfortran version < 7, you should delete the first patch in $ORAC_DIR/feedstocks/hdfeos2/cmake.patch.
There is a v3.0 of this library, but they removed the Cmake install so we instead follow the feedstocks and use,
tar -xzf $TAR_DIR/hdf-eos2-3.0-src.tar.gz
cd hdf-eos2-3.0
patch -p1 -i $ORAC_DIR/feedstocks/hdfeos2/SWapi.patch
patch -p1 -i $ORAC_DIR/feedstocks/hdfeos2/0001-fix-automake-files-for-linux-compatibility.patch
patch -p1 -i $ORAC_DIR/feedstocks/hdfeos2/configure3.0.patch
autoreconf -vfi
./configure --prefix="${LIB_DIR}" \
--with-hdf4="${LIB_DIR}" \
--with-zlib \
--with-szlib="${LIB_DIR}" \
--with-jpeg \
--enable-fortran
# Don't use -j as the dependencies are mightly buggered
make
make install
cd include
make install-includeHEADERS
This skips make check as that doesn't define all of the required libraries. For this version, building the Fortran hooks disables the shared libraries. However, I have never built a functional version of this library and neither has anyone at Goddard so we recommend you ignore it.
Hierarchical Data Format 5 is an HDF Group library for opening manipulating scientific data. It can be installed via your package manager but that tends to be horribly out-of-date and doesn't necessarily install the Fortran hooks. It is installed via,
tar -xzf $TAR_DIR/CMake-hdf5-1.12.1.tar.gz
cd CMake-hdf5-1.12.1
mv HDF5options.cmake HDF5options.cmake.example
cp $ORAC_DIR/feedstocks/hdf5/HDF5options.cmake HDF5options.cmake
ctest -S HDF5config.cmake,BUILD_GENERATOR=Unix,LIB_DIR="$LIB_DIR" -C Release -VV -O hdf5.log
cd build
make install
If using a local build of libz, you require the following patch,
--- a/HDF5options.cmake 2021-12-07 11:19:35.199204953 +0000
+++ b/HDF5options.cmake 2019-10-18 15:10:19.056288000 +0100
@@ -59,8 +59,8 @@ set (ADD_BUILD_OPTIONS "${ADD_BUILD_OPTI
#set (ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ALLOW_EXTERNAL_SUPPORT:STRING=GIT")
### ext libs on system
set (ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ALLOW_EXTERNAL_SUPPORT:STRING=NO")
-#set (ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DZLIB_LIBRARY:FILEPATH=${LIB_DIR}/lib/libz.a -DZLIB_INCLUDE_DIR:PATH=${LIB_DIR}/include")
-set (ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DSZIP_LIBRARY:FILEPATH=${LIB_DIR}/lib/libsz.a -DSZIP_INCLUDE_DIR:PATH=${LIB_DIR}/include")
+set (ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DZLIB_LIBRARY:FILEPATH=${LIB_DIR}/lib/libz.a -DZLIB_INCLUDE_DIR:PATH=${LIB_DIR}/include")
+set (ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DSZIP_LIBRARY:FILEPATH=${LIB_DIR}/lib/libszip.a -DSZIP_INCLUDE_DIR:PATH=${LIB_DIR}/include")
### disable using ext zlib
#set (ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=OFF")
network common data format is a scientific data format that ORAC uses for it's outputs. The cmake build for v4.8.1 doesn't appear to work, but the git repository is fine so we'll use that,
git clone [email protected]:Unidata/netcdf-c.git
cd netcdf-c
git checkout v4.8.1
mkdir build
cd build
cmake -DCMAKE_PREFIX_PATH="$LIB_DIR" \
-DCMAKE_BUILD_TYPE="Release" \
-DENABLE_DAP=OFF \
-DBUILD_SHARED_LIBS=ON \
-DENABLE_HDF4=ON \
-DNC_EXTRA_DEPS=jpeg \
-DCMAKE_INSTALL_PREFIX="$LIB_DIR" \
-G "Unix Makefiles" -Wno-dev ..
make
make test
make install
For v4.8.1, the test ncdump_test_rcmerge fails, but doesn't appear to be something we care about.
For v4.9.0 with gfortran, you must remove the CMAKE_BUILD_TYPE line as compiler optimisations break the library. This is a known issue and will hopefully be resolved.
Compiling in a virtual environment is a bit of a pain. Various solutions exist and the one I'm currently using is,
cmake -DCMAKE_PREFIX_PATH="$CONDA_PREFIX:$LIB_DIR" \
-DLIBXML2_LIBRARY="$LIB_DIR/lib/libxml2.so" \
-DCMAKE_BUILD_TYPE="Release" \
-DENABLE_DAP=OFF \
-DBUILD_SHARED_LIBS=ON \
-DENABLE_HDF4=ON \
-DNC_EXTRA_DEPS=jpeg \
-DCMAKE_INSTALL_PREFIX="$LIB_DIR" \
-G "Unix Makefiles" -Wno-dev ${CMAKE_ARGS} ..
The problem is with cmake using a single path for all links, such that you cannot use any libraries in a location alongside ones you do not want to use. (For example, you want to use your system's libm but not its libnetcdf; that appears to not be possible here you'll need to ensure libm is in your virtual environment.) Troubleshooting involves looking carefully through the libraries linked output of cmake for any folders you don't want and adding variables of the form NAME_LIBRARY=LOCATION to force cmake to use it.
These are the Fortran hooks for libnetcdf. It builds by,
git clone [email protected]:Unidata/netcdf-fortran.git
cd netcdf-fortran
git checkout v4.5.4
mkdir build
cd build
cmake -DCMAKE_PREFIX_PATH="$LIB_DIR" \
-DnetCDF_LIBRARIES="$LIB_DIR/lib/libnetcdf.so" \
-DnetCDF_INCLUDE_DIR="$LIB_DIR/include" \
-DCMAKE_BUILD_TYPE="Release" \
-DBUILD_SHARED_LIBS=ON \
-DNC_EXTRA_DEPS=jpeg \
-G "Unix Makefiles" -Wno-dev ..
make
make test
make install
where we've specified the location of libnetcdf.so precisely as there is some bug in the use of CMAKE_PREFIX_PATH. To build with gfortran from a conda environment, I had to append $LIB_DIR to $LD_LIBRARY_PATH.
Traditionally, this is installed in a separate directory as it clobbers the C library. It appears to be possible to avoid that clobber by replacing the last line above with,
mv $LIB_DIR/include/netcdf.inc $LIB_DIR/include/netcdf.c.inc
make install
mv $LIB_DIR/include/netcdf.inc $LIB_DIR/include/netcdff.inc
mv $LIB_DIR/include/netcdf.c.inc $LIB_DIR/include/netcdf.inc
It is left as an exercise for the future to work out what patch would implement the renaming of this single file.
Greg McGarragh put together this small Fortran library for reading SEVIRI data files. It is installed by,
git clone git://github.com/gmcgarragh/seviri_util
cd seviri_util
cp $ORAC_DIR/feedstocks/seviri_util/make.inc make.inc
make all
cp libseviri_util.a $LIB_DIR/lib
cp seviri_util.mod $LIB_DIR/include
Compiling in a virtual environment desires,
patch -p1 -i $ORAC_DIR/feedstocks/seviri_util/conda.patch
PREFIX=$CONDA_PREFIX make all -j8
instead of the make line above.
Simon Proud put together this small Fortran library for reading Himawari data files. It is installed by,
git clone [email protected]:simonrp84/Himawari_HSD_Reader.git
cd Himawari_HSD_Reader
cp $ORAC_DIR/feedstocks/hsd_reader/make.inc make.inc
cp $ORAC_DIR/feedstocks/hsd_reader/Makefile Makefile
make
cp *.a $LIB_DIR/lib
cp *.mod $LIB_DIR/include
cp AHI $LIB_DIR/bin
The repository doesn't include a Makefile or installation commands, so we do those ourselves. When using ifort, you will instead desire CCFLAGS = -g -O3 -I. -qopenmp and F90FLAGS = -g -O3 -I. -cpp -qopenmp. When using a conda compiler, you will conda.patch in the feedstocks folder.
ecCodes is a replacement for GRIB-API, a set of tools for manipulating GRID and other file formats developed by ECMWF. It is installed by,
tar -zxf $TAR_DIR/eccodes-2.26.0-Source.tar.gz
cd eccodes-2.26.0-Source
mkdir build
cd build
cmake -DCMAKE_PREFIX_PATH="$LIB_DIR" \
-DENABLE_JPG=ON \
-DENABLE_FORTRAN=ON \
-DCMAKE_INSTALL_PREFIX="$LIB_DIR" \
-G "Unix Makefiles" -Wno-dev ..
make
ctest
make install
The ECMWF interpolation library is a discontinued pain in everyone's backside and I can never apologise enough for adding it to ORAC. We will one day get rid of it. It is installed via,
tar -zxf $TAR_DIR/libemos-4.5.9-Source.tar.gz
cd libemos-4.5.9-Source
patch -p1 -i $ORAC_DIR/feedstocks/libemos/cmake.patch
patch -p1 -i $ORAC_DIR/feedstocks/libemos/init.patch
mkdir build
cd build
cmake -DCMAKE_PREFIX_PATH="$LIB_DIR" \
-DECCODES_PATH="$LIB_DIR" \
-DENABLE_GRIBEX_ABORT=OFF \
-DCMAKE_INSTALL_PREFIX="$LIB_DIR" \
-G "Unix Makefiles" -Wno-dev ..
make
make test
make install
More recent versions of gfortran also require -DCMAKE_Fortran_FLAGS='-fallow-argument-mismatch'.
These libraries have associated terms and conditions that you must agree to before attempting any compilation or other use of the code. We accept no liability for you not doing that.
Numerical Recipes are an exceedingly useful series of textbooks outlining algorithms for common computational tasks. You should go buy a copy - it's really useful. As we're academic users that own many copies of these books, the license permits us to use their code within ORAC. If you're a commercial user, this dependency is optional and you can ignore it.
We have packaged the set of routines from the book into a library. Please ask the ORAC development team for details of how to download or create that. To install,
mkdir nr
cd nr
tar -zxf $TAR_DIR/nr.tar.gz
patch -p1 -i $ORAC_DIR/feedstocks/nr/data.patch
patch -p1 -i $ORAC_DIR/feedstocks/nr/shoot.patch
make -j8
cp libnr.a $LIB_DIR/lib
The Makefile included in that package is hardcoded for gfortran compilation. A patch is included in the feedstocks directory for compilation in a virtual environment (i.e. conda). The following patch allows ifort compilation,
patch -p1 <<EOF
Subject: Replace compiler with ifort
--- nr/Makefile 2019-10-21 15:35:27.793714147 +0100
+++ new/Makefile 2019-10-21 15:36:30.409410000 +0100
@@ -1,6 +1,5 @@
-F77 = gfortran
-F77FLAGS = -O3 -Dfloat=double -std=legacy -Wno-conversion -fno-range-check \\
- -Wno-unused-dummy-argument -Wno-unused-function -fopenmp
+F77 = ifort
+F77FLAGS = -O3 -autodouble -warn unused -warn uncalled -qopenmp
OBJECTS = addint.o \\
airy.o \\
EOF
where the double-slashes ensure your terminal does not interpret those as continuation characters.
SZip is a compression library made by the HDF Group. It is free for non-commercial use and isn't technically required by ORAC but is needed for the compression level we recommend. It has been superceded by libAEC but if you want the old version, installation is done with,
tar -zxf $TAR_DIR/szip-2.1.1.tar.gz
cd szip-2.1.1
mkdir build
cd build
cmake -C ../config/cmake/cacheinit.cmake \
-DCMAKE_INSTALL_PREFIX=$LIB_DIR \
-DCMAKE_C_FLAGS="-fPIC" \
-G "Unix Makefiles" -Wno-dev ..
make -j8
make test
make install
cd $LIB_DIR/lib
ln -s libszip-static.a libszip.a
ln -s libszip-shared.so libszip.so
We note that compilation can fail for icc with the error _LIB_VERSION_TYPE is undefined, which appears to be a known issue in ifort's 2017 version and an early release of the 2018 version. If you must use it, you will need to use gcc instead.
An alternative method is,
autoreconf -vfi
./configure --prefix="${PREFIX}"
make -j${CPU_COUNT}
make check
make install -j${CPU_COUNT}
You may wish to remove the file psdes.f from the compilation as it fills 4-bit integers with 16-bit hex variables and some compilers do not like that.
--- a/Makefile 2021-11-30 16:40:20.000000000 +0000
+++ b/Makefile 2022-07-06 16:52:13.027626717 +0100
@@ -230,3 +230,2 @@
probks.o \\
- psdes.o \\
pwt.o \\
Radiative Transfer for TOVs is a fast radiative transfer code developed by the Met Office for NWP SAF. You must agree to appropriate licenses by creating an account through their portal to download the source. It is installed by,
mkdir rttov131
cd rttov131
tar -xf $TAR_DIR/rttov131.tar.xz
patch -p1 -i $ORAC_DIR/feedstocks/rttov/local.patch
rm -rf rtcoef_rttov13
ln -s $RTTOV_DIR rtcoef_rttov13
REL_PATH=`python -c "import os.path; print(os.path.relpath('$LIB_DIR', '$PWD'))"`
cd src
../build/Makefile.PL RTTOV_HDF=1 RTTOV_F2PY=1
make ARCH=ifort-openmp INSTALLDIR=$REL_PATH
cd ../rttov_test
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$LIB_DIR/lib ./test_fwd.sh ARCH=ifort-openmp BIN=$REL_PATH/bin
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$LIB_DIR/lib ./test_rttov12.sh ARCH=ifort-openmp BIN=$REL_PATH/bin
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$LIB_DIR/lib ./test_solar.sh ARCH=ifort-openmp BIN=$REL_PATH/bin
Installation using conda compilers requires a second patch, $ORACDIR/feedstocks/rttov/arch.patch.
These libraries should be included within your system's default install, and kept up-to-date by your package-management system. These instructions are for when that doesn't work.
The Fastest Fourier Transform in the West is an inventively named numerical library that is required by EMOS (to evaluate the spectral harmonics of ECMWF's grid). Installation is done with,
tar -zxf $TAR_DIR/fftw-3.3.10.tar.gz
cd fftw-3.3.10
./configure --exec-prefix=$LIB_DIR --prefix=$LIB_DIR --enable-shared
make -j8
make check
make install
We note that compilation fails with icc for the error _LIB_VERSION_TYPE is undefined, which appears to be a known issue in ifort's 2017 and 2018 versions. If you must use one of those, you will need to use gcc instead. Our [feedstock]((https://github.com/ORAC-CC/orac/blob/master/feedstocks/fftw/build.sh) is a fork of the conda-forge installation.
libjpeg is an image compression library used by netCDF and HDF. Installation is done with,
tar --warning=no-unknown-keyword -zxf $TAR_DIR/JPEG8d.tar.gz
cd JPEG8d
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX="$LIB_DIR" \
-DCMAKE_BUILD_TYPE="Release" \
-DBUILD_SHARED_LIBS="ON" \
-DJPEG_PACKAGE_EXT="" \
-DBUILD_TESTING="ON" \
-G "Unix Makefiles" -Wno-dev ..
make install -j8
This uses an older version of the library as they removed the cmake build suite. Some versions of the installation require the setting of symlinks as the libraries are appended with -static and -shared.
XML2 is a parsing library used by HDF. Installation is done with,
tar -xf $TAR_DIR/libxml2-2.9.14.tar.xz
pushd libxml2-2.9.14/
./configure --prefix="$LIB_DIR" \
--with-python="$CONDA_PREFIX/bin/python3" \
--with-history \
--disable-static
make -j8
make check
make install
You should replace $CONDA_PREFIX with an appropriate path for your system.
Z is a compression library used by netCDF and HDF. Installation is done with,
cd $SCRATCH
tar -zxf $TAR_DIR/zlib-1.2.11.tar.gz
cd zlib-1.2.11
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX="$LIB_DIR" \
-DCMAKE_C_FLAGS="-fPIC" \
-G "Unix Makefiles" -Wno-dev ..
make -j8
make test
make install
- User Guide
- ORAC
- Developer Guide