Skip to content

Installation

Jump to bottom
Adam Povey edited this page Nov 30, 2023 · 23 revisions

Licensing

Some dependencies of the ORAC library require users to obtain a license. Before installing the software, please review the terms and conditions at the following sites:

You must have obtained these licenses before using any pre-compiled binaries related to this project. If this is a problem, please contact us on DEVORAC at jiscmail.ac.uk to discuss alternatives.

Installation

There are three methods to install the ORAC processor. We currently only support Linux. Due to licensing issues, we cannot distribute the library through a public channel but it is available on the JASMIN cluster; just apply for access to the Aerosol CCI group workspace. If you are not eligible for access, please contact us on DEVORAC at jiscmail.ac.uk and we can direct you to the files.

conda

Conda is a package manager to create virtual environments and install/update (largely Python) software. It can be installed in the folder $PREFIX with,

PREFIX=/the/place/you/want/this/to/go
TMPDIR=$(mktemp -d)
pushd $TMPDIR
wget "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh"
chmod 0700 Miniconda3-latest-Linux-x86_64.sh
./Miniconda3-latest-Linux-x86_64.sh -b -p "$PREFIX"
source "${PREFIX}/etc/profile.d/conda.sh"
popd
rm -rf $TMPDIR
conda activate

The script should edit your .bashrc to automatically load conda when you start an interactive shell. We recommend also installing mamba as it is substantially faster at installing packages. This is done with conda install -y mamba and, thereafter, you use mamba install or update rather than conda install or update.

Default channels

Conda distributes software through "channels", which can be a folder or web address. We can save effort pointing to that folder every time by setting up a symlink,

ln -s /gws/nopw/j04/aerosol_cci/conda_channel $CONDA_PREFIX/conda-bld

where you must have access to the Aerosol CCI group workspace to access that folder. If not, replace that path with wherever you decompressed the ORAC channel. Thereafter, you can use -c local to tell conda to took in the ORAC channel for packages. (This is a bad idea if you ever intend to build your own packages with conda.) The instructions to follow assume you did this.

We can further save effort by creating the file .condarc in your home folder to define the default channels that should be searched. We recommend the file contain,

channels:
  - conda-forge
  - avalentino
  - defaults
  - local

14 Feb 2022: At the moment, mamba cannot use the avalentino channel as it has unexpected formatting in its metadata. As such, the third line of the above should be removed if you wish to use mamba.

Minimal use: Pre-compiled binaries

If you wish to use the code as-is (e.g. without compiling or changing anything), the ORAC software library and it's dependencies can be installed using conda. We recommend installing ORAC into a fresh environment as many of the dependencies clash with common packages from popular conda channels and (until they fix feature management) our installation is likely to break any existing environment.

To install ORAC call,

conda create -y -n orac_release --override-channels -c local -c conda-forge pyorac

You may replace orac_release with any other name you desire and local with the path to the ORAC channel. If you would like multi-threaded processing instead call,

conda create -y -n orac_release --override-channels -c local -c conda-forge pyorac "orac=*=mpi_openmpi*"

Auxiliary files

ORAC requires a variety of satellite, meteorological, and geographical data to run. A brief summary of them can be found here. A more detailed summary SHOULD BE DRAFTED.

The code requires a number of Static Application Data (SAD) files,

  • instrument channel characteristics (one file per instrument channel),
  • aerosol or cloud radiative properties for each supported cloud class in each instrument channel (Look Up Tables). These are stored in the Aerosol and Cloud CCI group workspaces or can be requested from us.

Python scripts

The recommended method to run the ORAC suite is with the provided Python script, orac.py, which is installed by pyorac and is described here. To avoid typing lengthy paths with every call, these scripts use a file called local_defaults.py to list the default locations of all auxiliary files. The version of that file distributed with the binaries assumes that a user:

If either of these are not true, or you wish to control other optional information, please edit the file $CONDA_PREFIX/envs/orac_release/lib/python{pyversion}/site-packages/pyorac/tools/pyorac/local_defaults.py, where ${pyversion} is the main release number you have installed, of the form 3.X, and can be determined with python --version`

An example local defaults is kept at here. For this minimal installation, you will desire:

ORAC_DIR = os.environ["CONDA_PREFIX"] + "/bin"
ORAC_LIB  = os.environ["CONDA_PREFIX"] + '/share/orac/lib.inc'

which points the Python scripts to an ORAC settings file.

Typical use: Compile ORAC from source

ORAC can also be compiled from source using the conda dependencies by adapting the instructions above. Make a clone of the ORAC repository and prepare the configuration with,

git clone https://github.com/ORAC-CC/orac.git
cd orac
mkdir common/obj
mkdir pre_processing/obj
mkdir src/obj
mkdir post_processing/obj
mkdir derived_products/broadband_fluxes/obj
mkdir derived_products/broadband_fluxes/obj/bugsrad
mkdir derived_products/broadband_fluxes/obj/fu_liou
export ORACDIR=$PWD

Note that we exported the path to the ORAC software for use later in these instructions. You will want to add this definition to your local environment (e.g. by putting it in your .login or .bashrc file) as the activation scripts below use it (you could hardcode the location but this allows you to easily use multiple ORAC versions, such as a release and development).

We create the build environment with,

conda activate
conda env create -y --file ${ORACDIR}/feedstocks/dependencies.openmpi.yml

You may select the name of the environment, which defaults to orac_git, using -n YOUR_NAME. If the ORAC conda channel is not local to your anaconda installation, use --override-channels -c PATH_TO_CHANNEL -c conda-forge. You may replace the second conda above with mamba, which is faster. If you desire single-threaded compilation (which is required on JASMIN's LOTUS cluster), replace openmpi with nompi.

We then copy over the activation/deactivation scripts,

SCRIPT_DIR=$CONDA_PREFIX/envs/orac_git/etc/conda
for CHANGE in "activate" "deactivate"; do
    mkdir -p "${SCRIPT_DIR}/${CHANGE}.d"
    cp "${ORACDIR}/feedstocks/orac/${CHANGE}.sh" "${SCRIPT_DIR}/${CHANGE}.d/${CHANGE}-orac.sh"
done
cat <<EOF >> $SCRIPT_DIR/activate.d/activate-orac.sh
# Point to this environment's libraries
if [ -n "\${ORAC_LIB}" ]; then
    export ORAC_LIB_SAVE="\${ORAC_LIB}"
fi
export ORAC_LIB="\${ORACDIR}/config/lib.conda.inc"
if [ -n "\${ORAC_ARCH}" ]; then
    export ORAC_ARCH_SAVE="\${ORAC_ARCH}"
fi
export ORAC_ARCH="\${ORACDIR}/config/arch.conda.inc"

if [ -n "\${PYTHONPATH}" ]; then
    export PYTHONPATH_SAVE="\${PYTHONPATH}"
fi
export PYTHONPATH="\${ORACDIR}/tools:\${CONDA_PREFIX}/lib:\${PYTHONPATH}"
EOF
cat <<EOF >> $SCRIPT_DIR/deactivate.d/deactivate-orac.sh
if [ -n "\${ORAC_LIB_SAVE}" ]; then
    export ORAC_LIB="\${ORAC_LIB_SAVE}"
    unset ORAC_LIB_SAVE
else
    unset ORAC_LIB
fi
if [ -n "\${ORAC_ARCH_SAVE}" ]; then
    export ORAC_ARCH="\${ORAC_ARCH_SAVE}"
    unset ORAC_ARCH_SAVE
else
    unset ORAC_ARCH
fi

if [ -n "\${PYTHONPATH_SAVE}" ]; then
    export PYTHONPATH="\${PYTHONPATH_SAVE}"
    unset PYTHONPATH_SAVE
else
    unset PYTHONPATH
fi
EOF

You'll see that we have added three clauses to the activation files. These define three environment variables,

  • ORAC_LIB is used by the makefiles and Python scripts to direct ORAC to the location of the external libraries. Those are all in the conda installation.
  • ORAC_ARCH is used by the makefiles to define the arguments given to the compiler.
  • PYTHONPATH tells Python where to go looking for modules. We add the pyORAC directory to it, as well as the conda library folder where RTTOV's Python wrapper is stored. Other means of getting pyORAC into Python are available if you know what you're doing.

You then need to create a local defaults file in tools/pyorac. A generic example can be found in that directory while the specific example distributed with the binaries can be found in feedstocks/orac.

Compiling

To compile the code,

cd $ORACDIR
conda activate orac_git
make

Intense use: Compile all dependencies from source

This is the previous method of installing ORAC and is not recommended for most users as many of these dependencies can be difficult to compile. If you intend to run on JASMIN, you may find this page useful as it outlines the existing libraries and resources.

Prerequisites

A Fortran 2003 compiler is required. The ORAC code is developed and tested on GNU/Linux platforms using gfortran (versions 7.5.0, 12.1.0), the Intel ifort compiler (versions 17, 18.0.5), and PGI (version 21.9-0). We expect other compilers will work with minimal effort and the code is also known to compile and run on MacOS using gfortran.

The bison and flex tools are required, both of which are standard development tools available on GNU/Linux. Version 3 of bison is preferred. Version 2 will work, but you must comment line 17 and uncomment line 22 here.

Libraries

Numerous libraries are required by the ORAC code and their installation is explained here. For each, there is a directory in feedstocks. The location of the source code is specified in the meta.yaml file under the source: tag with sub-headers,

  • git_url pointing to a git repository;
  • url pointing to a downloadable tarball;
  • path pointing to files in the feedstocks directory which should be manually copied across (to the destination specified by folder or the top directory);
  • patch specifying patch files that should be called with patch -p1 -i $PATCHFILE from the top directory.

The build.sh file outlines how we expect the library to be compiled, but should not be taken as definitive instructions for installation. The exact flags and patches required appear to be highly dependent on your machine architecture. If you are having difficulty, please email the ORAC Developer list (DEVORAC at jiscmail.ac.uk) for advice.

Some specific notes that may be of use to you but may have been fixed in later version:

  • Your Linux distribution may include some of these libraries already. If you are compiling them yourself (e.g. to include additional features), make doubly certain that your linker points to your compiled library and not the system default. This is our most common cause of compilation difficulties.
  • EMOS library The ECMWF EMOS library is currently required but has proven to be difficult to install (recommendations for an alternative happily accepted). It can be found at https://software.ecmwf.int/wiki/display/EMOS/Releases.
    • This library conflicts with the HDF library as both contain a routine called init, which is repaired with a patch.
    • You must include -DENABLE_GRIBEX_ABORT=OFF as we still use the old syntax.
    • The default configuration for ifort is incorrect. The LARGE_FILE flag should be set to -D_LARGEFILE64_SOURCE, -Vaxlib has been depreciated, the DEBUG flag should be set to -O2, and you may wish to use the icc compiler rather than gcc. (See the attached script.)
    • The default configuration for gfortran is partially incorrect. The FFLAGS variable should be set to -ffixed-line-length-0. (See the attached script.)
    • Verbose output from the library, for debugging, can be turned on by setting an environment variable JDCNDBG=1.
    • This links against the FFTW library, which is usually available in your distribution, but can be downloaded at http://www.fftw.org. Please build both the single and double precision libraries.
  • NetCDF The NetCDF library is required to read/write input/output files. The code has been successfully compiled with NetCDF versions 3 and 4 although NetCDF 4 is recommended. The core NetCDF C library and API is required along with the NetCDF Fortran API, which is shipped separately. They can be downloaded from the Unidata website: http://www.unidata.ucar.edu/downloads/netcdf/index.jsp
    • The library should be compiled from source with Fortran support enabled. If ORAC is to be run on a GNU/Linux machine, using the gfortran compiler, a pre-compiled version the NetCDF library can usually be installed directly using the distributions package manager. Both the NetCDF library and the associated development package, a.k.a "dev" package, should be installed.
  • RTTOV RTTOV is used to calculate the clear-sky transmittances and radiances and can be downloaded from http://nwpsaf.eu/site/software/rttov/downloads. The current version of ORAC requires versions 13.
    • You will need to edit Makefile.local to point to your netCDF4 or HDF5 library directory. We recommend using HDF5 rather than netCDF4. The rttov_compile.sh script can be used to simplify compilation, we recommend compiling with OpenMP support (if appropriate to your architecture) as this enables a significant improvement in execution time.
    • You will also require the emissivity and BRDF atlases along with the RT coefficient files for any sensors you wish to process. At the time of writing, these could be found (at the same website) in:
      • uw_ir_emis_atlas_hdf5.tar
      • uw_ir_emis_atlas_covariances_hdf5.tar
    • The RT coefficient files for ORAC-supported sensors are included as part of the RTTOV distribution. Any future coefficient updates will be available from https://nwpsaf.eu/site/software/rttov/download/coefficients/coefficient-download/
  • No issues have yet been encountered installing the following libraries with their default settings:
  • Optional Dependencies The following libraries are optional, but recommended, in the current ORAC code:

Building the code

All parts of ORAC are compiled using Makefiles. This is controlled by two files:

  • The archfile, which sets the Fortran and C compilers to be used and the settings for those compilers. It is specified by the environment variable ORAC_ARCH.
  • The libfile, which lists the paths to all the libraries required by the compilers. It is specified by the environment variable ORAC_LIB.

Examples can be found in the config folder.

The OBJS variable controls where the compiled object and module files are stored. We recommend setting and, when checking out a fresh copy from the repository, execute the following,

mkdir common/obj
mkdir pre_processing/obj
mkdir src/obj
mkdir post_processing/obj
mkdir derived_products/broadband_fluxes/obj
mkdir derived_products/broadband_fluxes/obj/bugsrad
mkdir derived_products/broadband_fluxes/obj/fu_liou

You can also set OBJS to a full path and all object and module files will be stored in a single folder.

Calling make from the trunk folder should compile all of ORAC. A single part may be compiled by calling make from within it's folder, though all parts of ORAC are dependent on the contents of the common folder. To remove the compilation, call make clean. The command make depend is available to automatically generate file dependencies, which is useful if you add modules to the code.

Note that ORAC responds very well to compiler optimisations, so if you are planing to do extensive processing with the code, you should consider using the appropriate flags (e.g. -O2 or -O3 will work wonders on most compilers). ORAC is parallelized using the OpenMP API for parallel execution on multiple cores. The flags are -qopenmp on ifort and -fopenmp on gfortran.

The -cpp compiler flag is used to enable pre-processing via the C pre-processor. This flag may vary depending on your compiler and version.

Optional modes

A number of optional features are included within the ORAC code. These are implemented using C-pre-processor statements and can be activated by adding -DX to the INC variable defined in your archfile, where X is the desired macro.

  • DEBUG Debug mode, which prints additional warning messages and text outputs.`
  • BKP Breakpoint mode, where details of most mathematical calculations are printed to stdout. While not depreciated, this feature is poorly maintained. It is possible to compile only a single routine with BKP on, but you must re-compile the executable as well or the breakpoint file will not be opened. Note that breakpoint output from different routines within ORAC can be switched in and out by changing the breakpoint level set in OracConstants_m or setting the breakpoint level in the driver file. Only run in the breakpoints configuration for a small number of pixels as the code runs slowly and produces substantial output.
  • WRAPPER The default installation of ORAC creates three executables. Wrapper mode compiles these programs as subroutines that can be called from a single executable.
  • USE_TIMING Prints details of how long each step of the main processor took to execute.`
  • USE_SLATEC Uses the SLATEC package for matrix inversion rather than LAPACK.
  • COMPATIBILITY_MODE Is a debugging mode for the Cox-Munk sea surface parameterisation, using sin and cos to calculate all angles directly.
  • INCLUDE_RTTOV_OPENMP Use multithreading in RTTOV, which must have been compiled with multithreading support.
  • The optional dependencies each have an appropriate macro: INCLUDE_ATSR_SUPPORT, INCLUDE_NR, INCLUDE_FU_LIOU_SUPPORT, INCLUDE_HIMARAWI_SUPPORT, INCLUDE_SEVIRI_SUPPORT.

Running the code

By default ORAC is compiled to use dynamic linking for HDF, HDF5, NetCDF and EPR-API and therefore the LD_LIBRARY_PATH must contain paths to these libraries. The following python one-liner may be of use,

LD_LIBRARY_PATH=$(cd $ORACDIR/trunk/tools; python -c "import os; import orac_utils as ou; print(ou.build_orac_library_path(ou.read_orac_libraries(os.environ['\''ORAC_LIB'\''])))")

History

  • 23-May-2011, Andy Smith
  • 13-Dec-2011, Matthias Jerg
  • 12-Jan-2012, Gareth Thomas
  • 14-Jun-2012, Caroline Poulsen
  • 08-Nov-2013, Adam Povey
  • 25-Jul-2016, Adam Povey
  • 26-Jul-2016, Greg McGarragh
  • 16-Oct-2016, Caroline Poulsen
  • 24-Mar-2017, Simon Proud
  • 19-Feb-2019, Adam Povey
  • 14-Feb-2022, Adam Povey
Clone this wiki locally