INSTALL

Installation
============

This documentation provides installation instructions for HARP.

HARP is a toolset for ingesting, processing and inter-comparing satellite data
against correlative data. The toolset is composed of a set of command line
tools and a library of analysis functions.


Supported platforms
-------------------

HARP is designed to run on most Unix-based operating systems (including Linux
and macOS) and Windows. The platforms that are supported include at least
Linux, Windows, and macOS.

What you will need
------------------

  - The HARP package : You can choose between a source installation
    (``harp-x.y.z.tar.gz``) or a binary installation (``harp-x.y.z-win64.msi``
    (Windows only)).
    Note that x.y.z stands for the version number of the package.
    The source package contains the source code for the HARP components (C
    Library and command line tools) together with all documentation.
    You will need this package if you want to use HARP on a Unix-based system.
    For Windows you will only need this package if you want to have access to
    the HARP source (if, for instance, you want to extend/improve HARP).
    The binary package, which is only available for Windows, contains pre-built
    versions of everything from the source distribution (without the sources)
    and all documentation.
    If you do not have the HARP package, you can download it from GitHub:

        https://github.com/stcorp/harp/releases/latest

  The following items are needed if you use the HARP source distribution:

  - A C compiler : Most Unix platforms come with their own C compiler so this
    should not be a problem. For macOS you should make sure you have installed
    the Developer Tools.

  - A recent version of CODA. You will need a version of CODA that is compiled
    using the same HDF4 and HDF5 libraries as you will use for building HARP.
    CODA can be downloaded from https://github.com/stcorp/coda/releases/latest

  - A recent version of HDF4. You will also need the additional libraries
    libjpeg, zlib, and szlib depending on whether HDF4 was build against them.
    In order to prevent conflicts of the HDF4 library with the netCDF interface
    that HARP uses, make sure you build the HDF4 software from source and use
    the following options when configuring the package::

       ./configure --enable-shared --disable-fortran --disable-netcdf

    Make sure that after installation the location of the libraries is in your
    shared library path (e.g. update your ``LD_LIBRARY_PATH`` environment
    variable).

  - A recent version of HDF5 (version 1.8.0 or higher). You will also need the
    additional libraries zlib, and szlib depending on whether HDF5 was build
    against them.

  - If you plan on using the Python interface you need either Python 2
    (version 2.7) or Python 3 (version 3.6 or higher).
    You will also need the Python packages ``numpy`` (version 1.4 or higher)
    and ``cffi`` (version 1.5.2 or higher).

  - If you plan on using the R interface you need R version 3 or higher.


Using GitHub
------------

HARP is also available as source code directly from GitHub, but this is only
recommended to be used by users who want to co-develop HARP and it will only
work on Unix-based systems.
You will need to have additional software installed and need to perform
additional steps if you want to build HARP from the GitHub git repository.

Additional software that you will need:
  - autoconf + automake + libtool
  - flex + bison

When you clone and checkout the HARP git repository make sure to also
initialize the git submodules::

    $ git clone --recurse-submodules https://github.com/stcorp/harp.git

This will pull in several entries in the definitions subdirectory.

Then, run the bootstrap script in the root of the source directory::

    $ cd harp
    $ ./bootstrap

After that you can follow the steps as usual for building the source package.

Note that the CMake build scripts that come with HARP can also be used to
build HARP, but they do not allow regeneration of automatically generated
files (such as the documentation), so they are not recommended for performing
co-development of HARP.

Using Conda
-----------

HARP is also available as a conda package for Windows, Linux, and macOS
(only 64bit and Python3). You can install HARP using conda with::

    $ conda install -c conda-forge harp

The R interface of HARP is provided as a separate conda package (that has
the core HARP package as dependency). You can install this using::

    $ conda install -c conda-forge r-harp

Note that these installations of HARP do not include the MATLAB or IDL
interfaces.

Also be aware that if you use conda within an Anaconda installation to first
create a separate environment using e.g. `conda create -n harp` and
`conda activate harp`, otherwise the conda installation will likely fail.
The recommended installation of conda is Miniforge or Mambaforge as this will
ensure that all packages are taken from the conda-forge channel by default.


Building the source package (Unix)
----------------------------------

To build the source package, make sure to download the official HARP source
package, which is named harp-x.y.z.tar.gz. Don't use the packages called
'Source code' from GitHub as these are contents of the git repository (see
'Using GitHub' above).

The HARP source package comes with both an autotools and a CMake build system.
For Unix-based systems the recommended approach is to use the autotools based
system (using the configure script), which is what is described below.
Building with CMake is also supported, but not documented here.

The following steps will guide you through the process of building the HARP
library and executables:

 1) Go to the directory that contains the downloaded harp-x.y.z.tar.gz
    file and unpack this package::

        $ gzip -d harp-x.y.z.tar.gz
        $ tar xf harp-x.y.z.tar

    Make sure you replace ``x.y.z`` with the appropriate version number.

 2) You should now have a ``harp-x.y.z`` directory. Go to this directory::

        $ cd harp-x.y.z

 3) Next you will have to execute the ``configure`` script that checks what
    system you are on and sets up the build environment. There are several
    options you can pass to this script. The most important ones are:

    ``--prefix <HARP installation directory>`` :
        By default, if you perform an installation of the HARP package
        (see further below on how to do this) all
        files are installed in subdirectories under ``/usr/local``.
        Executables will appear under ``/usr/local/bin``, libraries under
        ``/usr/local/lib`` and all data files (e.g. documentation) under
        ``/usr/local/share/harp``.
        However, installing the files into the default places requires you to
        have administrator privileges, which you may not have. In order to
        install HARP in a different location where you do have permission to
        copy files to, you can use this option to provide a different
        installation directory. For instance, you can use
        ``--prefix=$HOME/harp`` to install the software in the subdirectory
        ``harp`` of your home directory.

    ``CODA_LIB=<CODA library directory>`` :
        The configure script will automatically try to locate the CODA library
        in the default locations for libraries (``/usr/local/lib`` is usually
        not considered a default location!).
        If ``./configure`` complains that it can't find the ``coda`` library
        files, pass this option to ``./configure`` with the location of this
        library file.

    ``CODA_INCLUDE=<CODA include file directory>`` :
        The configure script will automatically try to locate the CODA include
        files in the default locations for include files
        (``/usr/local/include`` is usually not considered a default location!).
        If ``./configure`` complains that it can't find the ``coda.h`` include
        file, pass this option to ``./configure`` with the location of this
        include file.

    ``HDF4_LIB=<HDF4 library directory>`` :
        The configure script will automatically try to locate the HDF4
        libraries in the default locations for libraries (``/usr/local/lib``
        is usually not considered a default location!).
        If ``./configure`` complains that it can't find the ``df``, ``hdf``,
        or ``mfhdf`` library files, pass this option to ``./configure`` with
        the location of these library files.

    ``HDF4_INCLUDE=<HDF4 include file directory>`` :
        The configure script will automatically try to locate the HDF4 include
        files in the default locations for include files
        (``/usr/local/include`` is usually not considered a default
        location!).
        If ``./configure`` complains that it can't find the ``hdf.h`` or
        ``mfhdf.h`` include files, pass this option to ``./configure`` with
        the location of these include files.

    ``HDF5_LIB=<HDF5 library directory>`` :
        The configure script will automatically try to locate the HDF5 library
        in the default locations for libraries (``/usr/local/lib`` is usually
        not considered a default location!).
        If ``./configure`` complains that it can't find the ``hdf5`` library
        files, pass this option to ``./configure`` with the location of this
        library file.

    ``HDF5_INCLUDE=<HDF5 include file directory>`` :
        The configure script will automatically try to locate the HDF5 include
        files in the default locations for include files
        (``/usr/local/include`` is usually not considered a default location!).
        If ``./configure`` complains that it can't find the ``hdf5.h`` include
        file, pass this option to ``./configure`` with the location of this
        include file.

    ``--enable-python`` :
        By default HARP is built without the Python interface. Use this option
        to enable building of the interface to Python. Make sure that you choose
        the install prefix option (``--prefix``) such that the target location
        of the python package (e.g. ``<prefix>/lib/python3.7/site-packages``)
        ends up in your python path. Also, if you enable the Python interface
        then make sure you have installed the numpy and cffi packages for Python.

    ``PYTHON=<Python executable>`` :
        If you want to build the Python interface you should also use this option
        to tell the configuration script where your Python executable is located
        (e.g. ``PYTHON=/usr/bin/python``). Make sure that you provide an absolute
        path for the Python executable (i.e. the path should start with a ``/``).

    ``--enable-R`` :
        By default HARP is built without the R interface. Use this option to
        enable building of the interface to R. Make sure that you choose the
        install prefix option (``--prefix``) such that the target location
        of the R package (e.g. ``<prefix>/lib/R/site-library``) ends up in your R
        path.

    ``R=<R executable>`` :
        If you want to build the R interface you might have to use this option to
        tell the configuration script where your R executable is located
        (e.g. ``R=/usr/bin/R``). Make sure that you provide an absolute path for
        the R executable (i.e. the path should start with a ``/``).

    ``--enable-idl`` :
        By default HARP is built without the IDL interface.
        Use this option to enable building of the interface to IDL.

    ``IDL=<IDL root directory>`` :
        If you want to build the IDL interface you should also use this
        option to tell the configuration script where you have installed IDL.
        The <IDL root directory> is the root directory of your IDL
        installation. It should contain for instance the ``bin`` directory with
        the idl executable and an ``external`` directory containing the file
        ``export.h``. Also make sure that you provide an absolute path for the
        IDL root directory (i.e. starting with a ``/``).

    ``--enable-matlab`` :
        By default HARP is built without the MATLAB interface.
        Use this option to enable building of the interface to MATLAB.

    ``MATLAB=<MATLAB root directory>`` :
        If you want to build the MATLAB interface you should also use this
        option to tell the configuration script where you have installed
        MATLAB. The <MATLAB root directory> is the root directory of your
        MATLAB installation. It should contain for instance the ``bin``
        directory with the matlab and mex executables (or symbolic links to
        them) and an ``extern/include`` subdirectory containing the file
        ``mex.h``. Also make sure that you provide an absolute path for the
        MATLAB root directory (i.e. starting with a ``/``).

    You should now call the configure script with the options that are
    appropriate for you. For instance, if you want to install HARP in the
    ``harp`` subdirectory of your home directory then your invocation of
    configure would be::

        $ ./configure --prefix=$HOME/harp

 4) If this completes successfully, you are now able to build the library by
    executing the ``make`` command::

        $ make

    If everything was successful, you are now ready to install the package
    (this installation step is not optional! You will have to install HARP
    before you can use it). If you encountered any problems while building
    HARP and if you need help contact us (see the Feedback section at the
    bottom for contact information).

 5) In order to make use of the HARP library and tools, you should install
    HARP as build in the previous step. Please make sure you have provided
    the appropriate installation directory option (``--prefix=<installdir>``)
    to the configure script, as explained in the previous section, if you do
    not want to install HARP in its default location ``/usr/local``. After
    running the configure script, issue the following command::

        $ make install

 6) If you enabled the IDL interface then you will have to add
    ``<HARP installdir>/lib/harp/idl`` to your ``DLM_PATH``. You do this by
    setting an ``IDL_DLM_PATH`` environment variable and add this setting to
    your system shell startup script (if you don't now how to set environment
    variables or add these to your shell startup script, please ask your
    system administrator).
    The variable should be set to ``<IDL_DEFAULT>`` appended with the HARP DLM
    directory. If, for instance, you have installed HARP in ``/usr/local``
    then you should set the ``IDL_DLM_PATH`` environment variable to
    ``<IDL_DEFAULT>:/usr/local/lib/harp/idl``.

    Note that you should ideally not use the IDL ``pref_set`` function to set
    the ``IDL_DLM_PATH``. The HARP DLM file will still load, but HARP will not
    be able to find the coda format definition files or udunits2 xml files
    this way. You can work around this by setting the coda definition path and
    udunits2 xml path explicitly instead of having HARP/CODA automatically
    determine this location based on the ``IDL_DLM_PATH``.
    See points 8 and 9 below for how to set the environment variables globally.
    You can also set the environment variables inside IDL before you call any
    HARP or CODA functions using e.g.::

        IDL> SETENV, 'CODA_DEFINITION=/path/to/codadefs/'
        IDL> SETENV, 'UDUNITS2_XML_PATH=/path/to/udunits2.xml'

 7) If you enabled the MATLAB interface then you should create a ``startup.m``
    file in a ``matlab`` directory in your home directory
    (``~/matlab/startup.m``) that contains the line::

        addpath <HARP installdir>/lib/harp/matlab

 8) Make sure that the CODA definition path is correctly set.
    If you followed the instructions normally, then HARP and CODA should be
    able to find the coda definition files (.codadef files) inside the
    installation path automatically.
    If you are getting 'unsupported product' errors then you may have to set
    the coda definition path explicitly. You can do this by setting the
    ``CODA_DEFINITION`` environment variable and have it point to the
    directory that contains the .codadef files. On Linux this would be e.g.::

        export CODA_DEFINITION=/usr/local/share/coda/definitions

    On macOS you can set environment variables via a `~/Library/LaunchAgents/my.startup.plist
    <https://apple.stackexchange.com/questions/106355/setting-the-system-wide-path-environment-variable-in-mavericks>`_
    file.

    Also make sure that you do not already have a CODA_DEFINITION
    environment variable set that points to a different location than where
    the .codadef files that come with HARP are located.

 9) Make sure that the udunits2 xml path is correctly set.
    If you followed the instructions normally, then HARP should be able to
    find the udunits2 xml files inside the installation path automatically.
    If you are getting 'invalid unit' errors then you may have to set the
    udunits2 xml path explicitly. You can do this by setting the
    ``UDUNITS2_XML_PATH`` environment variable and have it point to the
    ``udunits2.xml`` file inside the installation path.
    On Linux this would be e.g.::

        export UDUNITS2_XML_PATH=/usr/local/share/harp/udunits2.xml

    On macOS you can set environment variables via a `~/Library/LaunchAgents/my.startup.plist
    <https://apple.stackexchange.com/questions/106355/setting-the-system-wide-path-environment-variable-in-mavericks>`_
    file.


Installing the binary package (Windows)
---------------------------------------

First make sure that you have the `Microsoft Visual C++ Redistributable
Packages for Visual Studio 2015
<https://www.microsoft.com/en-us/download/details.aspx?id=48145>`_ installed.

To install the binary package of HARP for Windows just run the msi executable
which will guide you through the installation process. Make sure to install
HARP in a directory that does not have any spaces in its path. Otherwise
you might encounter problems with e.g. the Python interface.

For Python you will have to copy the ``C:\Program Files\HARP\python\harp``
directory to the site-packages folder of your Python installation.
However, if you want to use the Python interfaces on Windows, it is recommended
to just install the Anaconda package of HARP.

For R you will either have to copy the ``C:\Program Files\HARP\R\harp``
directory to the site-library folder of your R installation or you will have to
add the ``C:\Program Files\HARP\R`` directory to your R library path.
However, if you want to use the R interfaces on Windows, it is recommended
to install the Anaconda package of HARP.

For IDL you will have to add ``<HARP installdir>\lib\harp\idl`` to your
``DLM_PATH``. You do this by setting an ``IDL_DLM_PATH`` environment variable.
On Windows open the 'System' control panel of your Windows operating system
and go to the 'Advanced' tab. Then click on 'Environment Variables' and create a
new system variable with the name ``IDL_DLM_PATH`` and value
``<IDL_DEFAULT>;C:\Program Files\HARP\lib\harp\idl``.
If you have installed HARP in a location different from
``C:\Program Files\HARP`` then replace this part in the value by the
installation directory you chose when installing HARP.
Note that you should ideally not use the IDL ``pref_set`` function to set the
``IDL_DLM_PATH``. The HARP DLM file will still load, but HARP will not be able
to find the coda format definition files or udunits2 xml files this way.
You can work around this by setting the coda definition path and udunits2
xml path explicitly instead of having HARP/CODA automatically determine this
location based on the ``IDL_DLM_PATH``.
This is done by setting the ``CODA_DEFINITION`` and ``UDUNITS2_XML_PATH``
environment variables. This can be done outside IDL by setting the environment
variable globally, but you can also do this inside IDL before you call any HARP
or CODA functions using::

    IDL> SETENV, 'CODA_DEFINITION=/path/to/codadefs/'
    IDL> SETENV, 'UDUNITS2_XML_PATH=/path/to/udunits2.xml'

For MATLAB you will have to start MATLAB and go to the 'Set Path...' menu item
in the 'File' menu. Here you should add the directory
``C:\Program Files\HARP\lib\harp\matlab``. If you have installed HARP in a
location different from ``C:\Program Files\HARP`` then replace this part of the
directory by the installation directory you had chosen when you installed HARP.

Note: The binary installer for Windows comes with DLLs for all dependencies,
which are CODA, HDF4, HDF5, and their dependencies.

Building the source package (Windows)
-------------------------------------

To build the source package, make sure to download the official HARP source
package, which is named ``harp-x.y.z.tar.gz``. Don't use the packages called
'Source code' from GitHub as these are contents of the git repository (see
'Using GitHub' above).

The official and only supported build system on Windows is CMake.
You will need to have builds of CODA, HDF4, HDF5 (and their dependencies)
available as well in order to build HARP. The locations of include files
and libraries for the third-party dependencies can be set using
``<package>_INCLUDE_DIR`` and ``<package>_LIBRARY_DIR`` CMake options.

The generation of the Windows binary installer is done using CPack and WIX.
So, in order to recreate the Windows binary installer, you will also have to
make sure that you have CMake (3.0 or later) and WIX installed.

Documentation location
----------------------

Both the source and binary HARP packages come with documentation in HTML.
For the source package you can access the documentation from within the
unpacked HARP source package directory by going to the ``doc/html``
subdirectory and opening the file index.html in your browser. If you perform
an install of the source package all documentation will also be installed. You
can find the documentation under the subdirectory ``share/harp/doc/html`` of
your installation directory (``/usr/local`` by default).

Feedback
--------

If you encounter any problems while trying to build, install or run one or more
components of the HARP package then create a topic on the Atmospheric Toolbox
Forum:

    https://forum.atmospherictoolbox.org/

If you are using the source package on a Unix based system, please provide a
copy of the config.log file that is created when you run ``./configure`` and a
copy of the output of ``make`` with your e-mail.

Apart from problems, we would also appreciate to hear from you if you have any
ideas, suggestions, or comments that may help us to improve the quality or
functionality of HARP.