horovod installation failed, cannot find NCCL

[lizhen62017]

Hi,

I installed tensorflow[and-cuda] v 2.18.0 as instructed, then I installed deepmd using

DP_VARIANT=cuda CUDAToolkit_ROOT=/cm/shared/apps/cuda12.1/toolkit/12.1.1 pip install . --ignore-installed

and dp -h runs fine. But when I install horovod using

HOROVOD_NCCL_HOME=/mnt/isilon/w_ccls/merzklab/Zhen/tf/lib/python3.9/site-packages/nvidia/nccl HOROVOD_GPU_OPERATIONS=NCCL pip install --no-cache-dir horovod

It gives this error:

      -- Linking against static NCCL library
      -- Could not find static NCCL library. Trying to find shared lib instead.
      CMake Error at /cm/shared/apps/cmake/3.23.2/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
        Could NOT find NCCL (missing: NCCL_LIBRARY)
      Call Stack (most recent call first):
        /cm/shared/apps/cmake/3.23.2/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
        cmake/Modules/FindNCCL.cmake:53 (find_package_handle_standard_args)
        CMakeLists.txt:223 (find_package)

Even though I did this already:

export NCCL_LIBRARY="/mnt/isilon/w_ccls/merzklab/Zhen/tf/lib/python3.9/site-packages/nvidia/nccl/lib:$NCCL_LIBRARY"

I cannot really put NCCL in /usr/local since I am using a supercomputer. Any help? Thanks!

[njzjz]

Even though I did this already:

export NCCL_LIBRARY="/mnt/isilon/w_ccls/merzklab/Zhen/tf/lib/python3.9/site-packages/nvidia/nccl/lib:$NCCL_LIBRARY"

According to horovod/horovod#2408 (comment), a different environment variable should be used. Since it's a horovod issue, I suggest earching across the horovod repository.

[lizhen62017]

Thanks Jinzhe!

I found the answer and it was a bit of dumb... in PyPI's nvidia-nccl-cu12 (2.21.5) it names the shared library as libnccl.so.2 but HOROVOD code will only search for libnccl.so. I renamed the file and the issue got resolved.