Unable to disable CURL library

[gsjaardema]

We have a need to be able to build netCDF with minimal dependencies. There are options to turn off the filters, remote functionality, and many other options. However, it looks like the latest main branch has a mandatory dependency on CURL. I don't seen anyway to turn it off other than removing the curl-related lines from the cmake/dependencies.cmake file.

Can we add an ENABLE_CURL option. I'm OK with it defaulting to ON, but would like the capability to turn it off...

This is with main branch as of February 1, 2024.

[gsjaardema]

Just for mor information. One of the main reasons is that libcurl pulls in lots of encryption and ssl and kerberos libraries which can cause issues with some python's that are linked to different versions of those libraries and we have python interface to our library which uses netCDF library so when we load the library, we get unsatisfied externals...

[WardF]

Agreed. The logic should be that netcdf is always built with minimal dependancies. It should be the case that disabling anything that would require libcurl (DAP, Zarr support) should result in libcurl not being linked against. I'll take a look, @K20shores has been helping modernize our CMake infrastructure (in addition to the great help @ZedThree has been providing!), so let me take a look and see what needs to be adjusted, without interfering with any of the PR backlog we're working through (on top of everything else). Thanks!

[WardF]

@DennisHeimbigner Besides Zarr and DAP2/DAP4, is there anything making use of libcurl-provided functionality?

[gsjaardema]

I can provide a starting PR if you would like... I have an ENABLE_CURL option and it defaults to on, but is set to off if ENABLE_REMOTE_FUNCTIONALITY=OFF or can be set manually also...

bash-4.2$ git diff
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 602f3dc3..7a8a02c0 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -435,6 +435,7 @@ if(NOT ENABLE_HDF5 OR NOT ENABLE_NETCDF4 OR NOT ENABLE_NETCDF_4)
   set(ENABLE_HDF5 OFF CACHE BOOL "Use HDF5" FORCE)
 endif()
 option(ENABLE_HDF4 "Build netCDF-4 with HDF4 read capability(HDF4, HDF5 and Zlib required)." OFF)
+option(ENABLE_CURL "Enable CURL." ON)
 option(ENABLE_DAP "Enable DAP2 and DAP4 Client." ON)
 option(ENABLE_NCZARR "Enable NCZarr Client." ON)
 option(ENABLE_PNETCDF "Build with parallel I/O for CDF-1, 2, and 5 files using PnetCDF." OFF)
@@ -452,9 +453,10 @@ endif()
 option(ENABLE_REMOTE_FUNCTIONALITY "Enable|disable all forms remote data access (DAP, S3, etc)" ON)
 message(">>> ENABLE_REMOTE_FUNCTIONALITY=${ENABLE_REMOTE_FUNCTIONALITY}")
 if(NOT ENABLE_REMOTE_FUNCTIONALITY)
-message(WARNING "ENABLE_REMOTE_FUNCTIONALITY=NO => ENABLE_DAP[4]=NO")
+message(WARNING "ENABLE_REMOTE_FUNCTIONALITY=NO => ENABLE_DAP[4]=NO, ENABLE_CURL=NO")
 set(ENABLE_DAP OFF CACHE BOOL "ENABLE_REMOTE_FUNCTIONALITY=NO => ENABLE_DAP=NO" FORCE)
 set(ENABLE_DAP4 OFF CACHE BOOL "ENABLE_REMOTE_FUNCTIONALITY=NO => ENABLE_DAP4=NO" FORCE)
+set(ENABLE_CURL OFF CACHE BOOL "ENABLE_REMOTE_FUNCTIONALITY=NO => ENABLE_CURL=NO" FORCE)
 endif()
 
 # Option to Build DLL
diff --git a/cmake/dependencies.cmake b/cmake/dependencies.cmake
index 0cd31fc5..2d966d61 100644
--- a/cmake/dependencies.cmake
+++ b/cmake/dependencies.cmake
@@ -63,7 +63,11 @@ if(ENABLE_HDF4)
   # Option to enable HDF4 file tests.
   option(ENABLE_HDF4_FILE_TESTS "Run HDF4 file tests.  This fetches sample HDF4 files from the Unidata ftp site to test with (requires curl)." ON)
   if(ENABLE_HDF4_FILE_TESTS)
-    find_program(PROG_CURL NAMES curl)
+    if(ENABLE_CURL)
+      find_program(PROG_CURL NAMES curl)
+    else()
+       set(PROG_CURL OFF)
+    endif()
     if(PROG_CURL)
       set(USE_HDF4_FILE_TESTS ON )
     else()
@@ -409,6 +413,7 @@ endif(USE_HDF5)
 # Curl
 ################################
 # See if we have libcurl
+if(ENABLE_CURL)
 find_package(CURL)
 ADD_DEFINITIONS(-DCURL_STATICLIB=1)
 include_directories(${CURL_INCLUDE_DIRS})
@@ -490,7 +495,7 @@ else()
   #include <curl/curl.h>
   int main() {int x = CURLOPT_TCP_KEEPALIVE;}" HAVE_CURLOPT_KEEPALIVE)
 endif()
-
+endif()
 ################################
 # Math
 ################################

[gsjaardema]

If only DAP, DAP4, and/or Zarr use curl, then it would be good to change such that ENABLE_CURL defaults to ON only if one or more of those are enabled...

[DennisHeimbigner]

I am confused. We did a PR to add an option to shut off all remote access.
Is the problem that we are still checking for the availability of libcurl
even if remote access is disabled. If so, then the right thing to do
is to make the check on libcurl also be dependent on shutting off all remote access.

[WardF]

Since curl is not used directly, but is rather used by functionality, I think rather than ENABLE_CURL being an option, it should be properly determined, programatically, if libcurl is needed or not at configure time (along with all other dependencies). Restricting options to largely refer to features, we can figure out which libraries need to be linked against. If DAP and Zarr functionality aren't enabled, then libcurl need never be checked for in the first place. That said, I want to make sure that the extant PR's that are reorganizing where the various options live aren't going to get fouled up (or rather, I want to get them merged, first) before adjusting this.

[WardF]

Or, what @DennisHeimbigner said XD.

[DennisHeimbigner]

The necessary change is probably about line 410 in dependencies.cmake.

[gsjaardema]

It looks like ENABLE_HDF4_FILE_TESTS also has an affect near line 64 of dependencies.cmake. That option is explicitly disabled if ENABLE_NETCDF_4 is set, but based on a quick look, it might be enabled if not using netCDF-4...

[DennisHeimbigner]

Agreed, the hdf4 file tests should be conditional on remote access.

[WardF]

Taking a look at this tomorrow for the v4.9.3 release.

[WardF]

I've put up a fix; I've not turned off the HDF4 tests, since they can already be toggled with a different option, and the check that is made when NETCDF_ENABLE_HDF4_FILE_TESTS is run is for the curl executable, not the libcurl development package, nor does it add any dependencies. It just wants to find curl so that sample files can be downloaded.