Can't install gdal into isis3 environment

[foobarbecue]

Conda fails if requested to install both isis3 and gdal into the same environment.

This was mentioned in closed issues #615 and #648 , but is still a problem, so I thought it would be worth opening a specific issue.

Description

How to reproduce
Create test.yml (on Dec 5 2019) with:

name: test
channels:
  - usgs-astrogeology/label/RC
  - conda-forge
  - defaults
dependencies:
  - isis3
  - gdal

Run conda env create -f test.yml

expect things to install, instead conda goes into confliction conniptions for hours.

[afrigeri]

Tried a fresh isis3 conda install, and still gdal seems not being installed in the same conda environment (conda install gdal gets suck in a neverending conflict solution tentatives).

In #615 @thareUSGS suggested to switch forth and back from isis3 and gdal envs. Is this still the best solution so far?

Alessandro

[jessemapel]

@afrigeri Yes that is still the current work around. We're working towards getting both installable sid-by-side but there are a handful of things that need to be updated like #3627

[foobarbecue]

@afrigeri my solution is to install ISIS with conda, then use a GDAL environment but set the ISIS environment variables manually (use the old install instructions). That seems to work well and I don't have to switch back and forth; all the ISIS tools are available in my GDAL environment.

[afrigeri]

Thanks @jessemapel and @foobarbecue!

[rbeyer]

Bleh, I added a similar comment to #615, but I can't seem to reopen it (probably for the best).

I have written up some instructions for how to accomplish the multiple env solution two different ways (one real easy with conda activate --stack, and the other with writing some stuff in the activate.d and deactivate.d directories). Where should I PR these instructions? On the main README with the Installation stuff? Elsewhere?

[oleg-alexandrov]

I tried to create a GDAL 3.1.0 environment and see how many ISIS dependencies I can add to it before it breaks. On the Mac I could add all ISIS dependencies except ALE but which I could compile by hand. I tried to compile ISIS too then, but it appears that some work will be needed to bump ISIS to boost 1.72 from 1.68, which would need upgrading from boost::signals to boost::signals2. I don't know how much work that would be. All other dependencies did not change much, with Qt going from 5.9 to 5.12 being one of the more notable ones. Then jpeg went from 9b to 9d, and python from 3.6 to 3.8.

On Linux I could fetch all dependencies except xalan-c, but this looks like a small library so maybe something could be done about, even building it as a custom conda package.

There is a very good chance I am missing something and that can't be found until ISIS is compiled.

For reference, here are the envs I ended up getting. (On Linux there is the extra danger of linking error because of the GCC ABI change in GCC 5, but that was long enough ago that I think any recent dependencies are safely past that).

OSX env:

channels:

• conda-forge/label/gcc7
• conda-forge
• usgs-astrogeology
• defaults
  dependencies:
• armadillo=9.200.7=h3e4a27e_0
• arpack=3.7.0=hbb27dc6_1
• blas=2.17=openblas
• boost=1.72.0=py38hbf1eeb5_0
• boost-cpp=1.72.0=hdf9ef73_0
• bullet=2.86.1=0
• bz2file=0.98=py_0
• bzip2=1.0.8=h0b31af3_2
• ca-certificates=2020.6.20=hecda079_0
• cairo=1.16.0=hec6a9b0_1003
• certifi=2020.6.20=py38h32f6830_0
• cfitsio=3.470=hddc9c06_5
• cmake=3.17.0=hd28f656_0
• cspice=66=h0b31af3_1007
• curl=7.71.0=hbdca1c0_0
• doxygen=1.8.18=h72f44dd_0
• eigen=3.3.7=ha1b3eb9_1001
• embree=2.16.0=h6834224_0
• expat=2.2.9=hb1e8313_2
• ffmpeg=4.2.3=hd0c0d6a_0
• flann=1.9.1=h1a820e9_1008
• fontconfig=2.13.1=h6b1039f_1001
• freetype=2.10.2=ha233b18_0
• freexl=1.0.5=h1de35cc_1002
• gdal=3.1.0=py38h9f7df5a_3
• geos=3.8.1=h4a8c4bd_0
• geotiff=1.6.0=hd8796ba_0
• gettext=0.19.8.1=h46ab8bc_1002
• giflib=5.2.1=h0b31af3_2
• glib=2.65.0=h577aef8_0
• gmm=5.0=h6aef312_0
• gmp=6.2.0=h4a8c4bd_2
• gnutls=3.6.13=hc269f14_0
• graphite2=1.3.14=h38d11af_0
• gsl=2.6=ha2d443c_0
• harfbuzz=2.4.0=hd8d2a14_3
• hdf4=4.2.13=h84186c3_1003
• hdf5=1.10.6=nompi_h3e39495_100
• icu=64.2=h6de7cb9_1
• jama=125=0
• jasper=1.900.1=h636a363_1006
• jpeg=9d=h0b31af3_0
• json-c=0.13.1=h575e443_1002
• kakadu=1=0
• kealib=1.4.13=h40102fb_1
• krb5=1.17.1=h14dd6a4_1
• lame=3.100=h1de35cc_1001
• libblas=3.8.0=17_openblas
• libcblas=3.8.0=17_openblas
• libcurl=7.71.0=he6690cf_0
• libcxx=10.0.0=h1af66ff_2
• libdap4=3.20.6=h8c15375_0
• libedit=3.1.20191231=hed1e85f_0
• libffi=3.2.1=h4a8c4bd_1007
• libgdal=3.1.0=he0622db_3
• libgfortran=4.0.0=2
• libiconv=1.15=h0b31af3_1006
• libkml=1.3.0=h169b8f9_1011
• liblapack=3.8.0=17_openblas
• liblapacke=3.8.0=17_openblas
• libnetcdf=4.7.4=nompi_ha11d67f_104
• libopenblas=0.3.10=h3d69b6c_0
• libopencv=4.3.0=py38_1
• libpng=1.6.37=hbbe82c9_1
• libpq=12.2=h489d428_1
• libprotobuf=3.12.3=hd174df1_0
• libspatialite=4.3.0a=h658e6c1_1038
• libssh2=1.9.0=h39bdce6_2
• libtiff=4.1.0=h2ae36a8_6
• libuv=1.38.0=h0b31af3_0
• libwebp-base=1.1.0=h0b31af3_3
• libxcb=1.13=h1de35cc_1002
• libxml2=2.9.10=h53d96d6_0
• llvm-openmp=10.0.0=h28b9765_0
• lz4-c=1.9.2=h4a8c4bd_1
• make=4.3=h0b31af3_0
• mesalib=18.0.0=hb6cfc13_1
• metis=5.1.0=h4a8c4bd_1005
• mysql=5.7.20=h0a44026_1001
• mysql-connector-c=6.1.11=had4075f_1006
• nanoflann=1.3.1=0
• ncurses=6.1=h0a44026_1002
• nettle=3.4.1=h3efe00b_1002
• ninja=1.10.0=ha1b3eb9_0
• nlohmann_json=3.7.3=h4a8c4bd_1
• nn=1.86.0=h1de35cc_2003
• numpy=1.18.5=py38h1f821a2_0
• opencv=4.3.0=py38_1
• openh264=2.1.1=hd174df1_0
• openjpeg=2.3.1=h254dc36_3
• openssl=1.1.1g=h0b31af3_0
• pcl=1.9.1=h3d0f52c_1005
• pcre=8.44=hb1e8313_0
• pip=20.1.1=py38_1
• pixman=0.38.0=h01d97ff_1003
• poppler=0.89.0=h3232a60_1
• poppler-data=0.4.9=1
• postgresql=12.2=h62ab893_1
• proj=7.0.0=hbce2d5c_4
• protobuf=3.12.3=py38hc84c608_0
• pthread-stubs=0.4=h1de35cc_1001
• py-opencv=4.3.0=py38h23f93f0_1
• python=3.8.3=cpython_hd5f0129_0
• python_abi=3.8=1_cp38
• qhull=2015.2=ha1b3eb9_1001
• qt=5.9.7=h8cf7e54_3
• qwt=6.1.4=hc6ac302_1003
• readline=8.0=h1de35cc_0
• rhash=1.3.8=ha12b0ac_0
• setuptools=47.3.1=py38_0
• six=1.15.0=py_0
• sqlite=3.32.3=h93121df_0
• suitesparse=5.7.2=h0e59142_0
• superlu=5.2.1=hc209f47_1207
• tbb=2020.0=h04f5b5a_0
• tiledb=2.0.5=h84aa2a7_0
• tk=8.6.10=hb0a8c7a_0
• tnt=126=0
• tzcode=2020a=h0b31af3_0
• wheel=0.34.2=py_1
• x264=1!152.20180806=h1de35cc_0
• xalan-c=1.11=h1922a5c_0
• xerces-c=3.2.2=h8f8adb3_1004
• xorg-fixesproto=5.0=h1de35cc_1002
• xorg-inputproto=2.3.2=h1de35cc_1002
• xorg-kbproto=1.0.7=h1de35cc_1002
• xorg-libice=1.0.10=h01d97ff_0
• xorg-libsm=1.2.3=h01d97ff_1000
• xorg-libx11=1.6.9=h0b31af3_0
• xorg-libxau=1.0.9=h1de35cc_0
• xorg-libxdmcp=1.1.3=h01d97ff_0
• xorg-libxext=1.3.4=h01d97ff_0
• xorg-libxfixes=5.0.3=h01d97ff_1004
• xorg-libxi=1.7.10=h01d97ff_0
• xorg-xextproto=7.3.0=h1de35cc_1002
• xorg-xproto=7.0.31=h1de35cc_1007
• xz=5.2.5=h1de35cc_0
• zlib=1.2.11=h0b31af3_1006
• zstd=1.4.4=h1990bb4_3

Linux env:

channels:

• conda-forge/label/gcc7
• usgs-astrogeology
• conda-forge
• defaults
  dependencies:
• _libgcc_mutex=0.1=conda_forge
• _openmp_mutex=4.5=1_llvm
• armadillo=9.200.7=hf4e8f56_0
• arpack=3.7.0=hc6cf775_1
• blas=2.17=openblas
• boost=1.72.0=py36h9de70de_0
• boost-cpp=1.72.0=h8e57a91_0
• bullet=2.88=py36hb3f55d8_1001
• bz2file=0.98=py_0
• bzip2=1.0.8=h516909a_2
• ca-certificates=2020.6.20=hecda079_0
• cairo=1.16.0=hcf35c78_1003
• certifi=2020.6.20=py36h9f0ad1d_0
• cfitsio=3.470=h3eac812_5
• cmake=3.17.0=h28c56e5_0
• cspice=66=h516909a_1009
• curl=7.71.0=he644dc0_0
• dbus=1.13.6=he372182_0
• doxygen=1.8.18=hd1b7508_0
• eigen=3.3.7=hc9558a2_1001
• embree=2.14.0=0
• expat=2.2.9=he1b5a44_2
• ffmpeg=4.2.3=h167e202_0
• flann=1.9.1=h44f99b7_1008
• fontconfig=2.13.1=h86ecdb6_1001
• freetype=2.10.2=he06d7ca_0
• freexl=1.0.5=h14c3975_1002
• gdal=3.1.0=py36hd60729c_3
• geos=3.8.1=he1b5a44_0
• geotiff=1.6.0=h05acad5_0
• gettext=0.19.8.1=hc5be6a0_1002
• giflib=5.2.1=h516909a_2
• glib=2.65.0=h6f030ca_0
• gmm=5.0=0
• gmp=6.2.0=he1b5a44_2
• gnutls=3.6.13=h79a8f9a_0
• graphite2=1.3.13=he1b5a44_1001
• gsl=2.6=h294904e_0
• gst-plugins-base=1.14.5=h0935bb2_2
• gstreamer=1.14.5=h36ae1b5_2
• harfbuzz=2.4.0=h9f30f68_3
• hdf4=4.2.13=hf30be14_1003
• hdf5=1.10.6=nompi_h3c11f04_100
• icu=64.2=he1b5a44_1
• jama=125=0
• jasper=1.900.1=h07fcdf6_1006
• jpeg=9d=h516909a_0
• json-c=0.13.1=hbfbb72e_1002
• kakadu=1=0
• kealib=1.4.13=h33137a7_1
• krb5=1.17.1=hfafb76e_1
• lame=3.100=h14c3975_1001
• ld_impl_linux-64=2.34=h53a641e_5
• libblas=3.8.0=17_openblas
• libcblas=3.8.0=17_openblas
• libclang=9.0.1=default_hde54327_0
• libcurl=7.71.0=hcdd3856_0
• libdap4=3.20.6=h1d1bd15_0
• libedit=3.1.20191231=h46ee950_0
• libffi=3.2.1=he1b5a44_1007
• libgcc=7.2.0=h69d50b8_2
• libgcc-ng=9.2.0=h24d8f2e_2
• libgdal=3.1.0=h4ed0dcf_3
• libgfortran-ng=7.5.0=hdf63c60_6
• libiconv=1.15=h516909a_1006
• libkml=1.3.0=hb574062_1011
• liblapack=3.8.0=17_openblas
• liblapacke=3.8.0=17_openblas
• libllvm9=9.0.1=he513fc3_1
• libnetcdf=4.7.4=nompi_h84807e1_104
• libopenblas=0.3.10=h5ec1e0e_0
• libopencv=4.2.0=py36_7
• libpng=1.6.37=hed695b0_1
• libpq=12.2=h5513abc_1
• libprotobuf=3.5.2=h6bb024c_0
• libspatialite=4.3.0a=h2482549_1038
• libssh2=1.9.0=hab1572f_2
• libstdcxx-ng=9.2.0=hdf63c60_2
• libtiff=4.1.0=hc7e4089_6
• libuuid=2.32.1=h14c3975_1000
• libuv=1.38.0=h516909a_0
• libwebp=1.1.0=h56121f0_4
• libwebp-base=1.1.0=h516909a_3
• libxcb=1.13=h14c3975_1002
• libxkbcommon=0.10.0=he1b5a44_0
• libxml2=2.9.10=hee79883_0
• llvm-openmp=10.0.0=hc9558a2_0
• lz4-c=1.9.2=he1b5a44_1
• make=4.3=h516909a_0
• mesalib=18.3.1=h590aaf7_0
• metis=5.1.0=he1b5a44_1005
• mysql=5.7.20=hf484d3e_1001
• mysql-connector-c=6.1.11=h876a3cc_1006
• nanoflann=1.3.1=0
• ncurses=6.1=hf484d3e_1002
• nettle=3.4.1=h1bed415_1002
• ninja=1.10.0=hc9558a2_0
• nlohmann_json=3.7.3=he1b5a44_1
• nn=1.86.0=h14c3975_2003
• nspr=4.26=he1b5a44_0
• nss=3.47=he751ad9_0
• numpy=1.18.5=py36h7314795_0
• opencv=4.2.0=py36_7
• openh264=2.1.1=h8b12597_0
• openjpeg=2.3.1=h981e76c_3
• openssl=1.1.1g=h516909a_0
• pcl=1.9.1=h2dfa329_1005
• pcre=8.44=he1b5a44_0
• pip=20.1.1=py_1
• pixman=0.38.0=h516909a_1003
• poppler=0.89.0=h4190859_1
• poppler-data=0.4.9=1
• postgresql=12.2=h8573dbc_1
• proj=7.0.0=h966b41f_4
• protobuf=3.5.2=py36h6bb024c_0
• pthread-stubs=0.4=h14c3975_1001
• py-opencv=4.2.0=py36h95af2a2_7
• python=3.6.10=h8356626_1011_cpython
• python_abi=3.6=1_cp36m
• qhull=2015.2=hc9558a2_1001
• qt=5.12.5=hd8c4c69_1
• qwt=6.1.5=h73816c6_0
• readline=8.0=hf8c457e_0
• rhash=1.3.6=h14c3975_1001
• setuptools=47.3.1=py36h9f0ad1d_0
• six=1.15.0=pyh9f0ad1d_0
• sqlite=3.32.3=hcee41ef_0
• suitesparse=5.7.2=h717dc36_0
• superlu=5.2.1=hfe2efc7_1207
• tbb=2020.1=hc9558a2_0
• tiledb=2.0.5=h8efa9f0_0
• tk=8.6.10=hed695b0_0
• tnt=126=0
• tzcode=2020a=h516909a_0
• wheel=0.34.2=py_1
• x264=1!152.20180806=h14c3975_0
• xerces-c=3.2.2=h8412b87_1004
• xorg-fixesproto=5.0=h14c3975_1002
• xorg-inputproto=2.3.2=h14c3975_1002
• xorg-kbproto=1.0.7=h14c3975_1002
• xorg-libice=1.0.10=h516909a_0
• xorg-libsm=1.2.3=h84519dc_1000
• xorg-libx11=1.6.9=h516909a_0
• xorg-libxau=1.0.9=h14c3975_0
• xorg-libxdmcp=1.1.3=h516909a_0
• xorg-libxext=1.3.4=h516909a_0
• xorg-libxfixes=5.0.3=h516909a_1004
• xorg-libxi=1.7.10=h516909a_0
• xorg-libxrender=0.9.10=h516909a_1002
• xorg-renderproto=0.11.1=h14c3975_1002
• xorg-xextproto=7.3.0=h14c3975_1002
• xorg-xproto=7.0.31=h14c3975_1007
• xz=5.2.5=h516909a_0
• zlib=1.2.11=h516909a_1006
• zstd=1.4.4=h6597ccf_3

[github-actions]

Thank you for your contribution!

Unfortunately, this issue hasn't received much attention lately, so it is labeled as 'stale.'

If no additional action is taken, this issue will be automatically closed in 180 days.

[oleg-alexandrov]

Here, making it not stale. This is a long-term important issue.

[jessemapel]

Still an issue.

[tmwilliams9999]

This would be very nice to have. It is error prone to switch between two environments for terminal sessions, and very difficult and slow in a script.

[github-actions]

Thank you for your contribution!

Unfortunately, this issue hasn't received much attention lately, so it is labeled as 'stale.'

If no additional action is taken, this issue will be automatically closed in 180 days.

[oleg-alexandrov]

Keeping this alive. Installing GDAL in the ISIS conda env is a very important goal. Especially that GDAL 3 improved when it comes to handling converting between projections and between vertical datums. It is the case for NASA ASP, and may be for ISIS as well, that homegrown solutions were implemented to deal with projections, datums, and large images. Some of those may become obsolete if recent GDAL is used instead.

In other words, given the widespread use and good maintenance that GDAL has, having ISIS depend on it and offloading some raster layer, vector layer, and projection processing to it could make ISIS easier to maintain.

[jlaura]

@acpaquette @scsides Has this been completed under the dependency work or are we still having issues here because GEOS work has not been finished?

[oleg-alexandrov]

I think GDAL provides an API for polygons via OGR, which under the hood uses GEOS. Given that the GEOS API C++ seems unstable, calling it through GDAL and letting GDAL maintain the GEOS interface may avoid future work keeping up with GEOS. My apologies if I got something wrong, I don't know where the status of things is when it comes to this issue.

[oleg-alexandrov]

It looks that the issue of installing GDAL in the ISIS env is still not solved in ISIS 8.0.0.

This ISIS version uses geos 3.9.1, but proj 9.1.0. No version of GDAL available on conda-forge is compatible with these. For proj 9.1.1 need at least geos 3.11.1.

I tried to install GDAL in a freshly fetched ISIS 8.0.0 environment, and it got into conflicts, as expected.

[acpaquette]

@oleg-alexandrov I can look into getting geos updated. Might be something we can just do but it's likely a little more complicated.

[acpaquette]

Looks like bumping geos to 3.11.1 is totally do able! Getting success with just building locally

[oleg-alexandrov]

This is very encouraging. Thank you for looking into this.

[oleg-alexandrov]

The very nice thing would be to see if ISIS, geos=3.11.1, proj=9.1.0 and libgdal=3.4.3 can live in the same conda environment.

Alternative versions of libgdal that look as if they may work are 3.5.2, 3.5.3, 3.6.0, 3.6.1, 3.6.2, as it appears from doing conda search.

Any of this conda work can require a lot of work, so not sure if it is feasible. The nice longer-term thing would be to ship a new ISIS version with GDAL already in.

[acpaquette]

I was able to get geos up to 3.11.1 and that natural brought proj up to 9.1.0. I know there are plans to work with gdal in ISIS in the future but nothing concrete yet. We should be able to get a ticked build out the door in the LTS

[oleg-alexandrov]

This sounds good, thanks. I plan to have an ASP release out in the near future. The plan for now is to use ISIS 8.0.0 and my own custom GDAL build.

If ISIS 8.0.1 with the new geos version happens to be out at some point I can likely piggy back on that one and tack onto it a conda-forge GDAL.

[acpaquette]

It should still be a 8.0.0 version just a +1 to the build number so 8.0.0_1 rather than 8.0.0_0

[oleg-alexandrov]

I'll take it whichever way you make it. :)

[Kelvinrr]

geos has been updated such that it is now possible to install gdal with ISIS.