Possibility to use system provided HDF5 binary

[ecsx1]

Pkg fails to install HDF5.jl on musl systems.
If I understand correctly it tries to either download precompiled binaries (which won't work because of musl) or compile HDF5 (which can be tricky because of musl).
I have a working HDF5 installation at /usr/bin/.

Would the problems be solved if Julia just used those binaries instead?

julia> using HDF5
[ Info: Precompiling HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f]
ERROR: LoadError: HDF5 not properly installed. Please run Pkg.build("HDF5")
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] top-level scope at /home/ecs/.julia/packages/HDF5/cCxZo/src/HDF5.jl:34
 [3] include at ./boot.jl:328 [inlined]
 [4] include_relative(::Module, ::String) at ./loading.jl:1094
 [5] include(::Module, ::String) at ./Base.jl:31
 [6] top-level scope at none:2
 [7] eval at ./boot.jl:330 [inlined]
 [8] eval(::Expr) at ./client.jl:432
 [9] top-level scope at ./none:3
in expression starting at /home/ecs/.julia/packages/HDF5/cCxZo/src/HDF5.jl:31
ERROR: Failed to precompile HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f] to /home/ecs/.julia/compiled/v1.2/HDF5/L7Dga.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1253
 [3] _require(::Base.PkgId) at ./loading.jl:1013
 [4] require(::Base.PkgId) at ./loading.jl:911
 [5] require(::Module, ::Symbol) at ./loading.jl:906

julia> Pkg.build("HDF5")
  Building CMake → `~/.julia/packages/CMake/nSK2r/deps/build.log`
  Building Blosc → `~/.julia/packages/Blosc/lzFr0/deps/build.log`
  Building HDF5 ─→ `~/.julia/packages/HDF5/cCxZo/deps/build.log`
┌ Error: Error building `HDF5`:
│ ERROR: LoadError: MethodError: no method matching iterate(::Nothing)
│ Closest candidates are:
│   iterate(!Matched::Core.SimpleVector) at essentials.jl:604
│   iterate(!Matched::Core.SimpleVector, !Matched::Any) at essentials.jl:604
│   iterate(!Matched::ExponentialBackOff) at error.jl:214
│   ...
│ Stacktrace:
│  [1] append_any(::Any, ::Vararg{Any,N} where N) at ./essentials.jl:722
│  [2] top-level scope at /home/ecs/.julia/packages/HDF5/cCxZo/deps/build.jl:41
│  [3] include at ./boot.jl:328 [inlined]
│  [4] include_relative(::Module, ::String) at ./loading.jl:1094
│  [5] include(::Module, ::String) at ./Base.jl:31
│  [6] include(::String) at ./client.jl:431
│  [7] top-level scope at none:5
│ in expression starting at /home/ecs/.julia/packages/HDF5/cCxZo/deps/build.jl:41
└ @ Pkg.Operations /julia-source/usr/share/julia/stdlib/v1.2/Pkg/src/backwards_compatible_isolation.jl:647
false

Note for @fredrikekre and @ararslan - this is with a fresh install using the recent julia-alpine
v1.2.0+0 from https://github.com/fredrikekre/julia-alpine/releases

[musm]

The error is a bit strange. Have you updated your packages lately?

[musm]

What's in build.log?

[ecsx1]

This is on a fresh install with almost no packages but I reran Pkg.update() just to be sure.
Trying using HDF5 and Pkg.build("HDF5") gives the exact same error.

The file ~/.julia/packages/HDF5/cCxZo/deps/build.log brings the same info, nothing new:

ERROR: LoadError: MethodError: no method matching iterate(::Nothing)
Closest candidates are:
  iterate(!Matched::Core.SimpleVector) at essentials.jl:604
  iterate(!Matched::Core.SimpleVector, !Matched::Any) at essentials.jl:604
  iterate(!Matched::ExponentialBackOff) at error.jl:214
  ...
Stacktrace:
 [1] append_any(::Any, ::Vararg{Any,N} where N) at ./essentials.jl:722
 [2] top-level scope at /home/ecs/.julia/packages/HDF5/cCxZo/deps/build.jl:41
 [3] include at ./boot.jl:328 [inlined]
 [4] include_relative(::Module, ::String) at ./loading.jl:1094
 [5] include(::Module, ::String) at ./Base.jl:31
 [6] include(::String) at ./client.jl:431
 [7] top-level scope at none:5
in expression starting at /home/ecs/.julia/packages/HDF5/cCxZo/deps/build.jl:41

[ecsx1]

Could this be related to a bad Pkg.add("CMake")?
It gave some errors at first: http://ix.io/1Thn
But it looked fine afterwards. using CMake runs silently without errors.
cmake then returns a binary path,

julia> cmake
"/home/ecs/.julia/packages/CMake/nSK2r/deps/usr/bin/cmake"

But that binary doesn't run on my system. readelf reports it's a GLIBC binary, but I'm on a musl system.

Could this be leading to the HDF5 error as well?

[musm]

Good investigating.

So if regular hdf5 builds work with cmake using musl. Then it must be a CMake.jl issue, since we are relying on them to build the hdf5 executable.

To test this
change $cmake_executable to cmake and make sure cmake is in PATH in
https://github.com/JuliaIO/HDF5.jl/blob/master/deps/compile.jl#L25
if the build works then the issue will need resolving from CMake.jl or we need to add override flags.

[musm]

Any status update? Did my recommendations works?

[ecsx1]

I manually edited all cases of $cmake_executable to /usr/bin/cmake but Pkg.build("HDF5") gives the same error exactly.

I'm feeling it will be a rough ride to use julia with musl right now due to the assumption julia makes that linux means glibc and the download of glibc binaries that won't work into the system.
On a deeper level I think it would be good to understand why julia doesn't just use the system's binaries. This would simplify porting the julia environment to a new system and make everything easier from my view.

In any case we are making the assumption that's the issue here.
The error message from Pkg.build("HDF5") is a bit cryptic to me.

[musm]

@ecsx1 those questions are better reserved to ask on Discourse. I suspect you may be downloading the wrong binaries, best to ask on DIscourse on slack.

[ecsx1]

Sorry for digressing. Let's get back on track, HDF5.jl.

Given that I hard-coded cmake as /usr/bin/cmake doesn't that rule out Pkg.build("HDF5") calling the wrong executable?
What else could I do to investigate the issue further?

[musm]

I'm feeling it will be a rough ride to use julia with musl right now due to the assumption julia makes that linux means glibc and the download of glibc binaries that won't work into the system.
On a deeper level I think it would be good to understand why julia doesn't just use the system's binaries. This would simplify porting the julia environment to a new system and make everything easier from my view.

You might want to ask on discourse regarding the glibc binaries.

[musm]

@ecsx1 Julia binaries for musl are now available. Want to double-check if this is still an issue? If not we may need to get the musl hdf5 lib uploaded.

[ecsx1]

Thanks for bringing this up again.
I tried the 1.5.0 official binary from julia website.
It's a different error now.

julia> using HDF5
[ Info: Precompiling HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f]
ERROR: LoadError: UndefVarError: libhdf5 not defined
Stacktrace:
 [1] h5_get_libversion() at /home/ecs/.julia/packages/HDF5/QXcoh/src/HDF5.jl:39
 [2] top-level scope at /home/ecs/.julia/packages/HDF5/QXcoh/src/HDF5.jl:45
 [3] include(::Function, ::Module, ::String) at ./Base.jl:380
 [4] include(::Module, ::String) at ./Base.jl:368
 [5] top-level scope at none:2
 [6] eval at ./boot.jl:331 [inlined]
 [7] eval(::Expr) at ./client.jl:467
 [8] top-level scope at ./none:3
in expression starting at /home/ecs/.julia/packages/HDF5/QXcoh/src/HDF5.jl:45
ERROR: Failed to precompile HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f] to /home/ecs/.julia/compiled/v1.5/HDF5/L7Dga_LVNRX.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1290
 [3] _require(::Base.PkgId) at ./loading.jl:1030
 [4] require(::Base.PkgId) at ./loading.jl:928
 [5] require(::Module, ::Symbol) at ./loading.jl:923

[musm]

Use the Env variable JULIA_HDF5_LIBRARY_PATH

Starting from Julia 1.3, the HDF5 binaries are by default downloaded using the HDF5_jll package. To use system-provided HDF5 binaries instead, set the environment variable JULIA_HDF5_LIBRARY_PATH to the HDF5 library path and then run Pkg.build("HDF5"). This is in particular needed for parallel HDF5 support, which is not provided by the HDF5_jll binaries.

For example, you can set JULIA_HDF5_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/hdf5/mpich/ if you're using the system package libhdf5-mpich-dev on Ubuntu 20.04.