h5repack issues including from static install

[bmribler]

(Created from a helpdesk issue)

I have been trying to use h5repack in HDF5 1.14.0 and 1.14.5 and in particular when we have a static-only build of libhdf5.a and associated binary tools.

I had the impression that a compression plugin should be able to work with a static-only install of HDF5 and, in particular, h5repack should be able to use the plugin when requested via the -f argument. However, that is turning out not to be my experience.

gcc -I. -I/usr/gapps/silo/zfp/1.0.1/blueos_3_ppc64le_ib_p9-gcc-8.3.1/include -I/usr/gapps/silo/hdf5/1.14.5/blueos_3_ppc64le_ib_p9_gcc.8.3.1/include -shared -fPIC -o plugin/libh5zzfp.so H5Zzfp.c -L/usr/gapps/silo/zfp/1.0.1/blueos_3_ppc64le_ib_p9-gcc-8.3.1/lib -lzfp -lsz -lm -lz -ldl -Wl,-Bsymbolic -Wl,-undefined,dynamic_lookup

I will identify a few issues I am seeing in bullets below…

1. About h5repack behavior when it cannot load a mandatory filter:
   * Specifying -f argument with mandatory filter flag AND naming a specific dataset to apply the filter to DOES NOT issue any error message or error code when it fails. Specfically, I tried…

env HDF5_PLUGIN_PATH=pwd/foo /g/g11/miller86/tmp/build-hdf5-1.14.5-blueos_3_ppc64le_ib_p9-gcc-8.3.1/my_install/bin/h5repack -f .silo/#15:UD=32013,0,4,1,0,0,1074528256 /usr/gapps/visit/data/wave0000.silo foo.silo

 *   note that `pwd`/foo is an invalid path. So, it can find no plugin with id 32013. the command, I specifically request dataset .silo/#000015 be compressed with that filter and I give the optional flag a 0 value…meaning mandatory. I get no error message or error code from h5repack
 *   It should probably provide an error message that it could not load the requested filter since the filter is mandatory in this case.
 *   I get that it may be problematic for h5repack to error if -f is specified but without targeting specific objects because then it is up to the filter’s can_apply method to decide which objects it can be applied to. OTOH, what does a 0 optional flag value mean in that case? It could potentially mean to list all the objects the filter could NOT be applied to upon completion but WITHOUT generating any error return value from h5repack. Where as, specifying a 1 optional flag means you don’t care to get that list. That’s just an idea on my part and not an actual request. So, take it or leave it.
 *   This likely has nothing to do with static-only or shared/static builds.

1. About static-only h5repack behavior with either mandatory or optional filter:
   * This is very strange but I’ve attempted to compress a specific dataset with the filter both as mandatory and as optional. And, I get behavior that represents kinda sorta the reverse of what I would expect. And, it appears to be in a “limbo” state…h5ls reports that the dataset has been filtered. But, it doesn’t show the filter by name, only by id and the dataset is the same size as uncompressed. I show 2 runs below. The first uses 0 for optional flag…meaning mandatory. The second uses 1 for optional flag meaning optional. Note however that h5ls on that dataset for the first shows that the dataset was never filtered whereas for the second case, it reports it was filtered but then it doesn’t report it the way I would normally see it if it had truly loaded the plugin correct. I think it failed to load the plugin.

lassen708{miller86}835: env HDF5_PLUGIN_PATH=pwd/plugin /g/g11/miller86/tmp/build-hdf5-1.14.5-blueos_3_ppc64le_ib_p9-gcc-8.3.1/my_install/bin/h5repack -f .silo/#15:UD=32013,0,4,1,0,0,1074528256 /usr/gapps/visit/data/wave0000.silo foo.silo

lassen708{miller86}836: h5ls -vlr foo.silo/.silo/#15 Opened "foo.silo" with sec2 driver.

.silo/#15 Dataset {101/101, 11/11, 16/16}

Location:  1:491964

Links:     1

Storage:   71104 logical bytes, 71104 allocated bytes, 100.00% utilization

Type:      native float

lassen708{miller86}837: env HDF5_PLUGIN_PATH=pwd/plugin /g/g11/miller86/tmp/build-hdf5-1.14.5-blueos_3_ppc64le_ib_p9-gcc-8.3.1/my_install/bin/h5repack -f .silo/#15:UD=32013,1,4,1,0,0,1074528256 /usr/gapps/visit/data/wave0000.silo foo.silo

lassen708{miller86}838: h5ls -vlr foo.silo/.silo/#15 Opened "foo.silo" with sec2 driver.

.silo/#15 Dataset {101/101, 11/11, 16/16}

Location:  1:491964

Links:     1

Chunks:    {101, 11, 16} 71104 bytes

Storage:   71104 logical bytes, 71104 allocated bytes, 100.00% utilization

Filter-0:  method-32013 OPT {1, 0, 0, 1074528256}

Type:      native float

1. About static-only h5repack behavior: I am not able to get this to work. I have added an abort() call in my compression plugin as part of the can_apply() method and it never aborts. So, I believe it is never loading the plugin.

[bmribler]

(Update from the author)

FWIW, I believe I have successfully compiled a static test-writer HDF5 client that does properly load and use my shared library plugin. So, I believe it is possible to get a statically compiled h5repack to behave as Quincey suggested it should. If you want more details, let me know. I’d be happy to jump on a zoom or Webex or Teams meeting too if useful.

I am also able to successfully use the h5ls binary tool from a static-only build of HDF5 and it is able to read zfp compressed data using my plugin. So, whatever is going on seem like it may be somewhat specific to h5repack.

[markcmiller86]

I will link here to some a relevant comment on H5Z-ZFP repo...LLNL/H5Z-ZFP#137 (comment)

[markcmiller86]

I've been exchanging emails with @lindstro about this. One thing he reminded me of is that in a static-only scenario, when an executable like h5repack is linked to libhdf5.a, the executable will get baked into it only those symbols defined among all the .o files in the archive that contain referenced symbols (either directly from the executable or indirectly through other .o files in the libhdf5.a archive).

Now, I suspect anything as sophisticated as an h5repack executable is likely to wind up loading all .o files from the libhdf5.a when it is linked. However, there is no guarantee of that unless you take action to engineer h5repack to ensure that does happen. Maybe you do? I don't honestly know.

But, if you do not, that means there is a non-zero chance that a compression plugin developer could use some symbol in HDF5 that is obscure enough that h5repack never loaded the .o object file from libhdf5.a that contains that symbol when it was linked meaning that the symbol the compression plugin needs will not be there. h5repack would SEGV when that symbol is referenced in the plugin.

I suspect the chances of this happening are small. But, I don't like that we might just be getting lucky a lot of the time.

I think that one of two possible choices may need to be made here for static-only builds. Either forgo support for dlopen()ing plugins or ensure that any tools/executables that may need to do that are engineered to ensure all HDF5 API symbols are loaded into them when they are linked...maybe just call the ...init() method of each interface?

[markcmiller86]

I did some analysis on h5repack from a static build on lassen. I am showing the list of .o files from libhdf5.a. The left column is what h5repack actually got (295 .o files) and the right is what libhdf5.a has available (336 .o files).

lassen709{miller86}999: diff -y used.txt avail.txt
H5.o								H5.o
H5A.o								H5A.o
H5AC.o								H5AC.o
H5ACdbg.o							H5ACdbg.o
H5ACproxy_entry.o						H5ACproxy_entry.o
H5Abtree2.o							H5Abtree2.o
H5Adense.o							H5Adense.o
							      >	H5Adeprec.o
H5Aint.o							H5Aint.o
							      >	H5Atest.o
H5B.o								H5B.o
H5B2.o								H5B2.o
H5B2cache.o							H5B2cache.o
							      >	H5B2dbg.o
H5B2hdr.o							H5B2hdr.o
H5B2int.o							H5B2int.o
H5B2internal.o							H5B2internal.o
H5B2leaf.o							H5B2leaf.o
H5B2stat.o							H5B2stat.o
H5B2test.o							H5B2test.o
H5Bcache.o							H5Bcache.o
H5Bdbg.o							H5Bdbg.o
H5C.o								H5C.o
H5CX.o								H5CX.o
H5Cdbg.o							H5Cdbg.o
H5Centry.o							H5Centry.o
H5Cepoch.o							H5Cepoch.o
H5Cimage.o							H5Cimage.o
H5Cint.o							H5Cint.o
H5Clog.o							H5Clog.o
H5Clog_json.o							H5Clog_json.o
H5Clog_trace.o							H5Clog_trace.o
H5Cprefetched.o							H5Cprefetched.o
H5Cquery.o							H5Cquery.o
H5Ctag.o							H5Ctag.o
							      >	H5Ctest.o
H5D.o								H5D.o
H5Dbtree.o							H5Dbtree.o
H5Dbtree2.o							H5Dbtree2.o
H5Dchunk.o							H5Dchunk.o
H5Dcompact.o							H5Dcompact.o
H5Dcontig.o							H5Dcontig.o
							      >	H5Ddbg.o
							      >	H5Ddeprec.o
H5Dearray.o							H5Dearray.o
H5Defl.o							H5Defl.o
H5Dfarray.o							H5Dfarray.o
H5Dfill.o							H5Dfill.o
H5Dint.o							H5Dint.o
H5Dio.o								H5Dio.o
H5Dlayout.o							H5Dlayout.o
H5Dnone.o							H5Dnone.o
H5Doh.o								H5Doh.o
H5Dscatgath.o							H5Dscatgath.o
H5Dselect.o							H5Dselect.o
H5Dsingle.o							H5Dsingle.o
							      >	H5Dtest.o
H5Dvirtual.o							H5Dvirtual.o
H5E.o								H5E.o
H5EA.o								H5EA.o
H5EAcache.o							H5EAcache.o
							      >	H5EAdbg.o
H5EAdblkpage.o							H5EAdblkpage.o
H5EAdblock.o							H5EAdblock.o
H5EAhdr.o							H5EAhdr.o
H5EAiblock.o							H5EAiblock.o
H5EAint.o							H5EAint.o
H5EAsblock.o							H5EAsblock.o
H5EAstat.o							H5EAstat.o
H5EAtest.o							H5EAtest.o
							      >	H5ES.o
H5ESevent.o							H5ESevent.o
H5ESint.o							H5ESint.o
H5ESlist.o							H5ESlist.o
H5Edeprec.o							H5Edeprec.o
H5Eint.o							H5Eint.o
H5F.o								H5F.o
H5FA.o								H5FA.o
H5FAcache.o							H5FAcache.o
							      >	H5FAdbg.o
H5FAdblkpage.o							H5FAdblkpage.o
H5FAdblock.o							H5FAdblock.o
H5FAhdr.o							H5FAhdr.o
H5FAint.o							H5FAint.o
H5FAstat.o							H5FAstat.o
H5FAtest.o							H5FAtest.o
H5FD.o								H5FD.o
H5FDcore.o							H5FDcore.o
H5FDfamily.o							H5FDfamily.o
H5FDint.o							H5FDint.o
H5FDlog.o							H5FDlog.o
H5FDmulti.o							H5FDmulti.o
H5FDonion.o							H5FDonion.o
H5FDonion_header.o						H5FDonion_header.o
H5FDonion_history.o						H5FDonion_history.o
H5FDonion_index.o						H5FDonion_index.o
H5FDperform.o							H5FDperform.o
H5FDsec2.o							H5FDsec2.o
H5FDspace.o							H5FDspace.o
H5FDsplitter.o							H5FDsplitter.o
H5FDstdio.o							H5FDstdio.o
							      >	H5FDtest.o
							      >	H5FDwindows.o
H5FL.o								H5FL.o
H5FO.o								H5FO.o
H5FS.o								H5FS.o
H5FScache.o							H5FScache.o
							      >	H5FSdbg.o
H5FSint.o							H5FSint.o
H5FSsection.o							H5FSsection.o
H5FSstat.o							H5FSstat.o
							      >	H5FStest.o
H5Faccum.o							H5Faccum.o
H5Fcwfs.o							H5Fcwfs.o
							      >	H5Fdbg.o
							      >	H5Fdeprec.o
H5Fefc.o							H5Fefc.o
H5Ffake.o							H5Ffake.o
H5Fint.o							H5Fint.o
H5Fio.o								H5Fio.o
H5Fmount.o							H5Fmount.o
H5Fquery.o							H5Fquery.o
H5Fsfile.o							H5Fsfile.o
H5Fspace.o							H5Fspace.o
H5Fsuper.o							H5Fsuper.o
H5Fsuper_cache.o						H5Fsuper_cache.o
							      >	H5Ftest.o
H5G.o								H5G.o
H5Gbtree2.o							H5Gbtree2.o
H5Gcache.o							H5Gcache.o
H5Gcompact.o							H5Gcompact.o
H5Gdense.o							H5Gdense.o
H5Gdeprec.o							H5Gdeprec.o
H5Gent.o							H5Gent.o
H5Gint.o							H5Gint.o
H5Glink.o							H5Glink.o
H5Gloc.o							H5Gloc.o
H5Gname.o							H5Gname.o
H5Gnode.o							H5Gnode.o
H5Gobj.o							H5Gobj.o
H5Goh.o								H5Goh.o
H5Groot.o							H5Groot.o
H5Gstab.o							H5Gstab.o
							      >	H5Gtest.o
H5Gtraverse.o							H5Gtraverse.o
H5HF.o								H5HF.o
H5HFbtree2.o							H5HFbtree2.o
H5HFcache.o							H5HFcache.o
							      >	H5HFdbg.o
H5HFdblock.o							H5HFdblock.o
H5HFdtable.o							H5HFdtable.o
H5HFhdr.o							H5HFhdr.o
H5HFhuge.o							H5HFhuge.o
H5HFiblock.o							H5HFiblock.o
H5HFiter.o							H5HFiter.o
H5HFman.o							H5HFman.o
H5HFsection.o							H5HFsection.o
H5HFspace.o							H5HFspace.o
H5HFstat.o							H5HFstat.o
							      >	H5HFtest.o
H5HFtiny.o							H5HFtiny.o
H5HG.o								H5HG.o
H5HGcache.o							H5HGcache.o
							      >	H5HGdbg.o
H5HGquery.o							H5HGquery.o
H5HL.o								H5HL.o
H5HLcache.o							H5HLcache.o
							      >	H5HLdbg.o
H5HLdblk.o							H5HLdblk.o
H5HLint.o							H5HLint.o
H5HLprfx.o							H5HLprfx.o
H5I.o								H5I.o
							      >	H5Idbg.o
H5Iint.o							H5Iint.o
							      >	H5Itest.o
H5L.o								H5L.o
							      >	H5Ldeprec.o
H5Lexternal.o							H5Lexternal.o
H5Lint.o							H5Lint.o
H5M.o								H5M.o
H5MF.o								H5MF.o
H5MFaggr.o							H5MFaggr.o
							      >	H5MFdbg.o
H5MFsection.o							H5MFsection.o
H5MM.o								H5MM.o
H5O.o								H5O.o
H5Oainfo.o							H5Oainfo.o
H5Oalloc.o							H5Oalloc.o
H5Oattr.o							H5Oattr.o
H5Oattribute.o							H5Oattribute.o
							      >	H5Obogus.o
H5Obtreek.o							H5Obtreek.o
H5Ocache.o							H5Ocache.o
H5Ocache_image.o						H5Ocache_image.o
H5Ochunk.o							H5Ochunk.o
H5Ocont.o							H5Ocont.o
H5Ocopy.o							H5Ocopy.o
H5Ocopy_ref.o							H5Ocopy_ref.o
H5Odbg.o							H5Odbg.o
							      >	H5Odeprec.o
H5Odrvinfo.o							H5Odrvinfo.o
H5Odtype.o							H5Odtype.o
H5Oefl.o							H5Oefl.o
H5Ofill.o							H5Ofill.o
H5Oflush.o							H5Oflush.o
H5Ofsinfo.o							H5Ofsinfo.o
H5Oginfo.o							H5Oginfo.o
H5Oint.o							H5Oint.o
H5Olayout.o							H5Olayout.o
H5Olinfo.o							H5Olinfo.o
H5Olink.o							H5Olink.o
H5Omessage.o							H5Omessage.o
H5Omtime.o							H5Omtime.o
H5Oname.o							H5Oname.o
H5Onull.o							H5Onull.o
H5Opline.o							H5Opline.o
H5Orefcount.o							H5Orefcount.o
H5Osdspace.o							H5Osdspace.o
H5Oshared.o							H5Oshared.o
H5Oshmesg.o							H5Oshmesg.o
H5Ostab.o							H5Ostab.o
							      >	H5Otest.o
H5Ounknown.o							H5Ounknown.o
H5P.o								H5P.o
H5PB.o								H5PB.o
							      >	H5PL.o
H5PLint.o							H5PLint.o
H5PLpath.o							H5PLpath.o
H5PLplugin_cache.o						H5PLplugin_cache.o
H5Pacpl.o							H5Pacpl.o
H5Pdapl.o							H5Pdapl.o
H5Pdcpl.o							H5Pdcpl.o
							      >	H5Pdeprec.o
H5Pdxpl.o							H5Pdxpl.o
H5Pencdec.o							H5Pencdec.o
H5Pfapl.o							H5Pfapl.o
H5Pfcpl.o							H5Pfcpl.o
H5Pfmpl.o							H5Pfmpl.o
H5Pgcpl.o							H5Pgcpl.o
H5Pint.o							H5Pint.o
H5Plapl.o							H5Plapl.o
H5Plcpl.o							H5Plcpl.o
H5Pmapl.o							H5Pmapl.o
H5Pmcpl.o							H5Pmcpl.o
H5Pocpl.o							H5Pocpl.o
H5Pocpypl.o							H5Pocpypl.o
H5Pstrcpl.o							H5Pstrcpl.o
							      >	H5Ptest.o
H5R.o								H5R.o
H5RS.o								H5RS.o
H5Rdeprec.o							H5Rdeprec.o
H5Rint.o							H5Rint.o
H5S.o								H5S.o
H5SL.o								H5SL.o
H5SM.o								H5SM.o
H5SMbtree2.o							H5SMbtree2.o
H5SMcache.o							H5SMcache.o
H5SMmessage.o							H5SMmessage.o
							      >	H5SMtest.o
H5Sall.o							H5Sall.o
H5Sdbg.o							H5Sdbg.o
							      >	H5Sdeprec.o
H5Shyper.o							H5Shyper.o
H5Snone.o							H5Snone.o
H5Spoint.o							H5Spoint.o
H5Sselect.o							H5Sselect.o
							      >	H5Stest.o
H5T.o								H5T.o
							      >	H5TS.o
H5Tarray.o							H5Tarray.o
H5Tbit.o							H5Tbit.o
H5Tcommit.o							H5Tcommit.o
H5Tcompound.o							H5Tcompound.o
H5Tconv.o							H5Tconv.o
H5Tconv_array.o							H5Tconv_array.o
H5Tconv_bitfield.o						H5Tconv_bitfield.o
H5Tconv_compound.o						H5Tconv_compound.o
H5Tconv_enum.o							H5Tconv_enum.o
H5Tconv_float.o							H5Tconv_float.o
H5Tconv_integer.o						H5Tconv_integer.o
H5Tconv_reference.o						H5Tconv_reference.o
H5Tconv_string.o						H5Tconv_string.o
H5Tconv_vlen.o							H5Tconv_vlen.o
H5Tcset.o							H5Tcset.o
H5Tdbg.o							H5Tdbg.o
							      >	H5Tdeprec.o
H5Tenum.o							H5Tenum.o
H5Tfields.o							H5Tfields.o
H5Tfixed.o							H5Tfixed.o
							      >	H5Tfloat.o
H5Tinit_float.o							H5Tinit_float.o
H5Tnative.o							H5Tnative.o
H5Toffset.o							H5Toffset.o
H5Toh.o								H5Toh.o
H5Topaque.o							H5Topaque.o
H5Torder.o							H5Torder.o
							      >	H5Tpad.o
H5Tprecis.o							H5Tprecis.o
H5Tref.o							H5Tref.o
H5Tstrpad.o							H5Tstrpad.o
H5Tvisit.o							H5Tvisit.o
H5Tvlen.o							H5Tvlen.o
H5UC.o								H5UC.o
H5VL.o								H5VL.o
H5VLcallback.o							H5VLcallback.o
H5VLdyn_ops.o							H5VLdyn_ops.o
H5VLint.o							H5VLint.o
H5VLnative.o							H5VLnative.o
H5VLnative_attr.o						H5VLnative_attr.o
H5VLnative_blob.o						H5VLnative_blob.o
H5VLnative_dataset.o						H5VLnative_dataset.o
H5VLnative_datatype.o						H5VLnative_datatype.o
H5VLnative_file.o						H5VLnative_file.o
H5VLnative_group.o						H5VLnative_group.o
H5VLnative_introspect.o						H5VLnative_introspect.o
H5VLnative_link.o						H5VLnative_link.o
H5VLnative_object.o						H5VLnative_object.o
H5VLnative_token.o						H5VLnative_token.o
H5VLpassthru.o							H5VLpassthru.o
							      >	H5VLtest.o
H5VM.o								H5VM.o
H5WB.o								H5WB.o
H5Z.o								H5Z.o
H5Zdeflate.o							H5Zdeflate.o
H5Zfletcher32.o							H5Zfletcher32.o
H5Znbit.o							H5Znbit.o
H5Zscaleoffset.o						H5Zscaleoffset.o
H5Zshuffle.o							H5Zshuffle.o
H5Zszip.o							H5Zszip.o
H5Ztrans.o							H5Ztrans.o
H5build_settings.o						H5build_settings.o
H5checksum.o							H5checksum.o
							      >	H5dbg.o
H5system.o							H5system.o
H5timer.o							H5timer.o
H5trace.o							H5trace.o

[markcmiller86]

So, using the information above, I constructed a bad case to see what would happen. I used H5Topen1 defined in H5Tdeprec.o and available in libhdf5.a in my compression plugin. That is a publicly available symbol in libhdf5.a that is in a .o file in the archive that is not listed as used by h5repack above. So, I figured it was a good chance that same symbol would not be used in my simple test_write_static test case. And...

lassen709{miller86}1030: env HDF5_PLUGIN_PATH=`pwd`/../src/plugin ./test_write_static zfpmode=1 rate=8 highd=1
    
Usage: test_write [opt1=value1 opt2=value2]...
    ifile=""                                  set input filename
    ofile="test_zfp.h5"                      set output filename
    
ZFP compression paramaters...
    zfpmode=1        (1=rate,2=prec,3=acc,4=expert,5=reversible)
    rate=8                                set rate for rate mode
    acc=0                         set accuracy for accuracy mode
    prec=11                     set precision for precision mode
    minbits=0                        set minbits for expert mode
    maxbits=4171                     set maxbits for expert mode
    maxprec=64                       set maxprec for expert mode
    minexp=-1074                      set minexp for expert mode
    
1D dataset generation arguments...
    npoints=1024             set number of points for 1D dataset
    noise=0.001         set amount of random noise in 1D dataset
    amp=17.7             set amplitude of sinusoid in 1D dataset
    chunk=256                      set chunk size for 1D dataset
    doint=0                                  requires ZFP>=0.5.1
                            
Advanced cases...
    highd=1                                4D w/2D chunk example
    sixd=0                                   requires ZFP>=0.5.4
    zfparr=0                requires ZFP>=0.5.4 with CFP enabled

4 cd_values=1,0,0,1075838976
    help=0                                     this help message
./test_write_static: symbol lookup error: /g/g11/miller86/tmp/H5Z-ZFP/test/../src/plugin/libh5zzfp.so: undefined symbol: H5Topen1

But, that example works fine when I don't reference that symbol.

[jhendersonHDF]

@markcmiller86 Just to chime in on

I think that one of two possible choices may need to be made here for static-only builds. Either forgo support for dlopen()ing plugins

When I last looked at trying to do this, I remember that it would be simple to do for CMake builds and a bit harder to do for Autotools builds. Though if we get rid of Autotools..

[markcmiller86]

For static-only builds, ChatGPT tells me there are simple ways to force an executable like h5repack to load all symbols in all .o files in libhdf5.a...

Yes, you can force the linker to load all object files from a static library (.a archive) even if the executable does not explicitly reference symbols from some of them. Here are several ways to do this:

1. Use --whole-archive (GNU ld, Clang, and LLD)

If you are using GNU ld, Clang's linker, or LLD, you can use the --whole-archive linker flag to force inclusion of all object files in libfoo.a:

gcc main.o -Wl,--whole-archive libfoo.a -Wl,--no-whole-archive -o my_executable

Explanation:

• -Wl,--whole-archive forces the linker to include all object files from libfoo.a, even if their symbols are not referenced.
• -Wl,--no-whole-archive restores the default behavior for any libraries that follow.

This method ensures that all .o files from libfoo.a are linked into my_executable.

2. Use -force_load (macOS, ld on Darwin)

If you are using macOS's ld (which does not support --whole-archive), use -force_load:

gcc main.o -Wl,-force_load,libfoo.a -o my_executable

This is the macOS equivalent of --whole-archive.

3. Extract and Link Individual .o Files

If --whole-archive or -force_load is not an option, you can manually extract and link all object files from libfoo.a:

mkdir tempdir
cd tempdir
ar x ../libfoo.a  # Extract all .o files
gcc ../main.o *.o -o my_executable
cd ..
rm -rf tempdir

This explicitly links every object file from libfoo.a.

4. Define a Dummy Symbol in Each .o File

If you have control over the source code of libfoo.a, another approach is to ensure each object file defines a weak symbol that is referenced from main.o, preventing them from being dropped.

For example, in each .c file:

__attribute__((weak)) void force_link_this_file() {}

Then, in main.c:

extern void force_link_this_file();
void (*dummy_ref)() = force_link_this_file;

This ensures at least one reference to every .o file.

Conclusion

• Linux, GCC, Clang, LLD: Use --whole-archive
• macOS (Darwin ld): Use -force_load
• Portable approach: Extract .o files manually with ar
• Code-based approach: Define weak symbols and reference them

Would you like help automating this for your build system (Makefile, CMake, etc.)?

[markcmiller86]

To summarize then...

• h5repack executable return code and error messaging need work, especially when requested filter is not-optional but still fails
• An hdf5 file produced by h5repack with an optional plugin (but which apparently failed to load the plugin), produces datasets in a sort of limbo state...it shows that the filter was applied and shows the filter's id but the dataset is unchanged from its original file.
• env HDF5_PLUGIN_PATH=<path-to-plugin-dir> h5repack for static-only builds is failing to load a plugin that another static-only libhdf5.a client uses fine and that h5ls is able to use fine to read a file requiring the plugin.
• static-only tools like h5repack need some software engineering work to ensure they will work properly when loading plugins. Right now, if they work, they are just getting lucky.