HDF5-DIAG warnings calling `open_mfdataset` with more than `file_cache_maxsize` datasets (hdf5 1.12.2) #7549

mx-moth · 2023-02-23T02:28:23Z

What happened?

Using ds = open_mfdataset(...) to open more than file_cache_maxsize files, then saving the combined dataset with ds.to_netcdf() prints many HDF5-DIAG warnings. This happens when using hdf5==1.12.2 as bundled with netcdf4~=1.6.0. Downgrading to netcdf4~=1.5.8 which bundles hdf5==1.12.0 stops the warnings being printed. The output file appears to be correct in either case.

What did you expect to happen?

No warnings from HDF5-DIAG. Either raise an error because of the number of files being opened at once, or behaving as open_mfdataset() normally does.

Minimal Complete Verifiable Example

import argparse
import pathlib
import tempfile
from typing import List

import netCDF4
import xarray

HERE = pathlib.Path(__file__).parent


def add_arguments(parser: argparse.ArgumentParser):
    parser.add_argument('count', type=int, default=200, nargs='?')
    parser.add_argument('--file-cache-maxsize', type=int, required=False)


def main():
    parser = argparse.ArgumentParser()
    add_arguments(parser)
    opts = parser.parse_args()

    if opts.file_cache_maxsize is not None:
        xarray.set_options(file_cache_maxsize=opts.file_cache_maxsize)

    temp_dir = tempfile.mkdtemp(dir=HERE, prefix='work-dir-')
    work_dir = pathlib.Path(temp_dir)
    print("Working in", work_dir.name)

    print("Making", opts.count, "datasets")
    dataset_paths = make_many_datasets(work_dir, count=opts.count)

    print("Combining", len(dataset_paths), "datasets")
    dataset = xarray.open_mfdataset(dataset_paths, lock=False)
    dataset.to_netcdf(work_dir / 'combined.nc')


def make_many_datasets(
    work_dir: pathlib.Path,
    count: int = 200
) -> List[pathlib.Path]:
    dataset_paths = []
    for i in range(count):
        variable = f'var_{i}'
        path = work_dir / f'{variable}.nc'
        dataset_paths.append(path)
        make_dataset(path, variable)

    return dataset_paths


def make_dataset(
    path: pathlib.Path,
    variable: str,
) -> None:
    ds = netCDF4.Dataset(path, "w", format="NETCDF4")
    ds.createDimension("x", 1)
    var = ds.createVariable(variable, "i8", ("x",))
    var[:] = 1
    ds.close()


if __name__ == '__main__':
    main()

MVCE confirmation

Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
Complete example — the example is self-contained, including all data and the text of any traceback.
Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

Working in work-dir-yi3jp5ly
Making 128 datasets
Combining 128 datasets
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 1:
  #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
    major: Attribute
    minor: Can't open object
  #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitGroomNumberOfSignificantDigits'
    major: Attribute
    minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 1:
  #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
    major: Attribute
    minor: Can't open object
  #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits'
    major: Attribute
    minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 1:
  #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
    major: Attribute
    minor: Can't open object
  #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitRoundNumberOfSignificantBits'
    major: Attribute
    minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 1:
  #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
    major: Attribute
    minor: Can't open object
  #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitGroomNumberOfSignificantDigits'
    major: Attribute
    minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 1:
  #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
    major: Attribute
    minor: Can't open object
  #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits'
    major: Attribute
    minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 1:
  #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
    major: Attribute
    minor: Can't open object
  #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitRoundNumberOfSignificantBits'
    major: Attribute
    minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 2:
  #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
    major: Attribute
    minor: Can't open object
  #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitGroomNumberOfSignificantDigits'
    major: Attribute
    minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 2:
  #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
    major: Attribute
    minor: Can't open object
  #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits'
    major: Attribute
    minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 2:
  #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
    major: Attribute
    minor: Can't open object
  #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitRoundNumberOfSignificantBits'
    major: Attribute
    minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 3:
  #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
    major: Attribute
    minor: Can't open object
  #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitGroomNumberOfSignificantDigits'
    major: Attribute
    minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 3:
  #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
    major: Attribute
    minor: Can't open object
  #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits'
    major: Attribute
    minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 3:
  #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
    major: Attribute
    minor: Can't open object
  #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitRoundNumberOfSignificantBits'
    major: Attribute
    minor: Object not found

Anything else we need to know?

The example is a script to run on the command line. Assuming the file is named test.py, invoke as:

# This will show the error. Defaults to making 200 files
$ python3 ./test.py
# This will not show the error - the number of files is less than file_cache_maxsize:
$ python3 ./test.py 127
# This will adjust file_cache_maxsize to show the error again, despite the lower number of files
$ python3 ./test.py 11 --file-cache-maxsize=10

The log output is from python3 ./test.py 128. The length of the log output depends on the number of files opened - the more files the longer the log. The log output from python3 ./test.py using the default of 200 datasets was too long to include in this issue!

All output files are restricted to directories created in the current working directory named work-dir-XXXXXXX, and are retained after the program exits. 200 datasets take up a total of 1.7MB.

Environment

Failing environment:

INSTALLED VERSIONS

commit: None
python: 3.8.10 (default, Nov 14 2022, 12:59:47)
[GCC 9.4.0]
python-bits: 64
OS: Linux
OS-release: 5.15.0-58-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_AU.UTF-8
LOCALE: ('en_AU', 'UTF-8')
libhdf5: 1.12.2
libnetcdf: 4.9.0

xarray: 2023.1.0
pandas: 1.5.3
numpy: 1.24.2
scipy: None
netCDF4: 1.6.2
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.6.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2023.2.0
distributed: None
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: 2023.1.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 44.0.0
pip: 23.0.1
conda: None
pytest: None
mypy: None
IPython: None
sphinx: None

Working environment:

INSTALLED VERSIONS

commit: None
python: 3.8.10 (default, Nov 14 2022, 12:59:47)
[GCC 9.4.0]
python-bits: 64
OS: Linux
OS-release: 5.15.0-58-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_AU.UTF-8
LOCALE: ('en_AU', 'UTF-8')
libhdf5: 1.12.0
libnetcdf: 4.7.4

xarray: 2023.1.0
pandas: 1.5.3
numpy: 1.24.2
scipy: None
netCDF4: 1.5.8
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.6.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2023.2.0
distributed: None
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: 2023.1.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 44.0.0
pip: 23.0.1
conda: None
pytest: None
mypy: None
IPython: None
sphinx: None

The text was updated successfully, but these errors were encountered:

dcherian · 2023-02-28T23:14:44Z

Not clear to me if this is a bug or just some verbose warnings.

@kmuehlbauer do you have any thoughts?

kmuehlbauer · 2023-03-01T07:01:40Z

@dcherian Thanks for the ping.

I can reproduce in a fresh conda-forge env with pip installed netcdf4, xarray and dask.

@mx-moth A search brought up this likely related issue over at netcdf-c, Unidata/netcdf-c#2458. The according PR with a fix Unidata/netcdf-c#2461 is milestoned for netcdf-c 4.9.1.

kmuehlbauer · 2023-03-01T09:19:26Z

I just tested this with netcdf-c 4.9.1 but still these errors show up, also using conda-forge only install.

To make this even weirder I've checked creation/reading with only hdf5/h5py/h5netcdf in the environment. Seems everything is working well.

import argparse
import pathlib
import tempfile
from typing import List

import h5netcdf.legacyapi as nc
import xarray

HERE = pathlib.Path(__file__).parent


def add_arguments(parser: argparse.ArgumentParser):
    parser.add_argument('count', type=int, default=200, nargs='?')
    parser.add_argument('--file-cache-maxsize', type=int, required=False)


def main():
    parser = argparse.ArgumentParser()
    add_arguments(parser)
    opts = parser.parse_args()

    if opts.file_cache_maxsize is not None:
        xarray.set_options(file_cache_maxsize=opts.file_cache_maxsize)

    temp_dir = tempfile.mkdtemp(dir=HERE, prefix='work-dir-')
    work_dir = pathlib.Path(temp_dir)
    print("Working in", work_dir.name)

    print("Making", opts.count, "datasets")
    dataset_paths = make_many_datasets(work_dir, count=opts.count)

    print("Combining", len(dataset_paths), "datasets")
    dataset = xarray.open_mfdataset(dataset_paths, lock=False, engine="h5netcdf")
    dataset.to_netcdf(work_dir / 'combined.nc', engine="h5netcdf")


def make_many_datasets(
    work_dir: pathlib.Path,
    count: int = 200
) -> List[pathlib.Path]:
    dataset_paths = []
    for i in range(count):
        variable = f'var_{i}'
        path = work_dir / f'{variable}.nc'
        dataset_paths.append(path)
        make_dataset(path, variable)

    return dataset_paths


def make_dataset(
    path: pathlib.Path,
    variable: str,
) -> None:
    ds = nc.Dataset(path, "w")
    ds.createDimension("x", 1)
    var = ds.createVariable(variable, "i8", ("x",))
    var[:] = 1
    ds.close()


if __name__ == '__main__':
    main()

# This will show no error. Defaults to making 200 files
$ python3 ./test.py
# This will also not show the error - the number of files is less than file_cache_maxsize:
$ python3 ./test.py 127
# This will adjust file_cache_maxsize to show the error again, despite the lower number of files, here we see another error issued by h5py
$ python3 ./test.py 11 --file-cache-maxsize=10

Working in work-dir-40mwn69y
Making 11 datasets
Combining 11 datasets
Traceback (most recent call last):
  File "/home/kai/python/gists/xarray/7549.py", line 65, in <module>
    main()
  File "/home/kai/python/gists/xarray/7549.py", line 36, in main
    dataset.to_netcdf(work_dir / 'combined.nc', engine="h5netcdf")
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/xarray/core/dataset.py", line 1911, in to_netcdf
    return to_netcdf(  # type: ignore  # mypy cannot resolve the overloads:(
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/xarray/backends/api.py", line 1226, in to_netcdf
    writes = writer.sync(compute=compute)
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/xarray/backends/common.py", line 172, in sync
    delayed_store = da.store(
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/dask/array/core.py", line 1236, in store
    compute_as_if_collection(Array, store_dsk, map_keys, **kwargs)
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/dask/base.py", line 341, in compute_as_if_collection
    return schedule(dsk2, keys, **kwargs)
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/dask/threaded.py", line 89, in get
    results = get_async(
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/dask/local.py", line 511, in get_async
    raise_exception(exc, tb)
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/dask/local.py", line 319, in reraise
    raise exc
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/dask/local.py", line 224, in execute_task
    result = _execute_task(task, data)
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/dask/array/core.py", line 126, in getter
    c = np.asarray(c)
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/xarray/core/indexing.py", line 459, in __array__
    return np.asarray(self.array, dtype=dtype)
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/xarray/core/indexing.py", line 623, in __array__
    return np.asarray(self.array, dtype=dtype)
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/xarray/core/indexing.py", line 524, in __array__
    return np.asarray(array[self.key], dtype=None)
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/xarray/backends/h5netcdf_.py", line 43, in __getitem__
    return indexing.explicit_indexing_adapter(
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/xarray/core/indexing.py", line 815, in explicit_indexing_adapter
    result = raw_indexing_method(raw_key.tuple)
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/xarray/backends/h5netcdf_.py", line 50, in _getitem
    return array[key]
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/h5netcdf/core.py", line 337, in __getitem__
    padding = self._get_padding(key)
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/h5netcdf/core.py", line 291, in _get_padding
    shape = self.shape
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/h5netcdf/core.py", line 268, in shape
    return tuple([self._parent._all_dimensions[d].size for d in self.dimensions])
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/h5netcdf/core.py", line 268, in <listcomp>
    return tuple([self._parent._all_dimensions[d].size for d in self.dimensions])
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/h5netcdf/dimensions.py", line 113, in size
    if self.isunlimited():
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/h5netcdf/dimensions.py", line 133, in isunlimited
    return self._h5ds.maxshape == (None,)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/kai/miniconda/envs/test-netcdf4/lib/python3.9/site-packages/h5py/_hl/dataset.py", line 588, in maxshape
    space = self.id.get_space()
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 299, in h5py.h5d.DatasetID.get_space
ValueError: Invalid dataset identifier (invalid dataset identifier)

INSTALLED VERSIONS ------------------ commit: None python: 3.9.16 | packaged by conda-forge | (main, Feb 1 2023, 21:39:03) [GCC 11.3.0] python-bits: 64 OS: Linux OS-release: 5.14.21-150400.24.46-default machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: de_DE.UTF-8 LOCALE: ('de_DE', 'UTF-8') libhdf5: 1.12.2 libnetcdf: None

xarray: 2023.2.0
pandas: 1.5.3
numpy: 1.24.2
scipy: None
netCDF4: None
pydap: None
h5netcdf: 1.1.0
h5py: 3.7.0
Nio: None
zarr: None
cftime: None
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2023.2.1
distributed: 2023.2.1
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: 2023.1.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 67.4.0
pip: 23.0.1
conda: None
pytest: None
mypy: None
IPython: None
sphinx: None

Update: added engine="h5netcdf" to call of open_mfdataset.

kmuehlbauer · 2023-03-01T09:37:43Z

This as far I can get for the moment. @mx-moth I'd suggest to go upstream (netCDF4/netcdf-c) with details about this issue. At least we can rule out an issue only related to hdf5=1.12.2.

Maybe @DennisHeimbigner can shed more light here?

( #005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitRoundNumberOfSignificantBits'
    major: Attribute
    minor: Object not found

trexfeathers · 2023-03-08T18:00:35Z

@pp-mo 👀

pp-mo · 2023-03-09T10:28:24Z

@pp-mo 👀

See our issues here : SciTools/iris#5187
It does look a lot like the "same" problem.
It is specifically related to the use of dask and multi-threading.
Here in python-netCDF4 land, we were protected from all of that before netCDF4 v1.6.1, since before that all netcdf operations held the Python GIL so could not be re-entered by threads.

trexfeathers · 2023-03-09T10:30:28Z

before netCDF4 v1.6.2

Before v1.6.1, I believe

pp-mo · 2023-03-09T10:43:50Z

@trexfeathers Before v1.6.1, I believe

Oops, fixed.

gewitterblitz · 2023-03-24T21:58:38Z

Thanks, @trexfeathers. I had the same problem of HDF5 DIAG warnings after upgrading to xarray v2023.3.0 yesterday. Your diagnosis in SciTools/iris#5187 helped isolate the issue with libnetcdf v4.9.1. Downgrading libnetcdf to v4.8.1 resulted in no HDF5 warnings.

Mikejmnez · 2023-03-26T00:41:10Z

Thanks everybody. Similar to @gewitterblitz and based on SciTools/iris#5187 , pinning libnetcdf to v4.8.1 did the trick

jhgoebbert · 2024-08-22T09:44:21Z

This merged pull request fixes the issue in netCDF: Unidata/netcdf-c#2675

mx-moth added bug needs triage Issue that has not been reviewed by xarray team member labels Feb 23, 2023

dcherian added topic-backends and removed needs triage Issue that has not been reviewed by xarray team member bug labels Feb 28, 2023

trexfeathers mentioned this issue Mar 8, 2023

HDF5-DIAG warnings from libnetcdf v4.9.1 SciTools/iris#5187

Closed

malmans2 mentioned this issue Mar 16, 2023

IGPwinter opening bug hainegroup/oceanspy#330

Closed

nbud mentioned this issue Mar 31, 2023

Many HDF5-DIAG warnings when used with xarray and dask Unidata/netcdf4-python#1242

Open

trexfeathers mentioned this issue Apr 17, 2023

Xarray does not support full range of netcdf-python compression options #7388

Closed

ekatef mentioned this issue May 5, 2023

Error in build_cutout for Ukraine pypsa-meets-earth/pypsa-earth#669

Open

2 tasks

keewis mentioned this issue Nov 3, 2023

Segmentation fault 139 (SIGSEGV) #8410

Closed

5 tasks

JamesRunnalls mentioned this issue Aug 15, 2024

Xarray producing many warnings when opening large numbers of files eawag-surface-waters-research/alplakes-fastapi#10

Closed

dcherian added the upstream issue label Aug 22, 2024

dcherian closed this as completed Aug 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDF5-DIAG warnings calling `open_mfdataset` with more than `file_cache_maxsize` datasets (hdf5 1.12.2) #7549

HDF5-DIAG warnings calling `open_mfdataset` with more than `file_cache_maxsize` datasets (hdf5 1.12.2) #7549

mx-moth commented Feb 23, 2023 •

edited

Loading

INSTALLED VERSIONS

INSTALLED VERSIONS

dcherian commented Feb 28, 2023

kmuehlbauer commented Mar 1, 2023

kmuehlbauer commented Mar 1, 2023 •

edited

Loading

kmuehlbauer commented Mar 1, 2023

trexfeathers commented Mar 8, 2023

pp-mo commented Mar 9, 2023 •

edited

Loading

trexfeathers commented Mar 9, 2023

pp-mo commented Mar 9, 2023

gewitterblitz commented Mar 24, 2023

Mikejmnez commented Mar 26, 2023

jhgoebbert commented Aug 22, 2024

HDF5-DIAG warnings calling open_mfdataset with more than file_cache_maxsize datasets (hdf5 1.12.2) #7549

HDF5-DIAG warnings calling open_mfdataset with more than file_cache_maxsize datasets (hdf5 1.12.2) #7549

Comments

mx-moth commented Feb 23, 2023 • edited Loading

What happened?

What did you expect to happen?

Minimal Complete Verifiable Example

MVCE confirmation

Relevant log output

Anything else we need to know?

Environment

INSTALLED VERSIONS

INSTALLED VERSIONS

dcherian commented Feb 28, 2023

kmuehlbauer commented Mar 1, 2023

kmuehlbauer commented Mar 1, 2023 • edited Loading

kmuehlbauer commented Mar 1, 2023

trexfeathers commented Mar 8, 2023

pp-mo commented Mar 9, 2023 • edited Loading

trexfeathers commented Mar 9, 2023

pp-mo commented Mar 9, 2023

gewitterblitz commented Mar 24, 2023

Mikejmnez commented Mar 26, 2023

jhgoebbert commented Aug 22, 2024

HDF5-DIAG warnings calling `open_mfdataset` with more than `file_cache_maxsize` datasets (hdf5 1.12.2) #7549

HDF5-DIAG warnings calling `open_mfdataset` with more than `file_cache_maxsize` datasets (hdf5 1.12.2) #7549

mx-moth commented Feb 23, 2023 •

edited

Loading

kmuehlbauer commented Mar 1, 2023 •

edited

Loading

pp-mo commented Mar 9, 2023 •

edited

Loading