Getting dataset attribute (getDatasetAttribute) on a 32 bit floating point in NODE/javascript returns a totally different value

[lioneldelmo]

Hi, I need some guidance in trying to get a 32bit floating point in an attribute, getting a different result in NODEJS --- ie. const scale = file.getDatasetAttribute('group_energy', 'scale_factor')

Actual Value (32bit floating point): 1.52597E-15
Result: 2.4441002025e-314

Is there something I need to do as far as conversion is concerned?

Attached is the actual NetCDF file used.

Thanks,

OR_GLM-L2-LCFA_G16_s20182632044200_e20182632044400_c20182632044426.nc.zip

[rimmartin]

Hi, I'll test; thanks for the example to try. Was the h5 created on the same machine you are serving?

[rimmartin]

added a check for size and now get 1.5259699826644982e-15.
pushed to the repository.

But I'm running thru your h5 looking for more types not handled; looks like a lot of good testing

[lioneldelmo]

Great! Thank you for your quick turnaround on this issue.

For now, I created a function that will convert Float64 to Float32 and this seems to work just fine. But will use your fix once it's available.
The following is the function I created to convert,

function convertFloat64ToFloat32(float64) { let a = new Float64Array(1); a[0] = float64; let b = new Float32Array(a.buffer); return b[0]; }

As for your question regarding the h5 file, I did not generate the file, I downloaded it from noaa.gov (publicly available). However, you may not be able to go the site until the government shutdown is over.
https://data.nodc.noaa.gov/cgi-bin/iso?id=gov.noaa.ncdc:C01527.
Let me know, if you need more sample files and I'll send them to you, I've got a bunch.

In a different topic in case you're interested, one thing I noticed is that few of the NetCDF / H5 files I downloaded, have an 'id' attribute/property and this causes conflict with the native 'id' property of the file, thus overwriting the original value of the 'id' with the value stored in the file. see code snippet below. The file.id gets overwritten with the value stored in the file after calling refresh(). So my workaround is to store the value of the id first before i call the refresh function.

const file = new hdf5.File(<path to the hd5 file>, Access.ACC_RDONLY); const fileId = file.id; file.refresh();

Attached is the sample NetCDF/H5 file that has an 'id' attribute.
(https://github.com/HDF-NI/hdf5.node/files/2771891/OR_GLM-L2-LCFA_G16_s20183051836400_e20183051837000_c20183051837027.nc.zip)

[rimmartin]

Hi @lioneldelmo,
the repository has a number of fixes committed; int 16's also weren't sized correctly.

The file and group id is the one gotcha I need to work around; all dataset attributes can be gotten without becoming javascript properties.

I'm adding a getAttributeNames and readAttribute on file and groups so the refresh and flush aren't the only way. One thing to be cautious about is flush might change types on attributes. Maybe I should add a big flashing red caution sign to the readme. Javascript doesn't have some types or that stay the same. Also it doesn't have a 64 bit integer; internally stores as a double and thus the mantis is only 53 bit. I custom made an Int64 for the id's but it doesn't have the math operators with others types such as Int32

if you add (synchronous) function callback for dataset options

            const readBuffer=h5lt.readDataset(group.id, 'Dielectric Constant', function(options){
                options.rank.should.equal(1);
                options.rows.should.equal(5);
            });

the options have the attributes instead of readBuffer having them.

The reason I asked about creation vs nodejs server is different OS's and js engines have endian differences which I'm looking into. Noone's filed an issue on it yet but I'm suspecting it.

Gong to do a bit of refactoring on internal datatype handling and clean up that part of the code

[rimmartin]

currently the group_energy atributes read as

_FillValue :  -1
long_name :  GLM L2+ Lightning Detection: group radiant energy
standard_name :  lightning_radiant_energy
_Unsigned :  true
valid_range :  0,-6
scale_factor :  1.5259699826644982e-15
add_offset :  0
units :  J
coordinates :  group_parent_flash_id event_parent_group_id group_id lightning_wavelength group_time_threshold group_time_offset group_lat group_lon
grid_mapping :  goes_lat_lon_projection
cell_measures :  area: group_area
cell_methods :  lightning_wavelength: sum group_time_offset: mean (times of occurrence of group's constituent events defined by variable event_parent_group_id) area: mean (centroid location of constituent events defined by variable event_parent_group_id weighted by their radiant energies) where cloud
ancillary_variables :  group_quality_flag
DIMENSION_LIST :  ->/number_of_groups

[rimmartin]

Can use an option and destructuring assignment to get the attributes on a group separate from the group object.

            var group, attrs;
            [group, attrs] = file.openGroup('pmc/refinement', {separate_attributes: true});

thinking yet how to write values back but this at least gets them separate. Haven't done it on a File object because it is a new instance constructor type; "new hdf5Lib.hdf5.File". Studying how this could be possibly achieved there too.

And working on handling many more datatypes