demprep segfaults

[ladoramkershner]

ISIS version(s) affected: 6.0.0 and 5.0.2; works in 3.9.2 (after setting ISISDATA)

Description

demprep seg faults with no helpful information. There are not amendments to the print.prt file

How to reproduce

demprep from= Lunar_LRO_LOLAKaguya_DEMmerge_60N60S_512ppd_radius.lbl to=Lunar_LRO_LOLAKaguya_DEMmerge_60N60S_512ppd_radius.cub

In the archived pds area, the label information is located MAP2/images/LolaKaguya_Topo/

Possible Solution

Any sort of error would be helpful. It is possible the demprep API changed between 3.9.2 and 6.0.0, but I wouldn't know where to seek that information, so a more direct error if the lbl is not of the correct form would be helpful.

Additional context

[ladoramkershner]

Additional information:

@lwellerastro pointed me to a Phobos DEM example (Phobos_ME_HRSC_DEM_Global_2ppd.lbl/ tif)
If you grab the relevant data and edit the lbl to have radius values (from the MAP2/images/ pds derived product archive) and edit the label to have a Base = raduis of Phobos (since ISIS doesn't operate on height DEMs)

cubeatt from= Phobos_ME_HRSC_DEM_Global_2ppd.lbl to= Phobos_ME_HRSC_DEM_Global_2ppd_radius.cub
editlab from= Phobos_ME_HRSC_DEM_Global_2ppd_radius.cub options=modkey grpname=pixels keyword=base value=11100.0

You can get a successful run of demprep

demprep from= Phobos_ME_HRSC_DEM_Global_2ppd_radius.cub to= Phobos_ME_HRSC_DEM_Global_2ppd_radius_demprep.cub

with output

Group = Results
  MinimumRadius = 8160.0 <meters>
  MaximumRadius = 14017.0 <meters>
End_Group

[lwellerastro]

The data the author references is no longer available in the location specified. I have copied the data into a work directory from an alternate local area that may cease to exist in the near future (would take hours to download from our site). See the project area for kaguya under lweller/Test/.

I have confirmed demprep segfaults under isis7.1.0

demprep from=Lunar_LRO_LOLAKaguya_DEMmerge_60N60S_512ppd_radius.lbl to=Lunar_LRO_LOLAKaguya_DEMmerge_60N60S_512ppd_radius.cub
Segmentation fault (core dumped)

I tested this under 3.9.1 and it gave a more meaningful error message that might be useful:

demprep from=Lunar_LRO_LOLAKaguya_DEMmerge_60N60S_512ppd_radius.lbl to=Lunar_LRO_LOLAKaguya_DEMmerge_60N60S_512ppd_radius.cub
terminate called after throwing an instance of 'Isis::IException'
  what():  **I/O ERROR** Unable to open [.../templates/labels/CubeFormatTemplate.pft] in Pvl.cpp at 86.
Abort (core dumped)

I removed the leading portion of the path name in the error above. It is looking in the isis3 data base area which does not exist. The data area available for isis3.9.1 does not have a templates directory at all. Pretty tough to describe this without path names... See the project area indicated above and the file error.txt for the full path information.

I know the templates area moved some time ago (and have no idea where it is now), and this breaking appears to be affected by it.

Maybe not super high priority because there are work arounds (in addition to OPs work around, buried on the serving site is info on how to convert to an isis cube using gdal), but it might be straightforward to address at some point.

Update: Found where the file is now located: $ISISROOT/appdata/templates/labels/CubeFormatTemplate.pft

[chkim-usgs]

Hi @lwellerastro,

I'm in the process of looking for the Kaguya data used above in order to reproduce the bug. I'm unable to find the test data under ../lweller/Test, do you possibly know where else I can find the data?

[lwellerastro]

Hi @chkim-usgs, the data are where I've cryptically indicated ;)
I'll send you the full path name via chat.

[chkim-usgs]

@lwellerastro, I was able to get the error messages thru instead of the segfault:

It looks like ^History being set in Lunar_LRO_LOLAKaguya_DEMmerge_60N60S_512ppd_radius.lbl causes ISIS to look for that particular file which does not exist:

There's a comment in the code stating that if the History was unable to be read from the input cube in order to be transferred to the output cube, a new History blob should be created. Could you possibly confirm if that's the correct way to go about a cube with no History or should the errors in the first image above be returned instead?

Manually removing the ^History keyword pair from the input label seems to run a successful demprep, btw.

[lwellerastro]

Oh, I see what's going on. This is not a pds file (like I thought), but a derived product from an ISIS cube made to look and act like a pds file (hence the .lbl and pointer to a tif file) and it's in a category of files where something about the label makes access to the History blob not possible.

I wonder if changes via post #4337 actually address the problem? Both the original poster and my tests many months later happened in versions of isis that don't yet have the change in 4337, I think. Looking at the change log I don't actually see that fix in any recent version of isis though.

I can test this under 7.2.0 and 8.0.0-RC1 to see if the results are different, but I won't be able to do so today since I have some time sensitive work to do.

[lwellerastro]

Hi @chkim-usgs, demprep continues to segfault on this particular file under 7.2.0 and 8.0.0-RC1, so if changes did make it to these versions via #4337, they are not helping here. I can't even run blobdump to get the History information and that's something that has worked for some older problematic files.

I think the best/correct way to move forward for this file and others like it is to create a new History blob. Occasional cases like this will continue to exist, but if the History entry is not readable, it seems like the only way to move forward is to create a new one.

[chkim-usgs]

That sounds good, I'll look into creating that conditional which results in a new History blob creation. Out of curiosity, could I see the blobdump command you used that failed to get the History info? Haven't worked with blobdump yet so figured this would be a good introduction.

[lwellerastro]

blobdump from=Lunar_LRO_LOLAKaguya_DEMmerge_60N60S_512ppd_radius.lbl to=dump.txt name=IsisCube type=History

It produced similar errors for those programs actually reporting problems:

Program = blobdump
    Class   = "I/O ERROR"
    Code    = 4
    Message = "Unable to read History [IsisCube]"
    File    = Blob.cpp
    Line    = 332

blobdump was recommended to use via #4337 (and personal communication) for cases where old data was failing cathist. I'll share this last example with you via chat. There are some characters in the History blob from the past that were tripping up things which is what #4337 was attempting to address. Old files are still in use so we still run up against the issues and blobdump sometimes help.