stereo_parse failing to read camera models in Ubuntu 18.04

[dbolan]

I know I emailed about this a while back, I think I've narrowed it down sufficiently for a bug report.
Basically there is some package in the base Ubuntu 18.04 installation that breaks stereo_parse.
The reproduction steps below are probably overkill because I was trying to prove to myself whether the problem was in ISIS or in ASP.

You will need:

• A thumb drive
• An external hard drive

Steps to reproduce:

• Before beginning, make sure you have your SPICE kernels and stereo pair on the external hard drive. I'm using M152445210LE and M152451994LE.
• It will also be helpful to have a copy of the miniconda installer and the tar file containing ASP 2.6.2.
• Create a bootable USB stick for Ubuntu 18.04. Boot into it.
• Mount your external hard drive.
• Install miniconda3 4.7.0 using default locations, and run conda init.
• Install ISIS 3.8.0 using conda. (I saw this same bug with 3.7.1, though I didn't use it for this test)
• Process your stereo pair with ISIS and then run stereo inside the environment you created for installing ISIS:

export ISIS3DATA=../isis3data
lronac2isis from=M152445210LE.IMG to=l0.cub
lronac2isis from=M152451994LE.IMG to=r0.cub
spiceinit from=l0.cub
spiceinit from=r0.cub
lronaccal from=l0.cub to=l1.cub
lronaccal from=r0.cub to=r1.cub
stereo l1.cub r1.cub output/results

• You should be able to reach the 'rough homography' pass.

• Make sure you save l1.cub and r1.cub to somewhere on your external hard drive.

• Next install a minimal installation of Ubuntu. Do not install any updates.

• Repeat the process you ran on the bootable USB stick. Install ISIS, and process your stereo pair and run stereo.

• You should get an error like the following:

...
Loading camera model: l1.cub 

None
Traceback (most recent call last):
  File "/home/dan/asptest/asp262/libexec/stereo", line 124, in <module>
    settings=run_and_parse_output( "stereo_parse", args, sep, opt.verbose )
  File "/home/dan/asptest/asp262/libexec/asp_system_utils.py", line 264, in run_and_parse_output
    raise Exception('Failed executing: ' + " ".join(call))
Exception: Failed executing: /home/dan/asptest/asp262/libexec/stereo_parse l1.cub r1.cub output/results --threads 12 --corr-seed-mode 1

• If you run this stereo_parse command, you should see the following:

...
Loading camera model: l1.cub 


Error: **ERROR** Unable to initialize camera model from group [Instrument].
**ERROR** Unsupported camera model, unable to find plugin for SpacecraftName [LUNAR RECONNAISSANCE ORBITER] with InstrumentId [NACL].
**ERROR** Unable to find plugin [LroNarrowAngleCameraPlugin] in shared library [/home/dan/miniconda3/envs/isis/lib/LroNarrowAngleCamera].

• If you deactivate the conda environment for ISIS and set ISISROOT to point to the directory containing ASP, you should see the "Failed to convert string [1,79769313486232e+308] to a double" issue I've been talking about in a few other places.
• If you install updates with sudo apt update; sudo apt upgrade, you should see the same errors.
• If you run stereo with the cubes that worked on the bootable USB, you should see the same errors.
• Conversely, if you run stereo on from the bootable USB on the cubes that were made on the desktop, it should work.

Comments
Phew!
Even with all of this, I'm still not 100% sure that the problem lies in ASP, it just seems quite a bit more likely, since (as far as I've seen) current versions of ISIS don't store their camera model plugins like this anymore.
According to cubediff the cubes generated by ISIS are identical, though comparison is slightly complicated by the fact that my username on the bootable USB was 'ubuntu' instead of 'dan'. Poor foresight on my part, sorry. After a few sed queries changing '/home/dan' to '/home/ubuntu' and fixing the Bytes field in the History object to account for the changing string lengths of the label, pvldiff also reports no differences.
There are more ~200 more packages installed on the bootable USB stick than the minimal desktop installation. It's well past quitting time for me here, but I'll attach the diffs of the dpkg -l outputs on each system in case you want to take a look, and I'll see if I can find a chance to look at them tomorrow as well.

Thank you again for maintaining such a wonderful tool! I'm very excited to finally use it properly!

Attachment: packages_diff.txt

[dbolan]

I was unable to reproduce this with Lubuntu 19.04, both on the bootable USB and on a proper installation.
I'll probably just use ASP on Lubuntu for now, but considering the popularity of Ubuntu as an operating system I'd say this is still an issue.

Attaching all three package lists for posterity:
packages-lubuntu-desktop.txt
packages-ubuntu-desktop.txt
packages-ubuntu-usb.txt

[oleg-alexandrov]

I believe this took you a lot of time. Sorry.

I finally built our code with ISIS 3.8, the latest. Maybe that was the problem. I went through the steps you mention, both on CentOS 7 and Ubuntu 18. Things work for me. I did not use the cubes with lronaccal applied to them, I think I was missing some calibration data and that command failed.

So here is one more attempt to straighten it out. You can try my build, at https://byss.arc.nasa.gov/oalexan1/StereoPipeline-x86_64-centos7.6.1810-2019-08-02_10-08-277997d78-dirty.tar.bz2

and maybe even with my own cubes (produced like you did) which are

https://byss.arc.nasa.gov/oalexan1/l0.cub

and

https://byss.arc.nasa.gov/oalexan1/r0.cub

and see how they compare with yours.

Note that I did not update my LRO NAC kernels for a while, but I don't think that is the problem. This is a software issue and not a data issue.

[dbolan]

Hi Oleg, thanks for giving this one more try. Unfortunately, this still doesn't work on a fresh installation. Tried with my own cubes and yours. Out of curiosity, were you able to reproduce this issue on your own machine? I found out a coworker ran into a similar issue (also on Ubuntu 18.04) some months ago, so I'm slightly more confident that it's not just an issue with my hardware or something -- though it working from the bootable USB would also make that possibility unlikely as well.

The main difference I'm seeing with your version running ISIS 3.8 is that I'm seeing Failed to convert string [1,79769313486232e+308] to a double before any other outputs. I've attached a
copy of the full output because it's tricky to describe how it's different.

As for lronaccal, if you used the web service for spiceinit you may have hit this issue: DOI-USGS/ISIS3/issues/1790

Thanks again for all your help.

[oleg-alexandrov]

No, I cannot reproduce the issue on my machine. I just tried it again with the build and cubes above on Ubuntu 18.04.2 LTS. I hope you did not change any paths and that it uses our own libraries. You can always edit the bin/stereo_pprc shell script and put there a line right before the last saying

ldd "${PROGRAM}"

to see if any libraries are picked from the wrong location. All libraries should be from our tarball except low level things like libc, libm, libX11, and just a few more.

As to that error about failing to convert a string, that is from ISIS indeed. The cub file itself has nothing so big as that number 1,79769313486232e+308, which is the largest double or close to it, the biggest value in the cub is the no-data value at -3.40282265508890445e+38.

Glad to hear at least Ubuntu 19 is working for you.

Since I can't reproduce it there is nothing I can do at this point.

[oleg-alexandrov]

Another user reported this, with HiRISE testcase ESP_042134_1985_RED.map.cub ESP_053962_1985_RED.map.cub, both on the Mac and Ubuntu 19 (referenced above). Hope to get to this when back from vacation in a few weeks and when not swamped by my other three projects at that time.

[oleg-alexandrov]

This issue came up again. This time I fetched the latest ISIS kernels and latest ISIS, and I still can't reproduce it. I ran things on Ubuntu 18.04. Adding my recipe here just for the record. It is odd that very few people run into this, but the issue occurs, and it is not clear what is going on.

Here's what I did with LRO NAC and it worked:
Ensure the data is up to date:

cd path/to/your/isis3data;

rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isis3data/data/base . --exclude '*/kernels'
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isis3data/data/lro . --exclude '*/kernels'

Install ISIS with conda, per https://github.com/USGS-Astrogeology/ISIS3#installation
Make a work directory. Go there. Do
wget https://byss.arc.nasa.gov/oalexan1/l0.cub
wget https://byss.arc.nasa.gov/oalexan1/r0.cub
wget https://byss.arc.nasa.gov/stereopipeline/daily_build/StereoPipeline-2.6.2_post-2019-12-16-x86_64-Linux.tar.bz2
Untar ASP.
export ISISROOT=/path/to/miniconda3/envs/isis3
export ISIS3DATA=path/to/your/isis3data
env | grep ISIS # to see if variables were set correctly
spiceinit from=l0.cub web = true
spiceinit from=r0.cub web = true
lronaccal from=l0.cub to=l0.cal.cub
lronaccal from=r0.cub to=r0.cal.cub
./StereoPipeline-2.6.2_post-2019-12-16-x86_64-Linux/bin/stereo l0.cal.cub r0.cal.cub out/run

[oleg-alexandrov]

I was told that regional formats for numbers, dates and currency is the problem, with the specific example of the Italian format.

I guess the ISIS folks should watch for it too, though maybe it is us who don't do something right about the locale and then ISIS gets fed bad info. To be studied at some point.

[dbolan]

Thanks for keeping up on this!
I do remember seeing this bug a while back which may a useful reference point. This particular instance was fixed in 3.9.0, but it seems very plausible that others are hanging around somewhere.
If I run into this again I'll be sure to check locale.

[rbeyer]

Riccardo Pozzobon over on OpenPlanetary helped track down the issue with this error:

Error: ERROR Unable to initialize camera model from group [Instrument].
ERROR Failed to convert string [1,79769313486232e+308] to a double.
Stereo step 0: Preprocessing failed

This may happen with using a binary version of ASP on a system with a locale setting that uses commas as the decimal separator. What happens is that you may have a newer version of ISIS that you use to prepare ISIS cubes before running stereo, and it uses your locale settings to write a number (somewhere, we still haven't tracked it down) as a string in your locale, like 1,79769313486232e+308, which is the largest value representable by a double. Then, when you run stereo, which uses a precompiled set of ISIS libraries, they try and read that string, and interprets it as 1.79769313486232e+322, which is larger than the allowable value representable by a double, and that leads to the error.

Essentially a version of ISIS that interprets written numbers one way tries to read in numbers written in a different way.

A work-around is to adjust the locale on your system, via editing your .bash_profile and/or
.bashrc (or their equivalents on your system) to set the following environment variables (again, syntax may vary with your shell):

export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8

When you do this, and then do your ISIS pre-processing on your data, your shiny new ISIS will use those locale settings to write numbers with a period as the decimal separator, and then the version of ISIS baked into ASP (by us here at Ames where we were just using our native locale), will be able to read them, and you shouldn't have this error.

Thanks Ricardo!

[oleg-alexandrov]

Now this is done for the user in ASP itself. It will be in the daily build soon and later when ASP 3.1.1 is released.