findFeaturesSegment.py TypeError

[lwellerastro]

ISIS version(s) affected: isis8.2.1-RC1

Description
The following error is generated for images where only one segment in each overlaps (there are two existing segments for each):

Traceback (most recent call last):
  File ".../isis8.2.0-RC1/bin/findFeaturesSegment.py", line 393, in <module>
    findFeaturesSegment(ui) 
    ^^^^^^^^^^^^^^^^^^^^^^^
  File ".../isis8.2.0-RC1/bin/findFeaturesSegment.py", line 324, in findFeaturesSegment
    segment_paths = [s["Path"] for sublist in list(from_segments.values()) for s in sublist]
                     ~^^^^^^^^
TypeError: tuple indices must be integers or slices, not str

Footprints (inspection of segmented cubes in qview show that segment1 for each image overlaps)

How to reproduce
This attempted network is under the scratch area at LRONAC/FFSeg/SmallRegion/FindFeatures_I8.2.0RC1/M105960889RE_Network/ and the command for running the script is in M105960889RE_matcher_ff.cmd. All images and segments are in this same directory.

The file LOG_neb1_46967131-4.out contains the standard output with error (near the end)

Contents of scratch Network directory can also now be found under my users work area at /Isis3Tests/FFSegmentScript/I8.2.0RC1/M105960889RE_Network/. Note that commands and lists will point to scratch though.

Additional context
Discovered this when running updated script on a larger set of data than what was used in #5416. Was likely there when I tested the changes, but was focussed on the fix for that issue and missed this error. Of the 18 images tested in this set, 3 of them encounter this problem where only two images overlap and only for one segment. It's a fairly common occurrence.

Have also determined that the tuple error has always been there, I just didn't notice the error in past LOGs until this isolated case failed.

[lwellerastro]

Just discovered that although image M105960889RE network generation encountered tuple errors (described above) and did not get passed to findfeatures, the image it overlaps with, M1342772956LE, overlaps many other images/segments and because of that, M105960889RE gets tied to it via M1342772956LE_Network. (This explains why when I tested these updates all 18 images in my test set were included in a merged network of the successful runs and everything seemed ok to me at a larger scale (quick check in qnet). But there is still obviously a problem to address because if I'm just working with two images I don't get a network in the end, just the tuple error.)

In trying to understand why M105960889RE works under the M1342772956LE Network run, I found that network had some errors as well despite getting a final network in the end. The following is an error produced by the findfeatures application (not a script error):

    Message = "Input cubes (5) failed to load. Must provide valid
               FROM/FROMLIST and MATCH cube or image filenames"

The MATCH cube, the FROMLIST file and the cubes in the FROMLIST file all exist, so this is a rather confusing message from findfeatures. After further inspection I understand the problem - none of the segments in the fromlist overlap the match file being worked on. Since I normally don't pass findfeatures data that do no overlap, I was not familiar with this particular error.

So although this isn't a script error, I'm wondering why findFeaturesSegment passed that data to the findfeatures program in the first place - I thought the preceding steps of findimageoverlaps and overlapstats would prevent this from happening. Best I can tell from looking at the log file, overlapstats does record a number of segments as NoOverlap in the output stats information. But it's honestly difficult for me to tell what all specifically is being worked because the call to overlapstats gets temporary files passed to it that are removed after everything runs. I'm inferring what's going on because all NoOverlap overlapstats results precede the findfeatures failure.

See my scratch directory LRONAC/FFSeg/SmallRegion/FindFeatures_I8.2.0RC1/M1342772956LE_Network/ or my user work area /Isis3Tests/FFSegmentScript/I8.2.0RC1/M1342772956LE_Network/ for data associated with this discussion.

[lwellerastro]

I tested an updated version of the script against the case under my users work area at /Isis3Tests/FFSegmentScript/I8.2.0RC1/M105960889RE_Network/Fix5489/ but it did not work as expected. See LOG.out in that directory for what went to screen.

It appears to have run findimageoverlaps 3 times, two of which resulted in "No overlaps were found" which may be accurate, but I don't know what overlaps were being worked on. The third time it was run it seemed to find overlaps according to what was captured in the print.prt

Group = Results
ErrorCount = 0
End_Group

then it ran overlapstats on the output but nothing gets passed to findfeatures. Instead the script ends and no output is generated.
I don't see the TypeError anymore. The only errors seem to be associated with findimageoverlaps, but the following should have had successful segment overlaps passed to findfeatures:

M105960889RE.lev1.segment1.cub (match image)
M1342772956LE.lev1.segment1.cub (fromlist image)

If I manually run findimageoverlaps and overlapstats on these two segments I appear to get the same results as what goes to screen:

ls M105960889RE.lev1.segment1.cub M1342772956LE.lev1.segment1.cub > seg1.lis
findimageoverlaps fromlist=seg1.lis overlaplist=overlaps_seg1.dat
overlapstats fromlist=seg1.lis overlaplist=overlaps_seg1.dat
    ThicknessMinimum            = 0.019281398318497
    AreaMinimum                 = 33124104.551025

I can then run findfeatures on these successful segments and get a network:

findfeatures algorithm=fast/brief maxthreads=7 match=M105960889RE.lev1.segment1.cub from=M1342772956LE.lev1.segment1.cub fastgeom=true geomtype=camera maxpoints=10000 epitolerance=9.0 ratio=0.9 hmgtolerance=9.0 filter=sobel networkid=M105960889RE pointid='M105960889RE_ff_?????' onet=M105960889RE_ff.net tolist=M105960889RE_cubes_ff.lis tonotmatched=M105960889RE_notmatched_ff.lis description='Create image-image control network' debug=true debuglog=M105960889RE_ff.log

These are the expected results for the images involved when run manually.

So although we have avoided the Tuple error, the successful overlaps are not being passed to findfeatures for some reason.

I'll test the script updates on a case that was successful before the changes were made to see if things still run as expected there.

[lwellerastro]

I ran the updated script on my user work area /Isis3Tests/FFSegmentScript/I8.2.0RC1/M1342772956LE_Network/Fix5489/ and Message = "No overlaps were found", but the match image M1342772956LE overlaps all the images in the fromlist. See LOG.out in that directory for details.

After overlapstats runs and generates statistics for overlapping images, the following messages go to standard out:

INFO:root:No overlaps were found.
DEBUG:root:From images overlapping Match: []
INFO:root:No overlaps were found.
DEBUG:root:From images overlapping Match: []
DEBUG:root:generate_cnet error

But there are 100% successful overlaps. The overlapping data never get passed to findimageoverlaps.

The match image being worked on is the blue footprint that is diagonal across the other image footprints.

Please don't allow this recent version of the script to go into the next version of ISIS since it doesn't seem to be working on any of the data any longer.

[Kelvinrr]

@lwellerastro

The PR was updated with fixes and performance improvements. Let me know if that one works for you.

[lwellerastro]

1. Specifically for /Isis3Tests/FFSegmentScript/I8.2.0RC1/M105960889RE_Network/ and the two displayed footprints in OP

Tuple error is gone and the two images that overlap now get points. Additionally, findfeatures is only called one time since only segment 1 for each image overlaps. Updates addressed via #5540 are also being handled properly. Yay!

2. Specifically for /Isis3Tests/FFSegmentScript/I8.2.0RC1/M1342772956LE_Network/ and the more complicated set of overlaps

    Message = "Input cubes (5) failed to load. Must provide valid
               FROM/FROMLIST and MATCH cube or image filenames"

This error still exists even though a valid final network is generated.

Thanks to the new WORKDIR parameter, I can see that segments that do not overlap the match image segment being evaluated are being passed into the findfeatures fromlist even though shouldn't be. This is odd since since in the case of M105960889RE_Network/ that issue is being handled properly.

Point 2 needs more investigation and a new post if necessary.

I think this post can ultimately be closed since the updates are addressing the original issue and others directly related to it.