MARCICAL Might Not be Calibrating Properly

[astrostu]

ISIS version(s) affected: 4.x+

Description
Regular pattern noise which should be removed if flats are properly applied is still present in MARCI images after running MARCICAL.

How to reproduce
After running MARCI2ISIS, then SPICEINIT, I run MARCICAL which runs without any announced errors. However, examination of the output shows that some pattern noise which is present in the flat, so should be removed if the flat is properly applied, is still in the post-MARCICAL images. This is most apparent in the UV images since those have very low signal-to-noise (S/N) so any detector issues are more readily apparent, but it also appears in lower S/N visible bands like the first (violet), especially when spatial summing = 2.

Possible Solution
Check under-the-hood to see if it's actually doing what it should be.

Additional context
This is related to Issue #3690 where new flats were put into the ISIS data directories in February 2020, and to Issue #3550 where the last comment raised the question of whether MARCICAL is actually doing what it should be doing; it doesn't look like there was a follow-up to that inquiry.

Per discussion on the astrodiscuss forum, @dpmayerUSGS noted: To summarize the state of the flatfiles in the ISIS data area:

• The most recent flats in the ISIS data area are derived from the most recent flats in the PDS, which MSSS has confirmed are canonical
• It is unknown whether the process to derive the flats in the ISIS data area is correct
• It is unknown whether there are issues besides the flats that make the output of marcical incorrect relative to the official algorithm description

[astrostu]

I've spent the evening creating a custom post-flat flat code to give you an idea of the residuals. While there are 12 different flats needed, I'm just attaching two: the UV-1 band and the violet band. Please note that that some of the signal here towards the edges is from Mars' limb, so ignore that curved brightening on both sides, and some of it is from the photometric effects of the wide viewing geometry; but, otherwise you can see the pattern noise of which I speak.

[github-actions]

Thank you for your contribution!

Unfortunately, this issue hasn't received much attention lately, so it is labeled as 'stale.'

If no additional action is taken, this issue will be automatically closed in 180 days.

[astrostu]

Might be stale, but needs to be checked.

[ladoramkershner]

Hey @astrostu, I have a few questions for my context! First, is 'violet band' synonymous to the UV-2 band?

Second, could you expand on what the "custom post-flat flat code" is? For example, is the output of this custom code calibrated images or a version of the flat files the account for the hot pixels that seem to show up in the UV images, but not the calibration files. Additionally what are the "residuals" you mention?

[astrostu]

Hi @ladoramkershner—

(1) By "violet" I mean the shortest-wavelength of the VIS images.

(2) After applying the MARCICAL code, so ostensibly removing optical issues by applying the built-in flats, what I did was design a script in GDAL to stack all the framelets in multiple images (per filter, per spatial summing mode) together. So, if I have 10 VIS images that are all ≈100 MB .IMG files (so all spatial-summing = 1), I would build five flats, which is one per color in that VIS cube. For all of the red/near-IR images (odd/even x10 images), for each framelet in them, I would stack them together and then take the median of that gigantic stack. (I'd then do stuff with that stack which is unimportant to your questions at the moment.) In the end, there would be five medians-of-stacks, one for each filter.

(3) If MARCICAL were applying correct flats correctly (so, two things are required there, then there should not be what appears to be residual sensor noise: There should be zero pattern remaining. I think that's what I meant above by "residuals."

So, it could be the flats are bad. It could be MARCICAL is not applying them properly.

Another fun possibility is that the optical properties change as MARCI scans, so that the flats could be correct for part of the scan, but not other parts. I came to this idea after talking with some people on the New Horizons LEISA team who said they had this problem, where some light bounced around the detector and illuminated some schmutz in different ways that affected how light fell on the detector, effectively making the flat field variable across a scan, which is just a "delightful" situation.

That said, I did point to two other Issues that were brought up here where folks have questioned whether MARCICAL is doing its stuff correctly, so it should be checked.

[ladoramkershner]

@astrostu Definitely agree with the checking! there have been a couple issues asking for that inspection and I am plenty happy to jump in a rabbit hole (shoot they can hardly keep me out of them). I have spent the past few days looking through the whole line of MARCI related functions / data (the flat file generation, ddd2isis, marci2isis, and marcical)

I did find one small update we needed to be fully congruent with the calibration technique outlined by the missions team (PR incoming). I also found that our calibration cubes were not being divided by the values outlined in the calibration description, so I am working on updating those as well.

I have tested the updated code on a UV image and I am still seeing some hot pixels / noise, which is disappointing. Additionally, since the flat file updates are a constant division I am not hopeful it will solve the issue we are seeing here with the noise.

I did want to point something out and get your opinion. So below is an example of what I would say is typical of what I am seeing of framlets in UV images. There are a few distinctly hot pixels and general noise.

However, the uv6flat file associated with this band looks like this

So, I am wondering if we should even expect the calibration to take care of these hot pixels (which I am interpreting these hot pixels as the issue here, I suppose)

[astrostu]

@ladoramkershner— Hmmm. I know I took a look at them when I was working on this, but I don't recall them being THAT far off from the noise I was finding. It's possible I'm thinking of the VIS flats, though. So, this is where, at least in this case and at least with the UV, it might be that the flats from the team – despite these being "v3" flats updated in 2020 – are not good.

I am glad to see that you've found some calibrations aren't being done correctly. Not glad that there's a bug, but glad that your deep dive into the MARCI-related programs is yielding something for your time. I've also never heard of the ddd2isis program.

[astrostu]

P.S. Something that is probably worth looking into, while you have the MARCI code open, is the whole variable exposure thing. That's a constant warning that gets generated when bringing MARCI images into ISIS, which for a n00b can be quite disconcerting. I'll also note there's an issue with that with the MOCCAL program, but that's probably a separate issue I should open.

[ladoramkershner]

@astrostu

So, this is where, at least in this case and at least with the UV, it might be that the flats from the team – despite these being "v3" flats updated in 2020 – are not good.

I think I agree, if I specify that "are not good" means that they are not accounting for these hot pixels we are seeing. And for some added context, the most recent version of the flatfiles were generated and released in the mrom_33 folder (they are released in every mrom, but 33 was the last update to the content of these files). I went and grabbed data from just before this time (mrom_30) and found that the most prominent hot pixel in the previous example is present in this data as well.

It is less intense, but the flatfiles still do not reflect that pixel being treated drastically different from the other pixels. What this tells me is that it is likely not the function of the flatfiles to remove these hot pixels.

I did create a PR to update the calibration and I have updated the flatfiles, this should hopefully be pushed out in the coming release.

[ladoramkershner]

P.S. Something that is probably worth looking into, while you have the MARCI code open, is the whole variable exposure thing. That's a constant warning that gets generated when bringing MARCI images into ISIS, which for a n00b can be quite disconcerting. I'll also note there's an issue with that with the MOCCAL program, but that's probably a separate issue I should open.

I think both of those are separate tickets. I could make the marci2isis ticket as I experienced a similar amount of "oh no" when I ingested some old cubes.

[astrostu]

P.S. Something that is probably worth looking into, while you have the MARCI code open, is the whole variable exposure thing. That's a constant warning that gets generated when bringing MARCI images into ISIS, which for a n00b can be quite disconcerting. I'll also note there's an issue with that with the MOCCAL program, but that's probably a separate issue I should open.

I think both of those are separate tickets. I could make the marci2isis ticket as I experienced a similar amount of "oh no" when I ingested some old cubes.

I'll do something with MOCCAL later, trying to get a paper out first on the data that I could process. If you could do the marci2isis one, that'd be great!

[astrostu]

And responding to the other point about the flats, that is really weird (to me) that they would not try to put hot pixels into the flats. That's the only way to remove them unless they have it in the calibration elsewhere, but it sounds like they don't.

[jessemapel]

I remember some calibration programs having separate hot pixel correction that happens outside of flatfiles. I agree though, flat correction seems like the appropriate place to handle this.

[ladoramkershner]

For the sake of completeness / closing the loop, the marci2isis warning issue is here #5012

[ladoramkershner]

Between the PR merge, flatfile updates, and our discussion surrounding the hot pixels is this ticket resolved? Don't want to close prematurely, but don't want to leave it hanging.

[astrostu]

Yes, I think it's resolved for now. I suppose someone contacting the MARCI team to ask why flats are the way they are might be in order, but that's a separate issue since it seems like ISIS is applying correctly what's supplied.