New Ctxcal Flat file Option

[acpaquette]

Description

Adds a new flat file option into ctxcal that allows users to calibrate CTX data using the newly generated monthly CTX flat field files. This removes a noticeable "Frown" across the image, where there was low saturation at the edges and gradually increasing saturation toward the center of the image (cross track).

Related Issue

#5279

How Has This Been Validated?

New test added and all old ctxcal tests pass

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Documentation change (update to the documentation; no code change)
• Breaking change (fix or feature that would cause existing functionality to change)
• Infrastructure change (changes to things like CI or the build system that do not impact users)

Checklist:

• I have read and agree to abide by the Code of Conduct
• I have read the CONTRIBUTING document.
• My change requires a change to the documentation and I have updated the documentation accordingly.
• I have added tests to cover my changes.
• I have added myself to the .zenodo.json document.
• I have added any user impacting changes to the CHANGELOG.md document.

Licensing

This project is mostly composed of free and unencumbered software released into the public domain, and we are unlikely to accept contributions that are not also released into the public domain. Somewhere near the top of each file should have these words:

This work is free and unencumbered software released into the public domain. In jurisdictions that recognize copyright laws, the author or authors of this software dedicate any and all copyright interest in the software to the public domain.

• I dedicate any and all copyright interest in this software to the public domain. I make this dedication for the benefit of the public at large and to the detriment of my heirs and successors. I intend this dedication to be an overt act of relinquishment in perpetuity of all present and future rights to this software under copyright law.

[jlaura]

What mechanism is in place to keep these updated? They are monthly and CTX is releasing additional data quarterly to the PDS, so as the files update. Does this need some sort of change to the ISIS data download script as well?

Also, is the reason to defer making these default because of this text in the API definition: "configuration or preference names and their defaults".

I am asking to understand the scope of that document as it relates to this PR.

[acpaquette]

@jlaura There shouldn't be a change to the ISIS download script but we will need to setup some other monthly task to pull new data. As for deferring till 9.0 to change the defaults it's related to the section you mentioned but explicitly that the API includes the "...argument defaults, configuration or preference names and their defaults...".

The new flag changes the functionality to use the new monthly CTX data and, if made default, would alter the output of the program for down stream users. We can make it a new feature in 8.X.0 and then the default in 9.0

[Kelvinrr]

@jlaura If we make it default and the flatfiles change, that would be API breaking. If we want to make these default, it can go into 9.0.

As far as a mechanism for updating these, assuming we are safe to replace everything in $ISISDATA/mro/calibration/ctxFlatFiles/, then we can update the rclone config file to include the flatfiles as part of the MRO union. That's assuming there exists an endpoint that rclone supports AFAIK downloading from a git repo isn't possible directly using rclone. We can download individual files with GETs (e.g. https://git.imp.fu-berlin.de/api/v4/projects/8879/repository/files/flats%2FB01%2Eflat%2Ecub/raw?ref=master), but idk about directories. If this is possible, then we can keep these up to date without any kind of complicated mechanisms. Otherwise, you would need some awkward python code to do it. In any case such a change should probably be included here.

[acpaquette]

@Kelvinrr 9.0 is not releasing for close to another year?

From our own release schedule
9.0.0 LTS Release 8.2.24

[Kelvinrr]

@Kelvinrr 9.0 is not releasing for close to another year?

From our own release schedule 9.0.0 LTS Release 8.2.24

yeah, I edited out my mistake.

[amystamile-usgs]

@acpaquette Need to fix conflicting Changelog before merging.

[Kelvinrr]

I think this can go in and we can make an issue about keeping the flatfiles up to date.

[Kelvinrr]

#5430

[jlaura]

The linked location for the flat files is a 404. Did you pull the flat file (singular, nice work by the authors) from here?

https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2023EA003491