removing azimuth burst ramps caused by ionosphere in topsStack

[CunrenLiang]

These updates are for removing azimuth burst ramps caused by ionosphere in the ionospheric correction in topsStack. The updates are only within this module.

[bjmarfito]

I would like to ask if there is a timeline in merging this PR. Thank you!

[yuankailiu]

I have tested this PR recently with a stack and did not encounter issues. Great contribution, thank you @CunrenLiang ! Sorry that it took me so long to run this PR on my datasets.

Looks like I need to find a Chilean region with even stronger burst ramp jumps to demonstrate this. But this is what I have now, and it is working. Perhaps we can merge it @hfattahi , @yunjunz unless more tests are needed?

The outputs were stored in the following directories.

• The ionosphere azimuth shift estimated in each interferogram:
  ion/date1_date2/ion_cal/azshift.ion

• The inverted time-series azimuth shift for each acquisition date:
  ion_azshift_dates

• The intra-burst ramps time series for all the bursts from 3 subswaths:
  ion_burst_ramp_dates

• The final merged burst ramp for each acquisition date:
  ion_burst_ramp_merged_dates/*.float

[yunjunz]

Looks good to me overall. Thank you @CunrenLiang for the PR and @yuankailiu for the testing!

I added a few suggestions below, trying to resolve the conflicts since stackSentinel.py has been updated (in #505, #702) since this PR. Or maybe you could do a git rebase to resolve yourself @CunrenLiang?

[chintals]

I am testing this PR to check if it would help with scalloping issue in my interferograms (workflow is correlation stack). In stackSentinel the function correlationStack is missing creation of generate_burst_igram run-file, and for merge_burst_igram run-file, its calling burstIgram_mergeBurst function instead of igram_mergeBurst

[yuankailiu]

I am testing this PR to check if it would help with scalloping issue in my interferograms (workflow is correlation stack). In stackSentinel the function correlationStack is missing creation of generate_burst_igram run-file, and for merge_burst_igram run-file, its calling burstIgram_mergeBurst function instead of igram_mergeBurst

Hi @chintals, as per my understanding, the -W correlation workflow is used to only generate a coherence stack of your network. So I think it will not create regular interferograms and ionosphere interferograms.

I think this PR is intended for -W interferogram workflow.

[chintals]

I am testing this PR to check if it would help with scalloping issue in my interferograms (workflow is correlation stack). In stackSentinel the function correlationStack is missing creation of generate_burst_igram run-file, and for merge_burst_igram run-file, its calling burstIgram_mergeBurst function instead of igram_mergeBurst

As per my understanding, the -W correlation workflow is used to only generate a coherence stack of your network. So I think it will not create regular interferograms and ionosphere interferograms.

I think this PR is intended for -W interferogram workflow.

Hi @yuankailiu, I apologize if my understanding is incorrect, but as far as I know, interferogram workflow creates interferogram and also performs SNAPHU or icu unwrapping. Correlation workflow only creates wrapped phase (filtered and unfiltered) and associated correlation file. And the ionospheric phase estimation creates lower and upper interferograms regardless of workflow, but for this PR to get more precise azshifts, coregistration should ideally be NESD.
The current ISCE2 version 2.6.3 stackSentinel creates runfile to generate_burst_Igram followed by runfile to merge_burst_Igram (for correlation workflow). This PR version is missing generate_Igram runfile, but creates merge_burst_Igram runfile, so how would you merge burst interferograms if you dont generate them?

[yuankailiu]

Hi, sorry I need to take back my words, and you are right, it should create those interferograms run files regardless of workflow.

I think what you mentioned has been fixed in #635 and committed. So, version 2.6.3 does not have this issue in correlationStack.

As @yunjunz said, once this PR is rebased on the current 2.6.3 version, it should be okay. I did it on my branch, but I don't have access to add my commit here I think.

After I do the rebase, it looks like the -W correlation workflow gets the run files properly. Is this what you would expect?

run_01_unpack_topo_reference
run_02_unpack_secondary_slc
run_03_average_baseline
run_04_extract_burst_overlaps
run_05_overlap_geo2rdr
run_06_overlap_resample
run_07_pairs_misreg
run_08_timeseries_misreg
run_09_fullBurst_geo2rdr
run_10_fullBurst_resample
run_11_extract_stack_valid_region
run_12_merge_reference_secondary_slc
run_13_generate_burst_igram
run_14_merge_burst_igram
run_15_filter_coherence
run_16_subband_and_resamp
run_17_generateIgram_ion
run_18_mergeBurstsIon
run_19_unwrap_ion
run_20_look_ion
run_21_computeIon
run_22_filtIon
run_23_invertIon
run_24_filtIonShift
run_25_invertIonShift
run_26_burstRampIon
run_27_mergeBurstRampIon

[chintals]

Hi, sorry I need to take back my words, and you are right, it should create those interferograms run files regardless of workflow.

I think what you mentioned has been fixed in #635 and committed. So, version 2.6.3 does not have this issue in correlationStack.

As @yunjunz said, once this PR is rebased on the current 2.6.3 version, it should be okay. I did it on my branch, but I don't have access to add my commit here I think.

After I do the rebase, it looks like the -W correlation workflow gets the run files properly. Is this what you would expect?

run_01_unpack_topo_reference
run_02_unpack_secondary_slc
run_03_average_baseline
run_04_extract_burst_overlaps
run_05_overlap_geo2rdr
run_06_overlap_resample
run_07_pairs_misreg
run_08_timeseries_misreg
run_09_fullBurst_geo2rdr
run_10_fullBurst_resample
run_11_extract_stack_valid_region
run_12_merge_reference_secondary_slc
run_13_generate_burst_igram
run_14_merge_burst_igram
run_15_filter_coherence
run_16_subband_and_resamp
run_17_generateIgram_ion
run_18_mergeBurstsIon
run_19_unwrap_ion
run_20_look_ion
run_21_computeIon
run_22_filtIon
run_23_invertIon
run_24_filtIonShift
run_25_invertIonShift
run_26_burstRampIon
run_27_mergeBurstRampIon

Hi @yuankailiu Thank you! Yes, this is what I expect and am running at the moment. I manually corrected the file on my end and figured I should let folks know as well.

[bjmarfito]

Hi @CunrenLiang and @rtburns-jpl, I would like to ask if this PR can be merged with the main branch.

[bjmarfito]

Any news when this PR will be incorporated in ISCE2 since it also has some changes on ALOS-2 data processing.

[bjmarfito]

Hi @rtburns-jpl and @hfattahi, can this PR be merged since it contains important tools for estimating and correcting ionosphere in Sentinel-1 IFG stack? It also contains fixes in processing ALOS-2 data.

[hfattahi]

Hi @rtburns-jpl and @hfattahi, can this PR be merged since it contains important tools for estimating and correcting ionosphere in Sentinel-1 IFG stack? It also contains fixes in processing ALOS-2 data.

Thanks @bjmarfito for your patience. @rtburns-jpl let's make progress with this PR and few other open.
PRs. @CunrenLiang did you see @yunjunz 's comments? Would you please resolve them by addressing the comments or responding to them?

[CunrenLiang]

Hi @rtburns-jpl and @hfattahi, can this PR be merged since it contains important tools for estimating and correcting ionosphere in Sentinel-1 IFG stack? It also contains fixes in processing ALOS-2 data.

Thanks @bjmarfito for your patience. @rtburns-jpl let's make progress with this PR and few other open. PRs. @CunrenLiang did you see @yunjunz 's comments? Would you please resolve them by addressing the comments or responding to them?

Thanks @yunjunz for helping with revolving the conflicts. They look fine to me. I revolved another conflict. The pull request is ready to merge. I think now it's time to do it in case there are further conflicts in the future. What do you think @hfattahi ?

[yunjunz]

Looks all good to me to be merged!