Github integration, with 1.4 release state

* Github integration from Bitbucket * Github workflow * Version 1.4 state from the bitbucket
cgi-estonia-space · Jan 4, 2023 · 5d938e1 · 5d938e1
1 parent ee1d7bb
commit 5d938e1
Show file tree

Hide file tree

Showing 94 changed files with 5,409 additions and 255 deletions.
diff --git a/.alus-lfs b/.alus-lfs
@@ -7,6 +7,8 @@ set -e
 script_directory=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 resources_file="$script_directory/.alusresources"
 
+mkdir -p "$script_directory/resources"
+
 if [ "$1" == "--clean" ]; then
 
     echo "cleaning"

diff --git a/.alusresources b/.alusresources
@@ -1,12 +1,12 @@
-https://alus-goods.s3.eu-central-1.amazonaws.com/goods_resources/srtm_41_01.zip resources/srtm_41_01.zip
-https://alus-goods.s3.eu-central-1.amazonaws.com/goods_resources/srtm_42_01.zip resources/srtm_42_01.zip
-https://alus-goods.s3.eu-central-1.amazonaws.com/goods_resources/srtm_43_06.zip resources/srtm_43_06.zip
-https://alus-goods.s3.eu-central-1.amazonaws.com/goods_resources/srtm_44_06.zip resources/srtm_44_06.zip
-https://alus-goods.s3.eu-central-1.amazonaws.com/validation/land_destruction_at_beirut/S1A_IW_SLC__1SDV_20200805T034334_20200805T034401_033766_03E9F9_52F6.SAFE.zip     resources/S1A_IW_SLC__1SDV_20200805T034334_20200805T034401_033766_03E9F9_52F6.zip
-https://alus-goods.s3.eu-central-1.amazonaws.com/validation/land_destruction_at_beirut/S1B_IW_SLC__1SDV_20200730T034254_20200730T034321_022695_02B131_E8DD.SAFE.zip     resources/S1B_IW_SLC__1SDV_20200730T034254_20200730T034321_022695_02B131_E8DD.zip
-https://alus-goods.s3.eu-central-1.amazonaws.com/goods_resources/4_bands.tif    resources/4_bands.tif
-https://alus-goods.s3.eu-central-1.amazonaws.com/goods_resources/Beirut_IW1_6_VH_orb_stack_cor_deb_coh_data.zip resources/Beirut_IW1_6_VH_orb_stack_cor_deb_coh_data.zip
-https://alus-goods.s3.eu-central-1.amazonaws.com/goods_resources/S1A_IW_SLC__1SDV_20190715T160437_20190715T160504_028130_032D5B_58D6_Orb_Stack_coh_deb_data.zip resources/S1A_IW_SLC__1SDV_20190715T160437_20190715T160504_028130_032D5B_58D6_Orb_Stack_coh_deb_data.zip
-https://alus-goods.s3.eu-central-1.amazonaws.com/goods_resources/apply_orbit_file_op.zip        resources/apply_orbit_file_op.zip
-https://alus-goods.s3.eu-central-1.amazonaws.com/goods_resources/topsar_deburst_op.zip  resources/topsar_deburst_op.zip
-https://alus-goods.s3.eu-central-1.amazonaws.com/goods_resources/topsar_merge_test_data.zip  resources/topsar_merge_test_data.zip
+https://alus-goods-set.s3.eu-central-1.amazonaws.com/goods_resources/srtm_41_01.zip resources/srtm_41_01.zip
+https://alus-goods-set.s3.eu-central-1.amazonaws.com/goods_resources/srtm_42_01.zip resources/srtm_42_01.zip
+https://alus-goods-set.s3.eu-central-1.amazonaws.com/goods_resources/srtm_43_06.zip resources/srtm_43_06.zip
+https://alus-goods-set.s3.eu-central-1.amazonaws.com/goods_resources/srtm_44_06.zip resources/srtm_44_06.zip
+https://alus-goods-set.s3.eu-central-1.amazonaws.com/validation/land_destruction_at_beirut/S1A_IW_SLC__1SDV_20200805T034334_20200805T034401_033766_03E9F9_52F6.SAFE.zip     resources/S1A_IW_SLC__1SDV_20200805T034334_20200805T034401_033766_03E9F9_52F6.zip
+https://alus-goods-set.s3.eu-central-1.amazonaws.com/validation/land_destruction_at_beirut/S1B_IW_SLC__1SDV_20200730T034254_20200730T034321_022695_02B131_E8DD.SAFE.zip     resources/S1B_IW_SLC__1SDV_20200730T034254_20200730T034321_022695_02B131_E8DD.zip
+https://alus-goods-set.s3.eu-central-1.amazonaws.com/goods_resources/4_bands.tif    resources/4_bands.tif
+https://alus-goods-set.s3.eu-central-1.amazonaws.com/goods_resources/Beirut_IW1_6_VH_orb_stack_cor_deb_coh_data.zip resources/Beirut_IW1_6_VH_orb_stack_cor_deb_coh_data.zip
+https://alus-goods-set.s3.eu-central-1.amazonaws.com/goods_resources/S1A_IW_SLC__1SDV_20190715T160437_20190715T160504_028130_032D5B_58D6_Orb_Stack_coh_deb_data.zip resources/S1A_IW_SLC__1SDV_20190715T160437_20190715T160504_028130_032D5B_58D6_Orb_Stack_coh_deb_data.zip
+https://alus-goods-set.s3.eu-central-1.amazonaws.com/goods_resources/apply_orbit_file_op.zip        resources/apply_orbit_file_op.zip
+https://alus-goods-set.s3.eu-central-1.amazonaws.com/goods_resources/topsar_deburst_op.zip  resources/topsar_deburst_op.zip
+https://alus-goods-set.s3.eu-central-1.amazonaws.com/goods_resources/topsar_merge_test_data.zip  resources/topsar_merge_test_data.zip
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -0,0 +1,142 @@
+name: CI on AWS GPU instance
+
+on:
+  pull_request:
+    branches: [ "development" ]
+    max-concurrency: 1
+  # Allows to run this workflow manually from the Actions tab
+  workflow_dispatch:
+    max-concurrency: 1
+
+env:
+  CI_PACKAGE_NAME: alus_github_action_ci_package_${{github.run_number}}
+
+# For 'trigger_instance' - tty=$(readlink /proc/$$/fd/2) - possibly needed in the future, add $tty to tee command.
+jobs:
+  package-repo:
+    name: Package the repo
+    runs-on: ubuntu-latest
+    outputs:
+      package_filename: ${{ steps.prepare-and-archive.outputs.package_filename }}
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          submodules: 'recursive'
+      - name: Prepare and archive
+        id: prepare-and-archive
+        env:
+          CI_PACKAGE_FILENAME: ${{ env.CI_PACKAGE_NAME }}.tar.gz
+          CI_PACKAGE_FILENAME_PATH: /tmp/${{ env.CI_PACKAGE_NAME }}.tar.gz
+        run: |
+          $GITHUB_WORKSPACE/build-automation/create_ci_package.sh $GITHUB_WORKSPACE $CI_PACKAGE_FILENAME_PATH
+          echo "package_path=$CI_PACKAGE_FILENAME_PATH" >> $GITHUB_ENV
+          echo "package_filename=$CI_PACKAGE_FILENAME" >> $GITHUB_OUTPUT
+      - uses: actions/upload-artifact@v2
+        with:
+          name: ${{ env.CI_PACKAGE_NAME }}
+          path: ${{ env.package_path }}
+          retention-days: 3
+
+  start-gpu-instance:
+    name: Start GPU instance
+    runs-on: ubuntu-latest
+    outputs:
+      instance_ip: ${{ steps.trigger_instance.outputs.instance_ip }}
+    steps:
+    - name: Configure AWS credentials
+      uses: aws-actions/configure-aws-credentials@v1
+      with:
+        aws-access-key-id: ${{ secrets.SID_ACCESS_KEY_ID }}
+        aws-secret-access-key: ${{ secrets.SID_SECRET_ACCESS_KEY }}
+        aws-region: ${{ secrets.ALUS_SID_INSTANCE_REGION }}
+    - uses: actions/checkout@v3
+    - name: Trigger the instance
+      id: trigger_instance
+      run: |
+        set -o pipefail
+        $GITHUB_WORKSPACE/build-automation/trigger_ec2_instance.sh ${{ secrets.ALUS_SID_INSTANCE_ID }} | tee trigger_out.txt
+        ip_address=$(cat trigger_out.txt | tail -n 1)
+        echo "instance_ip=$ip_address" >> $GITHUB_OUTPUT
+
+  run-ci-process:
+    name: Build and run tests
+    permissions: write-all
+    runs-on: ubuntu-latest
+    needs: [start-gpu-instance, package-repo]
+    steps:
+    - name: Create SSH key
+      env:
+        SSH_PRIVATE_KEY: ${{ secrets.ALUS_CI_ACTIONS_SSH_KEY }}
+      run: |
+        key_location=~/.ssh/
+        mkdir -p $key_location
+        key_path=$key_location/id_rsa
+        echo "$SSH_PRIVATE_KEY" > $key_path
+        sudo chmod 600 $key_path
+        ssh-keygen -f $key_path -y > $key_path.pub
+    - uses: actions/download-artifact@v2
+      with:
+        name: ${{ env.CI_PACKAGE_NAME }}
+    - name: Transfer package
+      id: transfer_package
+      env:
+        INSTANCE_USERNAME: ${{ secrets.ALUS_CI_ACTIONS_USERNAME }}
+      run: |
+        instance_ip=${{ needs.start-gpu-instance.outputs.instance_ip }}
+        package_filename=${{ needs.package-repo.outputs.package_filename }}
+        instance_package_path=/tmp/$package_filename
+        scp -o "StrictHostKeyChecking no" $package_filename $INSTANCE_USERNAME@$instance_ip:$instance_package_path
+        echo "instance_package_path=$instance_package_path" >> $GITHUB_OUTPUT
+    - uses: actions/checkout@v3
+    - name: CI process
+      id: ci_process
+      env:
+        INSTANCE_USERNAME: ${{ secrets.ALUS_CI_ACTIONS_USERNAME }}
+      run: |
+        instance_ip=${{ needs.start-gpu-instance.outputs.instance_ip }}
+        echo "Instance IP: $instance_ip"
+        package_path=${{ steps.transfer_package.outputs.instance_package_path }}
+        artifacts_test_results_location="/tmp/${{github.run_number}}"
+        ssh -o "StrictHostKeyChecking no" $INSTANCE_USERNAME@$instance_ip "mkdir $artifacts_test_results_location"  
+        scp -o "StrictHostKeyChecking no" build-automation/build_and_run_in_container.sh $INSTANCE_USERNAME@$instance_ip:$artifacts_test_results_location/
+        set +e
+        ssh -o "StrictHostKeyChecking no" $INSTANCE_USERNAME@$instance_ip "cd $artifacts_test_results_location; ./build_and_run_in_container.sh $package_path $artifacts_test_results_location /home/$INSTANCE_USERNAME/alus/resources"
+        run_result=$?
+        set -e
+        set -x
+        results_location_on_runner=$artifacts_test_results_location
+        mkdir -p $results_location_on_runner
+        scp -o "StrictHostKeyChecking no" -r $INSTANCE_USERNAME@$instance_ip:$artifacts_test_results_location/alus\*/\* $results_location_on_runner/
+        echo "results_location=$results_location_on_runner" >> $GITHUB_OUTPUT
+        exit $run_result
+    - name: Test report generation
+      uses: dorny/test-reporter@v1
+      if: success() || failure()
+      with:
+        name: Test report
+        path: '${{ steps.ci_process.outputs.results_location }}/*.xml,${{ steps.ci_process.outputs.results_location }}/clang/*.xml'
+        reporter: java-junit
+
+  stop-gpu-instance:
+    name: Stop GPU instance
+    runs-on: ubuntu-latest
+    needs: [start-gpu-instance, run-ci-process]
+    if: |
+      always() &&
+      needs.start-gpu-instance.result == 'success'
+    steps:
+    - name: Configure AWS Credentials
+      uses: aws-actions/configure-aws-credentials@v1
+      with:
+        aws-access-key-id: ${{ secrets.SID_ACCESS_KEY_ID }}
+        aws-secret-access-key: ${{ secrets.SID_SECRET_ACCESS_KEY }}
+        aws-region: ${{ secrets.ALUS_SID_INSTANCE_REGION }}
+    - uses: actions/checkout@v3
+    - name: Stop GPU instance
+      run: |
+        $GITHUB_WORKSPACE/build-automation/stop_ec2_instance.sh ${{ secrets.ALUS_SID_INSTANCE_ID }} 
+
+
+
+
+
diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml
@@ -0,0 +1,134 @@
+name: End-to-end tests on AWS GPU instance
+
+on:
+  pull_request:
+    branches: [ "development" ]
+    max-concurrency: 1
+  # Allows to run this workflow manually from the Actions tab
+  workflow_dispatch:
+    max-concurrency: 1
+
+jobs:
+  release-package:
+    name: Build package
+    runs-on: ubuntu-20.04
+    outputs:
+      package_filename: ${{ steps.build_and_pack.outputs.package_filename }}
+      package_name: ${{ steps.build_and_pack.outputs.package_name }}
+    steps:
+      - name: Prepare current runner
+        run: |
+          sudo apt update
+          sudo apt install wget
+          wget https://raw.githubusercontent.com/cgi-estonia-space/ALUs-platform/main/ubuntu/20_04/focal/install_cuda_base.sh
+          wget https://raw.githubusercontent.com/cgi-estonia-space/ALUs-platform/main/ubuntu/20_04/focal/setup_runtime.sh
+          wget https://raw.githubusercontent.com/cgi-estonia-space/ALUs-platform/main/ubuntu/20_04/focal/setup_dev.sh
+          chmod +x install_cuda_base.sh setup_runtime.sh setup_dev.sh
+          DEBIAN_FRONTEND=noninteractive sudo ./install_cuda_base.sh
+          echo "/usr/local/cuda-11.2/bin/" >> $GITHUB_PATH
+          sudo ./setup_runtime.sh
+          sudo ./setup_dev.sh
+      - uses: actions/checkout@v3
+        with:
+          submodules: 'recursive'
+      - name: Build and pack
+        id: build_and_pack
+        run: |
+          ver_major=$(grep VERSION_MAJOR VERSION | cut -d' ' -f3)
+          ver_minor=$(grep VERSION_MINOR VERSION | cut -d' ' -f3)
+          ver_patch=$(grep VERSION_PATCH VERSION | cut -d' ' -f3)
+          release_package_name="alus-$ver_major-$ver_minor-$ver_patch"
+          release_package_filename="$release_package_name.tar.gz"
+          release_package_filename_path="/tmp/$release_package_filename"
+          $GITHUB_WORKSPACE/build-automation/create_release_package.sh $GITHUB_WORKSPACE $release_package_filename_path
+          echo "package_path=$release_package_filename_path" >> $GITHUB_ENV
+          echo "package_name=$release_package_name" >> $GITHUB_ENV
+          echo "package_filename=$release_package_filename" >> $GITHUB_OUTPUT
+          echo "package_name=$release_package_name" >> $GITHUB_OUTPUT
+      - uses: actions/upload-artifact@v2
+        with:
+          name: ${{ env.package_name }}
+          path: ${{ env.package_path }}
+          retention-days: 3
+
+  start-gpu-instance:
+    name: Start GPU instance
+    runs-on: ubuntu-latest
+    needs: release-package
+    outputs:
+      instance_ip: ${{ steps.trigger_instance.outputs.instance_ip }}
+    steps:
+      - name: Configure AWS credentials
+        uses: aws-actions/configure-aws-credentials@v1
+        with:
+          aws-access-key-id: ${{ secrets.SID_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.SID_SECRET_ACCESS_KEY }}
+          aws-region: ${{ secrets.ALUS_ALEX_INSTANCE_REGION }}
+      - uses: actions/checkout@v3
+      - name: Trigger the instance
+        id: trigger_instance
+        run: |
+          set -o pipefail
+          $GITHUB_WORKSPACE/build-automation/trigger_ec2_instance.sh ${{ secrets.ALUS_ALEX_INSTANCE_ID }} | tee trigger_out.txt
+          ip_address=$(cat trigger_out.txt | tail -n 1)
+          echo "instance_ip=$ip_address" >> $GITHUB_OUTPUT
+
+  end-to-end-testing:
+    name: End-to-end testing
+    runs-on: ubuntu-latest
+    needs: [start-gpu-instance, release-package]
+    steps:
+      - name: Create SSH key
+        env:
+          SSH_PRIVATE_KEY: ${{ secrets.ALUS_ALEX_SSH_KEY }}
+        run: |
+          key_location=~/.ssh/
+          mkdir -p $key_location
+          key_path=$key_location/id_rsa
+          echo "$SSH_PRIVATE_KEY" > $key_path
+          sudo chmod 600 $key_path
+          ssh-keygen -f $key_path -y > $key_path.pub
+      - uses: actions/download-artifact@v2
+        with:
+          name: ${{ needs.release-package.outputs.package_name }}
+      - name: Transfer package
+        id: transfer_package
+        env:
+          INSTANCE_USERNAME: ${{ secrets.ALUS_ALEX_USERNAME }}
+        run: |
+          instance_ip=${{ needs.start-gpu-instance.outputs.instance_ip }}
+          package_filename=${{ needs.release-package.outputs.package_filename }}
+          instance_package_dir=/tmp/${{ needs.release-package.outputs.package_name }} 
+          ssh -oStrictHostKeyChecking=no $INSTANCE_USERNAME@$instance_ip "rm -rf $instance_package_dir"
+          ssh -o "StrictHostKeyChecking no" $INSTANCE_USERNAME@$instance_ip "mkdir $instance_package_dir"
+          scp -o "StrictHostKeyChecking no" $package_filename $INSTANCE_USERNAME@$instance_ip:$instance_package_dir/
+          echo "instance_package_dir=$instance_package_dir" >> $GITHUB_OUTPUT
+      - uses: actions/checkout@v3
+      - name: End-to-end tests
+        env:
+          INSTANCE_USERNAME: ${{ secrets.ALUS_ALEX_USERNAME }}
+        run: |
+          instance_ip=${{ needs.start-gpu-instance.outputs.instance_ip }}
+          package_dir=${{ steps.transfer_package.outputs.instance_package_dir }}
+          build_id=${{ needs.release-package.outputs.package_name }}
+          scp -o "StrictHostKeyChecking no" build-automation/*.py build-automation/*.sh build-automation/*.txt $INSTANCE_USERNAME@$instance_ip:$package_dir/
+          ssh -oStrictHostKeyChecking=no $INSTANCE_USERNAME@$instance_ip "export NIGHTLY_GOLDEN_DIR=/home/$INSTANCE_USERNAME/e2e/golden; export LD_LIBRARY_PATH=/usr/local/cuda-11.2/lib64/:\$LD_LIBRARY_PATH; cd $package_dir; ./execute_e2e.sh $build_id e2e_resources.txt"
+
+  stop-gpu-instance:
+    name: Stop GPU instance
+    runs-on: ubuntu-latest
+    needs: [start-gpu-instance, end-to-end-testing]
+    if: |
+      always() &&
+      needs.start-gpu-instance.result == 'success'
+    steps:
+      - name: Configure AWS Credentials
+        uses: aws-actions/configure-aws-credentials@v1
+        with:
+          aws-access-key-id: ${{ secrets.SID_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.SID_SECRET_ACCESS_KEY }}
+          aws-region: ${{ secrets.ALUS_ALEX_INSTANCE_REGION }}
+      - uses: actions/checkout@v3
+      - name: Stop GPU instance
+        run: |
+          $GITHUB_WORKSPACE/build-automation/stop_ec2_instance.sh ${{ secrets.ALUS_ALEX_INSTANCE_ID }}
diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -2,7 +2,7 @@ cmake_minimum_required(VERSION 3.20 FATAL_ERROR)
 project(ALUs
         LANGUAGES CXX CUDA
         DESCRIPTION "Parallel ALUs for geospatial data processing"
-        VERSION 1.2.0
+        VERSION 1.4.0
         )
 
 set(CMAKE_CUDA_STANDARD 17)
@@ -105,6 +105,7 @@ add_subdirectory(algs/calibration-routine)
 add_subdirectory(algs/resample)
 add_subdirectory(algs/topsar-merge)
 add_subdirectory(algs/thermal-noise-removal)
+add_subdirectory(algs/palsar-focus)
 
 # Test helpers
 add_subdirectory(test/range-doppler-geocoding)

diff --git a/README.md b/README.md
@@ -29,7 +29,8 @@ Each algorithm is a separate executable. Currently available ones are (more info
 * Sentinel 1 coherence estimation timeline generation - ``alus-coht`` ([README](algs/coherence-estimation-routine/README.md))
 * Sentinel 1 calibration routine - ``alus-cal`` ([README](algs/calibration-routine/README.md))
 * Sentinel 2 and other raster resample and tiling - ``alus-resa`` ([README](algs/resample/README.md))
-* Gabor feature extraction - ``alus-gfe``
+* Gabor feature extraction - ``alus-gfe`` ([README](algs/feature-extraction-gabor/README.md))
+* PALSAR level 0 focuser - ``alus-palsar-focus`` ([README](algs/palsar-focus/README.md))
 
 When building separately, these are located at ``<build_dir>/alus_package``
 

diff --git a/RELEASE.md b/RELEASE.md
@@ -1,3 +1,48 @@
+# Release 1.4.0
+
+## Breaking changes
+
+## Known Caveats
+* Coherence estimation results can have missing pixels(or coherence 0 values) on north and east direction on coastlines when compared to SNAP.
+  Currently it is unresolved what is the correct way. See examples [A](docs/coh_missing_pixels.png) and [B](docs/beirut_iw1_b6_coastal.png).
+  See posts about the issues/bugs - [tile size affecting results](https://forum.step.esa.int/t/tile-size-is-affecting-coregistration-results/32193/4)
+  and [no-data value interpretation](https://forum.step.esa.int/t/coregistration-no-data-value/35304/2).
+* Due to the nature of floating point arithmetic there are some discrepancies when compared to SNAP, see [slideshow](docs/GPU-GSTP-MPR-0008.pdf)
+* There are two code paths to calculate terrain corrected outputs' dimensions and geocoordinates
+  ([see description](https://forum.step.esa.int/t/range-doppler-terrain-correction-raster-size-and-coordinates-inconsistency/35977)). Since ALUs
+  implements the SNAP UI way the end results probably differ when compared to SNAP GPT outputs.
+
+## Major Features and Improvements
+* ALOS PALSAR Zero-Doppler L0 focuser implemented
+* Subswath merge support for calibration routine
+
+## Bug Fixes and Other Changes
+* Tile access error in coherence operator fixed
+
+## Thanks to our Contributors
+
+# Release 1.3.0
+
+## Breaking changes
+
+## Known Caveats
+* Coherence estimation results can have missing pixels(or coherence 0 values) on north and east direction on coastlines when compared to SNAP.
+  Currently it is unresolved what is the correct way. See examples [A](docs/coh_missing_pixels.png) and [B](docs/beirut_iw1_b6_coastal.png).
+  See posts about the issues/bugs - [tile size affecting results](https://forum.step.esa.int/t/tile-size-is-affecting-coregistration-results/32193/4)
+  and [no-data value interpretation](https://forum.step.esa.int/t/coregistration-no-data-value/35304/2).
+* Due to the nature of floating point arithmetic there are some discrepancies when compared to SNAP, see [slideshow](docs/GPU-GSTP-MPR-0008.pdf)
+* There are two code paths to calculate terrain corrected outputs' dimensions and geocoordinates
+  ([see description](https://forum.step.esa.int/t/range-doppler-terrain-correction-raster-size-and-coordinates-inconsistency/35977)). Since ALUs
+  implements the SNAP UI way the end results probably differ when compared to SNAP GPT outputs.
+
+## Major Features and Improvements
+* Gabor feature extraction processor added
+* Preliminary ALOS PALSAR level 0 focusser added
+
+## Bug Fixes and Other Changes
+
+## Thanks to our Contributors
+
 # Release 1.2.0
 
 ## Breaking changes

diff --git a/VERSION b/VERSION
@@ -1,3 +1,3 @@
 #define VERSION_MAJOR 1
-#define VERSION_MINOR 2
+#define VERSION_MINOR 4
 #define VERSION_PATCH 0