Merge pull request #168 from KMarshallX/restructure_1023

setting up github test for boost.py

.github/workflows/test_boost.yml

@@ -0,0 +1,50 @@
+name: boost module test
+
+on:
+  push:
+    branches: [ master ]
+    paths:
+      - README.md
+      - boost.py
+      - boost_config.py
+      - environment.yml
+      - documentation/boost_readme.md
+      - utils/module_utils.py
+      - utils/train_utils.py
+      - utils/single_data_loader.py
+      - utils/unet_utils.py
+      - tests/test_boost_module.sh
+      - miniconda-setup.sh
+      - .github/workflows/test_boost.yml
+
+  pull_request:
+    branches: [ master ]
+    paths:
+      - README.md
+      - boost.py
+      - boost_config.py
+      - environment.yml
+      - documentation/boost_readme.md
+      - utils/module_utils.py
+      - utils/train_utils.py
+      - utils/single_data_loader.py
+      - utils/unet_utils.py
+      - tests/test_boost_module.sh
+      - miniconda-setup.sh
+      - .github/workflows/test_boost.yml
+
+jobs:
+  test_boost_module_pipeline:
+    runs-on: ubuntu-20.04
+    steps:
+    - uses: actions/checkout@v3
+    - uses: actions/setup-python@v3
+      with:
+        python-version: 3.9.5
+    - name: test boost module
+      env:
+        OSF_TOKEN_: ${{ secrets.OSF_KEY }}
+        OSF_USERNAME_: ${{ secrets.OSF_USERNAME }}
+        OSF_PROJECT_ID_: "abk4p" 
+      run: |
+        /bin/bash tests/test_boost_module.sh

documentation/boost_readme.md

@@ -0,0 +1,15 @@
+# **Boosting Module**
+This module takes a single subject data and its coarse segmentation to train a model from scratch, and then use the trained model to predict a refined segmentation of the subject data. 
+
+## Example test run of this script:
+If you set prep_mode to 4, which means no preprocessing will happen, then you don't have to set a path to store the preprocessed images:
+
+```bash
+python boost.py --ds_path $path_to_images --lb_path $path_to_labels --out_path $path_to_output --outmo $path_to_scratch_model --prep_mode 4 --ep $n_epochs --lr 1e-3
+```
+
+If you set prep_mode to 1,2 or 3, which means (1) N4 bias field correction, (2)denosing, or (3) both N4 biasfield correction and denoising will happen, then you have to set a path to store the preprocessed images. In the following example, we set the preprocessing mode to "applying N4 bias field correction only".
+
+```bash
+python boost.py --ds_path $path_to_images --ps_path $path_to_preprocessed_images --lb_path $path_to_labels --out_path $path_to_output --outmo $path_to_scratch_model --prep_mode 1 --ep $n_epochs --lr 1e-3
+```

documentation/train_readme.md

@@ -1,6 +1,6 @@
 # **Initial Training Module**
 You can use this module to train your own base model.
-## **Create a base modle from scratch**
+## **Create a base model from scratch**
 ### Prepare the training data
 If you are working outside of a container, you can store you data under *data* folder as the instruction below. If you are working inside a container, you can mount your data folder to *./data/train/* and  *./data/label/*.\
 Please make sure that the name of a segmentation image file should contain the FULL NAME of its corresponding MRI image. \

tests/test_boost_module.sh

@@ -0,0 +1,97 @@
+#!/usr/bin/env bash
+set -e
+
+cp -r . /tmp/vessel_code
+
+# test readme
+echo "[DEBUG]: testing the clone command from the README:"
+clone_command=`cat /tmp/vessel_code/README.md | grep https://github.com/KMarshallX/vessel_code.git`
+echo $clone_command
+$clone_command
+
+echo "[DEBUG]: testing the miniconda installation from the README:"
+get_command=`cat /tmp/vessel_code/README.md | grep miniconda-setup.sh`
+echo $get_command
+$get_command 
+
+export PATH="/home/runner/miniconda3/bin:$PATH"
+source ~/.bashrc
+
+echo "[DEBUG]: testing the conda env build from the README:"
+cd vessel_code
+condaenv_command=`cat ./README.md | grep environment.yml`
+echo $condaenv_command
+$condaenv_command
+
+# conda activate in a bash script
+source /home/runner/miniconda3/bin/activate
+conda init bash
+
+echo "[DEBUG]: testing conda activate command from the README:"
+condact_command=`cat ./README.md | grep activate`
+echo $condact_command
+$condact_command
+
+# settings for data download
+mkdir -p ./data/images/
+mkdir -p ./data/labels/
+mkdir -p ./data/preprocessed/
+mkdir -p ./data/predicted_labels/
+
+pip install osfclient
+osf -p nr6gc fetch /osfstorage/twoEchoTOF/raw/GRE_3D_400um_TR20_FA18_TE7p5_14_sli52_FCY_GMP_BW200_32.nii ./data/images/sub-001.nii
+osf -p nr6gc fetch /osfstorage/twoEchoTOF/seg/seg_GRE_3D_400um_TR20_FA18_TE7p5_14_sli52_FCY_GMP_BW200_32_biasCor_H75_L55_C10.nii ./data/labels/sub-001.nii
+
+path_to_images="./data/images/"
+echo "Path to images: "$path_to_images""
+
+path_to_labels="./data/labels/"
+echo "Path to labels: "$path_to_labels""
+
+path_to_output="./data/predicted_labels/"
+echo "Path to output: "$path_to_output""
+
+path_to_model="./data/predicted_labels/model_test"
+echo "Path to model: "$path_to_model""
+
+path_to_preprocessed="./data/preprocessed/"
+echo "Path to preprocessed data: "$path_to_preprocessed""
+
+n_epochs=5
+echo "Number of epochs: "$n_epochs""
+
+echo "[DEBUG]: testing boost module:"
+train_command1=`cat ./documentation/boost_readme.md | grep 'prep_mode 4'`
+echo $train_command1
+eval $train_command1
+
+train_command2=`cat ./documentation/boost_readme.md | grep 'prep_mode 1'`
+echo $train_command2
+eval $train_command2
+
+echo "[DEBUG]: osf setup"
+export OSF_TOKEN=$OSF_TOKEN_
+export OSF_USERNAME=$OSF_USERNAME_
+export OSF_PROJECT_ID=$OSF_PROJECT_ID_
+mkdir -p ~/.osfcli
+echo -e "[osf]\nproject = $OSF_PROJECT_ID\nusername = \$OSF_USERNAME" > ~/.osfcli/osfcli.config
+cd $path_to_output
+for dir in *; do 
+    if [ -d "$dir" ]; then 
+        echo $dir
+        cd $dir
+        for file in *; do
+            echo $file
+            osf -p abk4p remove /osfstorage/github_actions/boost/predicted_labels/$dir/$file
+        done
+        osf -p abk4p upload -r ./ /osfstorage/github_actions/boost/predicted_labels/$dir/
+        cd .. 
+    fi;
+    if [ -f "$dir" ]; then 
+        echo $dir
+        osf -p abk4p remove /osfstorage/github_actions/boost/predicted_labels/$dir
+        osf -p abk4p upload $dir /osfstorage/github_actions/boost/predicted_labels/$dir
+    fi;
+done
+
+echo "Testing done!"