swiss-territorial-data-lab · cleherny · Sep 15, 2023 · Mar 13, 2023 · Mar 13, 2023 · Mar 14, 2023
diff --git a/Dockerfile b/Dockerfile
@@ -0,0 +1,28 @@
+FROM nvidia/cuda:11.3.1-runtime-ubuntu20.04
+
+RUN apt update &&\
+    apt upgrade -y &&\
+    apt install -y libgl1 &&\
+    apt install -y libglib2.0-0 &&\
+    apt install -y gdal-bin &&\
+    apt install -y wget &&\
+    apt install -y python3-pip &&\
+    apt install -y python-is-python3
+
+WORKDIR /app
+
+ADD requirements.txt .
+RUN pip install -r requirements.txt --no-cache-dir
+
+ADD helpers/*.py helpers/
+ADD scripts/*.py scripts/
+
+ADD setup.py .
+RUN pip install .
+
+USER 65534:65534
+
+ENV MPLCONFIGDIR /tmp 
+
+ENTRYPOINT [""]
+CMD ["stdl-objdet", "-h"]
diff --git a/README.md b/README.md
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -0,0 +1,15 @@
+version: "3"
+
+services:
+  stdl-objdet:
+    build: .
+    volumes:
+      - ./examples:/app/examples
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: 1
+              capabilities: [gpu]
+    command: /bin/bash
diff --git a/examples/quarry-detection/README.md b/examples/quarry-detection/README.md
@@ -3,38 +3,42 @@
 A sample working setup is provided here, enabling the end-user to detect quarries and mineral extraction sites in Switzerland over several years. <br>
 It consists of the following elements:
 
-- the read-to-use configuration files:
-    - `config_trne.yaml`,
-    - `config_prd.yaml`,
-    - `detectron2_config_dqry.yaml`,
-- the input data in the `data` subfolder:
-    - quarries shapefile from the product [swissTLM3D](https://www.swisstopo.admin.ch/fr/geodata/landscape/tlm3d.html), revised and synchronised with the 2020 [SWISSIMAGE](https://www.swisstopo.admin.ch/fr/geodata/images/ortho/swissimage10.html) mosaic (**label**),
-    - the delimitation of the AOI to perform inference predictions (**AOI**),
-    - the swiss DEM raster is too large to be saved on this platform but can be downloaded from this [link](https://github.com/lukasmartinelli/swissdem) using the EPSG:4326 - WGS 84 coordinate reference system. The raster must be first reprojected to EPSG:2056 - CH1903+ / LV95, named `switzerland_dem_EPSG2056.tif`and located in the **DEM** subfolder.
-- a data preparation script (`prepare_data.py`) producing the files to be used as input to the `generate_tilesets.py`script.
-- a results post-processing script (`filter_prediction.py`) filtering the predictions, produced from `make_prediction.py`script, to the final shapefile 
-
-After creating and a new environment in python 3.8, the end-to-end workflow can be run by issuing the following list of commands, straight from this folder:
-
-```bash
-$ sudo apt-get install -y python3-gdal gdal-bin libgdal-dev gcc g++ python3.8-dev
-$ pip install -r ../../requirements.txt
-$ python3 prepare_data.py config_trne.yaml
-$ python3 ../../scripts/generate_tilesets.py config_trne.yaml
-$ python3 ../../scripts/train_model.py config_trne.yaml
-$ python3 ../../scripts/make_predictions.py config_trne.yaml
-$ python3 ../../scripts/assess_predictions.py config_trne.yaml
-$ python3 prepare_data.py config_prd.yaml
-$ python3 ../../scripts/generate_tilesets.py config_prd.yaml
-$ python3 ../../scripts/make_predictions.py config_prd.yaml
-$ python3 filter_detection.py config_prd.yaml
+- ready-to-use configuration files:
+    - `config_trne.yaml`;
+    - `config_prd.yaml`;
+    - `detectron2_config_dqry.yaml`.
+- Input data in the `data` subfolder:
+    - quarry **labels** issued from the [swissTLM3D](https://www.swisstopo.admin.ch/fr/geodata/landscape/tlm3d.html) product, revised and synchronized with the 2020 [SWISSIMAGE](https://www.swisstopo.admin.ch/fr/geodata/images/ortho/swissimage10.html) orthophotos;
+    - the delimitation of the **Area of Interest (AoI)**;
+    - the Swiss DEM raster is too large to be saved on this platform but can be downloaded from this [link](https://github.com/lukasmartinelli/swissdem) using the [EPSG:4326](https://epsg.io/4326) coordinate reference system. The raster must be re-projected to [EPSG:2056](https://epsg.io/2056), renamed as `switzerland_dem_EPSG2056.tif` and located in the **DEM** subfolder. This procedure is managed by running the bash script `get_dem.sh`. 
+- A data preparation script (`prepare_data.py`) producing the files to be used as input to the `generate_tilesets` stage.
+- A post-processing script (`filter_detections.py`) which filters detections according to their confidence score, altitude and area. The script also identifies and merges groups of nearby polygons.
+
+The workflow can be run end-to-end by issuing the following list of commands, from the root folder of this GitHub repository:
+
+```
+$ sudo chown -R 65534:65534 examples
+$ docker compose run --rm -it stdl-objdet
+nobody@<id>:/app# cd examples/quarry-detection
+nobody@<id>:/app# python prepare_data.py config_trne.yaml
+nobody@<id>:/app# stdl-objdet generate_tilesets config_trne.yaml
+nobody@<id>:/app# stdl-objdet train_model config_trne.yaml
+nobody@<id>:/app# stdl-objdet make_detections config_trne.yaml
+nobody@<id>:/app# stdl-objdet assess_detections config_trne.yaml
+nobody@<id>:/app# python prepare_data.py config_prd.yaml
+nobody@<id>:/app# stdl-objdet generate_tilesets config_prd.yaml
+nobody@<id>:/app# stdl-objdet make_detections config_prd.yaml
+nobody@<id>:/app# bash get_dem.sh
+nobody@<id>:/app# python filter_detections.py config_prd.yaml
+nobody@<id>:/app# exit
+$ sudo chmod -R a+w examples
 ```
 
 We strongly encourage the end-user to review the provided `config_trne.yaml` and `config_prd.yaml` files as well as the various output files, a list of which is printed by each script before exiting.
 
-The model is trained on the 2020 [SWISSIMAGE](https://www.swisstopo.admin.ch/fr/geodata/images/ortho/swissimage10.html) mosaic. Inference can be performed on SWISSIMAGE mosaics of the product SWISSIMAGE time travel by changing the year in `config_prd.yaml`. It should be noted that the model has been trained on RGB color images and might not perform as well on Black and White images.
+The model is trained on the 2020 [SWISSIMAGE](https://www.swisstopo.admin.ch/fr/geodata/images/ortho/swissimage10.html) mosaic. Inference can be performed on SWISSIMAGE mosaics of the product [SWISSIMAGE time travel](https://map.geo.admin.ch/?lang=en&topic=swisstopo&bgLayer=ch.swisstopo.pixelkarte-farbe&zoom=0&layers_timestamp=2004,2004,&layers=ch.swisstopo.swissimage-product,ch.swisstopo.swissimage-product.metadata,ch.swisstopo.images-swissimage-dop10.metadata&E=2594025.91&N=1221065.68&layers_opacity=1,0.7,1&time=2004&layers_visibility=true,true,false) by changing the year in `config_prd.yaml`. It should be noted that the model has been trained on RGB images and might not perform as well on B&W images.
 
-For more information about this project, you can consult [the associated repository](https://github.com/swiss-territorial-data-lab/proj-dqry) (not public yet).
+For more information about this project, see [this repository](https://github.com/swiss-territorial-data-lab/proj-dqry) (not public yet).
 
 ## Disclaimer
 

diff --git a/examples/quarry-detection/config_prd.yaml b/examples/quarry-detection/config_prd.yaml
@@ -2,25 +2,25 @@
 ####### Inference detection ####### 
 # Automatic detection of Quarries and Mineral Extraction Sites (MES) in images
 
-# 1-Produce tiles geometry according to the AOI extent and zoom level
+# 1-Produce tiles geometry according to the AoI extent and zoom level
 prepare_data.py: 
   srs: "EPSG:2056"                       # Projection of the input file
   datasets:
-    labels_shapefile: ./data/AOI/AOI_2020.shp
-  output_folder: ./output/output-prd
+    labels_shapefile: ./data/AoI/AoI_2020.shp
+  output_folder: ./output/output_prd
   zoom_level: 16   
 
 # 2-Fetch of tiles (online server) and split into 3 datasets: train, test, validation
 generate_tilesets.py:
   debug_mode: False # sample of tiles
   datasets:
-    aoi_tiles_geojson: ./output/output-prd/tiles.geojson
+    aoi_tiles_geojson: ./output/output_prd/tiles.geojson
     orthophotos_web_service:
       type: XYZ # supported values: 1. MIL = Map Image Layer 2. WMS 3. XYZ
       url: https://wmts.geo.admin.ch/1.0.0/ch.swisstopo.swissimage-product/default/2020/3857/{z}/{x}/{y}.jpeg
-  output_folder: ./output/output-prd
+  output_folder: ./output/output_prd
   tile_size: 256 # per side, in pixels
-  overwrite: True
+  overwrite: False
   n_jobs: 10
   COCO_metadata:
     year: 2021
@@ -36,29 +36,29 @@ generate_tilesets.py:
       supercategory: "Land usage"
 
 # 3-Perform the object detection based on the optimized trained model
-make_predictions.py:
-  working_folder: ./output/output-prd
+make_detections.py:
+  working_folder: ./output/output_prd
   log_subfolder: logs
   sample_tagged_img_subfolder: sample_tagged_images
   COCO_files: # relative paths, w/ respect to the working_folder
     oth: COCO_oth.json
   detectron2_config_file: '../../detectron2_config_dqry.yaml' # path relative to the working_folder
   model_weights:
-    pth_file: '../output-trne/logs/model_0002999.pth' # trained model minimizing the validation loss curve, monitor the training process via tensorboard (tensorboard --logdir </logs>) 
-  image_metadata_json: './output/output-prd/img_metadata.json'
+    pth_file: '../output_trne/logs/model_0002999.pth' # trained model minimizing the validation loss curve, monitor the training process via tensorboard (tensorboard --logdir </logs>) 
+  image_metadata_json: './output/output_prd/img_metadata.json'
   rdp_simplification: # rdp = Ramer-Douglas-Peucker
     enabled: True
     epsilon: 2.0 # cf. https://rdp.readthedocs.io/en/latest/
   score_lower_threshold: 0.3
 
-# 4-Filtering and merging prediction polygons to improve results
-filter_prediction.py:
+# 4-Filtering and merging detection polygons to improve results
+filter_detections.py:
   year: 2020
-  input: ./output/output-prd/oth_predictions_at_0dot3_threshold.gpkg
-  labels_shapefile: ./data/AOI/AOI_2020.shp
+  input: ./output/output_prd/oth_detections_at_0dot3_threshold.gpkg
+  labels_shapefile: ./data/AoI/AoI_2020.shp
   dem: ./data/DEM/switzerland_dem_EPSG2056.tif
   elevation: 1200.0 # m, altitude threshold
-  score: 0.95 # prediction score (from 0 to 1) provided by detectron2
+  score: 0.95 # detection score (from 0 to 1) provided by detectron2
   distance: 10 # m, distance use as a buffer to merge close polygons (likely to belong to the same object) together
   area: 5000.0 # m2, area threshold under which polygons are discarded
-  output: ./output/output-prd/oth_prediction_at_0dot3_threshold_year-{year}_score-{score}_area-{area}_elevation-{elevation}_distance-{distance}.geojson
+  output: ./output/output_prd/oth_detections_at_0dot3_threshold_year-{year}_score-{score}_area-{area}_elevation-{elevation}_distance-{distance}.geojson
diff --git a/examples/quarry-detection/config_trne.yaml b/examples/quarry-detection/config_trne.yaml
@@ -2,24 +2,24 @@
 ####### Model training and evaluation ####### 
 # Training of automatic detection of Quarries and Mineral Extraction Sites (MES) in images with a provided ground truth
 
-# 1-Produce tiles geometry according to the AOI extent and zoom level
+# 1-Produce tiles geometry according to the AoI extent and zoom level
 prepare_data.py:  
   srs: "EPSG:2056"
   datasets:
     labels_shapefile: ./data/labels/tlm-hr-trn-topo.shp
-  output_folder: ./output/output-trne
+  output_folder: ./output/output_trne
   zoom_level: 16 
 
 # 2-Fetch of tiles (online server) and split into 3 datasets: train, test, validation
 generate_tilesets.py:
   debug_mode: False # sample of tiles
   datasets:
-    aoi_tiles_geojson: ./output/output-trne/tiles.geojson
-    ground_truth_labels_geojson: ./output/output-trne/labels.geojson
+    aoi_tiles_geojson: ./output/output_trne/tiles.geojson
+    ground_truth_labels_geojson: ./output/output_trne/labels.geojson
     orthophotos_web_service:
       type: XYZ # supported values: 1. MIL = Map Image Layer 2. WMS 3. XYZ
       url: https://wmts.geo.admin.ch/1.0.0/ch.swisstopo.swissimage-product/default/2020/3857/{z}/{x}/{y}.jpeg
-  output_folder: ./output/output-trne
+  output_folder: ./output/output_trne
   tile_size: 256 # per side, in pixels
   overwrite: False
   n_jobs: 10
@@ -38,7 +38,7 @@ generate_tilesets.py:
 
 # 3-Train the model with the detectron2 algorithm
 train_model.py:
-  working_folder: ./output/output-trne
+  working_folder: ./output/output_trne
   log_subfolder: logs
   sample_tagged_img_subfolder: sample_tagged_images
   COCO_files: # relative paths, w/ respect to the working_folder
@@ -50,8 +50,8 @@ train_model.py:
     model_zoo_checkpoint_url: "COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_1x.yaml"
 
 # 4-Perform the object detection based on the optimized trained model
-make_predictions.py:
-  working_folder: ./output/output-trne
+make_detections.py:
+  working_folder: ./output/output_trne
   log_subfolder: logs
   sample_tagged_img_subfolder: sample_tagged_images
   COCO_files: # relative paths, w/ respect to the working_folder
@@ -61,20 +61,20 @@ make_predictions.py:
   detectron2_config_file: '../../detectron2_config_dqry.yaml' # path relative to the working_folder
   model_weights:
     pth_file: './logs/model_0002999.pth' # trained model minimizing the validation loss curve, monitor the training process via tensorboard (tensorboard --logdir </logs>)
-  image_metadata_json: './output/output-trne/img_metadata.json'
+  image_metadata_json: './output/output_trne/img_metadata.json'
   rdp_simplification: # rdp = Ramer-Douglas-Peucker
     enabled: true
     epsilon: 2.0 # cf. https://rdp.readthedocs.io/en/latest/
   score_lower_threshold: 0.05
 
-# 5-Evaluate the quality of the prediction for the different datasets with metrics calculation
-assess_predictions.py:
+# 5-Evaluate the quality of the detections for the different datasets with metrics calculation
+assess_detections.py:
   datasets:
-    ground_truth_labels_geojson: ./output/output-trne/labels.geojson
-    image_metadata_json: ./output/output-trne/img_metadata.json
-    split_aoi_tiles_geojson: ./output/output-trne/split_aoi_tiles.geojson # aoi = Area of Interest
-    predictions:
-      trn: ./output/output-trne/trn_predictions_at_0dot05_threshold.gpkg
-      val: ./output/output-trne/val_predictions_at_0dot05_threshold.gpkg
-      tst: ./output/output-trne/tst_predictions_at_0dot05_threshold.gpkg
-  output_folder: ./output/output-trne
+    ground_truth_labels_geojson: ./output/output_trne/labels.geojson
+    image_metadata_json: ./output/output_trne/img_metadata.json
+    split_aoi_tiles_geojson: ./output/output_trne/split_aoi_tiles.geojson # aoi = Area of Interest
+    detections:
+      trn: ./output/output_trne/trn_detections_at_0dot05_threshold.gpkg
+      val: ./output/output_trne/val_detections_at_0dot05_threshold.gpkg
+      tst: ./output/output_trne/tst_detections_at_0dot05_threshold.gpkg
+  output_folder: ./output/output_trne
diff --git a/examples/quarry-detection/data/AoI/AoI_2020.cpg b/examples/quarry-detection/data/AoI/AoI_2020.cpg
@@ -0,0 +1 @@
+UTF-8
diff --git a/examples/quarry-detection/data/AoI/AoI_2020.dbf b/examples/quarry-detection/data/AoI/AoI_2020.dbf
diff --git a/examples/quarry-detection/data/AoI/AoI_2020.prj b/examples/quarry-detection/data/AoI/AoI_2020.prj
@@ -0,0 +1 @@
+PROJCS["CH1903+_LV95",GEOGCS["GCS_CH1903+",DATUM["D_CH1903+",SPHEROID["Bessel_1841",6377397.155,299.1528128]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Hotine_Oblique_Mercator_Azimuth_Center"],PARAMETER["False_Easting",2600000.0],PARAMETER["False_Northing",1200000.0],PARAMETER["Scale_Factor",1.0],PARAMETER["Azimuth",90.0],PARAMETER["Longitude_Of_Center",7.43958333333333],PARAMETER["Latitude_Of_Center",46.9524055555556],UNIT["Meter",1.0]]
diff --git a/examples/quarry-detection/data/AoI/AoI_2020.shp b/examples/quarry-detection/data/AoI/AoI_2020.shp
diff --git a/examples/quarry-detection/data/AoI/AoI_2020.shx b/examples/quarry-detection/data/AoI/AoI_2020.shx
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		PROJCS["CH1903+_LV95",GEOGCS["GCS_CH1903+",DATUM["D_CH1903+",SPHEROID["Bessel_1841",6377397.155,299.1528128]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Hotine_Oblique_Mercator_Azimuth_Center"],PARAMETER["False_Easting",2600000.0],PARAMETER["False_Northing",1200000.0],PARAMETER["Scale_Factor",1.0],PARAMETER["Azimuth",90.0],PARAMETER["Longitude_Of_Center",7.43958333333333],PARAMETER["Latitude_Of_Center",46.9524055555556],UNIT["Meter",1.0]]