Initial commit

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 zgojcic
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,185 @@
+# Learning Multiview 3D Point Cloud Registration repository
+This repository provides code and data to train and evaluate the LMPCR, the first end-to-end algorithm for multiview registration of raw point clouds in a globally consistent manner. It represents the official implementation of the paper:
+
+### [Learning Multiview 3D Point Cloud Registration (CVPR 2020).](https://arxiv.org/pdf/2001.05119.pdf)
+\*[Zan Gojcic](https://www.ethz.ch/content/specialinterest/baug/institute-igp/geosensors-and-engineering-geodesy/en/people/scientific-assistance/zan-gojcic.html),\* [Caifa Zhou](https://ch.linkedin.com/in/caifa-zhou-7a461510b), [Jan D. Wegner](http://www.prs.igp.ethz.ch/content/specialinterest/baug/institute-igp/photogrammetry-and-remote-sensing/en/group/people/person-detail.html?persid=186562), [Leonidas J. Guibas](https://geometry.stanford.edu/member/guibas/), [Tolga Birdal](http://tbirdal.me/)\
+|[EcoVision Lab ETH Zurich](https://prs.igp.ethz.ch/ecovision.html) | [Guibas Lab Stanford University](https://geometry.stanford.edu/index.html)|\
+\* Equal contribution
+
+We present a novel, end-to-end learnable, multiview 3D point cloud registration algorithm. Registration of multiple scans typically follows a two-stage pipeline: the initial pairwise alignment and the globally consistent refinement. The former is often ambiguous due to the low overlap of
+neighboring point clouds, symmetries and repetitive scene parts. Therefore, the latter global refinement aims at establishing the cyclic consistency across multiple scans and helps in resolving the ambiguous cases. In this paper we propose, to the best of our knowledge, the first end-to-end algorithm for joint learning of both parts of this two-stage problem. Experimental evaluation on well accepted benchmark datasets shows that our approach outperforms the state-of-the-art by a significant margin, while being end-to-end trainable and computationally less costly. Moreover, we present detailed analysis and an ablation study that validate
+the novel components of our approach. 
+
+![LM3DPCR](figures/LM3DPCR.jpg?raw=true)
+
+### Citation
+
+If you find this code useful for your work or use it in your project, please consider citing:
+
+```shell
+@inproceedings{gojcic2020LearningMultiview,
+	title={Learning Multiview 3D Point Cloud Registration},
+	author={Gojcic, Zan and Zhou, Caifa and Wegner, Jan D and Guibas, Leonidas J and Birdal, Tolga},
+	booktitle={International conference on computer vision and pattern recognition (CVPR)},
+	year={2020}
+}
+```
+
+### Contact
+If you have any questions or find any bugs, please let us know: Zan Gojcic {[email protected]}
+
+## Current state of the repository
+Currently the repository contains only part of the code connected to the above mentioned publication and will be consistently updated in the course of the following days. The whole code will be available the following weeks. 
+
+**NOTE**: The published model is not the same as the model used in the CVPR paper. The results can therefore slightly differ from the ones in the paper. The models will be updated in the following days (with the fully converged ones).
+
+## Instructions
+The code was tested on Ubuntu 18.04 with Python 3.6, pytorch 1.5, CUDA 10.1.243, and GCC 7.
+
+### Requirements
+After cloning this repository, you can start by creating a virtual environment and installing the requirements by running:
+
+```bash
+conda create --name lmpr python=3.6
+source activate lmpr
+conda config --append channels conda-forge
+conda install --file requirements.txt
+conda install -c open3d-admin open3d=0.9.0.0
+conda install -c intel scikit-learn
+conda install pytorch torchvision cudatoolkit=10.1 -c pytorch
+```
+
+Our network uses [FCGF](https://github.com/chrischoy/FCGF) feature descriptor which is based on the [MinkowskiEnginge](https://github.com/StanfordVL/MinkowskiEngine) library for sparse tensors. In order to install Minkowski Engine run:
+
+```bash
+source activate lmpr
+git clone https://github.com/StanfordVL/MinkowskiEngine.git
+cd MinkowskiEngine
+conda install numpy mkl-include
+export CXX=g++-7; python setup.py install
+cd ../
+```
+
+Finally, our network supports furthest point sampling, when sampling the interest points. To this end we require the [PointNet++](https://github.com/erikwijmans/Pointnet2_PyTorch/tree/master/pointnet2/models) library that can be installed as follows: 
+
+```bash
+source activate lmpr
+git clone https://github.com/erikwijmans/Pointnet2_PyTorch.git
+cd Pointnet2_PyTorch
+pip install -r requirements.txt
+cd ../
+```
+
+## Pretrained models
+We provide the pretrained models for [FCGF]((https://github.com/chrischoy/FCGF)) feature descriptor, our pairwise registration block, and jointly trained pairwise registration model. They can be downloaded using:
+
+ ```bash
+bash scripts/download_pretrained_models.sh
+```
+
+## Datasets
+### Pairwise registration
+In order to train the pairwise registration model, you have to download the full [3DMatch](http://3dmatch.cs.princeton.edu/) dataset. To (pre)-train the registration blocks you can either download the preprocessed dataset (~160GB) using
+
+```bash
+bash scripts/download_3DMatch_train.sh preprocessed
+```
+
+This dataset contains the pointwise correspondences established in the [FCGF](https://github.com/chrischoy/FCGF) feature space for all point cloud pairs of 3DMatch dataset.
+
+We also provide the raw 3DMatch data (~4.5GB) and a script to generate the preprocess training data `./scripts/extract_data.py` that can be used either with the raw 3DMatch data or your personal dataset. The raw dataset can be downloaded using
+
+```bash
+bash scripts/download_3DMatch_train.sh raw
+```
+
+And then processed (extract FCGF feature descriptors, establish the correspondences, and save training data) using:
+
+```bash
+source activate lmpr
+python ./scripts/extract_data.py \
+			--source_path ./data/train_data/ \
+			--target_path ./data/train_data/ \
+			--dataset 3d_match \
+			--model ./pretrained/fcgf/model_best.pth \
+			--voxel_size 0.025 \
+			--n_correspondences 20000 \
+			--inlier_threshold 0.05 \
+			--extract_features \
+			--extract_correspondences \
+			--extract_precomputed_training_data \
+			--with_cuda \
+
+bash scripts/download_preprocessed_3DMatch.sh
+```
+## Training
+Current release supports only training the pairwise registration network. The repository will be further updated in the next weeks. In order to train the pairwise registration network from scratch using the precomputed data run 
+```bash
+source activate lmpr
+python train.py ./configs/pairwise_registration/OANet.yaml
+
+```
+The training parameters can be set in `./configs/pairwise_registration/OANet.yaml`. 
+
+In order to fine tune the pairwise registration network in an end-to-end manner (including the FCGF block), the raw data has to be used. The code to sample the batches will be released in the next weeks.
+
+## Evaluation 
+We provide the scripts for the automatic evaluation of our method on the 3DMatch and Redwood dataset. The results for our method can differ slightly from the results in the CVPR paper as we have retrained the model. Due to the different implementation the results for RANSAC might also differ slightly from
+the results of the [official evaluation script](https://github.com/andyzeng/3dmatch-toolbox/tree/master/evaluation/geometric-registration).
+
+### 3DMatch
+To evaluate on 3DMatch you can either download the raw evaluation data (~450MB) using
+```bash
+bash scripts/download_3DMatch_eval.sh raw
+```
+or the processed data together with the results for our method and RANSAC (~2.7GB) using 
+```bash
+bash scripts/download_3DMatch_eval.sh preprocessed
+```
+
+If you download the raw data you first have to process it (extract features and correspondences)
+
+```bash
+source activate lmpr
+python ./scripts/extract_data.py \
+			--source_path ./data/eval_data/ \
+			--target_path ./data/eval_data/ \
+			--dataset 3d_match \
+			--model ./pretrained/fcgf/model_best.pth \
+			--voxel_size 0.025 \
+			--n_correspondences 5000 \
+			--extract_features \
+			--extract_correspondences \
+			--with_cuda \
+
+bash scripts/download_preprocessed_3DMatch.sh
+```
+
+Then you can run the pairwise registration evaluation using our RegBlock (with all points) as
+
+```bash
+source activate lmpr
+python ./scripts/benchmark_pairwise_registration.py \
+		--source ./data/eval_data/3d_match/ \
+		--dataset 3d_match \
+		--method RegBlock \
+		--model ./pretrained/RegBlock/model_best.pt \
+		--only_gt_overlap \
+```
+
+This script assumes that the correspondences were already estimated using `./scripts/extract_data.py` and only benchmarks the registration algorithms. To improve efficiency the registration parameters will only be computed for the ground truth overlapping pairs, when the flag `--only_gt_overlap` is set. This does not change the registration recall, which is used as the primary evaluation metric. In order to run the estimation on all n choose 2 pairs simply omit the `--only_gt_overlap` (results in ~10 times longer computation time on 3DMatch dataset)
+
+
+### Redwood
+To evaluate the generalization performance of our method on redwood you can again either download the raw evaluation data (~1.3GB) using
+```bash
+bash scripts/download_redwood_eval.sh raw
+```
+or the processed data together with the results for our method and RANSAC (~2.9GB) using 
+```bash
+bash scripts/download_redwood_eval.sh preprocessed
+```
+
+The rest of the evaluation follow the same procedure as for 3DMatch, you simply have to replace the dataset argument with `--dataset redwood`
+
+**NOTE**: Using the currently provided model the performance is slightly worse then reported in the paper. The model will be updated in the following days.

configs/pairwise_registration/OANet.yaml

@@ -0,0 +1,69 @@
+method:
+  task: pairwise # Name of the task, one of [pairwise, multiview]
+  descriptor_module: null # Descriptor method to be used. If null precomputed correspondences are used for training.
+  filter_module: oanet # Filtering method to be used.
+
+misc:
+  run_mode: train # Mode to run the network in 
+  net_depth: 12 # Number of layers
+  clusters: 500 # Number of clusters
+  iter_num: 1   # Number of iterations in the iterative network
+  net_channel: 128 # Number of channels in a layer
+  trainer: 'PairwiseTrainer' # Which class of trainer to use. Can be used if multiple different trainers are defined.
+  use_gpu: True # If GPU should be used or not
+  best_validation_metric: loss # Which validation metric to use. 0 Transformation loss, 1 Classification loss
+  log_dir: ./logs/pairwise_registration # Path to the folder where the models and logs will be saves
+  normalize_weights: True # If the inferred per point weights should be normalized to sum to 1 before SVD
+  trans_loss_margin: 0.1 # Value for clipping the transformation loss
+  inlier_weight_threshold: 0.5 # Threshold for determining the inlier/outlier class
+
+data:
+  dataset: Precomputed3DMatchExample
+  root: ./data/train_data/3d_match/
+  dist_th: 0.05 # Distance threshold for ground truth labels
+  shuffle_examples: True # Shuffle training examples in the data loader
+  augment_data: True # If data should be augmented by being transformed with random transformation parameters 
+  jitter: True # If jitter should be applied to input point clouds
+  use_mutuals: False # Use only mutual nearest neighbors or all correspondences. 0 do not use, 1 use as filter, 2 use as side info
+  max_num_points: 2000 #Number of keypoints to use per training example
+
+loss:
+  trans_loss_type: 3 # Type of transformation loss: 0 Eucl. distance to correspondence, 1 Fro. norm, 2 Eucl. distance to GT, 3 L1 distance to GT
+  loss_class: 1 # Weight of the classification loss
+  loss_trans: 0.2 # Weight of the transformation loss
+  loss_desc: 0.0 # Weight of the descriptor loss (only used if descriptor module is specified)
+  trans_loss_iter: 15000 # Iteration at which the transformation and confidence loss are added
+  inlier_threshold: 0.05 # GT inlier threshold (if gt inlier ratio is lower than this threshold the transformation loss will not be backpropagated)
+
+train:
+  batch_size: 8 # Training batch size
+  num_workers: 8 # Number of workers used for the data loader
+  max_epoch: 500 # Max number of training epochs
+  stat_interval: 5 # Interval at which the stats are printed out and saved for the tensorboard (if positive it denotes iteration if negative epochs)
+  chkpt_interval: 100 # Interval at which the model is saved (if positive it denotes iteration if negative epochs)
+  val_interval: 300 # Interval at which the validation is performed (if positive it denotes iteration if negative epochs)
+  model_selection_metric: loss # Metric used to determine best model on the validation dataset
+  model_selection_mode: minimize # Metric used to determine best model on the validation dataset
+  samp_type: rand # Sampler type on of [rand, fps]
+  corr_type: soft # Feature matching type on of [soft, hard, gumbel_soft]
+  st_grad_flag: True # If true and soft or gumbel_soft corr are selected, the gradients are propagated straight through (https://arxiv.org/abs/1308.3432)
+  compute_precision: True # Compute precision and recall of the per correspondence classification 
+
+val:
+  batch_size: 8 # Validation batch size
+  num_workers: 1 # Number of workers for the validation data set
+
+test:
+  results_path: '' # Path to where to save the test results
+  batch_size: 8 # Test batch size
+  num_workers: 1 # Num of workers to use for the test data set
+
+optimizer:
+  alg: Adam # Which optimizer to use
+  learning_rate: 0.0001 # Initial learning rate
+  weight_decay: 0.0 # Weight decay weight
+  momentum: 0.9 #Momentum
+
+model:
+  init_from: null # Path to the pretrained model
+  init_file_name: model_best.pt # Pretrained model filename