This repository contains code of our CVPR 2021 paper - "Learning Camera Localization via Dense Scene Matching" by Shitao Tang, Chengzhou Tang, Rui Huang, Siyu Zhu and Ping Tan.

This paper presents a new method for scene agnostic camera localization using dense scene matching (DSM), where a cost volume is constructed between a query image and a scene. The cost volume and the corresponding coordinates are processed by a CNN to predict dense coordinates. Camera poses can then be solved by PnP algorithms.

If you find this project useful, please cite:

@inproceedings{Tang2021Learning,
  title={Learning Camera Localization via Dense Scene Matching},
  author={Shitao Tang, Chengzhou Tang, Rui Huang, Siyu Zhu and Ping Tan},
  booktitle={Computer Vision and Pattern Recognition (CVPR)},
  year={2021}
}

Usage

Environment

• The codes are tested along with
  • pytorch=1.4.0
  • lmdb (optional)
  • yaml
  • skimage
  • opencv
  • numpy=1.17
  • tensorboard

Installation

• Build PyTorch operations

    cd libs/model/ops
    python setup.py install
  

• Build PnP algorithm

    cd libs/utils/lm_pnp
    mkdir build
    cd build
    cmake ..
    make all
  

Train and Test

• Download

  You can download the trained models and label files for 7scenes, Cambridge, Scannet.

  For 7scenes, you can use the prepared data in the following.

  Chess	Fire	Heads	Office	Pumpkin	Kitchen	Stairs

  For Cambridge landmarks, you can download image files here, and depths here.

• Test

  Please refer to configs/7scenes.yaml for detailed explaination of how to set label file path and image file path.

  • 7scenes

    python tools/video_test.py --config configs/7scenes.yaml
    

  • Camrbrige

    python tools/video_test.py --config configs/cambridge.yaml
    

• Train

  We use ResNet-FPN pretrained model.

    python tools/train_net.py
  

License

The MIT License (MIT)

Copyright (c) 2022 Shitao Tang

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.