Update on 2023/11: SePiCo is selected as 🏆 ESI Highly Cited Paper!!
Update on 2023/02/15: Code release for Cityscapes → Dark Zurich.
Update on 2023/01/14: 🥳 We are happy to announce that SePiCo has been accepted in an upcoming issue of the TPAMI.
Update on 2022/09/24: All checkpoints are available.
Update on 2022/09/04: Code release.
Update on 2022/04/20: ArXiv Version of SePiCo is available.
- Overview
- Installation
- Datasets Preparation
- Model Zoo
- SePiCo Evaluation
- SePiCo Training
- Tips on Code Understanding
- Acknowledgments
- Citation
- Contact
In this work, we propose Semantic-Guided Pixel Contrast (SePiCo), a novel one-stage adaptation framework that highlights the semantic concepts of individual pixel to promote learning of class-discriminative and class-balanced pixel embedding space across domains, eventually boosting the performance of self-training methods.
This code is implemented with Python 3.8.5 and PyTorch 1.7.1 on CUDA 11.0.
To try out this project, it is recommended to set up a virtual environment first:
# create and activate the environment
conda create --name sepico -y python=3.8.5
conda activate sepico
# install the right pip and dependencies for the fresh python
conda install -y ipython pipThen, the dependencies can be installed by:
# install required packages
pip install -r requirements.txt
# install mmcv-full, this command compiles mmcv locally and may take some time
pip install mmcv-full==1.3.7 # requires other packeges to be installed firstAlternatively, the mmcv-full package can be installed faster with official pre-built packages, for instance:
# another way to install mmcv-full, faster
pip install mmcv-full==1.3.7 -f https://download.openmmlab.com/mmcv/dist/cu110/torch1.7.0/index.htmlThe environment is now fully prepared.
- GTAV: Download all zipped images, along with their zipped labels, from here and extract them to a custom directory.
- Cityscapes: Download leftImg8bit_trainvaltest.zip and gtFine_trainvaltest.zip from here and extract them to a custom directory.
- Dark Zurich: Download Dark_Zurich_train_anon.zip, Dark_Zurich_val_anon.zip and Dark_Zurich_test_anon_withoutGt.zip from here and extract them to a custom directory.
Symlink the required datasets:
ln -s /path/to/gta5/dataset data/gta
ln -s /path/to/cityscapes/dataset data/cityscapes
ln -s /path/to/dark_zurich/dataset data/dark_zurichPerform preprocessing to convert label IDs to the train IDs and gather dataset statistics:
python tools/convert_datasets/gta.py data/gta --nproc 8
python tools/convert_datasets/cityscapes.py data/cityscapes --nproc 8Ultimately, the data structure should look like this:
SePiCo
├── ...
├── data
│ ├── cityscapes
│ │ ├── gtFine
│ │ ├── leftImg8bit
│ ├── dark_zurich
│ │ ├── corresp
│ │ ├── gt
│ │ ├── rgb_anon
│ ├── gta
│ │ ├── images
│ │ ├── labels
├── ...We provide pretrained models of both Domain Adaptive Semantic Segmentation tasks through Google Drive and Baidu Netdisk (access code: pico).
| variants | model name | mIoU | checkpoint download |
|---|---|---|---|
| DistCL | sepico_distcl_gta2city_dlv2.pth | 61.0 | Google / Baidu (acc: pico) |
| BankCL | sepico_bankcl_gta2city_dlv2.pth | 59.8 | Google / Baidu (acc: pico) |
| ProtoCL | sepico_protocl_gta2city_dlv2.pth | 58.8 | Google / Baidu (acc: pico) |
| variants | model name | mIoU | checkpoint download |
|---|---|---|---|
| DistCL | sepico_distcl_gta2city_daformer.pth | 70.3 | Google / Baidu (acc: pico) |
| BankCL | sepico_bankcl_gta2city_daformer.pth | 68.7 | Google / Baidu (acc: pico) |
| ProtoCL | sepico_protocl_gta2city_daformer.pth | 68.5 | Google / Baidu (acc: pico) |
| variants | model name | mIoU | checkpoint download |
|---|---|---|---|
| DistCL | sepico_distcl_syn2city_dlv2.pth | 58.1 | Google / Baidu (acc: pico) |
| BankCL | sepico_bankcl_syn2city_dlv2.pth | 57.4 | Google / Baidu (acc: pico) |
| ProtoCL | sepico_protocl_syn2city_dlv2.pth | 56.8 | Google / Baidu (acc: pico) |
| variants | model name | mIoU | checkpoint download |
|---|---|---|---|
| DistCL | sepico_distcl_syn2city_daformer.pth | 64.3 | Google / Baidu (acc: pico) |
| BankCL | sepico_bankcl_syn2city_daformer.pth | 63.3 | Google / Baidu (acc: pico) |
| ProtoCL | sepico_protocl_syn2city_daformer.pth | 62.9 | Google / Baidu (acc: pico) |
| variants | model name | mIoU | checkpoint download |
|---|---|---|---|
| DistCL | sepico_distcl_city2dark_dlv2.pth | 45.4 | Google / Baidu (acc: pico) |
| BankCL | sepico_bankcl_city2dark_dlv2.pth | 44.1 | Google / Baidu (acc: pico) |
| ProtoCL | sepico_protocl_city2dark_dlv2.pth | 42.6 | Google / Baidu (acc: pico) |
| variants | model name | mIoU | checkpoint download |
|---|---|---|---|
| DistCL | sepico_distcl_city2dark_daformer.pth | 54.2 | Google / Baidu (acc: pico) |
| BankCL | sepico_distcl_city2dark_daformer.pth | 53.3 | Google / Baidu (acc: pico) |
| ProtoCL | sepico_distcl_city2dark_daformer.pth | 52.7 | Google / Baidu (acc: pico) |
Our trained model (sepico_distcl_city2dark_daformer.pth) is also tested for generalization on the Nighttime Driving and BDD100k-night test sets.
| Method | model name | Dark Zurich-test | Nighttime Driving | BDD100k-night | checkpoint download |
|---|---|---|---|---|---|
| SePiCo | sepico_distcl_city2dark_daformer.pth | 54.2 | 56.9 | 40.6 | Google / Baidu (acc: pico) |
To evaluate the pretrained models on Cityscapes, please run as follows:
python -m tools.test /path/to/config /path/to/checkpoint --eval mIoUExample
For example, if you download sepico_distcl_gta2city_dlv2.pth along with its config json file sepico_distcl_gta2city_dlv2.json into folder ./checkpoints/sepico_distcl_gta2city_dlv2/, then the evaluation script should be like:
python -m tools.test ./checkpoints/sepico_distcl_gta2city_dlv2/sepico_distcl_gta2city_dlv2.json ./checkpoints/sepico_distcl_gta2city_dlv2/sepico_distcl_gta2city_dlv2.pth --eval mIoUTo evaluate on Dark Zurich, please get label predictions as follows and submit them to the official test server.
Get label predictions for the test set locally:
python -m tools.test /path/to/config /path/to/checkpoint --format-only --eval-options imgfile_prefix=/path/to/labelTrainIdsExample
For example, if you download sepico_distcl_city2dark_daformer.pth along with its config json file sepico_distcl_city2dark_daformer.json into folder ./checkpoints/sepico_distcl_city2dark_daformer/, then the evaluation script should be like:
python -m tools.test ./checkpoints/sepico_distcl_city2dark_daformer/sepico_distcl_city2dark_daformer.json ./checkpoints/sepico_distcl_city2dark_daformer/sepico_distcl_city2dark_daformer.pth --format-only --eval-options imgfile_prefix=dark_test/distcl_daformer/labelTrainIdsNote that the test server only accepts submission with the following directory structure:
submit.zip
├── confidence
├── labelTrainIds
├── labelTrainIds_invalidSo we need to construct the confidence and labelTrainIds_invalid directory by hand (as they are not necessary to SePiCo evaluation).
Our practice is listed below for reference (check the example above for directory name):
cd dark_test/distcl_daformer
cp -r labelTrainIds labelTrainIds_invalid
cp -r labelTrainIds confidence
zip -q -r sepico_distcl_city2dark_daformer.zip labelTrainIds labelTrainIds_invalid confidence
# Now submit sepico_distcl_city2dark_daformer.zip to the test server for results.To begin with, download SegFormer's official MiT-B5 weights (i.e., mit_b5.pth) pretrained on ImageNet-1k from here and put it into a new folder ./pretrained.
The training entrance is at run_experiments.py. To examine the setting for a specific task, please take a look at experiments.py for more details. Generally, the training script is given as:
python run_experiments.py --exp <exp_id>Tasks 1~6 are run on GTAV → Cityscapes, and the mapping between <exp_id> and tasks is:
<exp_id> |
variant | backbone | feature |
|---|---|---|---|
1 |
DistCL | ResNet-101 | layer-4 |
2 |
BankCL | ResNet-101 | layer-4 |
3 |
ProtoCL | ResNet-101 | layer-4 |
4 |
DistCL | MiT-B5 | all-fusion |
5 |
BankCL | MiT-B5 | all-fusion |
6 |
ProtoCL | MiT-B5 | all-fusion |
Tasks 7~8 are run on Cityscapes → Dark Zurich, and the mapping between <exp_id> and tasks is:
<exp_id> |
variant | backbone | feature |
|---|---|---|---|
7 |
DistCL | ResNet-101 | layer-4 |
8 |
DistCL | MiT-B5 | all-fusion |
After training, the models can be tested following SePiCo Evaluation. Note that the training results are located in ./work_dirs. The config filename should look like: 220827_1906_dlv2_proj_r101v1c_sepico_DistCL-reg-w1.0-start-iter3000-tau100.0-l3-w1.0_rcs0.01_cpl_self_adamw_6e-05_pmT_poly10warm_1x2_40k_gta2cs_seed76_4cc9a.json, and the model file has suffix .pth.
- Class-balanced cropping (CBC) strategy is implemented as
RandomCropclass in mmseg/models/utils/ours_transforms.py. - The projection head can be found in mmseg/models/decode_heads/proj_head.py.
- The semantic prototypes used for feature storage are implemented in mmseg/models/utils/proto_estimator.py, where all three variants of prototypes are included. For detailed usage, please refer to
mmseg/models/uda/sepico.py. - The losses in correspondence to the three variants of our framework, along with the regularization term, are implemented in mmseg/models/losses/contrastive_loss.py.
This project is based on the following open-source projects. We thank their authors for making the source code publicly available.
- MMSegmentation (Apache License 2.0, license details)
- SegFormer (NVIDIA Source Code License, license details)
- DAFormer (Apache License 2.0, license details)
- DACS (MIT License, license details)
- DANNet (Apache License 2.0, license details)
If you find our work helpful, please star🌟 this repo and cite📑 our paper. Thanks for your support!
@article{xie2023sepico,
title={Sepico: Semantic-guided pixel contrast for domain adaptive semantic segmentation},
author={Xie, Binhui and Li, Shuang and Li, Mingjia and Liu, Chi Harold and Huang, Gao and Wang, Guoren},
journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
year={2023},
publisher={IEEE}
}For help and issues associated with SePiCo, or reporting a bug, please open a [GitHub Issues], or feel free to contact [email protected].