Towards Natural Image Matting in the Wild via Real-Scenario Prior

In this paper, we propose the COCO-Matting dataset and SEMat framework to revamp training datasets, network architecture, and training objectives. Solving the problem of inappropriate synthetic training data, unaligned features and mattes from a frozen SAM, and end-to-end matting loss lacking generalization, we greatly improve the interactive matting performance on diverse datasets.

🌞 COCO-Matting

Our COCO-Matting features both natural and complex scenes. Training SEMat with COCO-Matting enables a substantial lead on HIM2K than using RefMatte and P3M-10K.

✨ SEMat

Our SEMat trained on COCO-Matting significantly outperforms the SoTA methods like MAM and SmartMat on several datasets, especially in transparency, details, and semantics.

⭐ Visual Results

🤗 Gradio Demo

Example

To use our gradio demo locally:

python demo.py

You can also try our online demo.

🛠️ Environment Setup

• Create a conda virtual env and activate it.

  conda create -n SEMat python==3.8.8
  conda activate SEMat
  

• Install packages.

  cd path/to/SEMat
  pip install -r requirements.txt
  

• Install detectron2 , follow its documentation. For instance:

  python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
  

🔩 Preparing the Testing Datasets

1. Download the testing dataset: AIM-500, RefMatte-RW100, AM-2K, P3M-500, RWP-636, and SIM

2. Modify the dataset path in line 4-12 of ./configs/semantic_enhanced_matting/dataloader.py

3. The data folder structure should look like this:

path/to/datasets
├── AIM-500
│   ├── mask
│   ├── original
│   ├── ...
├── RefMatte_RW_100
│   ├── image
│   ├── mask
│   ├── ...
├── AM-2K
│   ├── validation
│   │   ├── mask
│   │   ├── original
│   │   ├── ...
├── P3M-10k
│   ├── validation
│   │   ├── P3M-500-NP
│   │   │   ├── mask
│   │   │   ├── original_image
│   │   │   ├── ...
├── RealWorldPortrait-636
│   ├── alpha
│   ├── image
│   ├── ...
├── SIMD
│   ├── generated_testset
│   │   ├── defocus
│   │   │   ├── alpha
│   │   │   ├── merged
│   │   ├── ...

Please note that:

• The generated_testset in SIMD should be generated following SIM

▶️ Inference with SEMat

Download the trained model of SEMat (SAM) or SEMat (HQ-SAM) or SEMat (SAM2) and put them in the checkpoints folder. Then, you can inference with different SEMat versions:

Choose one testing dataset by replace [dataset] with RW100 / AIM500 / AM2K / P3M500 / RWP636 / SIM in the following commands.

1. SEMat (SAM)

python main.py --config-file configs/SEMat_SAM.py --eval_dataset [dataset] --init_from ./checkpoints/SEMat_SAM.pth --num-gpus 1 --eval-only 

2. SEMat (HQ-SAM)

python main.py --config-file configs/SEMat_HQ-SAM.py --eval_dataset [dataset] --init_from ./checkpoints/SEMat_HQ-SAM.pth --num-gpus 1 --eval-only

3. SEMat (SAM2)

python main.py --config-file configs/SEMat_SAM2.py --eval_dataset [dataset] --init_from ./checkpoints/SEMat_SAM2.pth --num-gpus 1 --eval-only

🏝️ COCO-Matting Dataset

Our COCO-Matting is available now. We only provide the concatenated trimap and alpha in COCO-Matting_trimap_alpha. Images can be downloaded from train2017.zip and unlabeled2017.zip. The image-label correspondence can be obtained from the json file we provided.

🏷️ TODO

• Hugging Face Demo.
• Release our COCO-Matting dataset.
• Testing on the HIM2K dataset.
• Release training codes.

♥️ Acknowledgements

Thanks Detectron2, ViTMatte, DiffMatte, GroundingDINO, SAM, HQ-SAM and SAM2 for their public code and released models.

✒️ Citation

If you find this project useful, please consider citing:

@article{SEMat,
  title={Towards Natural Image Matting in the Wild via Real-Scenario Prior},
  author={Xia, Ruihao and Liang, Yu and Jiang, Peng-Tao and Zhang, Hao and Sun, Qianru and Tang, Yang and Li, Bo and Zhou, Pan},
  journal={arXiv:2410.06593},
  year={2024}
}