Awesome-Panoptic-Segmentation

This repo is a collection of the challenging panoptic segmentation, including papers, codes, and benchmark results, etc.

Outline

• Panoptic Segmentation
• Datasets
• Evaluation
• Benchmark Results
• Papers
• Tutorials
• Blogs

Panoptic Segmentation

Summarize in one sentence : Panoptic Segmentation proposes to solve the semantic segmentation(*Stuff*) and instance segmentation(*Thing*) in a unified and general manner.

Structure Overview

from UPSNet.

Datasets

Generally, the datasets which contains both semantic and instance annotations can be used to solve the challenging panoptic task.

• COCO-Panoptic
• Cityscapes
• Mapillary Vistas
• ADE20K
• IDD20K

Evaluation

Metrics

• PQ are the standard metrics described in Panoptic Segmentation.

• PC are the standard metrics described in DeeperLab.

Evaluation Code

• cocodataset/panopticapi
• mcordts/cityscapesScripts

Competition

• AutoNUE 2019 Panoptic Segmentation Challenge (ICCV 2019 Workshop, Closed)
• COCO 2019 Panoptic Segmentation Task (ICCV 2019 Workshop, Closed)
• Mapillary 2019 Panoptic Segmentation Task (ICCV 2019 Workshop, Closed)
• Cityscapes Panoptic Semantic Labeling Task (Open)
• COCO 2018 Panoptic Segmentation Task (ECCV 2018 Workshop, Closed)
• Mapillary Vistas 2018 Panoptic Segmentation Task (ECCV 2018 Workshop, Closed)

Benchmark Results

COCO val Benchmark

Method	Backbone	PQ	PQ-Thing	PQ-Stuff	SQ	RQ	mIoU	AP-Mask	PC	e2e
SOGNet	ResNet-50	43.7	50.6	33.2	78.7	53.5	54.56	34.2	-	✅
UPSNet	ResNet-50	42.5	48.6	33.4	-	-	54.3	34.3	-	✅
OANet	ResNet-101	41.3	50.4	27.7	-	-	-	-	-	✅
OCFusion	ResNet-50	41.0	49.0	29.0	77.1	50.6	-	-	-	✅
Panoptic FPN	ResNet-101	40.9	48.3	29.7	-	-	-	-	-	✅
AUNet	ResNet-50	39.6	49.1	25.2	-	-	45.1	34.7	-	✅
AdaptIS	ResNet-101	37.0	41.8	29.9	-	-	-	-	-	✅
DeeperLab	Xception-71	34.3	37.5	29.6	77.1	43.1	-	-	56.8	✅

Cityscapes valBenchmark

Method	Backbone	PQ	PQ-Thing	PQ-Stuff	SQ	RQ	mIoU	AP-Mask	PC	e2e
Panoptic(Merge)	-	61.2	66.4	54.0	80.9	74.4	-	-	-	❎
AdaptIS	ResNet-101	60.6	58.7	64.4	-	-	79.2	36.3	-	✅
SOGNet	ResNet-50	60.0	56.7	62.5	-	-	-	-	-	✅
Seamless	ResNet-50	59.8	53.4	64.5	-	-	75.4	31.9	-	✅
UPSNet	ResNet-50	59.3	54.6	62.7	79.7	73.0	75.2	33.3	-	✅
TASCNet	ResNet-101	59.2	56	61.5	-	-	77.8	37.6	-	✅
AUNet	ResNet-101	59.0	54.8	62.1	-	-	75.6	34.4	-	✅
Panoptic FPN	ResNet-101	58.1	52.0	62.5	-	-	75.7	33.0	-	✅
DeeperLab	Xception-71	56.5	-	-	-	-	-	-	75.6	✅

Mapillary val Benchmark

Method	Backbone	PQ	PQ-Thing	PQ-Stuff	SQ	RQ	mIoU	AP-Mask	PC	e2e
Panoptic(Merge)	-	38.3	41.8	35.7	73.6	47.7	-	-	-	❎
Seamless	ResNet-50	37.2	33.2	42.5	-	-	50.2	16.3	-	✅
AdaptIS	ResNet-101	33.4	28.3	40.3	-	-	-	-	-	✅
TASCNet	ResNet-101	32.6	31.3	34.4	-	-	35.0	18.5	-	✅
DeeperLab	Xception-71	32.0	-	-	-	-	-	-	55.3	✅

Papers

AAAI2020

• SOGNet: Yibo Yang, Hongyang Li, Xia Li, Qijie Zhao, Jianlong Wu, Zhouchen Lin.
  "SOGNet: Scene Overlap Graph Network for Panoptic Segmentation." AAAI (2020). [paper]

ICCV2019

• AdaptIS: Konstantin Sofiiuk, Olga Barinova, Anton Konushin.
  "AdaptIS: Adaptive Instance Selection Network." ICCV (2019). [paper]

• Cheng-Yang Fu, Tamara L. Berg, Alexander C. Berg.
  "IMP: Instance Mask Projection for High Accuracy Semantic Segmentation of Things." ICCV (2019). [paper]

• Bowen Cheng, Maxwell D. Collins, Yukun Zhu, Ting Liu, Thomas S. Huang, Hartwig Adam, Liang-Chieh Chen.
  "Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation Bowen." ICCVW (2019). [paper]

CVPR2019

• Panoptic Segmentation: Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, Piotr Dollár.
  "Panoptic Segmentation." CVPR (2019). [paper]

• Panoptic FPN: Alexander Kirillov, Ross Girshick, Kaiming He, Piotr Dollár.
  "Panoptic Feature Pyramid Networks." CVPR (2019 oral). [paper] [unofficial code][detectron2]

• AUNet: Yanwei Li, Xinze Chen, Zheng Zhu, Lingxi Xie, Guan Huang, Dalong Du, Xingang Wang.
  "Attention-guided Unified Network for Panoptic Segmentation." CVPR (2019). [paper]

• UPSNet: Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, Raquel Urtasun.
  "UPSNet: A Unified Panoptic Segmentation Network." CVPR (2019 oral). [paper] [code]

• DeeperLab: Tien-Ju Yang, Maxwell D. Collins, Yukun Zhu, Jyh-Jing Hwang, Ting Liu, Xiao Zhang, Vivienne Sze, George Papandreou, Liang-Chieh Chen.
  "DeeperLab: Single-Shot Image Parser." CVPR (2019 oral). [paper] [project] [code]

• OANet: Huanyu Liu, Chao Peng, Changqian Yu, Jingbo Wang, Xu Liu, Gang Yu, Wei Jiang.
  "An End-to-End Network for Panoptic Segmentation." CVPR (2019). [paper]

• Eirikur Agustsson, Jasper R. R. Uijlings, Vittorio Ferrari .
  "Interactive Full Image Segmentation by Considering All Regions Jointly." CVPR (2019). [paper]

• Seamless: Lorenzo Porzi, Samuel Rota Bulo, Aleksander Colovic, Peter Kontschieder.
  "Seamless Scene Segmentation." CVPR (2019) (Extended Version). [paper][code]

ECCV2018

• Qizhu Li, Anurag Arnab, Philip H.S. Torr.
  "Weakly- and Semi-Supervised Panoptic Segmentation." ECCV (2018). [paper] [code]

ArXiv

• Rui Hou, Jie Li, Arjun Bhargava, Allan Raventos, Vitor Guizilini, Chao Fang, Jerome Lynch, Adrien Gaidon.
  "Real-Time Panoptic Segmentation from Dense Detections." arXiv (2019). [paper]

• Mark Weber, Jonathon Luiten, Bastian Leibe.
  "Single-Shot Panoptic Segmentation." arXiv (2019). [paper]

• Qiang Chen, Anda Cheng, Xiangyu He, Peisong Wang, Jian Cheng.
  "SpatialFlow: Bridging All Tasks for Panoptic Segmentation." arXiv (2019). [paper]

• Sagi Eppel, Alan Aspuru-Guzik.
  "Generator evaluator-selector net: a modular approach for panoptic segmentation." arXiv (2019). [paper]

• Jasper R. R. Uijlings, Mykhaylo Andriluka, Vittorio Ferrari.
  "Panoptic Image Annotation with a Collaborative Assistant." arXiv (2019). [paper]

• OCFusion: Justin Lazarow, Kwonjoon Lee, Zhuowen Tu.
  "Learning Instance Occlusion for Panoptic Segmentation." arXiv (2019). [paper]

• PEN: Yuan Hu, Yingtian Zou, Jiashi Feng.
  "Panoptic Edge Detection." arXiv (2019). [paper]

• TASCNet: Jie Li, Allan Raventos, Arjun Bhargava, Takaaki Tagawa, Adrien Gaidon.
  "Learning to Fuse Things and Stuff." arXiv (2018). [paper]

• Daan de Geus, Panagiotis Meletis, Gijs Dubbelman.
  "Panoptic Segmentation with a Joint Semantic and Instance Segmentation Network." arXiv (2018). [paper]

• Daan de Geus, Panagiotis Meletis, Gijs Dubbelman.
  "Single Network Panoptic Segmentation for Street Scene Understanding." arXiv (2019). [paper]

• David Owen, Ping-Lin Chang.
  "Detecting Reflections by Combining Semantic and Instance Segmentation." arXiv (2019). [paper]

• Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji.
  "PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things." arXiv (2019, IROS). [paper]

Tutorials

• CVPR 2019 Tutorial on Visual Recognition and Beyond. [slides] [homepage]
• COCO 2017 Workshop. [slides]

Blogs

• Megvii(Face++) Detection Team. [zhihu]