Panoptic Segmentation
- 1. 김 성 철
- 2. Contents 1. Introduction 2. Panoptic Segmentation 3. Human Consistency Study 4. Machine Performance Baselines 5. Future of Panoptic Segmentation 2
- 3. Introduction & Related Work • Introduction • Semantic Segmentation • Stuff (such as grass, sky, road, ..) • Simply assign a class label to each pixel in an image (treat thing classes as stuff) • FCN으로 object instance를 나누기 힘듬 • Instance Segmentation • Things (such as people, animals, tools, ..) • Detect each object and delineate it with a bbox or mask • Region-based method로 overlapping을 피하기 힘듬 3
- 4. Introduction & Related Work • Introduction • Panoptic Segmentation • Encompass both stuff and thing classes → Panoptic : including every thing visible in one view • Use a simple but general output format → 각 pixel은 semantic label과 instance id를 할당받음 • Introduce a uniform evaluation metric → Semantic, Instance 모두 적용 가능한 Panoptic Quality 4
- 5. Panoptic Segmentation • Format • Task format • Semantic classes ℒ ≔ {0, … , ℒ − 1} • Each pixel 𝑖 is mapped to a pair 𝒍𝒊, 𝒛𝒊 ∈ 𝓛 × ℕ • 𝑙𝑖 : the semantic class of pixel 𝑖 • 𝑧𝑖 : its instance id. Group pixels of the same class into distinct segments • 애매하거나 out-of-class pixel들은 void label로 배정 → 모든 pixel이 semantic label을 갖는 건 아님 5
- 6. Panoptic Segmentation • Format • Stuff and thing labels • Semantic label set은 subset ℒ 𝑆𝑡와 ℒ 𝑇ℎ로 구성 • ℒ = ℒ 𝑆𝑡 ∪ ℒ 𝑇ℎ , ℒ 𝑆𝑡 ∩ ℒ 𝑇ℎ = ∅ • Pixel이 𝑙𝑖 ∈ ℒ 𝑆𝑡 로 레이블 될 때, 𝑧𝑖는 관계 없음 → 모든 stuff classes는 같은 instance를 갖기 때문 (e.g., the same sky) • 모든 pixel이 같은 𝒍𝒊, 𝒛𝒊 일 때 (𝑙𝑖 ∈ ℒ 𝑇ℎ), 같은 instance에 속함 (e.g., the same car) • Single instance에 속하는 경우 같은 𝒍𝒊, 𝒛𝒊 을 가짐 6
- 7. Panoptic Segmentation • Format • Relationship to semantic segmentation • Panoptic segmentation은 semantic segmentation의 strict generalization • 각 pixel에 semantic label을 할당하는 것은 동일, thing class가 들어가는 순간 달라짐 • Relationship to instance segmentation • Instance segmentation은 overlapping segment 발생 → Panoptic segmentation은 각 pixel에 semantic label과 instance id 하나씩 할당 → Overlapping 발생 X 7
- 8. Panoptic Segmentation • Metric • 기존 방법들은 semantic or instance segmentation에 특화 → stuff와 thing을 동시에 측정 불가 • Stuff와 thing을 동시에 측정할 수 있는 metric 제시 • Completeness : Treat stuff and thing classes in a uniform way • Interpretability : A metric with identifiable meaning • Simplicity : Simple to define and implement → Panoptic Quality (PQ) 8
- 9. Panoptic Segmentation • Metric • Segment Matching • 𝐼𝑜𝑈 𝑝𝑖, 𝑔 = |𝑝 𝑖∩𝑔| |𝑝 𝑖∪𝑔| ≤ 𝑝 𝑖∩𝑔 𝑔 𝑓𝑜𝑟 𝑖 ∈ 1, 2 • 𝑝1 ∩ 𝑝2 = ∅, |𝑝𝑖 ∪ 𝑔| ≥ |𝑔| • 𝐼𝑜𝑈 𝑝1, 𝑔 + 𝐼𝑜𝑈 𝑝2, 𝑔 ≤ 𝑝1∩𝑔 + 𝑝2∩𝑔 𝑔 ≤ 1 • 𝑝1 ∩ 𝑔 + 𝑝2 ∩ 𝑔 ≤ g • Simple and Interpretable! 9
- 10. Panoptic Segmentation • Metric • PQ Computation • Class별로 PQ 계산 후 평균 → class imbalance에 insensitive하게 만듬 • 𝑷𝑸 = σ 𝒑,𝒈 ∈𝑻𝑷 𝑰𝒐𝑼(𝒑,𝒈) 𝑻𝑷 + 𝟏 𝟐 𝑭𝑷 + 𝟏 𝟐 𝑭𝑵 • 1 |𝑇𝑃| σ 𝑝,𝑔 ∈𝑇𝑃 𝐼𝑜𝑈(𝑝, 𝑔) : the average 𝐼𝑜𝑈 of matched segments • 1 2 𝐹𝑃 + 1 2 𝐹𝑁 : to penalize segments without matches • All segments receive equal importance regardless of their area! 10
- 11. Panoptic Segmentation • Metric • PQ Computation • 𝑷𝑸 = σ 𝒑,𝒈 ∈𝑻𝑷 𝑰𝒐𝑼(𝒑,𝒈) 𝑻𝑷 + 𝟏 𝟐 𝑭𝑷 + 𝟏 𝟐 𝑭𝑵 = σ 𝒑,𝒈 ∈𝑻𝑷 𝑰𝒐𝑼(𝒑,𝒈) 𝑻𝑷 × |𝑻𝑷| 𝑻𝑷 + 𝟏 𝟐 𝑭𝑷 + 𝟏 𝟐 𝑭𝑵 (𝑺𝑸 × 𝑹𝑸) • σ 𝑝,𝑔 ∈𝑇𝑃 𝐼𝑜𝑈(𝑝,𝑔) 𝑇𝑃 : segmentation quality (SQ) the average 𝐼𝑜𝑈 of matched segments • |𝑇𝑃| 𝑇𝑃 + 1 2 𝐹𝑃 + 1 2 𝐹𝑁 : recognition quality (RQ) 𝐹1 score과 흡사 11
- 12. Panoptic Segmentation • Metric • PQ Computation • Void labels in the ground truth • Out of class pixels / ambiguous or unknown pixels → 평가하지 않음 • Ground truth에서 void labels인 pixel들은 제외하고 𝐼𝑜𝑈 계산 • Void pixel의 일부가 포함된 unmatched predicted segments는 제거 & FP로 처리X • Group labels • Cityscapes, COCO는 instance id 대신 group label을 사용 • Matching 중에는 group region 사용 X • Matching 이후, matching threshold를 넘는 same class의 일부를 포함하는 12
- 13. Panoptic Segmentation • Metric • Comparison to Existing Metrics • Semantic segmentation metrics • 𝐼𝑜𝑈 → pixel output/labels 기반 계산 & object-level labels 무시 • Instance segmentation metrics • Average Precision : Confidence score 필요 → semantic segmentation에 적합하지 않음 • Panoptic quality • Treat all classes (stuff and things) in a uniform way • SQ, RQ로 decomposing 할 수 있지만, PQ가 semantic과 instance의 metric은 아님 → 오히려 SQ와 RQ가 각각 semantic과 instance의 metric 13
- 14. Panoptic Segmentation • Datasets • Cityscapes • 5000 images (2975 / 500 / 1525) • 19 classes among which 8 have instance-level • ADE20k • 25k images (20k / 2k / 3k) • 100 thing and 50 stuff classes in the 2017 Places Challenge • Mapillary Vistas • 25k street-view images (18k / 2k / 5k) • 28 stuff and 37 thing classes in ‘research edition’ 14
- 15. Human Consistency Study • Human annotation • 하나를 ground truth, 다른 하나를 prediction으로 설정 후 PQ 측정 • Human consistency 15
- 16. Human Consistency Study • Stuff 𝒗𝒔. Things • 모든 stuff and things classes가 고르게 분포 • Small 𝒗𝒔. large objects • Small 25% | middle 50% | largest 25% 16
- 17. Human Consistency Study • 𝑰𝒐𝑼 threshold • SQ 𝒗𝒔. RQ balance • 𝑅𝑄 𝛼 = |𝑇𝑃| 𝑇𝑃 +𝛼 𝐹𝑃 +𝛼|𝐹𝑁| (default 𝛼 = 0.5) • default 𝛼 가 SQ와 RQ 사이에 균형을 맞춰 줌 • 상황에 따라 변경해주면 될 듯 17
- 18. Machine Performance Baseline • 세 가지 관점 • How do heuristic combinations of top-performing instance and semantic segmentation systems perform on panoptic segmentation? • How does PQ compare to existing metrics like 𝐴𝑃 and 𝐼𝑜𝑈? • How do the machine results compare to the human results that we presented previously? 18
- 19. Machine Performance Baseline • Algorithms and data • 각 dataset에서 algorithm output을 얻음 • Cityscapes : masks generated by PSPNet and Mask R-CNN • ADE20k : the winners of both the semantic and instance seg. in the 2017 Places Challenge • Vistas : 주최측으로 부터 1k testset에 대한 semantic & instance seg. 결과를 제공받음 19
- 20. Machine Performance Baseline • Instance segmentation • Overlapping segment를 얻음 → 해결 • Confidence score 순으로 sorting → 낮은 score는 제거 • 가장 confident한 것부터 시작해서 반복 • Remove pixels which have been assigned to previous segments • Segment의 충분한 부분이 남아있으면 non-overlapping, 그렇지 않으면 전체 segment 제거 20
- 21. Machine Performance Baseline • Semantic segmentation • No overlapping → 바로 PQ 계산 가능 • Multi-scale : skip connection (?) 21
- 22. Machine Performance Baseline • Panoptic segmentation • Instance + Semantic for resolving any overlap between thing and stuff 22
- 23. Machine Performance Baseline • Human 𝒗𝒔. machine panoptic segmentation 23
- 24. Future of Panoptic Segmentation • Stuff와 thing을 동시에 설명하는 end-to-end model • Instance segmentation에서 non-overlapping을 위한 실험들이 진행됨 • Some form of higher-level ‘reasoning’ may be beneficial • Extend learnable NMS to PS 24
- 25. Future of Panoptic Segmentation • Stuff와 thing을 동시에 설명하는 end-to-end model • Instance segmentation에서 non-overlapping을 위한 실험들이 진행됨 • Some form of higher-level ‘reasoning’ may be beneficial • Extend learnable NMS to PS 25
- 26. UPSNet (in CVPR 2019) 26
- 27. 감 사 합 니 다 27