Hierarchical Novelty Detection for Visual Object Recognition
2 likes243 views
The document proposes a hierarchical novelty detection framework that classifies inputs into known leaf classes or more general superclasses in a taxonomy. It evaluates two approaches: (1) a top-down method that performs multi-stage classification until a known leaf class or novel class is reached, and (2) a flatten method that represents all class probabilities in a single vector after adding virtual novel classes. Experimental results on ImageNet, AwA2 and CUB datasets show the combined top-down and leave-one-out method achieves the best performance in hierarchical novelty detection and generalized zero-shot learning tasks.
Hierarchical Novelty Detection for Visual Object Recognition
- 1. Hierarchical Novelty Detection for Visual Object Recognition Kibok Lee*, Kimin Lee†, Kyle Min*, Yuting Zhang*, Jinwoo Shin†, Honglak Lee*‡ University of Michigan*, KAIST†, Google Brain‡
- 2. Why Hierarchical Novelty Detection? • Conventional novelty detection framework does not provide more information than “novelty” of an object. • Suppose we have training data like … Pomeranian Welsh corgiPersian cat Siamese cat
- 3. Why Hierarchical Novelty Detection? • Then, suppose we have test images like … Test image: True label: Siamese cat Angora cat Dachshund Pika Prior works: Siamese cat novel novel novel Pomeranian Welsh corgiPersian cat Siamese cat
- 4. Why Hierarchical Novelty Detection? Ours: Siamese cat novel cat novel dog novel animal animal dogcat Pomeranian Welsh corgiPersian cat Siamese cat True label: Siamese cat Angora cat Dachshund Pika Test image: Prior works: Siamese cat novel novel novel
- 5. Why Hierarchical Novelty Detection? • Our hierarchical novelty detection framework aims to find the most specific class label of any data on the hierarchical taxonomy built with known labels. Ours: Siamese cat novel cat novel dog novel animal True label: Siamese cat Angora cat Dachshund Pika Test image: Prior works: Siamese cat novel novel novel
- 6. Why Hierarchical Novelty Detection? • This framework can be potentially useful for automatically or interactively organizing a customized taxonomy • Company’s product catalog • Wildlife monitoring • Personal photo library • by suggesting closest categories for an image from novel categories. • new consumer products • unregistered animal species • untagged scenes or places
- 7. Hierarchical Taxonomy • Our taxonomy has three types of classes. • Known leaf classes are seen during training • Super classes are ancestors of leaf classes, also known • Novel classes are unseen during training • Their expected prediction is the closest super class in the taxonomy in our task
- 8. Hierarchical Taxonomy Taxonomy of known classes Set of all known leaf classes Set of parents Set of novel classes whose closest known class is Set of children Set of known classes except and its descendants Set of ancestors including itself:
- 9. Hierarchical Taxonomy Taxonomy of known classes Set of all known leaf classes Set of parents Set of novel classes whose closest known class is Set of children Set of known classes except and its descendants Set of ancestors including itself:
- 10. Hierarchical Taxonomy Taxonomy of known classes Set of all known leaf classes Set of parents Set of novel classes whose closest known class is Set of children Set of known classes except and its descendants Set of ancestors including itself:
- 11. Hierarchical Taxonomy Taxonomy of known classes Set of all known leaf classes Set of parents Set of novel classes whose closest known class is Set of children Set of known classes except and its descendants Set of ancestors including itself:
- 12. Hierarchical Taxonomy Taxonomy of known classes Set of all known leaf classes Set of parents Set of novel classes whose closest known class is Set of children Set of known classes except and its descendants Set of ancestors including itself:
- 13. Hierarchical Taxonomy Taxonomy of known classes Set of all known leaf classes Set of parents Set of novel classes whose closest known class is Set of children Set of known classes except and its descendants Set of ancestors including itself:
- 14. Approach - Top-down (TD) Method • Multi-stage classification • Until arriving at a known leaf class .2 .8 .7 .1 .2
- 15. Approach - Top-down (TD) Method • Multi-stage classification • Until arriving at a known leaf class • Or classification is unconfident (novel) .5 .5
- 16. Approach - Top-down (TD) Method • Classification rule: • Classification is confident if
- 17. Approach - Top-down (TD) Method • Training objective:
- 18. Approach - Flatten Method • Represent all probabilities of known leaf and novel classes in a single vector • Add virtual novel classes
- 19. Approach - Flatten Method • Represent all probabilities of known leaf and novel classes in a single vector • Add virtual novel classes • And then flatten the structure
- 20. Approach - Flatten Method • Represent all probabilities of known leaf and novel classes in a single vector • Add virtual novel classes • And then flatten the structure
- 21. Approach - Flatten Method • Classification rule: • We propose two strategies to train this model.
- 22. Approach - Flatten Method • Data relabeling • Fill novel classes by hierarchical relabeling animal cat dog Persian cat Siamese cat Pomeranian Welsh corgi
- 23. Approach - Flatten Method • Data relabeling • Fill novel classes by hierarchical relabeling animal cat dog Persian cat Siamese cat Pomeranian Welsh corgi
- 24. Approach - Flatten Method • Data relabeling • Fill novel classes by hierarchical relabeling animal cat dog Persian cat Siamese cat Pomeranian Welsh corgi
- 25. Approach - Flatten Method • Data relabeling • Fill novel classes by hierarchical relabeling • Relabeling rate can be chosen by validation animal cat dog Persian cat Siamese cat Pomeranian Welsh corgi
- 26. Approach - Flatten Method • Data relabeling • Training objective: animal cat dog Persian cat Siamese cat Pomeranian Welsh corgi
- 27. Approach - Flatten Method • Leave-one-out (LOO) strategy • Generate deficient taxonomies • And then train the model with them animal cat dog Persian cat Siamese cat Pomeranian Welsh corgi
- 28. Approach - Flatten Method • Leave-one-out (LOO) strategy • e.g., when animal cat dog Persian cat Siamese cat Pomeranian Welsh corgi
- 29. Approach - Flatten Method • Leave-one-out (LOO) strategy • e.g., when animal novel animal dog Pomeranian Welsh corgi
- 30. Approach - Flatten Method • Leave-one-out (LOO) strategy • Training objective: animal novel animal dog Pomeranian Welsh corgi
- 31. Approach - Combined Method (TD+LOO) • Compute and enumerate the output of TD • And then feed it to LOO • Combined method utilizes their complementary benefits. • Top-down method leverages hierarchical structure information. • But it suffers from error aggregation over hierarchy. • Flatten method avoids error aggregation over hierarchy. • But it does not leverage hierarchical structure information.
- 32. Experiments - Hierarchical Novelty Detection • Quantitative results Method ImageNet AwA2 CUB Novel AUC Novel AUC Novel AUC DARTS 10.89 8.83 36.75 35.14 40.42 30.07 Relabel 15.29 11.51 45.71 40.28 38.23 28.75 LOO 15.72 12.00 50.00 43.63 40.78 31.92 TD+LOO 18.78 13.98 53.57 46.77 43.29 33.16
- 33. Experiments - Hierarchical Novelty Detection • Compared algorithms • Baseline: DARTS (Deng et al., 2012) • Ours: Relabel, LOO, TD+LOO Method ImageNet AwA2 CUB Novel AUC Novel AUC Novel AUC DARTS 10.89 8.83 36.75 35.14 40.42 30.07 Relabel 15.29 11.51 45.71 40.28 38.23 28.75 LOO 15.72 12.00 50.00 43.63 40.78 31.92 TD+LOO 18.78 13.98 53.57 46.77 43.29 33.16 J. Deng, J. Krause, A. C. Berg, and L. Fei-Fei. “Hedging your bets: Optimizing accuracy-specificity trade-offs in large scale visual recognition.” In CVPR, 2012
- 34. Experiments - Hierarchical Novelty Detection • Datasets • ImageNet: 1k known, 16k novel classes • AwA2: 40 known, 10 novel classes • CUB: 150 known, 50 novel classes Method ImageNet AwA2 CUB Novel AUC Novel AUC Novel AUC DARTS 10.89 8.83 36.75 35.14 40.42 30.07 Relabel 15.29 11.51 45.71 40.28 38.23 28.75 LOO 15.72 12.00 50.00 43.63 40.78 31.92 TD+LOO 18.78 13.98 53.57 46.77 43.29 33.16
- 35. Experiments - Hierarchical Novelty Detection • Metrics • Novel class accuracy @ known class accuracy = 50% • By adding an appropriate score bias to all novel classes • Area under known-novel class accuracy curve • By varying the novel class score bias Method ImageNet AwA2 CUB Novel AUC Novel AUC Novel AUC DARTS 10.89 8.83 36.75 35.14 40.42 30.07 Relabel 15.29 11.51 45.71 40.28 38.23 28.75 LOO 15.72 12.00 50.00 43.63 40.78 31.92 TD+LOO 18.78 13.98 53.57 46.77 43.29 33.16
- 36. Experiments - Hierarchical Novelty Detection • Metrics • Novel class accuracy @ known class accuracy = 50% • By adding an appropriate score bias to all novel classes • Area under known-novel class accuracy curve • By varying the novel class score bias (a) ImageNet (b) AwA2 (c) CUB
- 37. Experiments - Hierarchical Novelty Detection • Metrics • Novel class accuracy @ known class accuracy = 50% • By adding an appropriate score bias to all novel classes • Area under known-novel class accuracy curve • By varying the novel class score bias (a) ImageNet (b) AwA2 (c) CUB
- 38. Experiments - Hierarchical Novelty Detection • Metrics • Novel class accuracy @ known class accuracy = 50% • By adding an appropriate score bias to all novel classes • Area under known-novel class accuracy curve • By varying the novel class score bias (a) ImageNet (b) AwA2 (c) CUB
- 39. Experiments - Hierarchical Novelty Detection • Qualitative results Novel class: American foxhound GT foxhound
- 40. Experiments - Hierarchical Novelty Detection • Qualitative results Novel class: American foxhound Method ε A Word GT foxhound DARTS 2 N beagle Relabel 1 Y hound dog LOO 0 Y foxhound TD+LOO 0 Y foxhound
- 41. Experiments - Hierarchical Novelty Detection • Qualitative results Novel class: serval Method ε A Word GT wildcat DARTS 3 N Egyptian cat Relabel 2 N domestic cat LOO 2 Y feline TD+LOO 1 Y cat
- 42. Experiments - Hierarchical Novelty Detection • Qualitative results Novel class: song thrush Method ε A Word GT thrush DARTS 3 N hummingbird Relabel 2 Y bird LOO 1 Y oscine bird TD+LOO 0 Y thrush
- 43. Experiments - Hierarchical Novelty Detection • Qualitative results Novel class: ice-cream sundae Method ε A Word GT frozen dessert DARTS 4 Y food, nutrient Relabel 1 N ice cream LOO 1 Y dessert TD+LOO 0 Y frozen dessert
- 44. Experiments - Generalized Zero-Shot Learning • Quantitative results Embedding AwA1 AwA2 CUB Att Word Hier Unseen AUC Unseen AUC Unseen AUC √ 65.29 50.02 63.87 51.27 50.05 23.60 √ 51.87 39.67 54.77 42.21 27.28 11.47 √ √ 67.80 52.84 65.76 53.18 49.83 24.13 Path 42.57 30.58 44.34 33.44 24.22 8.38 √ Path 67.09 51.45 66.58 53.50 50.25 23.70 √ Path 52.89 40.66 55.28 42.86 27.72 11.65 √ √ Path 68.04 53.21 67.28 54.31 50.87 24.20 TD 33.86 25.56 31.84 24.97 13.09 7.20 √ TD 66.13 54.66 66.86 57.49 50.17 30.31 √ TD 56.14 46.28 59.67 49.39 29.05 16.73 √ √ TD 69.23 57.67 68.80 59.24 50.17 30.31
- 45. Experiments - Generalized Zero-Shot Learning • Semantic embeddings • Attributes (numeric attribute values) • Word vector (similarity among words in real coordinate space) • Hierarchical embedding Embedding AwA1 AwA2 CUB Att Word Hier Unseen AUC Unseen AUC Unseen AUC √ 65.29 50.02 63.87 51.27 50.05 23.60 √ 51.87 39.67 54.77 42.21 27.28 11.47 √ √ 67.80 52.84 65.76 53.18 49.83 24.13 Path 42.57 30.58 44.34 33.44 24.22 8.38 √ Path 67.09 51.45 66.58 53.50 50.25 23.70 √ Path 52.89 40.66 55.28 42.86 27.72 11.65 √ √ Path 68.04 53.21 67.28 54.31 50.87 24.20 TD 33.86 25.56 31.84 24.97 13.09 7.20 √ TD 66.13 54.66 66.86 57.49 50.17 30.31 √ TD 56.14 46.28 59.67 49.39 29.05 16.73 √ √ TD 69.23 57.67 68.80 59.24 50.17 30.31
- 46. Experiments - Generalized Zero-Shot Learning • Compared hierarchical embeddings • Baseline: Path (Akata et al., 2015) • Distance between classes on hierarchy • Ours: Top-down (TD) • Expected output of our top-down model Embedding AwA1 AwA2 CUB Att Word Hier Unseen AUC Unseen AUC Unseen AUC √ 65.29 50.02 63.87 51.27 50.05 23.60 √ 51.87 39.67 54.77 42.21 27.28 11.47 √ √ 67.80 52.84 65.76 53.18 49.83 24.13 Path 42.57 30.58 44.34 33.44 24.22 8.38 √ Path 67.09 51.45 66.58 53.50 50.25 23.70 √ Path 52.89 40.66 55.28 42.86 27.72 11.65 √ √ Path 68.04 53.21 67.28 54.31 50.87 24.20 TD 33.86 25.56 31.84 24.97 13.09 7.20 √ TD 66.13 54.66 66.86 57.49 50.17 30.31 √ TD 56.14 46.28 59.67 49.39 29.05 16.73 √ √ TD 69.23 57.67 68.80 59.24 50.17 30.31 Z. Akata, S. Reed, D. Walter, H. Lee, and B. Schiele. “Evaluation of output embeddings for fine-grained image classification.” In CVPR, 2015.
- 47. Experiments - Generalized Zero-Shot Learning • Datasets • AwA1,2: 40 known, 10 novel classes • CUB: 150 known, 50 novel classes Embedding AwA1 AwA2 CUB Att Word Hier Unseen AUC Unseen AUC Unseen AUC √ 65.29 50.02 63.87 51.27 50.05 23.60 √ 51.87 39.67 54.77 42.21 27.28 11.47 √ √ 67.80 52.84 65.76 53.18 49.83 24.13 Path 42.57 30.58 44.34 33.44 24.22 8.38 √ Path 67.09 51.45 66.58 53.50 50.25 23.70 √ Path 52.89 40.66 55.28 42.86 27.72 11.65 √ √ Path 68.04 53.21 67.28 54.31 50.87 24.20 TD 33.86 25.56 31.84 24.97 13.09 7.20 √ TD 66.13 54.66 66.86 57.49 50.17 30.31 √ TD 56.14 46.28 59.67 49.39 29.05 16.73 √ √ TD 69.23 57.67 68.80 59.24 50.17 30.31
- 48. Experiments - Generalized Zero-Shot Learning • Metrics • Unseen class accuracy (ZSL) • Area under seen-unseen curve (GZSL) • By varying the unseen class score bias Embedding AwA1 AwA2 CUB Att Word Hier Unseen AUC Unseen AUC Unseen AUC √ 65.29 50.02 63.87 51.27 50.05 23.60 √ 51.87 39.67 54.77 42.21 27.28 11.47 √ √ 67.80 52.84 65.76 53.18 49.83 24.13 Path 42.57 30.58 44.34 33.44 24.22 8.38 √ Path 67.09 51.45 66.58 53.50 50.25 23.70 √ Path 52.89 40.66 55.28 42.86 27.72 11.65 √ √ Path 68.04 53.21 67.28 54.31 50.87 24.20 TD 33.86 25.56 31.84 24.97 13.09 7.20 √ TD 66.13 54.66 66.86 57.49 50.17 30.31 √ TD 56.14 46.28 59.67 49.39 29.05 16.73 √ √ TD 69.23 57.67 68.80 59.24 50.17 30.31
- 49. Experiments - Generalized Zero-Shot Learning • Metrics • Unseen class accuracy (ZSL) • Area under seen-unseen curve (GZSL) • By varying the unseen class score bias (a) AwA1 (b) AwA2 (c) CUB
- 50. Experiments - Generalized Zero-Shot Learning • Metrics • Unseen class accuracy (ZSL) • Area under seen-unseen curve (GZSL) • By varying the unseen class score bias (a) AwA1 (b) AwA2 (c) CUB
- 51. Experiments - Generalized Zero-Shot Learning • Metrics • Unseen class accuracy (ZSL) • Area under seen-unseen curve (GZSL) • By varying the unseen class score bias (a) AwA1 (b) AwA2 (c) CUB
- 52. Conclusion • We propose a hierarchical novelty detection framework, which aims to find the most specific class label of any data on the hierarchical taxonomy built with known labels. • We propose two approaches. • Top-down method performs classification & novelty detection hierarchically. • Flatten method computes single probability vector of all candidate classes. • Their combination takes advantage of their complementary benefits. • Our model can be combined with other semantic embeddings to improve generalized zero-shot learning performance.