Exploiting User Interaction and Object Candidates for Instance Retrieval and Object Segmentation
0 likes1,004 views
The document discusses methodologies for interactive object segmentation and instance retrieval, highlighting algorithms and user interfaces designed for effective image analysis. It covers experiments conducted using datasets and provides insights into the performance of different segmentation and retrieval techniques. The work is supported by various collaborations and contributions from the authors and sponsors.
Exploiting User Interaction and Object Candidates for Instance Retrieval and Object Segmentation
- 1. Exploiting User Interaction and Object Candidates for Instance Retrieval and Object Segmentation Amaia Salvador Aguilera Advisors: Kevin McGuinness, Xavier Giró
- 2. Motivation Object segmentation Instance Search 1
- 3. Motivation From rectangles to regions Exhaustive search Regions Object proposals 2
- 4. Motivation From rectangles to regions Exhaustive search Regions Object proposals Arbeláez, P., Pont-Tuset, J., Barron, J. T., Marques, F., & Malik, J. (2014). Multiscale Combinatorial Grouping. CVPR. 3
- 5. Motivation 4
- 6. Motivation 5
- 7. Motivation 6
- 8. Outline ●Motivation ●Interactive Object Segmentation oSegmentation algorithm oUser Interface oExperiments and Results oConclusions ●Instance Retrieval 7
- 9. Interactive Object Segmentation Amaia Salvador Xavier GiróAxel Carlier Vincent Charvillat Oge Marques Collaborationsponsoredbya scholarshipfromtheSpanishMinistry 8
- 10. Outline ●Motivation ●Interactive Object Segmentation oSegmentation algorithm oUser Interface oExperiments and Results oConclusions ●Instance Retrieval 9
- 11. Object Candidates Algorithm 1: Finding the best mask Limitation: sometimes there is no mask with no errors! 10
- 12. Object Candidates Algorithm 2: Combination of masks Union of masks containing at least one foreground point and no conflicting background points 11
- 13. Outline ●Motivation ●Interactive Object Segmentation oSegmentation algorithm oUser Interface oExperiments and Results oConclusions ●Instance Retrieval 12
- 14. Click’n’Cut 13
- 15. Outline ●Motivation ●Interactive Object Segmentation oSegmentation algorithm oUser Interface oExperiments and Results oConclusions ●Instance Retrieval 14
- 16. Dataset 100 objects from BSDS500 Dataset with their corresponding binary masks 5 control images from Pascal VOC2012 (for gold standard test) 15
- 17. Click’n’Cut and Object Candidates Crowd users Expert volunteers 16
- 18. Jaccard Index VS Time (*) BPT and GrabCut curves show the results obtained by McGuinness, K., & O’connor, N. E. in A comparative evaluation of interactive segmentation algorithms. Pattern Recognition, 43(2), 434-444. VS McGuinness(BPT) McGuinness(GrabCut) 17
- 19. Outline ●Motivation ●Interactive Object Segmentation oSegmentation algorithm oUser Interface oExperiments and Results oConclusions ●Instance Retrieval 18
- 20. Conclusions 19
- 21. Conclusions e.g. GrabCut 20
- 22. Conclusions 21
- 23. Outline ●Motivation ●Interactive Object Segmentation ●Instance Retrieval oFramework oUser interface and Relevance Feedback oExperiments and Results oConclusions 22
- 24. TRECVID Instance Search 2014 Amaia Salvador Xavier Giró Carles Ventura Eva Mohedano Kevin McGuinness Noel O’Connor ContributionsponsoredbytheCatalanGovernment 23
- 25. Outline ●Motivation ●Interactive Object Segmentation ●Instance Retrieval oFramework oUser interface oExperiments and Results oConclusions 24
- 26. Framework query / query set Representation target database Representation Feature Matching Rank List ID score query images masks 25
- 27. Framework query / query set Representation target database Representation Feature Matching Rank List ID score query images masks 26
- 28. Query Set 27
- 29. Framework query / query set Representation target database Representation Feature Matching Rank List ID score query images masks 28
- 30. Target database ●Collection of 244 videos from BBC EastEnders 29
- 31. Target database Full dataset 647,628 keyframes Ground Truth Subset 23,614 keyframes 30
- 32. Framework query / query set Representation target database Representation Feature Matching Rank List ID score query images masks 31
- 33. Convolutional Neural Networks Caffe: a CNN implementation. http://caffe.berkeleyvision.org/ Figure Source: Babenko, A., Slesarev, A., Chigorin, A., & Lempitsky, V. (2014). Neural CodesforImageRetrieval.arXivpreprintarXiv:1404.1777. 32
- 34. Framework query / query set Representation target database Representation Feature Matching Rank List ID score query images masks 33
- 35. Outline ●Motivation ●Interactive Object Segmentation ●Instance Retrieval oFramework oUser interface oExperiments and Results oConclusions 34
- 37. Outline ●Motivation ●Interactive Object Segmentation ●Instance Retrieval oFramework oUser interface oExperiments and Results oConclusions 36
- 38. 1.Local Features Global CNN features are nice, but… ...what if we use binary masks to compute local CNN features? Problem: we don’t have binary masks for the target keyframes! 37
- 39. Local Features: Object Candidates Arbeláez, P., Pont-Tuset, J., Barron, J. T., Marques, F., & Malik, J. (2014). Multiscale Combinatorial Grouping. CVPR. 38
- 40. Local features Global Local Only(square) Local Only(square+ padding) Local only(crop) Global Only Crop Square Square + padding Resultsontoysubset 39
- 41. 2.Combination of features Results on Ground Truth Datasetwith N = 20 40
- 42. 2.Combination of features Problems: ~100 object candidates/frame ~ 600,000 keyframes 41
- 43. 2.Combination of features Solution: 42
- 44. 3.Relevance Feedback Simulation of users interaction with the UI using the Ground Truth Subset Annotations from the ranking are used with Relevance Feedback techniques taking different percentages of the ranking. 43
- 45. Ranking fromannotations Positive annotationsare usedto createthenew ranking 44
- 46. Ranking fromannotations Addresultsto theranking byre-queryingusingpositive annotations 45
- 47. Ranking fromannotations Linear SVM withpositive/negativeannotations 46
- 48. 4.Performance differences 9074: a cigarette 9075: a Vodka bottle 9096: this woman Bad performance for topics containing small objects or people 47
- 49. 4.Performance differences 9090: this wooden bench 9097: these spheres 9081: a black taxi Good performance for topics for which context is important 48
- 50. Outline ●Motivation ●Interactive Object Segmentation ●Instance Retrieval oFramework oUser interface oExperiments and Results oConclusions 49
- 51. Conclusions 50
- 52. Conclusions 51
- 53. General Conclusions ObjectSegmentation InstanceSearch 52