Co-filtering human interaction and object segmentation
1 like1,304 views
The document discusses different techniques for filtering human interactions in crowdsourced object segmentation tasks. It examines removing users based on error rates or Jaccard index, filtering clicks in the same superpixel, and using a foreground map algorithm to take advantage of all user interactions. The document also explores automatically categorizing users into types like painters, experts, or spammers based on analyzed click patterns. Key findings include Jaccard index being a better measure than error rate to filter users, and that partial click filtering outperforms total click filtering when combined with user filtering.
Co-filtering human interaction and object segmentation
- 1. Co-filtering human interaction and object segmentation Ferran Cabezas Supervised by: Vincent Charvillat Axel Carlier Xavier Giró-i-Nieto Amaia Salvador 1
- 2. 1. Motivation 2. Related Work 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates b) Taking advantage of all human interaction - Foreground map algorithm 4. Automatic categorization of the users 5. Conclusions 6. Future work Outline 2
- 4. Filtering out bad human interactions Correct human interaction GoalResult of a correct human interaction Result of an incorrect human interaction Incorrect human interaction 4
- 5. 1. Motivation 2. Related Work 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates b) Taking advantage of all human interaction - Foreground map algorithm 4. Automatic categorization of the users 5. Conclusions 6. Future work Outline 5
- 6. Click’n’Cut • Web tool for interactive object segmentation designed for crowdsourcing tasks. A. Carlier, V. Charvillat, A.Salvador, X.Giró-i-Nieto, O. Marques, Click’n’Cut: Crowdsourced Interactive Segmentation with Object Candidates. In CrowdMM’14, 2014 DEMO 6
- 7. Data 20 users that have fully realized the Click’n’Cut experiment 100 objects with associated ground truth from the Berkeley-DCU dataset. Testing set 5 images from Pascal VOC 2012 to perform gold standard techniques. Training set Training set 7
- 8. How are obtained the masks from the clicks? • Combination of different precomputed binary object candidates . • Foreground map algorithm ? A.Carlier, Combining Content Analysis with Usage Analysis to better understand visual contents, PHD Thesis, 2014. A. Carlier, V. Charvillat, A.Salvador, X.Giró-i-Nieto, O. Marques, Click’n’Cut: Crowdsourced Interactive Segmentation with Object Candidates. In CrowdMM’14, 2014 8
- 9. Information of users are not always reliable Bad user interaction Good user interaction 9
- 10. First approach - How are separated good from bad user interactions? 4th GS1st GS Error rate Error rate Error rate Error rate Error rate 2nd GS 3rd GS 5th GS Mean error rate • Removing users based on their error rate on the Gold standard images (training set) 10
- 11. Removing users based on their error rate Remove users based on an error rate threshold 5GS User20 5GS User18 5GS User19 . . . 5GS User3 5GS User1 5GS User2 Error rate Error rate Error rate Error rate Error rate Error rate 11
- 12. 1. Motivation 2. Related Work 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates b) Taking advantage of all human interaction - Foreground map algorithm 4. Automatic categorization of the users 5. Conclusions 6. Future work Outline 12
- 13. How are evaluated the obtained masks? clicks Object candidate technique Ground truth mask ? ? Foreground map algorithm 13
- 14. Jaccard index A ∪ B A ∩ B Measure of similarity between the mask obtained from the Click’n’Cut experiment and the ground truth mask 14
- 15. 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates • Removing users • Removing clicks • Removing clicks and users Outline 15
- 16. Impact of good and bad users in the resulting mask Image 1 user (good user) Image 12 users (Good users) • A lot of errors can be removed just by discarding bad users Image 20 users 16
- 17. Jaccard index= 0.0214 Error rate = 0 Jaccard index= 0.9402 Error rate = 0 Users filtering NO OBVIOUS CORRELATION 17
- 18. Jaccard index for each user 4th GS1st GS Jaccard index Jaccard index Jaccard index Jaccard index Jaccard index 2nd GS 3rd GS 5th GS Mean Jaccard index • Better idea of how it is the contribution of the user in the final result 18
- 19. Jaccard index for each user 5GS User20 5GS User18 5GS User19 . . . 5GS User3 5GS User1 5GS User2 Jaccard index Jaccard index Jaccard index Jaccard index Jaccard index Jaccard index Remove users based on a Jaccard index threshold 19
- 20. Image 100 Jaccard index 100 Image 1 Jaccard index 1 Image 2 Jaccard index 2 Image 3 Jaccard index 3 Image 98 Jaccard index 98 Image 99 Jaccard index 99 MEAN Jaccard index for the test set . . . Maintained users Removed users 20
- 21. Results for the test set 0 2 4 6 8 10 12 14 16 18 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of users Jaccard index by taking different number of users JaccardIndex Users sorted by its ascendent Jaccard index Users sorted by its descendent error rate descendent ascendant 21
- 22. 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates • Removing users • Removing clicks • Removing clicks and users Outline 22
- 23. Schematic Combination of Object Candidates Image with filtered clicks Obtaining mask Slic Felzenszwalb N-cuts nothin g Three different techniques for over- segment an image Two techniques for discarding the clicks in a same superpixel Image with non filtered clicks 23
- 24. Schematic Combination of Object Candidates Image with filtered clicks Obtaining mask Slic Felzenszwalb N-cuts nothing Three different techniques for over- segment an image Two techniques for discarding the clicks in a same superpixel Image with non filtered clicks 24
- 25. Superpixel techniques Three different techniques for over- segment an image Two techniques for discarding the clicks in a same superpixel Combination of Object Candidates Slic Felzenszwalb N-cuts nothing Image with filtered clicks Obtaining mask 25
- 26. Superpixel techniques • Felzenszwalb • K = 20 • σ = 0,5 • m = 20 • SLIC • Region size = 10 • Regularizer = 0.1• N-cuts 26
- 27. Filtering Clicks in a same superpixel Three different techniques for over- segment an image Two techniques for discarding the clicks in a same superpixel Combination of Object Candidates Slic Felzenszwalb N-cuts nothing Image with filtered clicks Obtaining mask 27
- 28. Filtering Clicks in a same superpixel 1) Total removal of conflict clicks : Discarding all clicks in conflicting superpixels 2) Partial removal of conflict clicks : Discarding the clicks in minority /equality inside conflicting superpixels nothingnothing 28
- 29. Results Without applying any technique of filtering clicks 0.14 Techniques of filtering clicks in a same sppxl. Partial removal of conflict clicks Total removal of conflict clicks SLIC 0.2109 0.2412 N-CUTS 0.2735 0.3330 FELZ 0.2104 0.2240 • Jaccard index for all users in the test set 29
- 30. 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates • Removing users • Removing clicks • Removing clicks and users Outline 30
- 31. Results • Users sorted by its descendent Jaccard index 0 2 4 6 8 10 12 14 16 18 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of Users sorted by its descended Jaccard index JaccardIndex Comparing results with partial filtering and without filtering Felz. sppxl. technique Ncuts spxxl. technique SLIC spxxl. technique With no filtering clicks 0 2 4 6 8 10 12 14 16 18 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of Users sorted by its descended Jaccard indexJaccardIndex Comparing results with total filtering and without filtering Felz. sppxl. technique Ncuts spxxl. technique SLIC spxxl. technique With no filtering clicks Partial filtering Total filtering 31
- 32. 3. Treatment of human interaction b) Taking advantage of all human interaction - Foreground map algorithm Outline 32
- 33. Foreground map algorithm Set of clicks 50 100 150 200 250 300 350 400 450 50 100 150 200 250 300 50 100 150 200 250 300 350 400 450 50 100 150 200 250 300 Felzenzwalb Superpixel segmentation with k=100 Felzenzwalb Superpixel segmentation with k=300 • Each click have a measure of confidence based on the user error on the 5GS. • Weight superpixel based on clicks 33
- 34. Foreground map algorithm • Superpixel combination • Slic: 6 levels • Felzenzwalb: 8 levels . . . . . . R.Vieux, J.Benois, J.Domenger, A.Braquelaire, Segmentation-based multi-class semantic object detection, Multimedia Tools and Applications, 2010 34
- 35. Parameters to adjust after the combination • Threshold • Structure element for hole filling ? ? 35
- 36. Combining all Felz. and Slic levels Threshold 0.56 Jaccard index = 0.8603 • Felz: k: 10,20,50,100,200,300,400,500 • SLIC: Regions side: 5,10,20,30,40,50 • SE =7 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 X: 0.56 Y: 0.8891 Threshold JaccardIndex Combining Slic and Felzenzwalb superpixels techniques in the train set 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 X: 0.56 Y: 0.8603 Threshold JaccardIndex Combining Slic and Felzenzwalb superpixels techniques in the test set 36
- 37. Results combining all Felz. and Slic levels Threshold = 0.56 SE = 7 37
- 38. 1. Motivation 2. Related Work 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates b) Taking advantage of all human interaction - Foreground map algorithm 4. Automatic categorization of the users 5. Conclusions 6. Future work Outline 38
- 39. Type of users and their particularities • Painter: Lot of foreground clicks inside the object to segment 39
- 40. Type of users and their particularities • Tired: Few clicks per image 40
- 41. Type of users and their particularities • Border guards: Most of the bg clicks are in the contour of the image. 41
- 42. Type of users and their particularities • Surrounders: Most of the fg clicks are in the contour of the image 42
- 43. Type of users and their particularities • Mirrors: Have understood the experiment upside-down 43
- 44. Type of users and their particularities • Spammers: Randomly placed foreground clicks over the image. 44
- 45. Type of users and their particularities • Experts: Have well-understood the experiment and just made few mistakes 45
- 46. Type of users and their particularities • Different pattern: Does not follow the same pattern of clicks in all images 46
- 47. Manually categorization • It is done a manually categorization by considering just the 5 gold standard images Users Manually categorization 1 Painter 2 Expert 3 Mirror 4 Expert 5 Border guard 6 Expert 7 Tired 8 Border guard 9 Expert 10 Different pattern 11 Different pattern 12 Expert 13 Expert 14 Expert 15 Expert 16 Expert 17 Tired 18 Surrounder 19 Spammer 20 Expert 47
- 48. Manual rules for automatic user categorization Features Painter The mirror The border guard The surrounder The spammer The tired The expert # clicks >150/image - - - - <5/image - fg clicks(%) >95% - <20% >95% >90% - - errors(%) <3% >90% - - >40% <20% - Jaccard index (%) - <10% - - - <80% >80% Contour fg(%) (fg contour clicks/total fg clicks) - - - >80% <80% - - Contour bg(%) (bg contour clicks/total bg clicks) - - >70% - - - - • According to the particularities of each type of user, a set of features and its rules are created: 48
- 49. Automatic categorization evaluation for the test set Prediction Painter Mirror Expert Spammer Surrounder Border Guard Tired Diff. Pattern Ground Truth Painter 1 0 0 0 0 0 0 0 Mirror 0 1 0 0 0 0 0 0 Expert 0 0 9 0 0 0 0 1 Spammer 0 0 0 1 0 0 0 0 Surrounder 0 0 0 0 1 0 0 0 Border guard 0 0 0 0 0 1 0 1 Tired 0 0 0 0 0 0 1 1 Diff. pattern 0 0 0 0 0 0 0 2 49
- 50. 1. Motivation 2. Related Work 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates b) Taking advantage of all human interaction - Foreground map algorithm 4. Automatic categorization of the users 5. Conclusions 6. Future work Outline 50
- 51. Conclusions • Jaccard index is a better measure compared to error rate to separate bad users from good ones 0 2 4 6 8 10 12 14 16 18 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of users Jaccard index by taking different number of users JaccardIndex Users sorted by its ascendent Jaccard index Users sorted by its descendent error rate 51
- 52. Conclusions • Better results with partial than with total filtering • Filtering clicks only makes sense when treating with bad users 0 2 4 6 8 10 12 14 16 18 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of Users sorted by its descended Jaccard index JaccardIndex Comparing results with partial filtering and without filtering Felz. sppxl. technique Ncuts spxxl. technique SLIC spxxl. technique With no filtering clicks 0 2 4 6 8 10 12 14 16 18 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of Users sorted by its descended Jaccard index JaccardIndex Comparing results with total filtering and without filtering Felz. sppxl. technique Ncuts spxxl. technique SLIC spxxl. technique With no filtering clicks Partial filtering Total filtering 52
- 53. Conclusions • In the foreground map algorithm it is reached the best result by combining Felzenzwalb and Slic superpixel techniques with different levels 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 X: 0.56 Y: 0.8891 Threshold JaccardIndex Combining Slic and Felzenzwalb superpixels techniques in the train set 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 X: 0.56 Y: 0.8603 Threshold JaccardIndex Combining Slic and Felzenzwalb superpixels techniques in the test set 53
- 54. Conclusions Images from User 11 • It is not possible to automatically categorize users that does not follow the same pattern of clicks in all images 54
- 55. 1. Motivation 2. Related Work 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates b) Taking advantage of all human interaction - Foreground map algorithm 4. Automatic categorization of the users 5. Conclusions 6. Future work Outline 55
- 56. Future work • Study different techniques for filtering clicks in a same superpixel. • Take advantage of the clicks of some users to create a better mask (e.g. Border guard and Surrounder users) • Train classifier for automatic user categorization 56