Graph-based multimodal clustering for social event detection in large collections of images
Download as PPT, PDF7 likes2,086 views
The document presents a scalable graph-based multimodal clustering approach for social event detection (SED) leveraging an item-to-item model. It evaluates the approach using a dataset from the 2012 MediaEval SED task, demonstrating high accuracy and clustering quality, while also discussing the importance of time and other features. Future work includes improving methods for handling non-event images and exploring multimedia data quality.
![Evaluation: SE accuracy & clustering quality
• Same event classification accuracy 98.58% (SVM)
– 10K pos/neg training, 10K pos/neg testing (random)
• Clustering quality (NMI): 30/119 training/testing events [10 random splits]
– Incremental same or better than batch
– Item-to-item better than item-to-cluster (significant at 0.95 confidence)
BATCH
INCREMENTAL
ITEM-TO-CLUSTER
AVG
0.924
0.934
0.898
STD
0.019
0.021
0.027
• When non-event photos enter the dataset, NMI degrades quickly
NON-EVENT
BATCH
INCREMENTAL
ITEM-TO-CLUSTER
5%
0.4824
0.5164
0.3954
10%
0.3421
0.3683
0.2899
*
* In the second table, results were obtained using sed2011 for training and sed2012 for testing.
MMM 2014
Georgios Petkos et al.](https://image.slidesharecdn.com/mmm2014presentation-140106072830-phpapp01/85/Graph-based-multimodal-clustering-for-social-event-detection-in-large-collections-of-images-16-320.jpg)
Graph-based multimodal clustering for social event detection in large collections of images
- 1. Graph-based multimodal clustering for social event detection in large collections of images Georgios Petkos, Symeon Papadopoulos, Emmanouil Schinas, Yiannis Kompatsiaris Information Technologies Institute (ITI) Centre for Research & Technologies Hellas (CERTH) MMM 2014
- 2. Overview • The problem of social event detection • Existing approaches • Proposed approach • Evaluation • Summary & future work MMM 2014 #2 Georgios Petkos et al.
- 3. the problem MMM 2014 Georgios Petkos et al.
- 4. Social events? Attended by people and represented by multimedia content shared online news demonstration / riot / speech personal wedding / birthday / drinks entertainment concert / play / sports MMM 2014 #4 Georgios Petkos et al.
- 5. Pope Benedict 2007: iPhone release 2008: Android release 2010: iPad release Pope Francis http://petapixel.com/2013/03/14/a-starry-sea-of-cameras-at-the-unveiling-of-pope-francis/ MMM 2014 #5 Georgios Petkos et al.
- 6. Social event detection Social event detection involves the automatic organization of a multimedia collection C into groups of items, each (group) of which corresponds to a distinct event. Can be treated as a multimodal clustering problem COLLECTION EVENT SET E1 EVENT DETECTION E2 EN MMM 2014 #6 Georgios Petkos et al.
- 7. existing approaches MMM 2014 Georgios Petkos et al.
- 8. Supervised event detection • Rationale: use a large number of “known” event assignments to “learn” how to identify “same event” / “same cluster” relationships Two variants: • Item-to-item: learn whether two items belong to the same event cluster or not. – Model Input: the set of per modality distances between two images. • Item-to-cluster: learn whether a new item belongs to a given event cluster or not. – Model input: the set of per modality distances between an image and a prototype representation of the event. MMM 2014 #8 Georgios Petkos et al.
- 9. Utilizing the “same event” model for clustering • Item-to-item: – (Incremental). For each incoming image, average all item-to-item SE scores for all items in each cluster. Assign to best-matching cluster if average above threshold or create new cluster (Becker et. al.). – (Batch). Compute all item-item SE scores between each image and all other images and form an indicator vector. Cluster indicator vectors (Petkos et. al.). • Item-to-cluster: – (Incremental). For each cluster maintain a multimodal representation. Compute SE score between each incoming item and the existing prototype event representations. Assign to best-matching cluster if above threshold or create new cluster (Becker et. al). Alternatively use a second model for deciding if a new cluster should be added or not (Reuter et. al.). MMM 2014 #9 Georgios Petkos et al.
- 10. proposed approach MMM 2014 Georgios Petkos et al.
- 11. Overview of proposed approach • Item-to-item SE model utilized. • Candidate neighbours selection step (first appears in (Reuter et. al)) using a set of per modality indexes. • Graph representation. • Community detection on graph. Two variants of the algorithm: • Batch: SCAN • Incremental: QCA MMM 2014 #11 Georgios Petkos et al.
- 12. Proposed approach: advantages • Item-to-cluster methods may suffer from incorrect prototype representations (due to averaging). • Candidate neighbours selection step makes the application of the method much more scalable. • Graph representation: in order to introduce a scalable item-to-item approach without averaging. MMM 2014 #12 Georgios Petkos et al.
- 13. evaluation MMM 2014 Georgios Petkos et al.
- 14. Evaluation setup • Used the dataset of the 2012 SED task of MediaEval • Ground truth: 7,779 photos clustered around 149 events (18 technical, 79 soccer, 52 Indignados) • Assess the following aspects: – accuracy of same-event classification – compare clustering quality between item-to-cluster and the two versions of item-to-item (batch & incremental) – measure contributions of different features – study generalization abilities of same event model MMM 2014 Georgios Petkos et al.
- 15. Evaluation setup Features: • Uploader identity. • Actual image content: – GIST – SURF, aggregated using the VLAD scheme • Textual features: title, description and tags. Either a TF-IDF or a BM25 weighting scheme is utilized. • Time of media creation. • Location, when available (geodesic distance). Appropriate indices are utilized in order to rapidly fetch the candidate neighbours for each modality. MMM 2014 Georgios Petkos et al.
- 16. Evaluation: SE accuracy & clustering quality • Same event classification accuracy 98.58% (SVM) – 10K pos/neg training, 10K pos/neg testing (random) • Clustering quality (NMI): 30/119 training/testing events [10 random splits] – Incremental same or better than batch – Item-to-item better than item-to-cluster (significant at 0.95 confidence) BATCH INCREMENTAL ITEM-TO-CLUSTER AVG 0.924 0.934 0.898 STD 0.019 0.021 0.027 • When non-event photos enter the dataset, NMI degrades quickly NON-EVENT BATCH INCREMENTAL ITEM-TO-CLUSTER 5% 0.4824 0.5164 0.3954 10% 0.3421 0.3683 0.2899 * * In the second table, results were obtained using sed2011 for training and sed2012 for testing. MMM 2014 Georgios Petkos et al.
- 17. Evaluation: contribution of features • Same experiments using limited sets of features FEATUERS BATCH INCREMENTAL VISUAL 0.8020 ∓ 0.0193 0.8179 ∓ 0.0151 TEXTUAL 0.7925 ∓ 0.0255 0.7792 ∓ 0.0310 VISUAL+TIME 0.9244 ∓ 0.0195 0.9360 ∓ 0.0183 TEXTUAL+TIME 0.9016 ∓ 0.0173 0.9049 ∓ 0.0209 • Repeating the same experiments without the use of blocking led to significantly worse results – e.g. 0.030 for visual, 0.7148 for textual • Time is an extremely important feature MMM 2014 Georgios Petkos et al.
- 18. Evaluation: generalizing same event model • Train using one event type > test on a different one • In most cases negative impact • In few cases, performance is very high! BATCH soccer technical Indignados soccer - 0.8658 0.8494 technical 0.7967 - 0.8977 Indignados 0.9645 0.8456 - INCREMENTAL soccer Indignados soccer - 0.8892 0.8667 technical 0.7661 - 0.7735 Indignados MMM 2014 technical 0.9845 0.8482 Georgios Petkos et al.
- 19. summary & future work MMM 2014 Georgios Petkos et al.
- 20. Summary • Scalable item-to-item multimodal clustering approach for SED • Key characteristics: – Item-to-item “same event” model – Candidate neighbor selection – Organization of “same event” relationships to a graph – Efficient graph clustering algorithms: SCAN (batch) / QCA (incremental) • In general though, item-to-item approaches are less scalable than item-to-cluster approaches MMM 2014 #20 Georgios Petkos et al.
- 21. Future work • Extend method so that non-event images are properly handled • Multiple sources of multimedia • The MediaEval datasets are somewhat limited. Investigate the effect of crawling / image collection to the quality of results MMM 2014 #21 Georgios Petkos et al.
- 22. thank you! questions? Acknowledgements MMM 2014 Georgios Petkos et al.
- 23. online clustering of same-event graph QCA maintains community structure incrementally following graph change operations: node & edge addition (removal operations not applicable in same event graph): based on the concept of community attraction forces Cz new edge new node force from Cu to Cz A D X force from Cz to Cu C Cw B Cu • Depending on a test (computed based on local graph structure), community structure could remain the same, X assigned to Cu or A to Cz. • If A is assigned to Cu, all its neighbours will be checked for potential reassignment. MMM 2014 #23 Georgios Petkos et al.
- 24. graph clustering :: SCAN hub (μ,ε)- core structural similarity outlier • resilient to spurious links (e.g. visual links that connect unrelated images) • very fast (scales linearly to the number of edges) • leaves less-/ and over-connected items out of the clustering MMM 2014 #24 Georgios Petkos et al.
- 25. References • • • • • Reuter, T., & Cimiano, P. (2012, June). Event-based classification of social media streams. In Proceedings of the 2nd ACM International Conference on Multimedia Retrieval (p. 22). ACM. Petkos, G., Papadopoulos, S., & Kompatsiaris, Y. (2012). Social event detection using multimodal clustering and integrating supervisory signals. In Proceedings of the 2nd ACM International Conference on Multimedia Retrieval (p. 23). ACM. Becker, H., Naaman, M. & Gravano, L.. Learning similarity metrics for event identification in social media. In Proceedings of the third ACM International Conference on Web search and Data Mining, WSDM ’10, pages 291–300, New York. Nguyen, N., Dinh, T., Xuan, Y., & Thai, M.. Adaptive algorithms for detecting community structure in dynamic social networks. In INFOCOM 2011. 30th IEEE International Conference on Computer Communications, Joint Conference of the IEEE Computer and Communications Societies, 10-15 April 2011, Shanghai, China, pages 2282–2290. IEEE, 2011. Xu, X., Yuruk, N., Feng, Z. & Schweiger, T.. SCAN: a structural clustering algorithm for networks. In Proceedings of the 13th ACM SIGKDD, KDD ’07, pages 824–833, NY, USA, 2007. ACM MMM 2014 #25 Georgios Petkos et al.