Exploring EEG for object detection and retrieval
2 likes932 views
The document discusses the use of EEG signals for object detection and image retrieval, comparing their effectiveness to traditional mouse interactions. It provides insights into the methodology, including dataset selection and EEG signal processing, as well as early results that suggest EEG can be useful for image retrieval but exhibit variations across users. The conclusions emphasize the potential of EEG in image retrieval and suggest further exploration of its integration with relevance feedback.
![Retrieval in a larger dataset
28
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in
neural information processing systems (pp. 1097-1105).
Software:
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., ... & Darrell, T. (2014, November). Caffe: Convolutional
architecture for fast feature embedding. In Proceedings of the ACM International Conference on Multimedia(pp. 675-678). ACM. [web]](https://image.slidesharecdn.com/icmr2015exploringeegforobjectdetectionandretrieval-150504083103-conversion-gate01/85/Exploring-EEG-for-object-detection-and-retrieval-28-320.jpg)
Exploring EEG for object detection and retrieval
- 1. Exploring EEG for Object Detection and Retrieval Eva Mohedano Amaia Salvador Sergi Porta Xavi Giró Graham O’Healy Kevin McGuinness Noel O’Connor Alan Smeaton ACM International Conference on Multimedia Retrieval (ICMR) 2015. June 23-26, 2015. Shanghai, China.
- 2. Outline ● Motivation ● Related Work ● Methodology ● Object Detection ● EEG vs Mouse for Retrieval ● Conclusions ● Future work 2
- 3. Motivation Are EEG signals useful for Image Retrieval? ? 3
- 4. Motivation If so, are they comparable to mouse interaction? VS 4
- 5. Outline ● Motivation ● Related Work ● Methodology ● Object Detection ● EEG vs Mouse for Retrieval ● Conclusions ● Future work 5
- 6. Related Work: Rapid Image Retrieval Wang, J., Pohlmeyer, E., Hanna, B., Jiang, Y. G., Sajda, P., & Chang, S. F. (2009, October). Brain state decoding for rapid image retrieval. In Proceedings of the 17th ACM international conference on Multimedia (pp. 945-954). ACM. 6
- 7. Related Work: Rapid Image Retrieval Jun Yang, “A General Framework for Classifier Adaptation and its Applications to Multimedia”. Phd thesis. Carnegie Mellon University (2009). Section 7.2: Adaptation of EEG-based Relevance Models Cross-user adaptation to avoid re-training EEG-based SVM classifiers for relevance prediction. 7
- 8. Related Work: EEG Object Detection 8
- 9. Related Work: EEG Object Detection Ref: Kapoor, Ashish, Pradeep Shenoy, and Desney Tan. "Combining brain computer interfaces with vision for object categorization." Computer Vision and Pattern Recognition, 2008. CVPR 2008. 9
- 10. Outline ● Motivation ● Related Work ● Methodology ● Object Detection ● EEG vs Mouse for Retrieval ● Conclusions ● Future work 10
- 11. Methodology: Dataset Subset of TRECVID INS 2013 Dataset ● 3 Instances (4 local regions/instance) ● For each topic, 1000 Images ○ 50 relevant (targets) and 950 non-relevant (distractors) 11
- 13. Methodology: Image Sorting Unknown 200=10/190 images @ 5 Hz = 40 s 200=10/190 images @ 10 Hz = 20 s ...Rest... Round Time between relevant (targets) 10/190 guarantees 5% of target, but they may not be apart. ...Rest Rest... Rest... ... 5 rounds x 200 = 1,000 images @ 5 Hz -> 200 seconds 13
- 14. Methodology: Feature Vectors Cut EEG activity related to visual event From 200 ms to 1 second after the target presentation. 14
- 15. Methodology: Feature Vectors 1 2 3 32 ... ... Band-pass 0.1-20Hz Downsample 1000 → 250 Hz Downsample 250 → 20Hz Channel concatenation 512D vector 15
- 16. Methodology: Classification Linear SVM Q1 Q2 Q3 Testing:Training: Leave-one-out Cross Validation 16
- 17. Outline ● Motivation ● Related Work ● Methodology ● Object Detection ● EEG vs Mouse for Retrieval ● Conclusions ● Future work 17
- 18. Object Detection: Early results Images with high classifier scores for queries 2 and 3. 18
- 19. Object Detection: Early results Images with high classifier scores (yet not relevant) for query 1. 19
- 20. Object Detection: User diversity Receiver Operating Characteric prove EEGs are valid, but with significant variations among users. 20
- 21. Object Detection: User diversity 21
- 22. Object Detection: 5 vs 10 Hz 22
- 23. Outline ● Motivation ● Related Work ● Methodology ● Object Detection ● EEG vs Mouse for Retrieval ● Conclusions ● Future work 23
- 24. EEG vs Mouse for Retrieval EEG RSVP Mouse Clicks on thumbnails Timer 24
- 25. Retrieval within our dataset Linear SVM EEG: Sorting according to SVM confidence score Mouse: Top: Clicked images (relevant) Bottom: Observed but not clicked images Middle: Remaining images randomly sorted 25
- 26. Retrieval within our dataset 26
- 27. Retrieval in a larger dataset Subset of TRECVid INS 2013 ( 23.614 images) 27
- 28. Retrieval in a larger dataset 28 Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097-1105). Software: Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., ... & Darrell, T. (2014, November). Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the ACM International Conference on Multimedia(pp. 675-678). ACM. [web]
- 29. Retrieval in a larger dataset EEG Top 10: Positives Bottom 100: Negatives Clicked: Positives Observed & unclicked: Negatives Linear SVM ... Testing: 29
- 30. Retrieval in a larger dataset 30
- 31. Retrieval in a larger dataset 31
- 32. Outline ● Motivation ● Related Work ● Methodology ● Object Detection ● EEG vs Mouse for Retrieval ● Conclusions ● Future work 32
- 33. Conclusions ● Are EEG signals useful for Image Retrieval? YES! 33
- 34. Conclusions ● Not all users are the same... 34
- 35. Conclusions EEG Mouse Comparable performance for Image Retrieval 35
- 36. Future work Can RSVP benefit from relevance feedback? 36
- 37. Outline ● Motivation ● Related Work ● Methodology ● Object Detection ● EEG vs Mouse for Retrieval ● Conclusions ● Future work 37
- 38. 38