Object segmentation in images using EEG signals
2 likes1,271 views
This document presents research on using EEG signals to perform interactive object segmentation in images. The researchers propose a system where an image is partitioned into windows that are rapidly displayed to a user wearing an EEG headset. The user's EEG data is analyzed to identify their response to "target" windows containing objects versus other windows. This brain response data is then used to guide an image segmentation algorithm to segment the objects from the image. Initial results show the approach can achieve reasonable segmentation accuracy, though further improvements are still needed before it would be practical compared to using a mouse. Future work may explore changing parameters and combining the BCI with eye tracking.
Object segmentation in images using EEG signals
- 1. Object Segmentation in Images using EEG Signals Eva Mohedano, Graham Healy, Kevin McGuinness, ! Xavier Giró-i-Nieto, Noel E. O’Connor and Alan F. Smeaton! !! November 6, 2014
- 2. Outline •Interactive Object Segmentation! •ACM MultiMedia High Risk High Reward 2014! •Related Work! •System Proposal! •Results! •Conclusions E. Mohedano 2
- 3. Interactive Object Segmentation •Object Segmentation 3 E. Mohedano
- 4. Interactive Object Segmentation 4 E. Mohedano
- 5. Interactive Object Segmentation 5 1) P. Arbelaez and L. Cohen. Constrained image segmentation from hierarchical boundaries. In CVPR'08, 2008.! 2) McGuinness, K., & O’Connor, N. E. (2010). A comparative evaluation of interactive segmentation algorithms. Pattern Recognition, 43(2), 434-444. E. Mohedano
- 6. Outline •Interactive Object Segmentation! •ACM MultiMedia High Risk High Reward 2014! •Related Work! •System Proposal! •Results! •Conclusions E. Mohedano 02 May 2014 6
- 7. Brain-Computer Interface (BCI) 7 E. Mohedano
- 8. Brain-Computer Interface (BCI) 8 E. Mohedano
- 9. Brain-Computer Interface (BCI) 9 Electroencephalography (EEG) signals E. Mohedano
- 10. Brain-Computer Interface (BCI) • Non invasive! • Well established tool within clinical practice 10 Strengths E. Mohedano
- 11. Brain-Computer Interface (BCI) • Mostly BCI applications remain prototypes not used outside laboratories! • Users need to be trained! • Poor BCI performances! • Low signal-to-noise ratio! • High dimensional data 11 Challenges HIGH RISK E. Mohedano
- 12. Potentially High Reward • Medical applications! • Locked in Syndrome (LIS)! • Prosthetics control, wheelchairs, spellers 12 •Healthy Users! • BCI with Virtual Reality technologies! • Augmenting gaze control E. Mohedano
- 13. Outline •Interactive Object Segmentation! •ACM MultiMedia High Risk High Reward 2014! •Related Work! •System Proposal! •Results! •Conclusions 02 May 2014 13 E. Mohedano
- 14. Related Work: RSVP 14 ! •A positive waveform occurring approximately 300-550ms after an infrequent task-relevant stimulus E. Mohedano
- 15. Related Work: RSVP 15 E. Mohedano
- 16. RSVP: Demo 16 E. Mohedano
- 17. Related Work •Image Retrieval 17 E. Mohedano
- 18. Related Work •Object Detection 18 E. Mohedano
- 19. Related Work •BCI speller 19 Ref: D. Fernández-Cañellas, “Modeling temporal dependency of brain responses to rapidly stimuli in ERP based BCIs” (2013) E. Mohedano
- 20. Index •Interactive Object Segmentation! •ACM MultiMedia High Risk High Reward 2014! •Related Work! •System Proposal! •Results! •Conclusions 02 May 2014 20 E. Mohedano
- 21. System Proposal •Local RSVP (5Hz visualisation windows) 21 E. Mohedano
- 22. System Proposal •Different Reaction after seeing a target 22 Targets Distractors E. Mohedano
- 23. System Proposal 23 E. Mohedano
- 24. System Proposal Data Acquisition! Set of 22 images with an associated ground truth mask 24 E. Mohedano
- 25. System Proposal Data Acquisition! Images were partitioned into 192 non overlapped windows! ! ! ! ! ! 25 • 15% Target windows! • RSVP windows at 5Hz! • User asked to count the target windows visualised E. Mohedano
- 26. System Proposal 26 EEG processing E. Mohedano
- 27. 27 ! 1) Down sample from 1000Hz to 250Hz! 2) Bandpass filter 0.1-70 Hz! 3) Cut EEG activity related to each visual event! 4) Down sample from 250Hz to 20Hz! 5) Concatene 31 channels (434D) ! Support Vector Machine Model (SVM) System Proposal EEG processing ! EEG feature vectors E. Mohedano
- 28. 28 E. Mohedano
- 29. 29 System Proposal Segmentation • GrabCut: Interactive Foreground Extraction OpenCV’s GrabCut Tutorial:! http://docs.opencv.org/trunk/doc/py_tutorials/py_imgproc/py_grabcut/py_grabcut.html ! E. Mohedano
- 30. 30 System Proposal Evaluation Metric: Jaccard Index Measure of similarity between the segmentation results and the ground truth mask E. Mohedano
- 31. Outline •Interactive Object Segmentation! •ACM MultiMedia High Risk High Reward 2014! •Related Work! •System Proposal! •Results! •Conclusions 02 May 2014 31 E. Mohedano
- 32. Results •Single User 32 Jaccard Index = 0.47 E. Mohedano
- 33. Results •Averaged Users 33 Jaccard Index = 0.72 E. Mohedano
- 34. Index •Interactive Object Segmentation! •ACM MultiMedia High Risk High Reward 2014! •Related Work! •System Proposal! •Results! •Conclusions 02 May 2014 34 E. Mohedano
- 35. 35 Conclusions The approach is feasible: it is possible to use BCI as an interactive segmentation method based on simple EEG processing. E. Mohedano
- 36. Conclusions 36 BCI Interaction for segmentation Mouse Interaction for segmentation E. Mohedano BCI is time consuming Mouse interaction provides better results
- 37. Future work 37 • Improvements in EEG processing! • Change resolution of windows! • Use object candidates instead of a grid! • Active search! • Combine local EEG with eye tracker E. Mohedano
- 38. Thank you! 38 Questions? ! This publication resulted from research conducted with the financial support of Science Foundation Ireland (SFI) under grant number SFI/12/ RC/2289 and partially funded by the Project TEC2013-43935-R BigGraph of the Spanish Government. E. Mohedano