SlideShare a Scribd company logo
NCTA2020
GenerativeModelingof SyntheticEye-TrackingData:
NLP-BasedApproachwithRecurrentNeuralNetworks
Mahmoud Elbattah, Jean-Luc GuĂŠrin, Romuald Carette, Federica Cilia, Gilles Dequen
Laboratoire MIS
UniversitĂŠ de Picardie Jules Verne (UPJV), France
mahmoud.elbattah@u-picardie.fr
https://www.scitepress.org/PublicationsDetail.aspx?ID=Gjb/MLVTCAw=&t=1
NCTA2020
Background:Eye-TrackingTechnology
2
Image Source: https://imotions.com/blog/eye-tracking/
J.H. Goldberg, and J.I. Helfman, “Visual scanpath representation”, In Proceedings of the 2010 Symposium on Eye-Tracking Research &
Applications, ACM, 2010, pp. 203-210.
Scan-path
NCTA2020
Motivation
Challenge:
• The lack or difficulty of acquiring eye-tracking datasets presents a key challenge for
eye-tracking studies.
Possible Solution:
• Generating synthetic eye-tracking data.
3
NCTA2020
RELATEDWORK
• Hand-crafted algorithmic methods, such as:
• (Ma and Deng, 2009).
• (Le, Ma, and Deng, 2012).
• (Duchowski et al. 2016)
• Machine Learning approaches, such as:
• (Assens et al. 2018), using convolutional-recurrent architecture
• (Zemblys, Niehorster, and Holmqvist, 2019), using sequence-to-sequenceLSTM-based
architecture
• (Fuhl, 2020), using variational autoencoder
4
NCTA2020
Our Approach:Key Ideas
• Considering Eye-tracking records as textual sequences.
• Applying methods from the Natural Language Processing (NLP) domain to
transform and model eye-tracking sequences.
• Using a long short-term memory (LSTM) model for the generative modeling
task.
5
NCTA2020
Data Description
• The eye-tracking dataset was constructed over 25 sessions.
• More than 2M records stored in structured CSV files.
6
Number of Participants(ASD, TD) 59 (29, 30)
Gender Distribution (M, F) 38 (≈ 64%), 21 (≈ 36%)
Age (Mean, Median) years 7.88, 8.1
Carette, R., Elbattah, M., Dequen, G., GuĂŠrin, J. L., & Cilia, F. (2018, September). Visualization of eye-
tracking patterns in autism spectrum disorder: method and dataset. In 2018 Thirteenth International
Conference on Digital Information Management (ICDIM) (pp. 248-253). IEEE.
NCTA2020
Data Transformation
7
Segmentation of sequences was partly inspired by the K-mer representation, widely used in Genomics.
NCTA2020
GenerativeModelArchitecture
8
NCTA2020
ExperimentalResults
9
≈ 75% accuracy of
prediction
NCTA2020
Summary
• We presented an NLP-based approach for generating synthetic eye-tracking data.
• Using a sequence-based representation of the saccadic eye movement, eye-tracking
records could be modelled as textual strings with an LSTM model.
• The lack of open-access eye-tracking datasets could make our approach attractive
for further studies.
• For instance, the generative model can serve as an alternative method for data
augmentation in a wide range of eye-tracking applications.
10
NCTA2020
Thank You!

More Related Content

PDF
NLP-Based Approach to Detect Autism Spectrum Disorder in Saccadic Eye Movement
PDF
Precision Physiotherapy & Sports Training: Part 1
PPTX
Herramientas y tĂŠcnicas para la GestiĂłn del Conocimiento Nuclear
PDF
Geometric Deep Learning
PDF
Raai 2019 clinical unmet needs and its solutions of deep learning in medicine3
PPT
CI image processing mns
PDF
MediaEval 2017 - Satellite Task: Flood detection using Social Media Data and ...
PDF
SWARM OPTIMIZED MODULAR NEURAL NETWORK BASED DIAGNOSTIC SYSTEM FOR BREAST CAN...
 
NLP-Based Approach to Detect Autism Spectrum Disorder in Saccadic Eye Movement
Precision Physiotherapy & Sports Training: Part 1
Herramientas y tĂŠcnicas para la GestiĂłn del Conocimiento Nuclear
Geometric Deep Learning
Raai 2019 clinical unmet needs and its solutions of deep learning in medicine3
CI image processing mns
MediaEval 2017 - Satellite Task: Flood detection using Social Media Data and ...
SWARM OPTIMIZED MODULAR NEURAL NETWORK BASED DIAGNOSTIC SYSTEM FOR BREAST CAN...
 

Similar to Generative Modeling of Synthetic Eye-Tracking Data: NLP-Based Approach with Recurrent Neural Networks (20)

PDF
Vision-Based Approach for Autism Diagnosis Using Transfer Learning and Eye-Tr...
PDF
RETINAL IMAGE CLASSIFICATION USING NEURAL NETWORK BASED ON A CNN METHODS
PDF
EYE DISEASE IDENTIFICATION USING DEEP LEARNING
PPTX
DR PPT[1]2[1].pptx - Read-Only.pptx
PDF
DIABETIC RETINOPATHY DETECTION USING MACHINE LEARNING TECHNIQUE
PPTX
Phase_1-SAMPLE_AMCEC.pptx
PDF
Prediction of Cognitive Imperiment using Deep Learning
PPTX
Diabetic Retinopathy.pptx
PPTX
Diabetic Retinopathy.pptx
PPTX
Hierarchical structure adaptive
PDF
Application of deep learning methods for automated analysis of retinal struct...
PDF
A Deep Learning Approach for the Detection and Identification of Neovasculari...
PDF
Prediction of Age by utilising Image Dataset utilising Machine Learning
PDF
IRJET - Deep Multiple Instance Learning for Automatic Detection of Diabetic R...
PDF
Diabetic Retinopathy Detection
PDF
A Literature Survey on Image Linguistic Visual Question Answering
PDF
Covid Mask Detection and Social Distancing Using Raspberry pi
PDF
A MACHINE LEARNING METHODOLOGY FOR DIAGNOSING CHRONIC KIDNEY DISEASE
PDF
Real-time eyeglass detection using transfer learning for non-standard facial...
PDF
FACE MASK DETECTION USING ML
Vision-Based Approach for Autism Diagnosis Using Transfer Learning and Eye-Tr...
RETINAL IMAGE CLASSIFICATION USING NEURAL NETWORK BASED ON A CNN METHODS
EYE DISEASE IDENTIFICATION USING DEEP LEARNING
DR PPT[1]2[1].pptx - Read-Only.pptx
DIABETIC RETINOPATHY DETECTION USING MACHINE LEARNING TECHNIQUE
Phase_1-SAMPLE_AMCEC.pptx
Prediction of Cognitive Imperiment using Deep Learning
Diabetic Retinopathy.pptx
Diabetic Retinopathy.pptx
Hierarchical structure adaptive
Application of deep learning methods for automated analysis of retinal struct...
A Deep Learning Approach for the Detection and Identification of Neovasculari...
Prediction of Age by utilising Image Dataset utilising Machine Learning
IRJET - Deep Multiple Instance Learning for Automatic Detection of Diabetic R...
Diabetic Retinopathy Detection
A Literature Survey on Image Linguistic Visual Question Answering
Covid Mask Detection and Social Distancing Using Raspberry pi
A MACHINE LEARNING METHODOLOGY FOR DIAGNOSING CHRONIC KIDNEY DISEASE
Real-time eyeglass detection using transfer learning for non-standard facial...
FACE MASK DETECTION USING ML
Ad

More from Mahmoud Elbattah (18)

PDF
Learning Embeddings from Free-text Triage Notes using Pretrained Transformer ...
PDF
NLP-Based Prediction of Medical Specialties at Hospital Admission Using Triag...
PDF
Multi-Channel ConvNet Approach to Predict the Risk of In-Hospital Mortality f...
PDF
Learning Clusters in Autism Spectrum Disorder: Image-Based Clustering of Eye-...
PDF
Learning to Predict Autism Spectrum Disorder Based on the Visual Patterns of ...
PDF
Designing Care Pathways Using Simulation Modeling and Machine Learning
PDF
Clustering-Aided Approach for Predicting Patient Outcomes with Application to...
PDF
Using Machine Learning to Predict Length of Stay and Discharge Destination fo...
PDF
Evaluation Criteria of ERP Systems
PDF
ML-Aided Simulation: A Conceptual Framework for Integrating Simulation Models...
PDF
The Economic Burden of Hip Fractures among Elderly Patients in Ireland: A Com...
PDF
Using Simulation Modeling to Design Value-Based Healthcare Systems
PDF
Large-Scale Ontology Storage and Query Using Graph Database-Oriented Approach
PDF
Towards Improving Modeling and Simulation of Clinical Pathways: Lessons Learn...
PDF
FrrbaseViz-A Tool for Exploring Freebase Using Query-Driven Visualisation
PDF
Supply Chains Modelling and Simulation Framework:Graph-Driven Approach Using ...
PDF
Coupling Simulation with Machine Learning:A Hybrid Approach for Elderly Disch...
PDF
Learning about Systems Using Machine Learning:Towards More Data-Driven Feedba...
Learning Embeddings from Free-text Triage Notes using Pretrained Transformer ...
NLP-Based Prediction of Medical Specialties at Hospital Admission Using Triag...
Multi-Channel ConvNet Approach to Predict the Risk of In-Hospital Mortality f...
Learning Clusters in Autism Spectrum Disorder: Image-Based Clustering of Eye-...
Learning to Predict Autism Spectrum Disorder Based on the Visual Patterns of ...
Designing Care Pathways Using Simulation Modeling and Machine Learning
Clustering-Aided Approach for Predicting Patient Outcomes with Application to...
Using Machine Learning to Predict Length of Stay and Discharge Destination fo...
Evaluation Criteria of ERP Systems
ML-Aided Simulation: A Conceptual Framework for Integrating Simulation Models...
The Economic Burden of Hip Fractures among Elderly Patients in Ireland: A Com...
Using Simulation Modeling to Design Value-Based Healthcare Systems
Large-Scale Ontology Storage and Query Using Graph Database-Oriented Approach
Towards Improving Modeling and Simulation of Clinical Pathways: Lessons Learn...
FrrbaseViz-A Tool for Exploring Freebase Using Query-Driven Visualisation
Supply Chains Modelling and Simulation Framework:Graph-Driven Approach Using ...
Coupling Simulation with Machine Learning:A Hybrid Approach for Elderly Disch...
Learning about Systems Using Machine Learning:Towards More Data-Driven Feedba...
Ad

Recently uploaded (20)

PPTX
1_Introduction to advance data techniques.pptx
PDF
Fluorescence-microscope_Botany_detailed content
PDF
168300704-gasification-ppt.pdfhghhhsjsjhsuxush
PPTX
DISORDERS OF THE LIVER, GALLBLADDER AND PANCREASE (1).pptx
PPTX
STUDY DESIGN details- Lt Col Maksud (21).pptx
PPTX
MODULE 8 - DISASTER risk PREPAREDNESS.pptx
PPTX
Computer network topology notes for revision
PDF
BF and FI - Blockchain, fintech and Financial Innovation Lesson 2.pdf
PPTX
The THESIS FINAL-DEFENSE-PRESENTATION.pptx
PDF
Lecture1 pattern recognition............
PPTX
AI Strategy room jwfjksfksfjsjsjsjsjfsjfsj
PPTX
Introduction to Basics of Ethical Hacking and Penetration Testing -Unit No. 1...
PPTX
Qualitative Qantitative and Mixed Methods.pptx
PDF
.pdf is not working space design for the following data for the following dat...
PPTX
Introduction to machine learning and Linear Models
PDF
TRAFFIC-MANAGEMENT-AND-ACCIDENT-INVESTIGATION-WITH-DRIVING-PDF-FILE.pdf
PPTX
iec ppt-1 pptx icmr ppt on rehabilitation.pptx
PPTX
Microsoft-Fabric-Unifying-Analytics-for-the-Modern-Enterprise Solution.pptx
PDF
Galatica Smart Energy Infrastructure Startup Pitch Deck
1_Introduction to advance data techniques.pptx
Fluorescence-microscope_Botany_detailed content
168300704-gasification-ppt.pdfhghhhsjsjhsuxush
DISORDERS OF THE LIVER, GALLBLADDER AND PANCREASE (1).pptx
STUDY DESIGN details- Lt Col Maksud (21).pptx
MODULE 8 - DISASTER risk PREPAREDNESS.pptx
Computer network topology notes for revision
BF and FI - Blockchain, fintech and Financial Innovation Lesson 2.pdf
The THESIS FINAL-DEFENSE-PRESENTATION.pptx
Lecture1 pattern recognition............
AI Strategy room jwfjksfksfjsjsjsjsjfsjfsj
Introduction to Basics of Ethical Hacking and Penetration Testing -Unit No. 1...
Qualitative Qantitative and Mixed Methods.pptx
.pdf is not working space design for the following data for the following dat...
Introduction to machine learning and Linear Models
TRAFFIC-MANAGEMENT-AND-ACCIDENT-INVESTIGATION-WITH-DRIVING-PDF-FILE.pdf
iec ppt-1 pptx icmr ppt on rehabilitation.pptx
Microsoft-Fabric-Unifying-Analytics-for-the-Modern-Enterprise Solution.pptx
Galatica Smart Energy Infrastructure Startup Pitch Deck

Generative Modeling of Synthetic Eye-Tracking Data: NLP-Based Approach with Recurrent Neural Networks