SlideShare a Scribd company logo

Vision-based Recognition of Human Behaviour for Intelligent Environments

A
Alexandros Andre Chaaraoui

This thesis by Alexandros Andre Chaaraoui focuses on the vision-based recognition of human behavior to enhance ambient-assisted living (AAL) services within intelligent environments. It proposes a robust research framework and analyzes human actions using multiple advancements in pose representation, action classification, and continuous recognition techniques. The results demonstrate high recognition rates, outperforming existing methods and addressing specific AAL demands such as privacy and adaptability in real-time environments.

TechnologyEducation
Related topics:
Pattern Recognition • Computer Vision Insights
1 of 52
Downloaded 106 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
Vision-based Recognition of Human Behaviour for Intelligent Environments Alexandros Andre Chaaraoui Departamento de Tecnolog´ Inform´tica y Computaci´n ıa a o Universidad de Alicante alexandros@ua.es Supervisor: Dr. Francisco Fl´rez-Revuelta o January 20, 2014 Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 1 / 45
Overview 1 Introduction 2 Research framework 3 Contributions 4 Results 5 Concluding remarks Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 2 / 45
Introduction Introduction Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 3 / 45
Introduction Motivation Introduction (I) Motivation Demographic ageing - Ambient-assisted living (AAL) Intelligent environments Human behaviour analysis Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 4 / 45
Introduction Objectives Introduction (II) In this thesis, our goal is to support the development of AAL services for smart homes with advances in human behaviour analysis. Main objectives 1 Establish the research framework 2 Propose a method for the recognition of human behaviour 3 Satisfy specific demands of AAL services 4 Reach robustness for different scenarios Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 5 / 45
Research framework Research framework Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 6 / 45
Research framework Related work Research framework Proposed taxonomy Figure 1: Human Behaviour Analysis levels — Classification proposed in Chaaraoui et al. (2012). Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 7 / 45
Research framework Proposal Conclusions of the analysis Motion Motion, pose and gaze estimation is the most resolved level. Action Action recognition is currently receiving the greatest interest both from research and industry. Activity-Behaviour Activity and long-term behaviour recognition is performed directly based on low-level sensor data, instead of using action recognition. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 8 / 45
Research framework Proposal Camera 1 Long-term Analysis Setup (activities, inhabitants, objects…), Profiles and Log Camera 2 ... Camera N Motion Detection Motion Detection ... Motion Detection Human Behaviour Analysis Human Behaviour Analysis ... Human Behaviour Analysis Multi-view Human Behaviour Analysis Environmental Sensor Information Reasoning System Privacy Event Alarm Actuators Caregiver Figure 2: Architecture of the intelligent monitoring system to promote independent living at home and support AAL services. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 9 / 45
Contributions Contributions Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 10 / 45
Contributions Outline Contributions Proposals have been made at different processing stages: 1 Pose representation 2 Fusion of multiple views 3 Action classification 4 Evolutionary-based optimisation 5 Continuous recognition Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 11 / 45
Contributions Pose representation Pose representation Images Silhouettes Pose Representations Figure 3: Outline of the pose representation process. Based on the recorded video frames, foreground segmentations are obtained. Holistic features can then be extracted relying on the shape of the human silhouettes. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 12 / 45
Contributions Pose representation Radial Summary (I) Figure 4: Overview of the feature extraction process: 1) All the contour points are assigned to the corresponding radial bin; 2) for each bin, a summary representation is obtained (example with 18 bins). Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 13 / 45
Contributions Pose representation Radial Summary (II) Figure 5: Graphical explanation of the statistical range frange of a sample radial bin. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 14 / 45
Contributions Fusion of multiple views Fusion of multiple views Multiple views of the same field of view provide Additional characteristic data Advantages with respect to occlusions Advantages with respect to unfavourable viewing angles (ambiguous actions) However, difficulties have been observed The recognition does not necessarily improve Performance issues (temporal and spatial) Burdensome, highly-restricted systems (3D pose estimation, calibrated and synchronised camera networks, ...) Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 15 / 45
Contributions Fusion of multiple views Weighted feature fusion scheme Weights are learnt for each view and action: Figure 6: Overview of the feature fusion process of the multi-view pose representation. This example shows five different views of a specific pose taken from the walk action class from the IXMAS dataset (Weinland et al., 2006). Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 16 / 45
Contributions Action classification Action classification Key Poses K Key Poses ... ... ... Bag of Key Poses K Key Poses Sequences of Key Poses Figure 7: Outline of the learning stage. Using the pose representations, key poses are obtained for each action. In this way, a bag-of-key-poses model is learnt. The temporal relation between key poses is modelled using sequences of key poses. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 17 / 45
Contributions Action classification Bag of key poses Action1 K Key Poses Action2 K Key Poses ... ... ... ... ActionA K Key Poses Bag of Key Poses Figure 8: Learning scheme of the bag-of-key-poses model. For each action class, K key poses are obtained separately and then joined together. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 18 / 45
Contributions Action classification Recognition Sequences of Key Poses Sequence Matching DTW Action Recognition Figure 9: Outline of the recognition stage. The unknown sequence of key poses is obtained and compared to the known sequences. Through sequence matching, the action of the video sequence can be recognised. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 19 / 45
Contributions Evolutionary-based optimisation Evolutionary-based optimisation Genetic feature subset selection By means of a genetic algorithm for binary feature selection, the interesting body parts can be selected, and redundant or noisy body parts can be ignored. Figure 10: Example of a result provided by genetic feature selection (dismissed radial bins are shaded in grey). Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 20 / 45
Contributions Evolutionary-based optimisation Evolutionary-based optimisation Genetic feature subset selection Coevolutionary optimisation Simultaneous selection of training instances, features and parameter values Coevolution enables to split the problem in subproblems of optimisation with a common goal (Wiegand, 2004) Cooperative coevolution allows to consider intrinsic dependencies among optimisation goals (Derrac et al., 2012) Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 21 / 45
Contributions Evolutionary-based optimisation Evolutionary-based optimisation Genetic feature subset selection Coevolutionary optimisation Adaptive learning Evolving bag of key poses Supports incremental and adaptive learning of new data Applies selection of training instances, features and parameter values Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 22 / 45
Contributions Continuous recognition Continuous recognition In AAL, human action recognition has to be applied to continuous video streams. This requires: To detect meaningful human actions online And to recognise the appropriate action in real-time We propose: To learn action zones, i.e. the most discriminative parts of action performances The usage of a sliding and growing window technique for recognition Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 23 / 45
Contributions Continuous recognition Action zones Figure 11: Sample silhouettes of a waving sequence of the DAI RGBD dataset. The action zone that should be extracted is highlighted. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 24 / 45
Contributions Continuous recognition Action zones Figure 12: Evidence values H(t) of each action class and the detected action zones are shown for a scratch head sequence of the IXMAS dataset. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 25 / 45
Results Results Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 26 / 45
Results Evaluation methodology Results Experimentation has been performed on: Single- and multi-view datasets (up to five views) Manually- and automatically-extracted silhouettes (including depth-based segmentation) Using the following cross validations: Leave one sequence out (LOSO) Leave one actor out (LOAO) Leave one view out (LOVO) Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 27 / 45
Results Benchmarks Weizmann dataset Table 1: Comparison of recognition rates and speeds obtained on the Weizmann dataset (Gorelick et al., 2007) with other state-of-the-art approaches. Approach ˙ Ikizler and Duygulu (2007) Tran and Sorokin (2008) Fathi and Mori (2008) Actions 9 10 10 Test LOSO LOSO LOSO Rate 100% 100% 100% fps N/A N/A N/A Hern´ndez et al. (2011) a Cheema et al. (2011) Sadek et al. (2012) 10 9 10 LOAO LOSO LOAO 90.3% 91.6% 97.8% 98 56 18 Our approach Our approach 10 10 LOSO LOAO 93.5% 97.8% 188 188 Optimised approach 10 LOAO 100% 210 Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 28 / 45
Results Benchmarks MuHAVi-14 dataset Table 2: Comparison of recognition rates and speeds obtained on the MuHAVi-14 dataset (Singh et al., 2010) with other state-of-the-art approaches. Approach LOSO LOAO LOVO fps Singh et al. (2010) Eweiwi et al. (2011) 82.4% 91.9% 61.8% 77.9% 42.6% 55.8% N/A N/A Cheema et al. (2011) 86.0% 73.5% 50.0% 56 98.5% 94.1% 59.6% 99 100% 100% - - Our approach Optimised approach Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 29 / 45
Results Benchmarks MuHAVi-8 dataset Table 3: Comparison of recognition rates and speeds obtained on the MuHAVi-8 dataset (Singh et al., 2010) with other state-of-the-art approaches. M´todo e LOSO LOAO LOVO fps Singh et al. (2010) Mart´ ınez-Contreras et al. (2009) Eweiwi et al. (2011) 97.8% 98.4% 98.5% 76.4% 85.3% 50.0% 38.2% N/A N/A N/A Cheema et al. (2011) 95.6% 83.1% 57.4% 56 Our approach 100% 100% 82.4% 98 Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 30 / 45
Results Benchmarks IXMAS dataset Table 4: Comparison with other multi-view human action recognition approaches of the state of the art. The rates obtained in the LOAO cross validation performed on the IXMAS dataset are shown. Approach Actions Actors Views Rate fps Yan et al. (2008) Wu et al. (2011) Cilla et al. (2012) Weinland et al. (2006) Cilla et al. (2013) Holte et al. (2012) 11 12 11 11 11 13 12 12 12 10 10 12 4 4 5 5 5 5 78% 89.4% 91.3% 93.3% 94.0% 100% N/A N/A N/A N/A N/A N/A Cherla et al. (2008) Weinland et al. (2010) 13 11 N/A 10 4 5 80.1% 83.5% 20 ∼500 Our approach 11 12 5 91.4% 207 Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 31 / 45
Results Benchmarks RGBD datasets DAI RGBD dataset Table 5: Cross validation results obtained on our multi-view depth dataset. Approach LOSO LOAO fps Our approach 94.4% 100% 80 DHA dataset Table 6: LOSO cross validation results obtained on the DHA dataset (Lin et al., 2012) (10 Weizmann actions). Approach LOSO Lin et al. (2012) Our approach Alexandros Andre Chaaraoui (UA) fps 90.8% N/A 95.2% 99 PhD Thesis January 20, 2014 32 / 45
Results Benchmarks Continuous recognition Table 7: Obtained results applying CHAR and segment analysis evaluation (LOAO). Results are detailed using the segmented sequences or the proposed action zones. Dataset Approach F1 -measure IXMAS IXMAS Segmented sequences Action zones 0.504 0.705 Weizmann Weizmann Segmented sequences Action zones 0.693 0.928 Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 33 / 45
Concluding remarks Concluding remarks Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 34 / 45
Concluding remarks Discussion Discussion Silhouettes 2D silhouettes can be difficult to obtain and they are view dependant Privacy Privacy concerns of indoor monitoring Intelligent monitoring system with privacy protection The method only relies on the boundary of the silhouette Validation of the proposed method The classification method based on the bag of key poses has also been validated for gaming and NUI (Chaaraoui et al., 2014, 2013; Climent-P´rez et al., 2013) e Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 35 / 45
Concluding remarks Conclusions Conclusions 1 Proposal of a 2D template-based non-parametric method for human action recognition & optimisations and extensions 2 Specific demands of AAL services have been satisfied: relaxed camera setup requirements, adaptive learning, continuous recognition and real-time execution 3 The HAR method based on a bag-of-key-poses model handles single- and multi-view recognition proficiently. 4 State-of-the-art recognition rates have been achieved, outperforming the best known rates in several benchmarks. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 36 / 45
Concluding remarks Future work Future work Future directions of this work: Pose representation: Other local and global features Key poses: Generation algorithms Bag-of-key-poses model: Applications Distance metrics: Key poses and sequences of key poses Evaluation and deployment However, two main future lines stand out: Recognition of complex activities based on action sequences and multi-modal data Feature fusion techniques, e.g. for recognition of subtle movements or gestures Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 37 / 45
Other information and details Other information and details Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 38 / 45
Other information and details Research projects and grants Research projects and grants Intelligent system for follow-up and promotion of personal autonomy: Computer vision system for the monitoring of activities of daily living at home – Sistema de visi´n para la o monitorizaci´n de la actividad diaria en el hogar. Spanish Ministry of Science and o Innovation and Valencian Ministry of Education, Culture and Sport (TALISMAN+, Technical University of Madrid, University of Deusto, University of Castile-La Mancha and University of Alicante) PhD. Research Fellowship – Programa VALi+d para investigadores en formaci´n. Valencian Ministry of Education, Culture and Sport (ACIF/2011/160) o Research Collaboration Stay – Digital Imaging Research Centre, Faculty of Science, Engineering and Computing, Kingston University. Kingston upon Thames, UK. (Funded by VALi+d, BEFPI/2013/015) Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 39 / 45
Other information and details Other activities Other activities Teaching collaboration – Information Technology. First year core subject of Degree in Sound and Image in Telecommunication Engineering Reviewer of international journals – Neurocomputing (Elsevier), Pervasive Computing (IEEE), EURASIP Journal on Image and Video Processing (Springer), Expert Systems With Applications (Elsevier) Conference session chair – IEEE/RSJ Intelligent Robots and Systems (IROS 2012), Genetic and Evolutionary Computation Conference (GECCO 2013) Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 40 / 45
Other information and details Intl. review International review This thesis has been approved by the following reviewers for the International PhD Honourable Mention Dr. Jean-Christophe Nebel (Kingston University, UK) Dr. Jes´s Mart´ u ınez del Rinc´n o (Queen’s University of Belfast, UK) Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 41 / 45
Other information and details Publications Publications Journals I Chaaraoui, A.A., Climent-P´rez, P., Fl´rez-Revuelta, F., 2012. A Review on e o Vision Techniques Applied to Human Behaviour Analysis for Ambient-Assisted Living. Expert Systems with Applications. Citations: 10 II Chaaraoui, A.A., Climent-P´rez, P., Fl´rez-Revuelta, F., 2013. e o Silhouette-based Human Action Recognition Using Sequences of Key Poses. Pattern Recognition Letters. Citations: 6 III Chaaraoui, A.A., Fl´rez-Revuelta, F., 2013. Optimizing Human Action o Recognition Based on a Cooperative Coevolutionary Algorithm. Engineering Applications of Artificial Intelligence. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 42 / 45
Other information and details Publications Publications IV Chaaraoui, A.A., Padilla-L´pez, J.R., Climent-P´rez, P., Fl´rez-Revuelta, F., o e o 2014. Evolutionary Joint Selection to Improve Human Action Recognition with RGB-D Devices. Expert Systems with Applications. V Chaaraoui, A.A., Fl´rez-Revuelta, F., 2014. A Low-Dimensional Radial o Silhouette-based Feature for Fast Human Action Recognition Fusing Multiple Views. Information Fusion. Under review VI Chaaraoui, A.A., Fl´rez-Revuelta, F., 2014. Adaptive Human Action o Recognition With an Evolving Bag of Key Poses. IEEE Transactions on Autonomous Mental Development. Under review Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 43 / 45
Other information and details Publications Publications Conferences and workshops I Intl. Symposium on Ambient Intelligence (ISAmI 2012) II 3rd Intl. Workshop on Human Behavior Understanding (HBU 2012), IEEE/RSJ Intl. Conference on Intelligent Robots and Systems (IROS 2012) III 11th Mexican Intl. Conference on Artificial Intelligence (MICAI 2012) IV Genetic and Evolutionary Computation Conference (GECCO 2013) V 5th Intl. Work-conference on Ambient Assisted Living (IWAAL 2013) VI 3rd Workshop on Consumer Depth Cameras for Computer Vision (CDC4CV13), IEEE Intl. Conference on Computer Vision (ICCV 2013) Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 44 / 45
Other information and details Publications “We can only see a short distance ahead, but we can see plenty there that needs to be done.” (Turing, 1950) Vision-based Recognition of Human Behaviour for Intelligent Environments PhD Thesis Alexandros Andre Chaaraoui Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 45 / 45
References References Chaaraoui, A. A., P. Climent-P´rez, and F. Fl´rez-Revuelta (2012). A review on vision e o techniques applied to human behaviour analysis for ambient-assisted living. Expert Systems with Applications 39 (12), 10873 – 10888. Chaaraoui, A. A., J. R. Padilla-L´pez, P. Climent-P´rez, and F. Fl´rez-Revuelta (2014). o e o Evolutionary joint selection to improve human action recognition with RGB-D devices. Expert Systems with Applications 41 (3), 786 – 794. Methods and Applications of Artificial and Computational Intelligence. Chaaraoui, A. A., J. R. Padilla-L´pez, and F. Fl´rez-Revuelta (2013). Fusion of skeletal o o and silhouette-based features for human action recognition with RGB-D devices. In IEEE 14th International Conference on Computer Vision Workshops, 2013. ICCV Workshops 2013. To be presented in 3rd Workshop on Consumer Depth Cameras for Computer Vision (CDC4CV13). Cheema, S., A. Eweiwi, C. Thurau, and C. Bauckhage (2011). Action recognition by learning discriminative key poses. In IEEE 13th International Conference on Computer Vision Workshops, 2011. ICCV Workshops 2011, pp. 1302 –1309. Cherla, S., K. Kulkarni, A. Kale, and V. Ramasubramanian (2008). Towards fast, view-invariant human action recognition. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2008. CVPRW 2008, pp. 1 – 8. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 46 / 45
References References Cilla, R., M. A. Patricio, A. Berlanga, and J. M. Molina (2012). A probabilistic, discriminative and distributed system for the recognition of human actions from multiple views. Neurocomputing 75 (1), 78 – 87. Brazilian Symposium on Neural Networks (SBRN 2010), International Conference on Hybrid Artificial Intelligence Systems (HAIS 2010). Cilla, R., M. A. Patricio, A. Berlanga, and J. M. Molina (2013). Human action recognition with sparse classification and multiple-view learning. Expert Systems. DOI 10.1111/exsy.12040. Climent-P´rez, P., A. A. Chaaraoui, J. R. Padilla-L´pez, and F. Fl´rez-Revuelta (2013). e o o Optimal joint selection for skeletal data from RGB-D devices using a genetic algorithm. In I. Batyrshin and M. Mendoza (Eds.), Advances in Computational Intelligence, Volume 7630 of Lecture Notes in Computer Science, pp. 163 – 174. Springer Berlin / Heidelberg. Derrac, J., I. Triguero, S. Garc´ and F. Herrera (2012). A co-evolutionary framework for ıa, nearest neighbor enhancement: Combining instance and feature weighting with instance selection. In E. Corchado, V. Sn´ˇel, A. Abraham, M. Wo´niak, M. Gra˜a, and S.-B. as z n Cho (Eds.), Hybrid Artificial Intelligent Systems, Volume 7209 of Lecture Notes in Computer Science, pp. 176 – 187. Springer Berlin / Heidelberg. Eweiwi, A., S. Cheema, C. Thurau, and C. Bauckhage (2011). Temporal key poses for human action recognition. In IEEE 13th International Conference on Computer Vision Workshops, 2011. ICCV Workshops 2011, pp. 1310 –1317. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 47 / 45
References References Fathi, A. and G. Mori (2008). Action recognition by learning mid-level motion features. In IEEE Conference on Computer Vision and Pattern Recognition, 2008. CVPR 2008, pp. 1 – 8. Gorelick, L., M. Blank, E. Shechtman, M. Irani, and R. Basri (2007). Actions as space-time shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence 29 (12), 2247 – 2253. Hern´ndez, J., A. Montemayor, J. Jos´ Pantrigo, and A. S´nchez (2011). Human action a e a ´ recognition based on tracking features. In J. Ferr´ndez, J. Alvarez S´nchez, F. de la Paz, a a and F. Toledo (Eds.), Foundations on Natural and Artificial Computation, Volume 6686 of Lecture Notes in Computer Science, pp. 471 – 480. Springer Berlin / Heidelberg. Holte, M., B. Chakraborty, J. Gonzalez, and T. Moeslund (2012). A local 3-D motion descriptor for multi-view human action recognition from 4-D spatio-temporal interest points. IEEE Journal of Selected Topics in Signal Processing 6 (5), 553 – 565. ˙ Ikizler, N. and P. Duygulu (2007). Human action recognition using distribution of oriented rectangular patches. In A. Elgammal, B. Rosenhahn, and R. Klette (Eds.), Human Motion - Understanding, Modeling, Capture and Animation, Volume 4814 of Lecture Notes in Computer Science, pp. 271 – 284. Springer Berlin / Heidelberg. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 48 / 45
References References Lin, Y.-C., M.-C. Hu, W.-H. Cheng, Y.-H. Hsieh, and H.-M. Chen (2012). Human action recognition and retrieval using sole depth information. In Proceedings of the 20th ACM international conference on Multimedia, MM ’12, New York, NY, USA, pp. 1053 – 1056. ACM. Mart´ ınez-Contreras, F., C. Orrite-Urunuela, E. Herrero-Jaraba, H. Ragheb, and S. Velastin (2009). Recognizing human actions using silhouette-based HMM. In IEEE Int. Conference on Advanced Video and Signal Based Surveillance, 2009. AVSS 2009, pp. 43 – 48. Sadek, S., A. Al-Hamadi, B. Michaelis, and U. Sayed (2012). A fast statistical approach for human activity recognition. International Journal of Intelligence Science 2 (1), 9 – 15. Singh, S., S. Velastin, and H. Ragheb (2010). MuHAVi: A multicamera human action video dataset for the evaluation of action recognition methods. In IEEE Int. Conference on Advanced Video and Signal Based Surveillance, 2010. AVSS 2010, pp. 48 – 55. IEEE. Tran, D. and A. Sorokin (2008). Human activity recognition with metric learning. In D. Forsyth, P. Torr, and A. Zisserman (Eds.), European Conference on Computer Vision. ECCV 2008, Volume 5302 of Lecture Notes in Computer Science, pp. 548 – 561. Springer Berlin / Heidelberg. Turing, A. M. (1950). Computing machinery and intelligence. Mind 59 (236), 433 – 460. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 49 / 45
References References ¨ Weinland, D., M. Ozuysal, and P. Fua (2010). Making action recognition robust to occlusions and viewpoint changes. In K. Daniilidis, P. Maragos, and N. Paragios (Eds.), European Conference on Computer Vision. ECCV 2010, Volume 6313 of Lecture Notes in Computer Science, pp. 635 – 648. Springer Berlin / Heidelberg. Weinland, D., R. Ronfard, and E. Boyer (2006). Free viewpoint action recognition using motion history volumes. Computer Vision and Image Understanding 104 (2-3), 249 – 257. Wiegand, R. P. (2004). An analysis of cooperative coevolutionary algorithms. Ph. D. thesis, George Mason University, Fairfax, VA, USA. Wu, X., D. Xu, L. Duan, and J. Luo (2011). Action recognition using context and appearance distribution features. In IEEE Conference on Computer Vision and Pattern Recognition, 2011. CVPR 2011, pp. 489 – 496. Yan, P., S. Khan, and M. Shah (2008). Learning 4D action feature models for arbitrary view action recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2008. CVPR 2008, pp. 1 – 7. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 50 / 45
Copyright Copyright This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. You are free to copy, distribute and transmit the work under the following conditions: 1) you must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work); 2) you may not use this work for commercial purposes; and 3) you may not alter, transform, or build upon this work. With the understanding that: 1) any of the above conditions can be waived if you get permission from the copyright holder; 2) where the work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license; and 3) in no way are any of the following rights affected by the license: your fair dealing or fair use rights, other applicable copyright exceptions and limitations and rights other persons may have either in the work itself or in how the work is used, such as publicity or privacy rights. Please see http://creativecommons.org/licenses/by-nc-nd/3.0/ for greater detail. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 51 / 45
Copyright Author’s contact details Alexandros Andre Chaaraoui alexandros@ua.es www.alexandrosandre.com Departamento de Tecnolog´ Inform´tica y Computaci´n ıa a o Universidad de Alicante Carretera San Vicente del Raspeig s/n E-03690 San Vicente del Raspeig (Alicante) - Spain Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 52 / 45

More Related Content

PPTX
Simulating Human Behaviour for Understanding and Managing Environmental Resou...
Specken.NL mens-computerinteractie
 
PDF
Talk 2010-monash-seminar-panic-driven-event-detection
Mahfuzul Haque
 
PDF
Dr. rustom kanga latest trends in video analytics for railways
imadhammoud
 
PDF
Architecting Human Behaviour
Andy Budd
 
PDF
cvpr2011: human activity recognition - part 1: introduction
zukun
 
PPT
Activity recognition for video surveillance
variable_orr
 
PPTX
human behaviour presentation
Joseph Wanjala Makomeri
 
PPTX
: Islamic Ethics Human behaviour interaction between nature and nurture
NaimAlmashoori
 
Simulating Human Behaviour for Understanding and Managing Environmental Resou...
Specken.NL mens-computerinteractie
 
Talk 2010-monash-seminar-panic-driven-event-detection
Mahfuzul Haque
 
Dr. rustom kanga latest trends in video analytics for railways
imadhammoud
 
Architecting Human Behaviour
Andy Budd
 
cvpr2011: human activity recognition - part 1: introduction
zukun
 
Activity recognition for video surveillance
variable_orr
 
human behaviour presentation
Joseph Wanjala Makomeri
 
: Islamic Ethics Human behaviour interaction between nature and nurture
NaimAlmashoori
 

Viewers also liked (18)

PPT
Human Behaviour In Public Space2
mariekegelissen
 
PPTX
Activity Recognition using Cell Phone Accelerometers
Ishara Amarasekera
 
PPTX
Human activity recognition
srikanthgadam
 
PPTX
HUMAN BEHAVIOUR
Bhãvíñ Jáïñ
 
PDF
Human activity recognition
Randhir Gupta
 
PPT
Basis Of Human Behaviour 1sem 1 in ob
avinash agarwal
 
PPT
Human behaviour
ArunKRai
 
PPTX
Lecture 10 intruders
rajakhurram
 
PPTX
HUMAN MOTION DETECTION AND TRACKING FOR VIDEO SURVEILLANCE
Aswinraj Manickam
 
PPT
Abnormal Behavior
Alex Holub
 
PPT
Human behaviour theories[1]
tammico
 
PPT
Human Behaviour
nayan_gaurav28
 
PPTX
Human Behavior in Organization by: Prof. Jenny
Jay Gonzales
 
PPTX
Fundamentals of human behavior
Aloy Salas
 
PPTX
Human behaviour.and personality development.ppt
Monika Sharma
 
PPTX
Human-behavior-in-Organization by Parts
International Colleges for Excellence
 
PPT
Human behavior in Organization
Twinkle Constantino
 
PPT
Human behaviour ppt
Nupoor Mehra
 
Human Behaviour In Public Space2
mariekegelissen
 
Activity Recognition using Cell Phone Accelerometers
Ishara Amarasekera
 
Human activity recognition
srikanthgadam
 
HUMAN BEHAVIOUR
Bhãvíñ Jáïñ
 
Human activity recognition
Randhir Gupta
 
Basis Of Human Behaviour 1sem 1 in ob
avinash agarwal
 
Human behaviour
ArunKRai
 
Lecture 10 intruders
rajakhurram
 
HUMAN MOTION DETECTION AND TRACKING FOR VIDEO SURVEILLANCE
Aswinraj Manickam
 
Abnormal Behavior
Alex Holub
 
Human behaviour theories[1]
tammico
 
Human Behaviour
nayan_gaurav28
 
Human Behavior in Organization by: Prof. Jenny
Jay Gonzales
 
Fundamentals of human behavior
Aloy Salas
 
Human behaviour.and personality development.ppt
Monika Sharma
 
Human-behavior-in-Organization by Parts
International Colleges for Excellence
 
Human behavior in Organization
Twinkle Constantino
 
Human behaviour ppt
Nupoor Mehra
 
Ad

Similar to Vision-based Recognition of Human Behaviour for Intelligent Environments (20)

PDF
Continuous human action recognition in ambient assisted living scenarios
Francisco (Paco) Florez-Revuelta
 
PPTX
Human action recognition optimization based on evolutionary feature
Francisco (Paco) Florez-Revuelta
 
PDF
IRJET- Recurrent Neural Network for Human Action Recognition using Star S...
IRJET Journal
 
PPTX
Iciap 2
Ionut Mironica
 
PPTX
Silhouette analysis based action recognition via exploiting human poses
AVVENIRE TECHNOLOGIES
 
PDF
A Multiple Kernel Learning Based Fusion Framework for Real-Time Multi-View Ac...
Francisco (Paco) Florez-Revuelta
 
PDF
Human Action Recognition Using Deep Learning
IRJET Journal
 
PPTX
A general survey of previous works on action recognition
zukun
 
PDF
A COMPARATIVE STUDY ON HUMAN ACTION RECOGNITION USING MULTIPLE SKELETAL FEATU...
mlaij
 
PDF
A COMPARATIVE STUDY ON HUMAN ACTION RECOGNITION USING MULTIPLE SKELETAL FEATU...
mlaij
 
PDF
Motion based action recognition using k nearest neighbor
eSAT Publishing House
 
PDF
E03404025032
theijes
 
PDF
IRJET- Content Based Video Activity Classifier
IRJET Journal
 
PPTX
seminar Islideshow.pptx
rakeshjain627294
 
PPTX
Review paper human activity analysis
Iftikhar Alam
 
PDF
Human action recognition using local space time features and adaboost svm
eSAT Publishing House
 
PDF
A SURVEY ON HUMAN POSE ESTIMATION AND CLASSIFICATION
IRJET Journal
 
PDF
IRJET- Tracking and Recognition of Multiple Human and Non-Human Activites
IRJET Journal
 
PDF
A Intensified Approach on Deep Neural Networks for Human Activity Recognition...
IRJET Journal
 
PPTX
Human Action Recognition in Videos Employing 2DPCA on 2DHOOF and Radon Transform
Fadwa Fouad
 
Continuous human action recognition in ambient assisted living scenarios
Francisco (Paco) Florez-Revuelta
 
Human action recognition optimization based on evolutionary feature
Francisco (Paco) Florez-Revuelta
 
IRJET- Recurrent Neural Network for Human Action Recognition using Star S...
IRJET Journal
 
Iciap 2
Ionut Mironica
 
Silhouette analysis based action recognition via exploiting human poses
AVVENIRE TECHNOLOGIES
 
A Multiple Kernel Learning Based Fusion Framework for Real-Time Multi-View Ac...
Francisco (Paco) Florez-Revuelta
 
Human Action Recognition Using Deep Learning
IRJET Journal
 
A general survey of previous works on action recognition
zukun
 
A COMPARATIVE STUDY ON HUMAN ACTION RECOGNITION USING MULTIPLE SKELETAL FEATU...
mlaij
 
A COMPARATIVE STUDY ON HUMAN ACTION RECOGNITION USING MULTIPLE SKELETAL FEATU...
mlaij
 
Motion based action recognition using k nearest neighbor
eSAT Publishing House
 
E03404025032
theijes
 
IRJET- Content Based Video Activity Classifier
IRJET Journal
 
seminar Islideshow.pptx
rakeshjain627294
 
Review paper human activity analysis
Iftikhar Alam
 
Human action recognition using local space time features and adaboost svm
eSAT Publishing House
 
A SURVEY ON HUMAN POSE ESTIMATION AND CLASSIFICATION
IRJET Journal
 
IRJET- Tracking and Recognition of Multiple Human and Non-Human Activites
IRJET Journal
 
A Intensified Approach on Deep Neural Networks for Human Activity Recognition...
IRJET Journal
 
Human Action Recognition in Videos Employing 2DPCA on 2DHOOF and Radon Transform
Fadwa Fouad
 
Ad

Recently uploaded (20)

PDF
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
PPTX
Spectroscopy.pptx food analysis technology
nawahassan45
 
PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PDF
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
PDF
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
PDF
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
DOCX
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PDF
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
PDF
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
PPT
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
PDF
Approach and Philosophy of On baking technology
30anthuong17
 
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PPTX
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
PPTX
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
Spectroscopy.pptx food analysis technology
nawahassan45
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
Approach and Philosophy of On baking technology
30anthuong17
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 

Vision-based Recognition of Human Behaviour for Intelligent Environments

  • 1. Vision-based Recognition of Human Behaviour for Intelligent Environments Alexandros Andre Chaaraoui Departamento de Tecnolog´ Inform´tica y Computaci´n ıa a o Universidad de Alicante alexandros@ua.es Supervisor: Dr. Francisco Fl´rez-Revuelta o January 20, 2014 Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 1 / 45
  • 3. Introduction Introduction Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 3 / 45
  • 4. Introduction Motivation Introduction (I) Motivation Demographic ageing - Ambient-assisted living (AAL) Intelligent environments Human behaviour analysis Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 4 / 45
  • 5. Introduction Objectives Introduction (II) In this thesis, our goal is to support the development of AAL services for smart homes with advances in human behaviour analysis. Main objectives 1 Establish the research framework 2 Propose a method for the recognition of human behaviour 3 Satisfy specific demands of AAL services 4 Reach robustness for different scenarios Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 5 / 45
  • 6. Research framework Research framework Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 6 / 45
  • 7. Research framework Related work Research framework Proposed taxonomy Figure 1: Human Behaviour Analysis levels — Classification proposed in Chaaraoui et al. (2012). Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 7 / 45
  • 8. Research framework Proposal Conclusions of the analysis Motion Motion, pose and gaze estimation is the most resolved level. Action Action recognition is currently receiving the greatest interest both from research and industry. Activity-Behaviour Activity and long-term behaviour recognition is performed directly based on low-level sensor data, instead of using action recognition. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 8 / 45
  • 9. Research framework Proposal Camera 1 Long-term Analysis Setup (activities, inhabitants, objects…), Profiles and Log Camera 2 ... Camera N Motion Detection Motion Detection ... Motion Detection Human Behaviour Analysis Human Behaviour Analysis ... Human Behaviour Analysis Multi-view Human Behaviour Analysis Environmental Sensor Information Reasoning System Privacy Event Alarm Actuators Caregiver Figure 2: Architecture of the intelligent monitoring system to promote independent living at home and support AAL services. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 9 / 45
  • 10. Contributions Contributions Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 10 / 45
  • 11. Contributions Outline Contributions Proposals have been made at different processing stages: 1 Pose representation 2 Fusion of multiple views 3 Action classification 4 Evolutionary-based optimisation 5 Continuous recognition Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 11 / 45
  • 12. Contributions Pose representation Pose representation Images Silhouettes Pose Representations Figure 3: Outline of the pose representation process. Based on the recorded video frames, foreground segmentations are obtained. Holistic features can then be extracted relying on the shape of the human silhouettes. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 12 / 45
  • 13. Contributions Pose representation Radial Summary (I) Figure 4: Overview of the feature extraction process: 1) All the contour points are assigned to the corresponding radial bin; 2) for each bin, a summary representation is obtained (example with 18 bins). Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 13 / 45
  • 14. Contributions Pose representation Radial Summary (II) Figure 5: Graphical explanation of the statistical range frange of a sample radial bin. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 14 / 45
  • 15. Contributions Fusion of multiple views Fusion of multiple views Multiple views of the same field of view provide Additional characteristic data Advantages with respect to occlusions Advantages with respect to unfavourable viewing angles (ambiguous actions) However, difficulties have been observed The recognition does not necessarily improve Performance issues (temporal and spatial) Burdensome, highly-restricted systems (3D pose estimation, calibrated and synchronised camera networks, ...) Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 15 / 45
  • 16. Contributions Fusion of multiple views Weighted feature fusion scheme Weights are learnt for each view and action: Figure 6: Overview of the feature fusion process of the multi-view pose representation. This example shows five different views of a specific pose taken from the walk action class from the IXMAS dataset (Weinland et al., 2006). Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 16 / 45
  • 17. Contributions Action classification Action classification Key Poses K Key Poses ... ... ... Bag of Key Poses K Key Poses Sequences of Key Poses Figure 7: Outline of the learning stage. Using the pose representations, key poses are obtained for each action. In this way, a bag-of-key-poses model is learnt. The temporal relation between key poses is modelled using sequences of key poses. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 17 / 45
  • 18. Contributions Action classification Bag of key poses Action1 K Key Poses Action2 K Key Poses ... ... ... ... ActionA K Key Poses Bag of Key Poses Figure 8: Learning scheme of the bag-of-key-poses model. For each action class, K key poses are obtained separately and then joined together. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 18 / 45
  • 19. Contributions Action classification Recognition Sequences of Key Poses Sequence Matching DTW Action Recognition Figure 9: Outline of the recognition stage. The unknown sequence of key poses is obtained and compared to the known sequences. Through sequence matching, the action of the video sequence can be recognised. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 19 / 45
  • 20. Contributions Evolutionary-based optimisation Evolutionary-based optimisation Genetic feature subset selection By means of a genetic algorithm for binary feature selection, the interesting body parts can be selected, and redundant or noisy body parts can be ignored. Figure 10: Example of a result provided by genetic feature selection (dismissed radial bins are shaded in grey). Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 20 / 45
  • 21. Contributions Evolutionary-based optimisation Evolutionary-based optimisation Genetic feature subset selection Coevolutionary optimisation Simultaneous selection of training instances, features and parameter values Coevolution enables to split the problem in subproblems of optimisation with a common goal (Wiegand, 2004) Cooperative coevolution allows to consider intrinsic dependencies among optimisation goals (Derrac et al., 2012) Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 21 / 45
  • 22. Contributions Evolutionary-based optimisation Evolutionary-based optimisation Genetic feature subset selection Coevolutionary optimisation Adaptive learning Evolving bag of key poses Supports incremental and adaptive learning of new data Applies selection of training instances, features and parameter values Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 22 / 45
  • 23. Contributions Continuous recognition Continuous recognition In AAL, human action recognition has to be applied to continuous video streams. This requires: To detect meaningful human actions online And to recognise the appropriate action in real-time We propose: To learn action zones, i.e. the most discriminative parts of action performances The usage of a sliding and growing window technique for recognition Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 23 / 45
  • 24. Contributions Continuous recognition Action zones Figure 11: Sample silhouettes of a waving sequence of the DAI RGBD dataset. The action zone that should be extracted is highlighted. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 24 / 45
  • 25. Contributions Continuous recognition Action zones Figure 12: Evidence values H(t) of each action class and the detected action zones are shown for a scratch head sequence of the IXMAS dataset. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 25 / 45
  • 26. Results Results Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 26 / 45
  • 27. Results Evaluation methodology Results Experimentation has been performed on: Single- and multi-view datasets (up to five views) Manually- and automatically-extracted silhouettes (including depth-based segmentation) Using the following cross validations: Leave one sequence out (LOSO) Leave one actor out (LOAO) Leave one view out (LOVO) Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 27 / 45
  • 28. Results Benchmarks Weizmann dataset Table 1: Comparison of recognition rates and speeds obtained on the Weizmann dataset (Gorelick et al., 2007) with other state-of-the-art approaches. Approach ˙ Ikizler and Duygulu (2007) Tran and Sorokin (2008) Fathi and Mori (2008) Actions 9 10 10 Test LOSO LOSO LOSO Rate 100% 100% 100% fps N/A N/A N/A Hern´ndez et al. (2011) a Cheema et al. (2011) Sadek et al. (2012) 10 9 10 LOAO LOSO LOAO 90.3% 91.6% 97.8% 98 56 18 Our approach Our approach 10 10 LOSO LOAO 93.5% 97.8% 188 188 Optimised approach 10 LOAO 100% 210 Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 28 / 45
  • 29. Results Benchmarks MuHAVi-14 dataset Table 2: Comparison of recognition rates and speeds obtained on the MuHAVi-14 dataset (Singh et al., 2010) with other state-of-the-art approaches. Approach LOSO LOAO LOVO fps Singh et al. (2010) Eweiwi et al. (2011) 82.4% 91.9% 61.8% 77.9% 42.6% 55.8% N/A N/A Cheema et al. (2011) 86.0% 73.5% 50.0% 56 98.5% 94.1% 59.6% 99 100% 100% - - Our approach Optimised approach Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 29 / 45
  • 30. Results Benchmarks MuHAVi-8 dataset Table 3: Comparison of recognition rates and speeds obtained on the MuHAVi-8 dataset (Singh et al., 2010) with other state-of-the-art approaches. M´todo e LOSO LOAO LOVO fps Singh et al. (2010) Mart´ ınez-Contreras et al. (2009) Eweiwi et al. (2011) 97.8% 98.4% 98.5% 76.4% 85.3% 50.0% 38.2% N/A N/A N/A Cheema et al. (2011) 95.6% 83.1% 57.4% 56 Our approach 100% 100% 82.4% 98 Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 30 / 45
  • 31. Results Benchmarks IXMAS dataset Table 4: Comparison with other multi-view human action recognition approaches of the state of the art. The rates obtained in the LOAO cross validation performed on the IXMAS dataset are shown. Approach Actions Actors Views Rate fps Yan et al. (2008) Wu et al. (2011) Cilla et al. (2012) Weinland et al. (2006) Cilla et al. (2013) Holte et al. (2012) 11 12 11 11 11 13 12 12 12 10 10 12 4 4 5 5 5 5 78% 89.4% 91.3% 93.3% 94.0% 100% N/A N/A N/A N/A N/A N/A Cherla et al. (2008) Weinland et al. (2010) 13 11 N/A 10 4 5 80.1% 83.5% 20 ∼500 Our approach 11 12 5 91.4% 207 Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 31 / 45
  • 32. Results Benchmarks RGBD datasets DAI RGBD dataset Table 5: Cross validation results obtained on our multi-view depth dataset. Approach LOSO LOAO fps Our approach 94.4% 100% 80 DHA dataset Table 6: LOSO cross validation results obtained on the DHA dataset (Lin et al., 2012) (10 Weizmann actions). Approach LOSO Lin et al. (2012) Our approach Alexandros Andre Chaaraoui (UA) fps 90.8% N/A 95.2% 99 PhD Thesis January 20, 2014 32 / 45
  • 33. Results Benchmarks Continuous recognition Table 7: Obtained results applying CHAR and segment analysis evaluation (LOAO). Results are detailed using the segmented sequences or the proposed action zones. Dataset Approach F1 -measure IXMAS IXMAS Segmented sequences Action zones 0.504 0.705 Weizmann Weizmann Segmented sequences Action zones 0.693 0.928 Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 33 / 45
  • 34. Concluding remarks Concluding remarks Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 34 / 45
  • 35. Concluding remarks Discussion Discussion Silhouettes 2D silhouettes can be difficult to obtain and they are view dependant Privacy Privacy concerns of indoor monitoring Intelligent monitoring system with privacy protection The method only relies on the boundary of the silhouette Validation of the proposed method The classification method based on the bag of key poses has also been validated for gaming and NUI (Chaaraoui et al., 2014, 2013; Climent-P´rez et al., 2013) e Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 35 / 45
  • 36. Concluding remarks Conclusions Conclusions 1 Proposal of a 2D template-based non-parametric method for human action recognition & optimisations and extensions 2 Specific demands of AAL services have been satisfied: relaxed camera setup requirements, adaptive learning, continuous recognition and real-time execution 3 The HAR method based on a bag-of-key-poses model handles single- and multi-view recognition proficiently. 4 State-of-the-art recognition rates have been achieved, outperforming the best known rates in several benchmarks. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 36 / 45
  • 37. Concluding remarks Future work Future work Future directions of this work: Pose representation: Other local and global features Key poses: Generation algorithms Bag-of-key-poses model: Applications Distance metrics: Key poses and sequences of key poses Evaluation and deployment However, two main future lines stand out: Recognition of complex activities based on action sequences and multi-modal data Feature fusion techniques, e.g. for recognition of subtle movements or gestures Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 37 / 45
  • 38. Other information and details Other information and details Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 38 / 45
  • 39. Other information and details Research projects and grants Research projects and grants Intelligent system for follow-up and promotion of personal autonomy: Computer vision system for the monitoring of activities of daily living at home – Sistema de visi´n para la o monitorizaci´n de la actividad diaria en el hogar. Spanish Ministry of Science and o Innovation and Valencian Ministry of Education, Culture and Sport (TALISMAN+, Technical University of Madrid, University of Deusto, University of Castile-La Mancha and University of Alicante) PhD. Research Fellowship – Programa VALi+d para investigadores en formaci´n. Valencian Ministry of Education, Culture and Sport (ACIF/2011/160) o Research Collaboration Stay – Digital Imaging Research Centre, Faculty of Science, Engineering and Computing, Kingston University. Kingston upon Thames, UK. (Funded by VALi+d, BEFPI/2013/015) Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 39 / 45
  • 40. Other information and details Other activities Other activities Teaching collaboration – Information Technology. First year core subject of Degree in Sound and Image in Telecommunication Engineering Reviewer of international journals – Neurocomputing (Elsevier), Pervasive Computing (IEEE), EURASIP Journal on Image and Video Processing (Springer), Expert Systems With Applications (Elsevier) Conference session chair – IEEE/RSJ Intelligent Robots and Systems (IROS 2012), Genetic and Evolutionary Computation Conference (GECCO 2013) Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 40 / 45
  • 41. Other information and details Intl. review International review This thesis has been approved by the following reviewers for the International PhD Honourable Mention Dr. Jean-Christophe Nebel (Kingston University, UK) Dr. Jes´s Mart´ u ınez del Rinc´n o (Queen’s University of Belfast, UK) Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 41 / 45
  • 42. Other information and details Publications Publications Journals I Chaaraoui, A.A., Climent-P´rez, P., Fl´rez-Revuelta, F., 2012. A Review on e o Vision Techniques Applied to Human Behaviour Analysis for Ambient-Assisted Living. Expert Systems with Applications. Citations: 10 II Chaaraoui, A.A., Climent-P´rez, P., Fl´rez-Revuelta, F., 2013. e o Silhouette-based Human Action Recognition Using Sequences of Key Poses. Pattern Recognition Letters. Citations: 6 III Chaaraoui, A.A., Fl´rez-Revuelta, F., 2013. Optimizing Human Action o Recognition Based on a Cooperative Coevolutionary Algorithm. Engineering Applications of Artificial Intelligence. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 42 / 45
  • 43. Other information and details Publications Publications IV Chaaraoui, A.A., Padilla-L´pez, J.R., Climent-P´rez, P., Fl´rez-Revuelta, F., o e o 2014. Evolutionary Joint Selection to Improve Human Action Recognition with RGB-D Devices. Expert Systems with Applications. V Chaaraoui, A.A., Fl´rez-Revuelta, F., 2014. A Low-Dimensional Radial o Silhouette-based Feature for Fast Human Action Recognition Fusing Multiple Views. Information Fusion. Under review VI Chaaraoui, A.A., Fl´rez-Revuelta, F., 2014. Adaptive Human Action o Recognition With an Evolving Bag of Key Poses. IEEE Transactions on Autonomous Mental Development. Under review Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 43 / 45
  • 44. Other information and details Publications Publications Conferences and workshops I Intl. Symposium on Ambient Intelligence (ISAmI 2012) II 3rd Intl. Workshop on Human Behavior Understanding (HBU 2012), IEEE/RSJ Intl. Conference on Intelligent Robots and Systems (IROS 2012) III 11th Mexican Intl. Conference on Artificial Intelligence (MICAI 2012) IV Genetic and Evolutionary Computation Conference (GECCO 2013) V 5th Intl. Work-conference on Ambient Assisted Living (IWAAL 2013) VI 3rd Workshop on Consumer Depth Cameras for Computer Vision (CDC4CV13), IEEE Intl. Conference on Computer Vision (ICCV 2013) Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 44 / 45
  • 45. Other information and details Publications “We can only see a short distance ahead, but we can see plenty there that needs to be done.” (Turing, 1950) Vision-based Recognition of Human Behaviour for Intelligent Environments PhD Thesis Alexandros Andre Chaaraoui Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 45 / 45
  • 46. References References Chaaraoui, A. A., P. Climent-P´rez, and F. Fl´rez-Revuelta (2012). A review on vision e o techniques applied to human behaviour analysis for ambient-assisted living. Expert Systems with Applications 39 (12), 10873 – 10888. Chaaraoui, A. A., J. R. Padilla-L´pez, P. Climent-P´rez, and F. Fl´rez-Revuelta (2014). o e o Evolutionary joint selection to improve human action recognition with RGB-D devices. Expert Systems with Applications 41 (3), 786 – 794. Methods and Applications of Artificial and Computational Intelligence. Chaaraoui, A. A., J. R. Padilla-L´pez, and F. Fl´rez-Revuelta (2013). Fusion of skeletal o o and silhouette-based features for human action recognition with RGB-D devices. In IEEE 14th International Conference on Computer Vision Workshops, 2013. ICCV Workshops 2013. To be presented in 3rd Workshop on Consumer Depth Cameras for Computer Vision (CDC4CV13). Cheema, S., A. Eweiwi, C. Thurau, and C. Bauckhage (2011). Action recognition by learning discriminative key poses. In IEEE 13th International Conference on Computer Vision Workshops, 2011. ICCV Workshops 2011, pp. 1302 –1309. Cherla, S., K. Kulkarni, A. Kale, and V. Ramasubramanian (2008). Towards fast, view-invariant human action recognition. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2008. CVPRW 2008, pp. 1 – 8. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 46 / 45
  • 47. References References Cilla, R., M. A. Patricio, A. Berlanga, and J. M. Molina (2012). A probabilistic, discriminative and distributed system for the recognition of human actions from multiple views. Neurocomputing 75 (1), 78 – 87. Brazilian Symposium on Neural Networks (SBRN 2010), International Conference on Hybrid Artificial Intelligence Systems (HAIS 2010). Cilla, R., M. A. Patricio, A. Berlanga, and J. M. Molina (2013). Human action recognition with sparse classification and multiple-view learning. Expert Systems. DOI 10.1111/exsy.12040. Climent-P´rez, P., A. A. Chaaraoui, J. R. Padilla-L´pez, and F. Fl´rez-Revuelta (2013). e o o Optimal joint selection for skeletal data from RGB-D devices using a genetic algorithm. In I. Batyrshin and M. Mendoza (Eds.), Advances in Computational Intelligence, Volume 7630 of Lecture Notes in Computer Science, pp. 163 – 174. Springer Berlin / Heidelberg. Derrac, J., I. Triguero, S. Garc´ and F. Herrera (2012). A co-evolutionary framework for ıa, nearest neighbor enhancement: Combining instance and feature weighting with instance selection. In E. Corchado, V. Sn´ˇel, A. Abraham, M. Wo´niak, M. Gra˜a, and S.-B. as z n Cho (Eds.), Hybrid Artificial Intelligent Systems, Volume 7209 of Lecture Notes in Computer Science, pp. 176 – 187. Springer Berlin / Heidelberg. Eweiwi, A., S. Cheema, C. Thurau, and C. Bauckhage (2011). Temporal key poses for human action recognition. In IEEE 13th International Conference on Computer Vision Workshops, 2011. ICCV Workshops 2011, pp. 1310 –1317. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 47 / 45
  • 48. References References Fathi, A. and G. Mori (2008). Action recognition by learning mid-level motion features. In IEEE Conference on Computer Vision and Pattern Recognition, 2008. CVPR 2008, pp. 1 – 8. Gorelick, L., M. Blank, E. Shechtman, M. Irani, and R. Basri (2007). Actions as space-time shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence 29 (12), 2247 – 2253. Hern´ndez, J., A. Montemayor, J. Jos´ Pantrigo, and A. S´nchez (2011). Human action a e a ´ recognition based on tracking features. In J. Ferr´ndez, J. Alvarez S´nchez, F. de la Paz, a a and F. Toledo (Eds.), Foundations on Natural and Artificial Computation, Volume 6686 of Lecture Notes in Computer Science, pp. 471 – 480. Springer Berlin / Heidelberg. Holte, M., B. Chakraborty, J. Gonzalez, and T. Moeslund (2012). A local 3-D motion descriptor for multi-view human action recognition from 4-D spatio-temporal interest points. IEEE Journal of Selected Topics in Signal Processing 6 (5), 553 – 565. ˙ Ikizler, N. and P. Duygulu (2007). Human action recognition using distribution of oriented rectangular patches. In A. Elgammal, B. Rosenhahn, and R. Klette (Eds.), Human Motion - Understanding, Modeling, Capture and Animation, Volume 4814 of Lecture Notes in Computer Science, pp. 271 – 284. Springer Berlin / Heidelberg. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 48 / 45
  • 49. References References Lin, Y.-C., M.-C. Hu, W.-H. Cheng, Y.-H. Hsieh, and H.-M. Chen (2012). Human action recognition and retrieval using sole depth information. In Proceedings of the 20th ACM international conference on Multimedia, MM ’12, New York, NY, USA, pp. 1053 – 1056. ACM. Mart´ ınez-Contreras, F., C. Orrite-Urunuela, E. Herrero-Jaraba, H. Ragheb, and S. Velastin (2009). Recognizing human actions using silhouette-based HMM. In IEEE Int. Conference on Advanced Video and Signal Based Surveillance, 2009. AVSS 2009, pp. 43 – 48. Sadek, S., A. Al-Hamadi, B. Michaelis, and U. Sayed (2012). A fast statistical approach for human activity recognition. International Journal of Intelligence Science 2 (1), 9 – 15. Singh, S., S. Velastin, and H. Ragheb (2010). MuHAVi: A multicamera human action video dataset for the evaluation of action recognition methods. In IEEE Int. Conference on Advanced Video and Signal Based Surveillance, 2010. AVSS 2010, pp. 48 – 55. IEEE. Tran, D. and A. Sorokin (2008). Human activity recognition with metric learning. In D. Forsyth, P. Torr, and A. Zisserman (Eds.), European Conference on Computer Vision. ECCV 2008, Volume 5302 of Lecture Notes in Computer Science, pp. 548 – 561. Springer Berlin / Heidelberg. Turing, A. M. (1950). Computing machinery and intelligence. Mind 59 (236), 433 – 460. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 49 / 45
  • 50. References References ¨ Weinland, D., M. Ozuysal, and P. Fua (2010). Making action recognition robust to occlusions and viewpoint changes. In K. Daniilidis, P. Maragos, and N. Paragios (Eds.), European Conference on Computer Vision. ECCV 2010, Volume 6313 of Lecture Notes in Computer Science, pp. 635 – 648. Springer Berlin / Heidelberg. Weinland, D., R. Ronfard, and E. Boyer (2006). Free viewpoint action recognition using motion history volumes. Computer Vision and Image Understanding 104 (2-3), 249 – 257. Wiegand, R. P. (2004). An analysis of cooperative coevolutionary algorithms. Ph. D. thesis, George Mason University, Fairfax, VA, USA. Wu, X., D. Xu, L. Duan, and J. Luo (2011). Action recognition using context and appearance distribution features. In IEEE Conference on Computer Vision and Pattern Recognition, 2011. CVPR 2011, pp. 489 – 496. Yan, P., S. Khan, and M. Shah (2008). Learning 4D action feature models for arbitrary view action recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2008. CVPR 2008, pp. 1 – 7. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 50 / 45
  • 51. Copyright Copyright This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. You are free to copy, distribute and transmit the work under the following conditions: 1) you must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work); 2) you may not use this work for commercial purposes; and 3) you may not alter, transform, or build upon this work. With the understanding that: 1) any of the above conditions can be waived if you get permission from the copyright holder; 2) where the work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license; and 3) in no way are any of the following rights affected by the license: your fair dealing or fair use rights, other applicable copyright exceptions and limitations and rights other persons may have either in the work itself or in how the work is used, such as publicity or privacy rights. Please see http://creativecommons.org/licenses/by-nc-nd/3.0/ for greater detail. Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 51 / 45
  • 52. Copyright Author’s contact details Alexandros Andre Chaaraoui alexandros@ua.es www.alexandrosandre.com Departamento de Tecnolog´ Inform´tica y Computaci´n ıa a o Universidad de Alicante Carretera San Vicente del Raspeig s/n E-03690 San Vicente del Raspeig (Alicante) - Spain Alexandros Andre Chaaraoui (UA) PhD Thesis January 20, 2014 52 / 45