Chapter 1_Introduction.docx
- 1. Chapter 1: Introduction This chapter presents the background of the research area, presenting an overview of Human Action Recognition (HAR) and the challenges and applications of HAR. Moreover, the need for HAR system, motivation behind this research, the research objectives, social contributions and the organization of the thesis are also discussed. 1.1. Introduction about Computer Vision The most superior sense of the human-being is vision. It canautomatically gain high level understanding of images and videos. In the sameway, computer vision research area focuses on understanding the content of imagesand videos by computers i.e, enabling computers to see. Computer vision involvesautomatically extracting the information from images or videos. This informationcan be collected from 2D or 3D models and detected objects. Finally, recognitionis performed by grouping or tracking of collected information. Computer vision isused in wide range of application fields due to their importance in real- world. The applications include Optical Character Recognition (OCR), machine inspection, 3D model building, medical imaging, automatic safety, sports pose estimation, motion capture, surveillance, finger print recognition, biometrics, etc. The typical security surveillance uses multiple cameras and requireintense man-power for monitoring the content of the video. The continuousmonitoring of multiple cameras by a human may cause error. On the other hand,smart surveillance systems aim at automatically detecting moving objects in videosand understanding the video content. This thesis focuses on actions performed by a single person. An action isa combination of multiple atomic actions. For instance, boxing, running, joggingand jumping are actions; the atomic actions involved in boxing are boxing position,left hand forward, right hand forward, right leg forward and left leg forward.Recognition of human actions find importance in various application areas suchas video surveillance, sign language processing, video indexing and sports videoanalysis. Automatic human activity Recognition is a challenging problemin computer vision. One of its main goals is the understanding of the complexhuman visual system and the knowledge of how humans represent the humanactivity to discriminate different identities with high accuracy.This kind of systems, namely human activity detection andrecognition of the detected human activity, must address two fundamental andconceptually independent problems. Work on the recognition stage takes thedetected human activity values as input to the algorithm. This stage can be separated in
- 2. two steps: feature extraction,where vital information for discrimination is saved, and the matching step,where the recognition result is given with the aid of a human activity database. 1.2. Human Activity Recognition The research in the fields of computer vision has dealt with the analysis of the temporal sequence of image data. The vast availability of low cost and high-quality digital cameras with massive storage capabilities, and high-speed bandwidths has all contributed to realize the role of video processing in various application domains. Also, human biological visual systems are well managed to handle spatiotemporal information, which is yet another reason for the popularity of video data analysis. Video-based tracking and analysis help to observe the motion patterns in a non-intrusive way. This task must be well modeled when dealt with human subjects. Video surveillance systems play a significant role in the circumstances where continuous observation by visual analysts is not possible. Typically, visual analysts continuously monitor and collect data from various cameras and report to the authorities when the necessity arises. However, it is manpower intensive. Hence, it is necessary to build an automatic human action recognition system and build a higher-level behavior modeling for the events occurring in the scene. In the computer vision community, ‘Action’ and ‘Activity’ are frequently interchangeable terms. In this thesis, ‘Action’ is defined as simple motion patterns which are usually exhibited by a single person for a concise duration. Examples of activities include running, walking, and waving. On the other hand, ‘Activity’ is defined as a complex sequence of actions where several humans are involved, and they interact with each other in a constrained manner. Activities are typically characterized by much longer temporal durations. Examples of activities are two persons shaking hands, a football team scoring a goal and a coordinated bank robbery multiple people. Recognition of human actions from multiple views needs high-level interactions with broad applications such as video surveillance, human-computer interaction, motion analysis, video indexing, and sports video analysis. The challenges of human action recognition come fromdifficulties such as significant intra-class variance, scaling, occlusion, and clutter. Human action recognition has been extensively researched through methods based on local and holistic representations. HAR is a highly dynamic system that finds the actions of persons based on videos acquired from observation and information about the context of the monitored activities. An activity is identified irrespective of its environment and the person who performs the action.
- 3. In HAR process, different steps are involved in gathering data from raw data to identification of the performed activity. The essential objective of HAR systems is to monitor and examine human activities for recognizing the ongoing events. HAR systems recover and develop contextual data with the aid of visual and non-visual sensory data to recognize the human behavior. A dictionary-based approach was implemented for sparse characterization of noisy and radio frequency identification (RFID) streaming signals. The human actions are detected with better representation of activities using optimal data. This is followed by orthogonal matching pursuit to address sparse optimization problems. In addition, features are mined from unrefined signal strength stream with the aid of rank-based feature selection method. However, robust features are not extracted for achieving effective activity recognition. Generally, HAR is classified into the following three stages of representations: a) Low-level core technology b) Mid-level human activity recognition systems c) High-level applications Low-level Core Technology The main processing stages involved in low level core technology are Object segmentation Feature extraction and representation Activity detection and classification algorithms Initially, human object is segmented and their features are extracted from the video sequence as a set of characteristics. Then, categorization approach is implemented on mined features for recognizing the different human actions. Mid-level human activity recognition systems The second level of human activity recognition systems includes the following three significant tasks. Single person activity recognition Multiple people interaction and crowd behavior Abnormal activity recognition High-level applications
- 4. The application in which results acquired through processing of raw data are explained in high-level applications of HAR systems. 1.2.1 Local Representations Methods based on local representation, also known as local methods, encode a video sequence as a collection of local spatio-temporal features (local descriptors). These local descriptors are extracted from spatiotemporal interest points (STIPs) which can be sparsely detected from video sequences by detectors. In contrast to holistic representations of human actions, local methods enjoy many advantages. Avoidance of some preliminary steps, e.g., background subtraction and target tracking required in holistic methods. Resistance to background variation and occlusions. However, local representations also have disadvantages, of which a fundamental limitation is that it can be too local, as it is not possible to capture adequate spatial and temporal information. Local descriptors can also be obtained from trajectories. It indicates that it is enough to distinguish human actions by the tracking of joint positions. One of the advantages of using trajectories is being discriminative. Nevertheless, the performance of trajectory-based methods depends on the quality of these trajectories, and in practice extracting trajectories from video sequences would be computationally expensive. To obtain the final representation of an action, the Bag-of-Words (BoW) model has been widely used and has achieved excellent results in human action recognition tasks. The BoW model is based on mapping local features of each video sequence onto a pre-learned dictionary, which unavoidably introduces quantization errors during its creation. The errors would be propagated to the final representation and harm the recognition performance. Additionally, the size of this dictionary needs to be empirically determined, and code words, i.e., the cluster centers, obtained by k-means, gather around dense regions of local feature space, resulting in less effective code words of action primitives. Sparse representation has recently been introduced for action representation based on local features. 1.2.2 Holistic Representations Methods based on holistic representation also called as global methods, treat a video sequence rather than applying sparse sampling using STIP detectors or extracting trajectories. In holistic representations, spatio-temporal features are directly learned from the raw frames
- 5. in video sequences. Holistic representations have recently drawn increasing attention because they can encode more visual information by preserving spatial and temporal structures of actions occurring in a video sequence. However, holistic representations are highly sensitive to partial occlusions and background variations. Additionally, they often require preprocessing steps, such as background subtraction, segmentation, and tracking, which makes it computationally expensive and even intractable in some realistic scenarios. 1.3. Challenges of Automatic Human Activity Recognition Some of the major Challenges related to HAR in videos are summarized as follows: • Spatial variations Recognition of human actions in the presence of spatial variations such as translation, scaling (Variation in sizes) and rotation is challenging. • Lighting variations Rapid changes in lighting cause erroneous human recognition both in indoor and outdoor videos. In indoor scenario, abrupt switch on or off of light may lead to difficulty in recognition. In outdoor scenario, drastic change of weather, which may include changes in lighting, sudden rain, cloudy weather, etc. may source false positive object recognition. • Occlusions Occlusions created when two or more objects appear in the same frame with an object hiding the other introduce difficulty in recognition. The object of interest can be either partially occluded or fully occluded. It affects the visibility of all the body parts of humans in the video. • Background clutter Recognition of moving object in outdoor environment is hard. For instance, swinging tree branches or leaves, birds in sky, clouds, waves, smog establish irregular movements or periodic changes in the background. • Moving shadow Reflection of the object of interest in moving environment causes problem in HAR, because shadow moves along with objects. Separating the shadow region from the object is hard.
- 6. • Appearance of moving object The appearance of the human may be transformed when the person is in motion. For instance, the front view of the human is different from the top view or side view. Appearance of moving human may be different. Automatic human activity Recognition is a challenging problem in computer vision. One of its main goals is the understanding of the complex human visual system and the knowledge of how humans represent the human activity to discriminate different identities with high accuracy. This kind of systems must address two fundamental and conceptually independentproblems such as Human activity detection and Recognition of the detected human activity. The focus of this work is on the recognition stage, i.e. taking the detected human activity as the input to the algorithm. This stage can be separated into two steps: 1. Feature extraction, where vital information for discrimination is saved, and 2. Matching step, where the recognition result is given with the aid of a human activity database. Vision is the most superior of human senses, and it is no surprise that images and videos convey more critical information during human perception. Image processing paved a broad spectrum of application fields due to their variety of light energy, namely visible, Ultraviolet, X-rays, Gamma rays, Infrared, Microwaves, and radio waves. In general, image processing is distinguished as low level, mid-level, and high-level processing. The lower level processing, also termed as image preprocessing, involves primitive operations on images such as noise removal, contrast enhancement, and image sharpening. The mid-level or intermediate processing on images involves tasks such as segmentation, object representation, description, and classification. The higher-level processing involves image recognition, image understanding, or computer vision. In this context, HAR has gained a lot of attention in recent times and has potential applications in various areas. 1.4. Application of HAR The remarkable perspective in research on HAR becomes evident when looking at the applications that may be benefited from the research works in this domain. Some application
- 7. areas that highlight the potential impact of vision-based action recognition system are listed in Table 1.1 and discussed in this section. Table 1.1 Applications of Human Motion Analysis. S.No. Domain Filed of application 1. Kinesiology Clinical orthopedic studies Biomechanics Person- centric learning 2. Computer Graphics and Animation Virtual reality Games Animated movies Teleconferencing Games 3. Behavioral Biometrics Gesture driven control Gait analysis Human behavior learning 4. Content-based Video Analysis Video indexing, Video retrieval Video summarization, Storyboarding Sports video analysis 5. Surveillance and Security Parking lots, Anomaly detection Vehicle tracking Human action recognition 6. Human Computer Interaction Human Computer Interface(HCI) Ambient intelligence Expression recognition i. Kinesiology: Kinesiology, the study of biomechanics, involves the development of human body models aiming to improve the efficiency of human movement. This is widely used in medical studies of orthopedic patients, and such studies need detailed information about the movement of body parts and joints. This information is gathered in an intrusive way by placing retro-reflective markers or light emitting diodes (LED) on the human body. ii. Computer Graphics and Animation: HAR can be used to develop a high-level description of movements of dance, ballet, and sports field. This has been used to study and synthesize realistic motion patterns of virtual world humans. iii. Behavioral Biometrics: Biometrics involves the study of approaches and algorithms for uniquely recognizing humans, based on human physical and behavioral cues. The appearance-based tracking approaches solve the problems of gait recognition and gesture recognition. The physical body parts are marked, and their motion trajectories are mapped to identify the human. This domain has a
- 8. broad spectrum of potential applications which have been discussed by researchers in the fields of hand gesture-driven control of robots, human gait, abandoned luggage detection, emotion recognition, and behavior learning. iv. Content-based Video Analysis: Since internet service providers (ISP) face persistent growth, it has become compulsory to develop efficient indexing and storage schemes to improve the user feedbacks for content-based video searching. This involves the learning of flow patterns from raw video and summarizing the content of the video. Discussed the enhanced content-based video summarization with corresponding advances in content-based image retrieval. The standard approach for indexing and retrieval is to query the video database using semantic action descriptors like ‘videos where the person kicks somebody.’ So, the retrieval is the kind of process to deliver the analogous videos according to action-based features in the query video. v. Surveillance and Security: Video surveillance and related security systems mostly rely on multiple video cameras, monitored by a human operator who should know the activity of interest in the field of view. The monitoring efficiency and accuracy of the human operators can be stretched for increasing the number of cameras and their deployments. Hence, security agencies are seeking vision-based solutions which can replace or assist a human operator. Many researchers concentrated on anomalies in automatic recognition problem. Generally, the existing surveillance techniques performed a post operation task and detected the abnormal events based on the actions that have been committed. These techniques, many times, need manual intervention for real-time detection. However, activity detection systems incorporated with ‘Smart Surveillance Systems’ (SSS) in real time detection are also possible without manual intervention and are used in banks, ATM centers, parking lots, supermarkets and department stores. So, the conflicts in such surveillance applications are the main factors of SSS which affect the privacy of human beings. vi. Human-Computer Interface: Understanding the interaction between a computer and a human remains one of the enduring challenges in designing human- computer interfaces. 1.5. Recognition of Actions in Human
- 9. Activity recognition is a new ground for the development of robust machine learning techniques, as applications in this field typically require to deal with high-dimensional, multimodal streams of data that are characterized by a large variability (e.g., due to changes in the user's behaviour or as a result of noise). However, unlike other applications, there is a lack of established benchmarking problems for activity recognition. Typically, each research group tests and reports the performance of their algorithms on their datasets using experimental setups specially conceived for that specific purpose.For this reason, it is difficult to compare the performance of different methods or to assess how a technique will perform if the experimental conditions change (e.g., in case of sensor failure or changes in sensor location). We intend to address this issue by setting up a challenge on activity recognition aimed to provide a common platform that allows the comparison of different machine learning algorithms on the very same conditions. This is called for methods for tackling critical questions in activity recognition such as classification based on multimodal recordings, activity spots, and robustness to noise — methods in activity recognition using semantic features. Unlike low-level features, semantic features describe inherent characteristics of activities. Therefore, semantics make the recognition task more reliable, especially when the same actions look visually different due to the variety of action executions.This defines a semantic space, including the most popular semantic features of action, namely the human body (pose and pose let), attributes, related objects, and scene context. This work presents methods exploiting these semantic features to recognize activities from still images and video data as well as four groups of activities: atomic actions, people interactions, human–object interactions, and group activities. 1.6. Motion Analysis The conventional Human Motion Analysis (HMA) system can detect, track, and identify the moving humans in a video sequence and may include recognizing their action. The types of interaction of the HMA system with the environment are as follows: Passive - It captures merely and stores the visual information in an organized fashion without performing any analysis. Active - It controls and adjusts the acquisition device parameters, namely pan, zoom, and tilt effects, depending on the external environment conditions. The HMA typically has the following three necessary steps:
- 10. I. Detection: It involves finding the answer, ‘Is there motion (corresponding to a human) present in the scene?’ and essentially requires low-level processing of images. II. Tracking: It answers the question: ‘Where is the human moving?’. The tracking is of significant importance to HMA. It needs history to be maintained for action recognition, and it involves the mid-level processing on the history of images. However, sometimes there may be considerable overlap between detection and tracking algorithms. III. Action Recognition / Behavior Understanding: It is a high-level vision step, which involves interpreting the information derived in the steps as mentioned above in order to answer the question ‘What is the human doing?’ 1.6.1 Properties of Shape Features The machine vision has super excellent performance towards complicated problem solving with small complexity. Also, choosing appropriate features for a shape recognition system must consider the kind of features suitable for the task. The techniques to describe the shape of a deformable object have been extensively studied by researchers for the past few decades compared many shape representation techniques to characterize a deformable object. These techniques belong to the following three main categories: (i) Boundary-based and region-based methods (ii) Spatial domain and frequency domain methods (iii) Information preserving and non-information preserving methods In a boundary-based shape description, the shape boundary points are only used to represent the shape. However, in the region-based methods, the combination of boundary and interior points are used to represent the shape. In the spatial domain methods, the point feature basis helps to compare two shapes, and the vector basis is used in frequency domain methods. The information preserving method provides an accurate shape reconstruction from the shape descriptors, and the non- information preserving methods offer only partial reconstruction with a compromise on the identifiable property of shape features. The identified shape features should have the following essential properties irrespective of the description strategy:
- 11. Identifiable: The shapes which are similar to human perception should have the closest feature correspondence. Spatial invariance: The spatial transformations such as translation, scaling, and rotation should not affect the extracted features. Noise resistance: Features must be as robust as possible against noise, i.e., the noise strength of specific range should not affect the pattern. Occultation invariance: The original shape features should not be affected during the occultation. Statistically independent: Two distinct features must be statistically independent and must ensure the compactness of the representation. Reliable: To ensure minimum intra class variability. The widely used methods for human shape representation in HMA are as follows: Bounding Box - This is the smallest rectangle that contains every point in the object shape. The shape can either have a global bounding box or a collection of bounding boxes which describe the human body parts, namely head, torso, legs, hands, and feet. However, this representation conveys the coarse features only. Moments - The magnitude of a set of orthogonal complicated moments of the image known as Zernike moments is most suitable for shape similarity-based image retrieval in terms of compact representation, robustness, and retrieval performance But, the presence of many factorial terms of Zernike moments leads to complex Computation. Chain code - introduced the chain code approach, which represented the complete boundary description of the shape that ensures retaining of the shape. The chain code cannot tolerate scale and rotation variations, which are common problems that occur in object tracking. Fourier descriptors – The high-frequency components of Fourier descriptor are associated with corners, and low frequencies are associated with shape’s border. The Fourier descriptor is used to transform a set of boundary pixels from the spatial domain representation into the frequency domain. The Fourier descriptors are also invariant to scale rotation and starting point of the shapes.
- 12. So, the accuracy of any pattern recognition system is based on the selection of appropriate shape feature. 1.7. Video Sequence analysis A sequence of images displayed at a specific rate of frequency (frame per second) is termed as a video sequence. It consists of information in the form of spatial changes concerning time and if someone wants to extract the information from a video sequence, then the spatial change concerning time must be perceived. For the video sequence analysis (VSA), the knowledge of core technologies of digital image processing is the fundamental requirement, which includes image enhancement, image segmentation, morphological operations, feature extraction and representation, image classification. The main aim of this analysis is to automatically detect and determine the spatiotemporal event in the video signal and further to interpret the nature of the event. In this analysis, the identification of appropriate objects/regions in the scene (segmentation), and most detailed characteristics of each object/region in the entire scene (feature extraction) is extracted. In both the cases, the segmentation, and feature extraction the spatial and temporal dimensions must be taken into consideration for the effective representation of object/region. Feature extraction is useful for both coding and indexing, and the adequate coding parameters for each of the object/region are set. A similar process is used for segmentation, to identify the objects/regions, which permits and distinct coding and unwraps enhanced interaction possibilities. Similarly, the capability to define content in an object/region-based technique increases the plenty of the description. Since last few decades, the VSA using intelligent techniques has emerged as a promising and exciting area of research in the field of computer vision and image processing due to the critical issues and numerous applications of this analysis. 1.7.1. Applications of VSA There are numerous applications of VSA, and these are broadly categorized as: Entertainment: One of important application which is directly related to the daily life of human beings like television (TV), movies, high definition television (HDTV) transmission, video games, live streaming of sports analysis.
- 13. Commercial: VSA can be used for commercial purposes such as smart CCTV, advertisement of the product, in the retail industry for tracking of shoppers inside the store. Security and Surveillance: Most widely used application for the security and safety purpose by military and police. It can be monitoring of crowd behavior at public functions, terrorist activities at public places like airports, railways stations, bus stands, robbery detection, home intrusion system. Human-Computer Interaction: Nowadays, the interaction of human with the machine is increasing rapidly due to the advancement of VSA technologies. The traditional way of interaction of human with the machine is through remote control, keyboard, mouse, joystick. But in the coming years, these modes of interactions may become obsolete due to the invention of various recognition systems based on body pose, hand gesture, facial expression. Motion Analysis: There are a variety of systems, where VSA is used to detect and determine the motion of the object. The effective detection, tracking, and recognition of an object leads to several critical applications like human activity recognition system for detecting various kind of human abnormal and regular activities, object tracking, intruder detection, and industrial monitoring. As it is highlighted that VSA has several applications in the various fields of science and technology, but the focus of this research is to design and develop a novel VSA algorithm for human activity recognition (HAR) system. 1.7.2. Challenges in VSA In the VSA, there are numerous factors which limit the performance of VSA system and these are the recording settings, illumination variations, camera motion, viewpoint variations, the complexity of the background, the similarity between foreground and background object, high dimensionality, and redundancy of the data. All these factors provide an open challenge to the researchers/technocrats, to design and develop such an algorithm which can deal with these issues. The environmental conditions play a significant role when the recording/acquisition of the video signal is done because the performance of vision based system is highly dependent on the weather conditions. The captured video signal in adverse environmental conditions leads to poor quality of video signal. In the poor quality of video signal, the object and background
- 14. of the scene may not be discernible, and due to this, the following task (segmentation, feature extraction) associated in video sequence analysis system leads to worst performance. Hence, proper illumination is needed to acquire a good quality video signal, where object and background must be discernible. The motion of the camera creates a blurred object in the scene, and due to this, the new de-blurring algorithm is needed to de-blur the object. Hence, to avoid this, there must be a proper installation of the camera. The complexity of the background introduces the problem of extracting an object from the scene. Due to the cluttered background, the object (foreground) and background may have more similarity, and due to this, the accurate segmentation of the object may not be possible. Hence, it is worthwhile that the recording of the scene must be performed based on the application to avoid these issues. 1.8. HAR Identification Using Artificial Intelligence Human action recognition and analysis one of the most active topics in computer vision, has drawn increasing attention and its applications can be found in video surveillance and security, video annotation and retrieval, behavioral biometrics and human-computer interaction. The term ‘Action’ refers to simple motion patterns usually executed by a single person and typically lasting for short durations of time, on the order of tens of seconds. Examples of actions include bending, walking, swimming, and so forth. The goal of action recognition is to analyze ongoing actions from an unknown video automatically. In the last few decades, action recognition has been extensively researched while there is still a long way to go for real applications. Human action recognition is a critical component of visual surveillance systems for event- based analysis, and modeling human action is a challenging task for recognizing the activities. The conventional human action recognition system for video surveillance has the following major subsystems: (i) Background subtraction (ii) Feature extraction (iii)Posture estimation (iv)Action recognition
- 15. Background subtraction is a simple solution in motion segmentation. A static image without any object of interest (OOI) would be considered as the background image. The difference in pixel level between the successive frames and the background image provides motion information. Feature extraction, by human perception, is a very complex task in shape recognition. The selection of appropriate shape feature would be considered for improving the accuracy of any pattern recognition system, and detailed experimental analysis could be used to measure the suitability of the feature to obtain the proper outcome. Human posture refers to the arrangement of the body and its limbs. This is one of the critical aspects of analyzing human behavior. Likewise, implementation of action recognition system meets many challenges at each subsystem. The background subtraction suffers from clutter, illumination changes, and camera movements, and the existence of either noise or partial occlusions during tracking affects the feature extraction stage. The typical task of a HAR system is to detect and analyze human activity in a video sequence. The challenges in vision-based HAR systems. Several factors that make the task challenging are the variations in body postures, the rate of performance, lighting conditions, occlusion, viewpoint, and cluttered background. A sound HAR system can adapt to these variations and efficiently recognizes the human activity class. The essential steps are involved in HAR systems are usually: a) Segmentation of foreground b) Efficient extraction and representation of feature vectors, and c) Classification or recognition. A practical and novel solution can be proposed at any step of the work individually or collectively for all the steps. Due to the variation in human body taxonomy and environmental conditions, every step is full of challenges, and therefore, one can only provide the best solution in terms of recognition accuracy and processing speed. The shape and motion feature-based descriptors are two widely used methods in HAR systems. The silhouette of the human body generally represents the shapebased descriptor, and silhouettes are the heart of the activity. Motion-based descriptors are based on the motion of the body, and the region of interest can be extracted using optical flow and the pixel-wise oriented difference between the subsequent frames. The motion-based descriptors are not efficient, especially when the object in the scene is moving with variable speed. The subsequent section highlights the details of issues related to HAR system and its functioning.The changes in camera viewpoint, anthropometry (body shapes and sizes of
- 16. different actors), different dressing styles, actions’ execution rate and styles of actor’s and so forth exhibit large intraclass variability while recognizing the actions. Human Behavior Understanding Automatic analysis in surveillance scenarios failed to fulfill the real-world need at the action recognition stage alone, which requires the detection of abnormal human actions based on the periodicity in their behaviour patterns. Abnormal actions are context dependent, that includes infrequent posture patterns. For example, in a supermarket, a person walking from one point to another is ‘normal’ whereas running is ‘abnormal’ behavior. In a visual surveillance scenario, identifying the occurrence of possibly dangerous actions helps the visual analyst to minimize the manual effort and mistakes.Hence, the behavior understanding, along with action recognition, could be used for surveillance purpose. Thus, combining human action recognition and behavior understanding model is necessary for obtaining practical video data analysis. Human activities are present during comparatively longer period of time than sensors’ sampling rate. A specific sample on a time instant does not offer adequate data for explaining the performed activity. The activities are required to be identified in a time window basis. Therefore, feature extraction approaches are utilized for acquiring significant information and quantitative measures. The integration of original features is a substitute method for choosing a division of significant features. Feature extraction is the method of integrating the original feature set for describing a new significant feature set. In order to perform effective human action recognition, PTD was designed from 3D joint locations sequence. A hierarchical temporal dividing algorithm is developed to subdivide a sequence into compact sub-sequences for encoding the temporal information. Through joint positions, posture feature and dynamical tendency discriminative features of human posture are obtained. Natural and intuitive dividing algorithms assist in encoding temporal dynamics. Further geometric structures of action snippets are also maintained. But, the minimization of computational complexity during the human activity recognition is not performed. In other words, feature extraction is the conversion of high-dimensional data into a significant illustration of minimum dimensionality data. The key benefit of feature extraction is that it assists in categorization and visualization of high-dimensional data. Feature extraction from time series data is executed in the following two ways:
- 17. Statistical Techniques Structural Techniques Statistical Techniques employ quantitative features of data to mine features while structural techniques consider the interrelationship among data. Depending on the nature of specified signals, these approaches are selected. A Hierarchical Spatio-Temporal Model (HSTM) was developed by for addressing implementation issues of spatial and temporal restraints. HSTM is a two-layer Hidden Conditional Random Field (HCRF) which assists in identifying spatial associations in each frame. Then, bottom-layer HCRF identifies discriminative representations and top-layer HCRF. 1.8.1. Machine Learning based Model Machine learning performance was examined for recognizing hand gestures. Gesture recognition accuracy is increased by using normalization approach. A non-parametric Wilcoxon signed-rank test is applied to investigate the gesture recognition accuracies. Then, accuracies achieved with the aid of normalization to AUC-RMS value method are also evaluated. Classification accuracy is increased and applications of bio-medical are also supported. However, complex hand gestures are not recognized efficiently using machine learning techniques. The learning approaches employed in machine learning are classified into the following three categories: Supervised Unsupervised Semi supervised Supervised Learning In supervised learning, a set of models of normal or abnormal behavior are constructed depending on labelled training samples. The video samples which do not suit for any model are categorized as abnormal. This type of learning approach is restricted to only known events which necessitate adequate training data. But, real world video samples comprise actions that are not infrequent which leads to the lack of adequate training samples. k-Nearest Neighbors (k-NN) is one of the supervised classification approach employed for direct classification of human activities without a learning process. K-NN necessitates only the data storage space. K-NN algorithm employs the principle of similarity between the training set and new data for categorization. The new data is allocated to the common class
- 18. by means of majority vote of k nearest neighbors. Then, the distance of neighbors is evaluated by using a distance measurement termed as similarity function or Euclidean distance. Unsupervised Learning Instead of depending on labelled training information for identifying the decision boundaries among classes, unsupervised learning designs employs the underlying organization of unlabelled data to carry out processes like clustering or dimensionality reduction. Markov chain characterizes a discrete time stochastic process which influences a finite number of states so that present state depends on previous state. Hidden Markov Model (HMM) is employed in two-level classification methods for differentiating various daily living actions. During human activity recognition, each activity is characterized with a state. A Markov chain is designed to construct sequential data and employed in general model called HMM. HMM presumes that the observed sequence is managed by a hidden state sequence. When HMM trained, frequent sequences of activities are estimated with the aid of Viterbi algorithm. However, HMMs fails in ensuring the convergence to global minimum and initialization of EM algorithm is to be considered. HMM is also trained with the aid of the posterior probabilities for restricting the complexity issues. Semi-supervised Learning Semi-supervised learning integrates a small amount of labelled training data with huge volumes of unlabelled data for executing tasks such as categorization or ranking to offer information about a sector’s structure. Then, labelled data is employed for achieving exact classification/regression outcomes. An activity recognition approach was developed by EneaCippitelli et al. (2016) for extracting skeleton data with the help of RGBD sensors. On the basis of key pose extraction, a feature vector is formed. Then, a multiclass Support Vector Machine is applied to perform classification and a clustering approach established to carry out evaluation and involvement of key poses. Activity recognition is improved but action segmentation and discovery of unidentified activities are not efficiently performed.
- 19. 1.8.1.1 Support Vector Machine Support Vector Machines (SVMs) are one of the most significant machine learning approaches used for analyzing the data and recognizing patterns. They are commonly employed for categorization and regression. SVMs assist in estimating the optimal hyperplane for dividing data into linear and non-linear classes. As SVMs belong to supervised learning, training samples are employed. Every training sample is a pair of an input object and a preferred output value. SVMs examine the training data and construct a conditional function for exact estimation of class label to an unobserved input object. SVMs is also applied in a number of pattern classification applications such as image detection, text labelling, face recognition and faulty card discovery. SVM are widely used in HAR by influencing kernel functions which process all the instances to a higher dimensional space to identify a linear decision boundary for categorizing the data. The placement-independent and subject independent trials are improved by avoiding different placement and subject including orientation differences. Following it, detection performance on placement-independent and subject-independent experiments is improved by using an online independent support vector machine (OISVM) algorithm. CT-PCA scheme fails in performing feature selection leading to decreased effectiveness of human activity recognition. 1.8.2 Deep Learning Methods Deep learning methods are potentially more suited for human activity identification. However, they fail in extracting robust features. A Long Short Term Memory (LSTM) and Recurrent Neural Network increases accuracy and performance. In addition, LSTM Recurrent Neural Network does not assume computational complexity. Convolutional Neural Network Model Convolutional Neural Network Model (CNN) is a machine learning technique of feed- forward artificial neural network, employed in a number of applications in robotics, computer vision and video surveillance. CNN approach assists in executing automatic feature extraction with more exact outcomes. CNNs help in minimizing the connections and parameters employed in artificial neural network model for flexible training. A CNN is a collection of two layers such as, Convolutional Layers Pooling Layers
- 20. Convolutional Layers Convolutional layers are employed as feature extractor for obtaining feature maps as input. A shifting window is employed for convoluting each feature map to generate one pixel in an output feature map. In addition, 3D convolutional layers are also employed for extracting the motion data from various stacked frames. Pooling Layers The main objective of pooling layers is to minimize the spatial size of characterization that is more effective to limited alterations in the position of features in the prior layer. CNNs establish a degree of locality to the patterns matched in input data and allow a translational invariance based on exact location of each pattern. The temporal convolution layer matches to the convolution of input sequence with various kernels. Max-pooling finds the maximum sequence and performs subsampling to implement translational invariance. The, output of each max-pooling layer is converted by employing a ReLU activation function. The CNN classifier includes a number of convolutional neural networks for mining the robust features and a classifier for classification. Figure 1.4 shows a CNN Classifier Framework for Human Action Identification. As features are taken from the input and processed for performing feature extraction. Then, convolution masks are executed to identify relevant discriminative local features through subsampling and feature pooling for classification of activities. After the local features are extracted, feature classification is performed through CNN classifier. Convolutional Neural Networks (CNNs) to resolve the issues of user-independent human activity recognition. An improved statistical feature is utilized by maintaining global features of accelerometer time series. Feature engineering and data preprocessing are not necessitated and also recognition interval was reduced. Mobile applications are also initiated in real time. Due to the inefficiency in extracting discriminative feature, human activity recognition accuracy is minimized. CNN in Data Processing The raw data is preprocessed and CNN is applied on the preprocessed data. Then, 2D convolution is performed and activation function is employed for nonlinear processing. It is followed by max pooling layer that is applied after every convolution layer for extracting
- 21. considerable features from sensor data. The output of convolution layer and pooling layer characterize the features of input sensor data. 1.9. Visual surveillance System Visual surveillance in machine understanding has been investigated worldwide during the last few decades. Human motion detection & tracking from video imagery is one of the most active research fields. The social interests in movement detection and tracking of people have enormously increased in recent years with numerous applications like human computer interface, surveillance, security, and many others. Moving object detection and tracking is a challenging computer vision task consisting of two closely associated video analysis processes. The first one object detection involves locating an image object in the frames of a video sequence, while object tracking performs the monitoring of the video object temporal and spatial changes during the sequence, including its presence, shape, size, position. 1.9.1 Need for Surveillance System Computers have become essential in daily lives. They perform repetitive, data-intensive and computational tasks, more accurately and efficiently than humans. It is very natural to try to extend their capabilities to carry out more intelligent tasks for example analysis of visual scenes or speech, logical inference and reasoning – in brief the high-level tasks that we, humans, perform subconsciously hundreds of times every day with so much ease that we do not usually even realize that we are performing them. Privacy remains one of the ethically crucial issues of the information age, and it will remain like this forever because of human nature. People desire the safety and the privacy, but our actions and behaviors are defined by interests, which sometimes can be achieved only by violating other humans safety or/and privacy. To guard one's safety, thousands of surveillance cameras were installed all over the world. Human operators are processing the information collected by the cameras inefficiently and slowly. One possible solution for this issue is to build an automatic system, which would fulfill the human operator’s job better and faster. This problem is difficult. First, the number of people must be determined, and then their location needs to be estimated. Detecting people and estimating their location is a challenging task as peoples move randomly, they wear different clothes and appear in different gestures. This work is research on object tracking and
- 22. how it can be applied to surveillance footage. Surveillance networks are typically monitored by one or more people, looking at several monitors displaying the camera feeds. Each person may potentially be responsible for monitoring hundreds of cameras, making it very difficult for the human operator to detect events as they happen effectively. During the last few decades, there are attempts by various researches to design a better surveillance system. However, the problems such as illumination change, motion blur complexity. Persist; hence, there is vast scope to implement the optimal system. The evaluation of human motion in image sequences involves different tasks, such as acquisition, detection motion segmentation, and target classification. 1.9.2 Challenges in Surveillance System Detecting and tracking people in scenes supervised by cameras is an essential step in many application scenarios such as surveillance, urban planning or behavioural. The amount of data produced by camera feeds is so large that it is also vital to be performed with the most exceptional computational efficiency and often even real-time. Human motion segmentation and tracking can be completed in two or three dimensions depending on the difficulty of analysis, representations of the human body shape range from volumetric models to basic stick figures. Tracking depends on the correspondence of image characteristics between consecutive frames of video, taking into concern information such as color, shape, position, and consistency. Boundary segmentation can be performed by comparing the contrast or/and color of neighbouring pixels, looking particularly for rapid changes or discontinuities. However, motion segmentation & tracking is still an open and significant problem due to dynamic environmental conditions such as illumination changes, shadows, waving tree branches in the wind. and difficulties with physical changes in the scene. However, several essential challenges still exist. 1.10 Internet of Things (IOT) The internet of things, or IOT, is a system of interrelated computing devices, mechanical and digital machines, objects, animals, or people that are provided with unique identifiers and the ability to transfer data over a network without requiring human-to-human or human-to- computer interaction.
- 23. Object detection is the main capability expected from a robot. After an image is acquired, it is processed. Machine vision image processing methods include Thresholding, Pixel counting, Segmentation, Edge detection, Neural network processing and Pattern recognition. 1.10.1. Benefits of IOT The internet of things offers several benefits to organizations enabling them to: Monitor their overall business processes; Improve the customer experience; Save time and money; Enhance employee productivity; Integrate and adapt business models; Make better business decisions; and Generate more revenue. 1.10.2. Implementation of IOT-Based Smart video Surveillance System Smart video surveillance is an IoT-based application as it uses the Internet for various purposes, also providing more security by recording the activity of that person. While leaving the premises, the user activates the system by entering a password. System working starts with the detection of motion refining to human detection followed by counting human in the room, and human presence also gets noticed to a neighbor by turning on alarm. Also, notification about the same is sent to the user through SMS and e-mail. 1.11. Problem statement Posture Tendency Descriptor (PTD) is implemented to conserve the geometric organization of relevant action snippets within skeleton. Encoding of temporal data is performed by using a dividing algorithm. Subsequently, an interpretable and discriminative descriptor is implemented to represent one action snippet. Complete skeleton sequence is integrated with different PTDs in a hierarchical and temporal order for distinguishing human activities. This aids in attaining an efficient human recognition However, it fails in detection of abnormal activities. A low-cost, unobtrusive and robust system is established for supporting independent living. The signal fluctuations are interpreted by radio-frequency identification (RFID) technology and machine learning approaches. A dictionary-based approach is established to
- 24. restrict noisy, streaming and unstable RFID signals. This helps in recognizing informative dictionaries of activities using unsupervised subspace decomposition. The embodied discriminative data is employed in utilizing sparse coefficient features of recognized dictionaries on activity detection task. In addition, activity representation also assisted in attaining efficient activity recognition. A machine learning approach does not resolve complex dynamic circumstances. A deep learning model is designed for differentiating human activities without usage of previous data. The accuracy of human activity recognition is enhanced by employing Long Short-Term Memory (LSTM) Recurrent Neural Network. However, variance on identifications and early stopping criteria is not considered in deep learning model. A user- independent deep learning-based approach is established for online human activity categorization. Local feature extraction is performed with uncomplicated statistical features by using CNN to preserve the global form of time series data. Subsequently, consequences of time series length on recognition accuracy are examined to perform constant real-time activity classification. This deep learning-based approachdoes not require any manual feature engineering. In addition, computational cost is also decreased. However, Scalability is not significantly improved. The above discussion reveals that there is a need of providing effective methods for automated surveillance system in order to develop fast, reliable intelligent vision algorithms that can track, follow and re-acquire arbitrary moving ground targets (e.g. people) autonomously as well as satisfy high real-time requirement to generate high value intelligence information to build a system that will automate the whole video processing. We concentrate on unusual activity recognition and predicting their behavior (i.e. action) in video surveillance system for indoor/outdoor public areas. This objective of research is further divided into following subproblems. (a) Eliminate moving cast shadow of the foreground object (human). (b) Recognize human action/behavior analysis or action classification (c) Detect unusual activity for single/multiple persons (d) Comparing performance of implemented algorithms with existing solutions in literature 1.12. Motivation of the Research Work
- 25. Human Activity Recognition (HAR) is performed for automatic identification of physical activities for addressing human action identification in mobile and ubiquitous computing. HAR system implements classification processes for distinguishing various human daily activities. Human activity identification and classification are essential in applications such as, observation of elderly people, robotics, surveillance and tracking of athletic actions. In other words, HAR aims to identify activities from a number of observations on subject actions and environmental circumstances. Action recognition is one of the essential factors for visual surveillance video retrieval and human-computer interaction. Detection of Human activities is denoted as a three-level categorization of inherent hierarchical structure for representing various levels. The bottom level represents action primitives comprising highly complex human actions. The second level characterizes simple actions. The last level indicates complex interactions of more than two persons and objects. Human Activity Recognition system is based on a hierarchical structure. The action recognition module is implemented by reasoning engines for encoding the context of actions. The increasing advances in hardware, computing and networking, smart phone and sensor based HAR schemes assist in classifying different types of human activities in real time with the aid of machine learning approaches. In addition, human activity detection also has applications in healthcare, personal biometric signature identification and navigation. 1.13. Objective of the Work The objective of this research work is to enhance the performance of HAR system. It aims at reducing the computational time of the HAR system by selecting representative framelets and improving recognition by extracting discriminative features that aid in effectively recognizing the different actions which involve single person. Hence, this research work can be applied in improving security in the surveillance environment. The main objective of the research work is described as follows: 1. A hybrid approach combining the frameworks of both handcrafted features and deep learning methodology to enhance the performance of HAR. 2. The major intention of the second research technique is to identify the human pose accurately and efficiently. The technique majorly focused on the detection and classification of human poses. The technique encompasses a series of processes namely pre-processing,
- 26. EfficientNetbased feature extraction, SSA-based hyperparameter tuning, and Modified SVM (M-SVM) based classification. 3. Finally, noise removal methodology is proposed for both inertial sensor data and video sensor data to remove different types of noises like salt and pepper noise, Gaussian noise, outliers and blurring of the boundaries using different types of filters like MOSSE filter, Kalman filter, Butterworth and J filter. By using this technique Peak of Side-lobe Ratio (PSR) is reduced from the raw data. 1.14. Methodologies used in the work In the first objective of the research work, the proposed study attempts to combine both handcrafted models and deep learning methods since they possess distinct individual merits and complement the effective and automated identification process. The models are targeted at enabling lifestyle tracking on a real-time basis through smartphones. This performance enhanced real-time analysis serves as an efficient and low-cost HAR model, finding its applications in health care management. The automated learning methodologies in HAR are either handcrafted or deep learning or a combination of both. Handcrafted models can be regional or wholesome recognition models such as RGB, 3D mapping and skeleton data models, and deep learning models are categorized into generative models such as LSTM (long short-term memory), discriminative models such as convolutional neural networks (CNNs) or a synthesis of such models. Several datasets are available for undertaking HAR analysis and representation. The hierarchy of processes in HAR is classified into gathering information, preliminary processing, property derivation and guiding based on framed models. The second methodology is focused on the Human pose estimation (HPE) that is a procedure for determining the structure of the body pose and it is considered a challenging issue in the computer vision (CV) communities. Despite the benefits of HPE, it is still a challenging process due to the variations in visual appearances, lighting, occlusions, dimensionality, etc. To resolve these issues, this second research work presents a squirrel search optimization with a deep convolutional neural network for HPE (SSDCNN-HPE) technique. The major intention of the SSDCNN-HPE technique is to identify the human pose accurately and efficiently. Primarily, the video frame conversion process is performed and pre-processing takes place via bilateral filtering-based noise removal process. Then, the EfficientNet model is applied to identify the body points of a person with no problem
- 27. constraints. Besides, the hyperparameter tuning of the EfficientNet model takes place by the use of the squirrel search algorithm (SSA). In the final stage, the multiclass support vector machine (M-SVM) technique was utilized for the identification and classification of human poses. The design of bilateral filtering followed by SSA based EfficientNet model for HPE depicts the novelty of the work. Finally, it is noted that HAR models are developed in an unconstrained environment, which have several limitations like Personal Interference (PI), Electromagnetic (EM) noise, in-band noise or human movements and outliers involved while capturing the input data. These noises are affecting the overall performance and robustness of the model. In order to improve the model performance, noise removal techniques are introduced in this work. Noises like salt and pepper noise, Gaussian noise, and blurring of the boundaries and outlier treatment are processed for the hybrid data acquired using video and sensors in line of sight mode in this paper. For removing these noises, a combination of filters like Kalman filter, MOSSE filter, Butter-worth and J filter, are applied to the input data. By doing this noise removal technique, it is observed that Peak to Sidelobe Ratio (PSR) is reduced from the raw data. After removing these noises, features are extracted using the top layers of CNN Inceptionv3 model for video data. Similarly, by using inertial sensor, features like tri-axial accelerometer, gyroscopes and magnetometers are collected and a feature vector is created. Pyramidal flow feature fusion (PFFF) technique is used to fuse the extracted features from video and inertial sensor data. Finally, the fused features are given to the Support Vector Machine (SVM) classifier to perform activity recognition. 1.15. Summary & Organization of the Thesis This chapter presents the introduction of automatic human activity recognition, Applications, Issues, and challenges of automatic human activity recognition applications. Also, about security, while video surveillance system trust computation and trust management systems. This chapter implicitly explains the need for security and automatic detection using a trust computation among video surveillance in the automatic human activity recognition. From this chapter, anyone can read and understand the functionalities of the entire thesis concept in a better manner. The remainder of the thesis is organized as follows:
- 28. Chapter 2 provides a comprehensive survey of the research already done by different researchers on various aspects of HAR. It provides a detailed survey of the state-of-art techniques in key frame extraction and feature extraction using hand-crafted features and deep learning features. This chapter also brings out the direction of this research work. Chapter 3 describes the framework of the HAR system that has been used in this research work. In addition, this chapter provides automatic representative framelets selection for HAR in videos, a contribution of this work for reducing the computational time. Chapter 4 discusses the descriptor proposed for extraction of features that is significant in discrimination of action recognition and the use of deep learning approach for reducing the dimensionality. Chapter 5 presents the contribution of this research work in feature fusion technique which overcomes the limitation of high dimensionality, and also handles the view and scale variance. Chapter 6 presents the conclusion derived from the proposed works and future enhancement.