Human Activity Recognition (HAR) using HMM based Intermediate matching kernel by representing video as sequence of sets of feature vectors

Presented By
Rupali Bhatnagar
14CSE2013
Under the guidance of
Dr. Veena T.
Assistant Professor
Project Presentation on
Human Activity Recognition using HMM based Intermediate
Matching Kernel by representing videos as sets of feature
vectors
Department of Computer Science And Engineering
National Institute of Technology Goa
8 July 2016

Outline
• Human Activity Recognition
• Types of patterns in a video
• Challenges to the task of classification of videos
• Problem Statement
• Related Work
• SVM based methods
• GMM based methods
• HMM based methods
• Proposed Solution
• Feature Extraction Module
• Classification using HIMK based SVM
• Results
• Conclusions and Future Directions
• References
January 10, 2017 Human Activity Recognition using HIMK by representing videos as sets of feature vectors 2

Human Activity Recognition
January 10, 2017 3
• Automatic detection of human activity events from videos by :-
• Detecting when the activity takes place
• Determining what activity has taken place
• APPLICATIONS:-
• Surveillance Systems
• Patient Monitoring Systems
• Crowd Behaviour Prediction Systems
• Sports play analysis
• Content based video search
Human Activity Recognition using HIMK by representing videos as sets of feature vectors

Classification of videos
January 10, 2017 4
• Video is composed of a sequence of frames.
• The number of frames depends on the duration of the video.
• The images are temporally related to one another.
• The images themselves have a local spatial correlations.
Time t= 0 1 2 3 4 t T-2 T-1 T
Figure : A video is composed of a sequence of frames

Classification of videos : Types of patterns
• A video has 2 categories of patterns:-
• SPATIAL PATTERNS
• TEMPORAL PATTERNS
January 10, 2017 5Human Activity Recognition using HIMK by representing videos as sets of feature vectors

• Local features of frames of videos.
• Appearance based features – Corners, Edges, Colors, etc
• Helps in detecting:-
• Edges
• Backgrounds
• Textures
• Objects
January 10, 2017 6
Figure : Spatial patterns in an image

• Capture the sequence of frames.
• Motion information embedded in the video can be taken out.
January 10, 2017
Human Activity Recognition using HIMK by
representing videos as sets of feature vectors
7
Figure : Motion information embedded in a video (Action = handclapping)

Classification of videos : Challenges
• Varying length representations[1]
• High dimensionality
• Intra-class variability
• Inter-class similarity

• Varying length representations[1]
January 10, 2017 9
Time t= 0 1 2 3 4 t1 T1 -2 T1 -1 T1
Time t= 0 1 2 3 4 t2 T2 -2 T2-1 T2
T1 frames =
Figure : Varying length representations for videos of different sizes
Video 1
Video 2
F1 F2 …… Ft1 ……. ……… FT1
T2 frames = F1 F2 …… Ft2 ……. …….. ……. ……… FT2

• Varying length representations
January 10, 2017
Human Activity Recognition using HIMK by
representing videos as sets of feature vectors
10
Time t= 0 1 2 3 4 t T-2 T-1 T
D-dimensional D-dimensional D-dimensional
D-dim D-dim D-dim D-dim
F1 F2 … … Ft … …. FT
Figure : High dimensionality of video data

January 10, 2017 11
Figure : Variations in the running class

January 10, 2017 12
Figure : (a) Similarity between Karate and Taekwondo classes
(b) Similarity between running and walking classes
(a) (b)

Problem Statement
• For the task of human activity recognition, we need to come up with a
methodology that does the following:-
• The model should capture the appearance based information in the video.
• It should also capture the temporal information of a video.
• The model captures the sequential information in video accurately.
• The model should have a definitive reason to classify a given video by using the
information we capture above.

Related Work
January 10, 2017 14

Related Work
• Method 1 : By Yegnanarayana et al.[2]
• Uses 3 kinds of features : Color Features, Shape features & Motion features
• Uses 1-vs-rest approach for SVM classification

Related Work
• Method 2 : Directed Acyclic Graph based SVM (DAGSVM) by Jiang et al.[3]
• Uses features based on video editing, color, texture and motion.
• Uses 1-vs-1 SVM classifiers arranged as a directed acyclic graph.
January 10, 2017 16
Figure : DAGSVM Approach

Related Work
• Method 3 : Hierarchical SVM by Yuan et al.[4]
• Uses Spatial features – face-frame ratio, brightness & entropy.
• Uses Temporal features - average shot length, cut percentage, average color diﬀerence & camera
motion.
• Creates 2 trees:
• Local optimal SVM binary tree
• Global optimal SVM binary tree

• Method 4 : String Kernel by Ballan et al.[5]
• Events are modeled as a sequence composed of histograms of visual features, computed using Bag of
Words(BoW) approach.
• The sequences are treated as strings (phrases) where each histogram is considered as a character.
• String kernel is based on Needleman-Wunsch edit distance which is computed as following:-
𝐾 𝑥, 𝑥′ = 𝑒−𝑑(𝑥,𝑥′)
Related Work
January 10, 2017 18
Figure: String Kernel Approach by Ballan et al.

Related Work
January 10, 2017 19
• Method 1: Approach by Xu et al.[6]
• They combine 3 video features and 1 audio feature to create a super vector and then apply
Principal Component Analysis(PCA) to reduce the dimensionality.
• They model the features for various classes using GMM and train the parameters of GMM using
Expectation-Maximization Algorithm(EM).

Related Work
January 10, 2017 20
• Method: ACTIVE(Activity Concept Transition in Video Events) by Nevatia et al.[7]
• Video event is defined as a sequence of activity concepts .
• A new concept is generated with certain probabilities based on the previous concept.
• An observation is a low level feature vector from a sub-clip and generated based on the concepts.
• The feature vector is obtained by using Fisher Kernel over the HMM.

Proposed Solution
January 10, 2017 21
Figure : Model of the proposed solution
Video
Dataset
Class
Labels
Video
Representation
using Bag of
Words of models
HoG Feature
Extraction
Feature Extraction Module
SVM Classifier
HIMK Kernel
Gram Matrix
Classification Module

Proposed Model: Feature Extraction
January 10, 2017 22
• Histogram of Oriented Gradients[8] is scale-invariant & rotation-invariant within a cell.
Normalization makes it illuminance-invariant.
• Useful for object detection.
Block B
C11 C12 C13 C14 C15
. .
. .
. Cell .
. .
C51 C52 C53 C54 C55
Figure : Image containing blocks which contain overlapping cells

January 10, 2017 23
• 2 methods to extract features:-
• Dense HoG features by using overlapping blocks
• Dense HoG features by using non-overlapping blocks
Method 1: Overlapping blocks based HoG algorithm by Dalal et al.[8]-
• Feature Vector Dimension = (no of blocks in image * no of pixels in image)
• Where no. of overlapping blocks for image =
𝑛𝑜 𝑜𝑓 𝑟𝑜𝑤𝑠−1 ∗(𝑛𝑜 𝑜𝑓 𝑐𝑜𝑙𝑠 −1)
(𝐵𝑙𝑜𝑐𝑘 𝑠𝑖𝑧𝑒)
• Due to the overlapping nature of the blocks in the image, the dimensionality of the local feature vector
increases.
• This resulted in a very huge training feature vector set.
• This feature vector set became computationally inefficient.
• Also, because of such a huge dimensional data, it is not possible to apply statistical methods of
dimensionality reduction (PCA)

January 10, 2017 24
• 2 methods to extract features:-
• Dense HoG features by using overlapping blocks
• Dense HoG features by using non-overlapping blocks
Method 2: Non-overlapping blocks based HoG algorithm by Dalal et al.[8]-
• Due to the overlapping nature of the blocks in the image, the dimensionality of the local feature vector
increases.
• We observe that dimensionality of the feature vector for each frame in the video reduces drastically when
we ignore the non-overlapping block data.
[266x36] dimensional ⟶ [70x36] dimensional

Video Representation: Bag of words model
January 10, 2017 25
Training dataset represented as a
set of HoG feature vectors taken
from each frame of each training
video
Clustering
A
B
D
E
F
Codebook Generation
Codewords generated by clustering Generated codebook
(extracted Features)
Figure: Codebook generated using codewords (Bag of words model)

Histogram Matching Score based K-medoid
clustering
January 10, 2017 26
• INTUITION-
• Features used = Histogram of Oriented Gradients(HoG).
• For calculating similarity between Histograms, we use Histogram Matching Score.
• HISTOGRAM MATCHING SCORE-
HMS h1,h2 =
𝟏
𝑵 𝒏=𝟏
𝑵
min 𝒉 𝟏𝒏, 𝒉 𝟐𝒏
where N= number of bins in Histograms h1 and h2.

clustering
January 10, 2017 27
HMS(H1,H2)=
4+3+2+4+6+1+1+6+5
9
= 3.44
Figure : Calculation of Histogram Matching Score

clustering
January 10, 2017 28
Algorithm : Histogram Matching Score based – K-medoid algorithm
Inputs: k := number of clusters
Initialize: {x1, x2 . . . xk} ← k random cluster centers
while {x1, x2 . . . xk} not converged do
for each data vector vi do
for each cluster centre xk do
Calculate Histogram Matching Score between vi and xk
Assign index of vi as:
index(vi) ← max(Histogram Matching Score w.r.t all the cluster centers)
for each cluster k do
New cluster center xnew = medoid of all the Histogram Scores in the cluster
if ( xnew == x ) then
return converged
else
x = xnew
return not converged
end

SVM Classifier
January 10, 2017 29
• SVM is a discriminative classifier with the following properties:-
It is a binary classifier.
It constructs an optimum hyperplane to divide the data.[9]
Maximum
Margin
Hyperplane
Figure : Maximum Margin Hyperplane for Linearly
Separable Data
Figure : Soft Margin Hyperplane for Non Linearly
Separable Data & Overlapping Data[10]

Kernel based methods for SVM
January 10, 2017 30
• Kernel method was proposed to handle the issues for non-linearly separable
data & overlapping data.
 Nonlinear transformation of data to a higher dimensional feature space induced by a
Mercer kernel.
 Construction of optimal linear solutions in the kernel feature space.
Figure: Illustration of Kernel method for non-linearly separable data

Sequence Kernel/Dynamic Kernel
January 10, 2017 31
• Videos are a sequence of frames. To capture the motion information, we model a video as a sequence of feature
vectors.
• ADVANTAGE- No need to convert varying length representations into a fixed length representation.
• Examples of Sequence Kernels:
• Fisher Kernel
• Probablistic Sequence Kernel
• GMM Supervector kernel
• CIGMM-IMK[11]
• HIMK[12]
F1 F2 …… Ft1 ……. ……… FT1
F1 F2 …… Ft2 ……. …….. ……. ……… FT2
Feature vector of size T1 (xi)
Feature vector of size T2 (xj)
Figure: Feature vector of 2 examples with different lengths
K(xi,xj)
SEQUENCE KERNEL

Intermediate Matching Kernel(IMK)
January 10, 2017 32
• Intermediate Matching Kernel makes use of virtual feature vectors to match 2 varying length
representations.
X1 X2 … … Xm Y1 Y2 … … … … Yn
K(X1*, Y1*) K(X2*, Y2*) … … … … K(XQ*, YQ*)
X1* X2* … … … … XQ* Y1* Y2* … … … … YQ*
Figure: Matching using virtual feature vectors
Mapping to virtual feature vector Mapping to virtual feature vector

HMM-based Intermediate Matching Kernel(HIMK)
January 10, 2017 33
• In its core, it uses an HMM that is an apt model for representing sequential information.
• Intermediate Matching Kernel makes use of virtual feature vectors to match 2 varying length
representations.
• Proposed by Dileep et al.[12], HIMK for speech is calculated as sum of base kernels of all the
components of all the GMMs that are present at each state of the HMM.
Figure: HMM based IMK calculation for speech signals [12]

HMM-based Intermediate Matching Kernel(HIMK)
January 10, 2017 34
Figure: HIMK for videos

Results
January 10, 2017 35
Boxing Handclapping Handwaving Jogging Running Walking
Boxing 63.7% 6.72% 2.61% 7.34% 15.9% 3.73%
Handclapping 11.31% 71.48% 8.41% 2.64% 5.11% 1.05%
Handwaving 18.26% 12.39% 65.34% 1.4% 2.03% 0.58%
Jogging 8.61% 1.26% 1.4% 49.54% 22.9% 16.29%
Running 4.65% 0.16% 0.67% 19.61% 62.18% 12.73%
Walking 5.13% 2.19% 4.31% 23.47% 12.29% 52.61%
Accuracy 60.81%
Table: Percent wise Confusion Matrix using the proposed method for k=32

Results
January 10, 2017 36
Representation Accuracy
String kernel with Chi Square Metric 52.5%
String kernel with Intersection metric 51.48%
String kernel with Kolomogrov Smirnov
metric
48.37%
Proposed method 60.81%
Table: Comparison of accuracy of classification

Conclusions & Future Directions
January 10, 2017 37
• Conclusion
• We proposed to use HIMK based SVM classifier for the task of human activity
recognition.
• We discussed the feature extraction process to get a varying length representation
for videos using the Bag of Features model using Histogram Match based K-medoids
algorithm.
• We then discussed about the HMM based IMK and how to use the HIMK for the task
of classification for videos.
• Future Work
• Use of motion features for better representation.
• Use of deep learning based feature representations for videos.

References
January 10, 2017 38
1. Roach, M., Mason, J. S., Evans, N. W., Xu, L. Q., & Stentiford, F. “Recent Trends in Video Analysis: A Taxonomy of Video
Classification Problems”, in IMSA, 2002, pp. 348-353.
2. V. Suresh, M. C Krishna, R. Swamy and B. Yegnanarayana, "Content-based video classification using support vector
machines", in International conference on neural information processing, 2004, pp. 726-731.
3. X. Jiang, T. Sun, and S. Wang, "An automatic video content classification scheme based on combined visual features model
with modified DAGSVM,“ in Multimedia Tools and Applications, 2010 ,vol. 52, no. 1, pp. 105–120.
4. Yuan, X., Lai, W., Mei, T., Hua, X. S., Wu, X. Q., & Li, S., ”Automatic video genre categorization using hierarchical SVM”, in
International Conference on Image Processing, 2006, pp. 2905-2908
5. L. Ballan, M. Bertini, A. Del Bimbo, and G. Serra, "Video event classification using string kernels, in "Multimedia Tools and
Applications, 2009 vol. 48, no. 1, pp. 69–87.
6. Xu, L. Q., & Li, Y. “Video classification using spatial-temporal features and PCA”, in In International Conference on
Multimedia and Expo ,2003, vol. 3, pp: 3-485.
7. Sun, Chen, and Ram Nevatia. "Active: Activity concept transitions in video event classification." In Proceedings of the IEEE
International Conference on Computer Vision, 2013 ,pp. 913-920.
8. Dalal, Navneet, and Bill Triggs. "Histograms of oriented gradients for human detection." In IEEE Computer Society
Conference on Computer Vision and Pattern Recognition,2005, vol. 1, pp. 886-893.

References
January 10, 2017 39
9. Vapnik, Vladimir N. "An overview of statistical learning theory“ in IEEE transactions on neural networks,1999, vol.
10,no 5, pp : 988-999.
10. Dileep, A. D., T. Veena, and C. Chandra Sekhar "A review of kernel methods based approaches to classification and
clustering of sequential patterns, part i: sequences of continuous feature vectors.“ in Data Mining: Concepts,
Methodologies, Tools, and Applications: Concepts, Methodologies, Tools, and Applications, 2012,vol. 1, pp: 1-251.
11. Dileep, Aroor Dinesh, and Chellu Chandra Sekhar "GMM-based intermediate matching kernel for classification of
varying length patterns of long duration speech using support vector machines“ in IEEE transactions on neural
networks and learning systems, 2014, vol. 25, no. 8, pp: 1421-1432.
12. Dileep, A. D., and C. Chandra Sekhar "HMM based intermediate matching kernel for classification of sequential
patterns of speech using support vector machines. in IEEE Transactions on Audio, Speech, and Language Processing,
2013, vol. 21, no. 12, pp: 2570-2582.

January 10, 2017 40
THANK YOU

Human Activity Recognition (HAR) using HMM based Intermediate matching kernel by representing video as sequence of sets of feature vectors

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Human Activity Recognition (HAR) using HMM based Intermediate matching kernel by representing video as sequence of sets of feature vectors (20)

Recently uploaded (20)

Human Activity Recognition (HAR) using HMM based Intermediate matching kernel by representing video as sequence of sets of feature vectors