Sign Language Classification Process By neural Network
Download as PPT, PDF1 like922 views
This document presents a contour-based approach for classifying hand postures using neural networks, addressing challenges in gesture recognition for various applications like autonomous driving and home appliances. The proposed methodology includes segmentation, preprocessing, feature extraction, and training a neural network to improve accuracy in dynamic backgrounds without special markers. Limitations of previous methods are discussed, and future work is suggested to enhance classification independence from background noise and improve user-friendliness.
![Previous Work and Limitation
In[6] template matching approach which require hand
band in the hand to normalize the image.
In [1] orientation Histogram based approach some times
map same posture in different class.
In [2] gesture classification by presence of number of
fingers and their respective distance with palm center
limit the number gesture to be classified
3 Department of CSE, CUET](https://image.slidesharecdn.com/0704026-140918011828-phpapp02/85/Sign-Language-Classification-Process-By-neural-Network-3-320.jpg)
![References
[1] William T. Freeman and Michel Roth, “Orientation Histograms for Hand Gesture
Recognition ” IEEE Intl. Workshop on Automatic Face and Gesture Recognition, Zurich,
June ,2006
[2] S.M Hassan Ahmed Todd C Alexender, “Real Time static and dynamic hand gesture
recognition for human computer Interaction”-Electrical Engineering, University of
Miami, FL.
[3]Priyanka Mekala, “Real-time Sign Language Recognition based on Neural Network
Architecture”, Florida International University, FL, U.S.A
[4] Klimis Symeonidis “Hand Gesture Recognition Using Neural Networks”, School of
Electronic and Electrical Engineering, August 23, 2009.
[5]Bowden & Sarhadi ” Building Temporal models for Gesture Recognition” in
preceding British Machine Vision Conference, pages 32-41,2002.
[6] Dr. Kaushik deb, Helena Parveen Mony & Sujan Chowdhury “Two Handed Sign
Language Recognition for Bangla Sign Character using Cross Correlation” Global
journal of Computer Science and Technology, Volume 12, Issue 3, February 2012.
26 Department of CSE, CUET](https://image.slidesharecdn.com/0704026-140918011828-phpapp02/85/Sign-Language-Classification-Process-By-neural-Network-26-320.jpg)
Sign Language Classification Process By neural Network
- 1. AA CCoonnttoouurr BBaasseedd AApppprrooaacchh ttoo CCllaassssiiffyy HHaanndd PPoossttuurree uussiinngg NNeeuurraall NNeettwwoorrkk Presented by Md.Tunvir Rahman ID:0704026 Supervised by Anik Saha Lecturer , CSE CUET
- 2. Motivation Touch less interaction with devices require fast, robust method to classify hand posture. Autonomous driving require to classify the hand gesture shown by traffic police or passengers. Home appliances like TV, Microwave oven etc. need posture classification. Giving command to a robot can be done by hand gesture which needs a good gesture classifier. 2 Department of CSE, CUET
- 3. Previous Work and Limitation In[6] template matching approach which require hand band in the hand to normalize the image. In [1] orientation Histogram based approach some times map same posture in different class. In [2] gesture classification by presence of number of fingers and their respective distance with palm center limit the number gesture to be classified 3 Department of CSE, CUET
- 4. Goal Classify gesture in an dynamic background. No special marker in the hand. Noise reduction from image frame. Implementation of neural network as classifier. Implement this approach to classify Bangla Sign character. 4 Department of CSE, CUET
- 5. Our Proposed Methodology RGB Image Hand Region Segmentation Preprocessing Connected Component labeling and Noise Removal Normalization and Contour Detection Feature Extraction Training Set Neural Network Classified Posture 5 Department of CSE, CUET Train Test
- 6. Recognition System 6 Department of CSE, CUET Detected sign
- 7. ROI Detection ROI Detection Based on Skin Color Some unwanted region appears in the frame 7 Department of CSE, CUET
- 8. Preprocessing Erosion and Dilation on Binary image to smooth the image contour and remove small holes. { 255 O ( i , j ) = if at least one neighbor is 255 d I ( i , j ) if at least one neighbor is 255 8 Department of CSE, CUET Dilation
- 9. Preprocessing e I i j if O i j = all 8 neighbors are 255 9 Department of CSE, CUET Erosion { 255 ( , ) ( , ) if at least one neighbor is 0
- 10. Noise Reduction Label the Connected Component using Flood Fill and Consider two big region containing maximum binary data. Other will be considered as noise. Color Segmented Image After Removing Noise 10 Department of CSE, CUET
- 11. Flood Fill Algorithm 2 2 2 2 2 P1 P2 p3 p4 p3 p4 2 11 Department of CSE, CUET p3 p4 P4 p5 2 p5 p6 p7 A Connected region label by 2
- 12. Normalization Hand Forearm follow an Non-increasing radius shape up to wrist of the hand . Forearm part is unwanted for classification. Contour pixel of Hand shape 12 Department of CSE, CUET
- 13. Feature Extraction 13 Department of CSE, CUET 90 0 Summing Up Number of Pixel’s lie in this Angle Total 19 histogram is extracted from the image 180
- 14. Feature Extraction Each Bin Contain count of Contour pixel . Taking ratio of bins count and pass this ratio as the feature vector to the neural network. First train the network by feature(input) and response (output). Then test gesture with the trained network. 14 Department of CSE, CUET
- 15. 1 2 Input Output 15 Department of CSE, CUET F1 F2 1 2 3 N 5 Our Proposed Network Hidden layer F3 Fn
- 16. 1 w01 Input = -3.93 Target 16 Department of CSE, CUET X1 x2 w11= 3 w12= 6 w21= 4 w22= 5 w10= 1 w20= -6 w21= -1 w22= 1 1 0 Neural Network and Back propagation 1 w02= 1
- 17. Output Y1=0.982 Input Target Output Y2=0.50 17 Department of CSE, CUET I1 1 I2 H1 H2 O1 X1 x2 -6 1 0 Neural Network and Back propagation Activation a=1*3+0*4+3*1*1=4 3 4 1 6 5 2 4 Activation a=0 -3.93 1 1 Output 0.51 Target-Output=0.49 O2 1 2 4 -3. Output 0.72 Target-Output=0.27 Total Error=0.49+0.27=0.76
- 18. Calculating the Delta values for Output and Hidden Neuron Dout= out*(1-out)*(target-out) Do1 =0.51*(1-0.51)*(1- Input Target 18 Department of CSE, CUET I1 1 I2 H1 H2 O1 X1 x2 -6 1 0 3 4 1 6 5 2 4 -3.93 1 1 O2 1 2 4 -3. 0.51)=0.1225 Do2 =0.27*(1-0.27)*(1- 0.27)=0.14 DH= out*(1-out)*W* Doi Dh11 =0.982*(1- 0.982)*2*0.1225=0.0043 Dh12 =0.982*(1- 0.982)*2*0.14=0.0049 Dh21 =0.51*(1- 0.51)*4*0.1225=0.1225 Dh22 =0.51*(1- 0.51)*4*0.14=0.139
- 19. DWij = η*Yi*Dj Input D0.1*1*0.0043=0.0004 w(03)= Target 19 Department of CSE, CUET I1 1 I2 H1 H2 O1 X1 x2 -6 1 0 Neural Network and Back propagation 3 4 1 6 5 2 4 -3.93 1 1 O2 1 2 4 -3. η= Learning Constant=0.1 Do1 =0.1225 Do1 =0.14 Dh11 =0.0043 Dh12 =0.0049 Dh21 =0.1225 Dh22 =0.139 1.0004 w (ij)new =wij(old)+ Dw(ij)
- 20. Input Target 20 Department of CSE, CUET I1 1 I2 H1 H2 O1 X1 x2 -5.987 1 0 Neural Network and Back propagation 3.0004 4 1 6.0123 5 2.012 4.006 -3.918 1 1 O2 1 2.013 4.012 -2.98 1.0004
- 21. 21 Department of CSE, CUET I1 1 I2 H1 H2 O1 X1 x2 -5.987 1 Input 0 Target Network response after Weight adjustment 3.0004 4 1 6.0123 5 2.012 4.006 -3.918 1 1 O2 1 2.013 4.012 -2.98 1.0004 New OutH1=0.9820 New OutH2=0.5063 New OutO1=0.5214 New OutO2=0.736 1-0.5214=0.4786 1-0.736=0.264 Total Error=0.4786+0.264 =0.7426 In First Iteration Error reduced from 0.76 to 0.74 Iteration Continues until the desired error goal is achieved
- 22. Experimental Analysis • Performance depends on the no of training set • Train: Test ratio significantly effects successful classification. 22 Department of CSE, CUET • Defining 100 neurons in hidden layer requires around 450 epochs to reach the error goal.
- 23. Performance Analysis 23 Department of CSE, CUET Train Successful Classification Rate 10 Sample for each Sign character
- 24. Limitation Background fully skin-colored the classification system fail. Noise component is larger than the hand ROI. Angular distortion cause the system failure. 24 Department of CSE, CUET
- 25. Future Works Shape based hand region segmentation can make the classification independent of background. Dynamic hand gesture can be extracted from video and make the system user friendly. 25 Department of CSE, CUET
- 26. References [1] William T. Freeman and Michel Roth, “Orientation Histograms for Hand Gesture Recognition ” IEEE Intl. Workshop on Automatic Face and Gesture Recognition, Zurich, June ,2006 [2] S.M Hassan Ahmed Todd C Alexender, “Real Time static and dynamic hand gesture recognition for human computer Interaction”-Electrical Engineering, University of Miami, FL. [3]Priyanka Mekala, “Real-time Sign Language Recognition based on Neural Network Architecture”, Florida International University, FL, U.S.A [4] Klimis Symeonidis “Hand Gesture Recognition Using Neural Networks”, School of Electronic and Electrical Engineering, August 23, 2009. [5]Bowden & Sarhadi ” Building Temporal models for Gesture Recognition” in preceding British Machine Vision Conference, pages 32-41,2002. [6] Dr. Kaushik deb, Helena Parveen Mony & Sujan Chowdhury “Two Handed Sign Language Recognition for Bangla Sign Character using Cross Correlation” Global journal of Computer Science and Technology, Volume 12, Issue 3, February 2012. 26 Department of CSE, CUET
- 27. Thanks 27 Department of CSE, CUET