Indian-Sign-Language-Recognition
1 like759 views
This document proposes a system for recognizing Indian sign language using a region of interest convolutional neural network. It begins with an introduction describing the need for such a system to help deaf people communicate. It then reviews related work on sign language recognition systems. The document outlines the objectives of developing an easy to use and low cost system. It describes the proposed use of a convolutional neural network with region of interest to recognize signs, rather than conventional feature extraction and classification approaches. Details of the system design, architecture and results are provided. The goal is to help deaf individuals communicate more easily with others.
Indian-Sign-Language-Recognition
- 1. See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/331859356 INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTION NEURAL NETWORK Presentation · November 2018 CITATIONS 0 READS 29 2 authors: Some of the authors of this publication are also working on these related projects: Indian Sign Language Numeral Recognition Using Yolo Architecture. View project Intelligent Data Analytics Platform for a Metro Rail Transport System View project Beena M V Vidya Academy of Science & Technology 4 PUBLICATIONS 5 CITATIONS SEE PROFILE Sajanraj T D Jyothi Engineering College 5 PUBLICATIONS 5 CITATIONS SEE PROFILE All content following this page was uploaded by Sajanraj T D on 19 March 2019. The user has requested enhancement of the downloaded file.
- 2. INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTION NEURAL NETWORK SAJANRAJ T D1 BEENA M V2 1,2Vidya Academy of Science Technology Department of Computer Science Thalakkottukara P O, Thrissur - 680501 Nov, 2018 SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 1 / 39
- 3. Outline 1 Introduction 2 Literature 3 Existing System 4 Objectives 5 Design Proposed Work Block Diagram System Architecture Region of Interest 6 Results 7 Summary 8 Publications SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 2 / 39
- 4. Introduction Sign Language Recognition (SLR) system is a method which allow deaf people to communicate with society. In the world , deaf people needs a translator to communicate with a society. The intermediate person is always needed for this people. The involvement of second person can be removed if there is an intelligent system to recognize a sign language and translate to normal people. So the intelligent system for sign language recognition will help the people to communicate among them. Many systems are developed for sign language recognition and there is no exact system for recognize the complete signs. SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 3 / 39
- 5. Literature Real Time Indian Sign Language Recognition System to aid Deaf and Dumb people Rajam, P. Subha and Dr G Bala krishnan (2011) Cam shift and HSV approach for recognizing gesture SignPro-An Application Suite for Deaf and Dumb Ashish Sethi, Hemanth, Kuldeep kumar (2012) Skin segmentation and region growth approach is used for image recognition Skin segmentation and region growth approach is used for image recognition Er.Aditi Kalsh, Dr.N.S.Garewal (2013) Canny edge and peak detection is used for feature extraction. SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 4 / 39
- 6. Literature Real Time Sign Language Recognition Using PCA Shreyashi Narayan Sawant ; M. S. Kumbhar(2014) The Principle Component Analysis (PCA) algorithm. American Sign Language Alphabet Recognition Using Microsoft Kinect. Cao Dong ; Ming C. Leu ; Zhaozheng Yin(2015) Random Forest (RF) classifier is built to recognize ASL signs using the joint angles. 24 static ASL alphabet signs. Using Kinect sensor Multi-modality American Sign Language Recognition. Chenyang Zhang ; Yingli Tian ; Matt Huenerfauth(2016) Using depth map, facial expression, joint and hand position .SVM for classification. Using Kinect sensor SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 5 / 39
- 7. Literature Survey on Real Time Sign Language Recognition System: An LDA Approach Suriya M , Sathyapriya N ,Srinithi M ,Yesodha V (2016) Linear discriminant analysis (LDA) used which is an improved version. Continuous Chinese sign language recognition with CNN-LSTM Su Yang, Qing Zhu (2017) convolutional neural network (CNN) combined with Long Short-Term Memory (LSTM) network, SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 6 / 39
- 8. Conventional Approach Extract relevant features from images. SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 7 / 39
- 9. Conventional Approach Extract relevant features from images. Pass features on to a classifier (e.g. SVM). SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 7 / 39
- 10. Conventional Approach Extract relevant features from images. Pass features on to a classifier (e.g. SVM). Conventionally, hand tailored features are extracted. SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 7 / 39
- 11. Conventional Approach Extract relevant features from images. Pass features on to a classifier (e.g. SVM). Conventionally, hand tailored features are extracted. Classification performance highly dependent on task specific preprocessing SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 7 / 39
- 12. Conventional Approach Extract relevant features from images. Pass features on to a classifier (e.g. SVM). Conventionally, hand tailored features are extracted. Classification performance highly dependent on task specific preprocessing Conventional feature detection methods include: SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 7 / 39
- 13. Conventional Approach Extract relevant features from images. Pass features on to a classifier (e.g. SVM). Conventionally, hand tailored features are extracted. Classification performance highly dependent on task specific preprocessing Conventional feature detection methods include: HOG (histogram of oriented gradient, FFT, Hu Invariant Moments SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 7 / 39
- 14. Convolution Neural Network Biologically inspired approach on image processing. SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 8 / 39
- 15. Convolution Neural Network Biologically inspired approach on image processing. Instead of exhaustive pre-processing, let system extract features on its own. From the literature the most applicable and efficient way for the hand gesture recognition method is found to be CNN(convolution neural network). SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 8 / 39
- 16. Convolution Neural Network Biologically inspired approach on image processing. Instead of exhaustive pre-processing, let system extract features on its own. Similar to basic neural networks but with additional properties: From the literature the most applicable and efficient way for the hand gesture recognition method is found to be CNN(convolution neural network). SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 8 / 39
- 17. Convolution Neural Network Biologically inspired approach on image processing. Instead of exhaustive pre-processing, let system extract features on its own. Similar to basic neural networks but with additional properties: Weight sharing, Receptive fields, Sub-sampling From the literature the most applicable and efficient way for the hand gesture recognition method is found to be CNN(convolution neural network). SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 8 / 39
- 18. Convolution Neural Network Biologically inspired approach on image processing. Instead of exhaustive pre-processing, let system extract features on its own. Similar to basic neural networks but with additional properties: Weight sharing, Receptive fields, Sub-sampling Layers involved: From the literature the most applicable and efficient way for the hand gesture recognition method is found to be CNN(convolution neural network). SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 8 / 39
- 19. Convolution Neural Network Biologically inspired approach on image processing. Instead of exhaustive pre-processing, let system extract features on its own. Similar to basic neural networks but with additional properties: Weight sharing, Receptive fields, Sub-sampling Layers involved: Input layer Convolutional layer Sub-sampling layer (i.e. max-pooling), Output layer (usually fully connected) From the literature the most applicable and efficient way for the hand gesture recognition method is found to be CNN(convolution neural network). SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 8 / 39
- 20. Objectives Develop a system for the easy communication of deaf and dumb people with the outside world in Indian sign language using low cost device Figure: video with subtitle SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 9 / 39
- 21. Outline 1 Introduction 2 Literature 3 Existing System 4 Objectives 5 Design Proposed Work Block Diagram System Architecture Region of Interest 6 Results 7 Summary 8 Publications SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 10 / 39
- 22. Proposed Work Indian sign language is a complex system which include the involvement of two hands. The efficient way is to perform convolution neural network on the image to increase the efficiency of classification and for real life application. . SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 11 / 39
- 23. Proposed Work Indian sign language is a complex system which include the involvement of two hands. The efficient way is to perform convolution neural network on the image to increase the efficiency of classification and for real life application. . 17460 Training 2644 Testing SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 11 / 39
- 24. Proposed Work Indian sign language is a complex system which include the involvement of two hands. The efficient way is to perform convolution neural network on the image to increase the efficiency of classification and for real life application. . 17460 Training 2644 Testing Steps: SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 11 / 39
- 25. Proposed Work Indian sign language is a complex system which include the involvement of two hands. The efficient way is to perform convolution neural network on the image to increase the efficiency of classification and for real life application. . 17460 Training 2644 Testing Steps: Input the image (video frame). Find the hand object. Extract the feature . Classification and prediction . SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 11 / 39
- 26. Outline 1 Introduction 2 Literature 3 Existing System 4 Objectives 5 Design Proposed Work Block Diagram System Architecture Region of Interest 6 Results 7 Summary 8 Publications SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 12 / 39
- 27. Block Diagram Figure: SLR System SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 13 / 39
- 28. Outline 1 Introduction 2 Literature 3 Existing System 4 Objectives 5 Design Proposed Work Block Diagram System Architecture Region of Interest 6 Results 7 Summary 8 Publications SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 14 / 39
- 29. System Architecture Figure: System Architecture SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 15 / 39
- 30. Convolution Figure: convolution on rgb channel. SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 16 / 39
- 31. Filters Figure: Convolutional Filters SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 17 / 39
- 32. Activations Figure: Activation maps SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 18 / 39
- 33. Activation Functions Figure: Activation Functions SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 19 / 39
- 34. Pooling Figure: Pooling Functions SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 20 / 39
- 35. Fully Connected Layer Figure: Fully connected layer SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 21 / 39
- 36. Flatten matrix Figure: Flatten matrix SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 22 / 39
- 37. Softmax Classification In the Softmax classifier, the function mapping f ( x i; W ) = W x i SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 23 / 39
- 38. Back Propagation Figure: Back Propagation SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 24 / 39
- 39. Data-set Figure: Data-set for Indian Sign Language SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 25 / 39
- 40. Outline 1 Introduction 2 Literature 3 Existing System 4 Objectives 5 Design Proposed Work Block Diagram System Architecture Region of Interest 6 Results 7 Summary 8 Publications SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 26 / 39
- 41. Steps Capture frames and convert to RGB to LAB . lab = cv2.cvtColor(img, cv2.COLORBGR2LAB) Apply CLAHE (Contrast Limited Adaptive Histogram Equalization).The method is used for enhancing the lightness value. Gaussian Blur I have used Gaussian Blurring on the original image. Apply Thresholding for the skin segmentation on the HSV image. Find the largest two contours in the frame. Draw bounding box and forward the region to CNN. SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 27 / 39
- 42. Import Python Package SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 28 / 39
- 43. Training on Network SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 29 / 39
- 44. Results SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 30 / 39
- 45. Confusion Matrix SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 31 / 39
- 46. Precision and Recall SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 32 / 39
- 47. Main Screen SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 33 / 39
- 48. File Chooser SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 34 / 39
- 49. Predicted Window SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 35 / 39
- 50. Real-Time System SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 36 / 39
- 51. Summary The system will recognize the static signs with perfect cropped/Bounding Box image . The training using higher resolution image needs higher end GPU to process since the memory is a factor which effect the training capability. The object detection approach is costly due to labeling of each image with bounding box before the training. The thresholding approach with segmentation is one of the best method to overcome the cost of labelling. SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 37 / 39
- 52. Publications Mr. Sajanraj T D, Beena M V ,” Human Activity Recognition by Smartphone using Machine Learning Algorithm for Remote Monitoring” ,International Journal of Innovations and Advancement in Computer Science with ISSN 2347 8616, DIIF Impact Factor Value 2.65) with ISBN: 978-81-934288-4- 9. Ms. Keerthana Rathan K, Mr. Sajanraj T D, Ms. Ayana Ajith, and Ms. Beena M V .,” Smartphone Movement Based Robotic Arm Control ” published in Journal of Advanced Research in Dynamical and Control Systems, ISSN: 1943-023X, ELSEVIER SCOPUS, Multidisciplinary Journal. Mr. Sajanraj T D, Ms. Beena M V,” Indian Sign Language Numeral Recognition Using Region Of Interest Convolutional Neural Network ” International Conference On Inventive Communication And Computational Technologies(ICICCT) On 21 22 April, 2018, Coimbatore, India. , IEEE Xplore Proceedings. SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 38 / 39
- 53. For Further Reading I Ross Girshick. R-CNN. Arixiv, 30 Apr 2015. Stanford Lecture CS231n Convolutional Neural Networks for Visual Recognition lectures, 2(1):50–100, 2000. SAJANRAJ T D, BEENA M V (Universities of Somewhere and Elsewhere)INDIAN SIGN LANGUAGE RECOGNITION USING REGION OF INTEREST CONVOLUTINov, 2018 39 / 39View publication statsView publication stats