![• Vgg-Face to create
embeddings!
• (224,224) Target Image
• (1,2262) Dimensional
Tensor
• Output layer
• vgg_face=Model(inputs
=model.layers[0].input
,outputs=model.layers
[-2].output)](https://image.slidesharecdn.com/smartenvironmentforindustry4-200728101116/85/Smart-environment-for-industry-4-0-54-320.jpg)
Smart environment for industry 4.0
- 1. GROUP MEMBERS Jawad Sajid FA16-BCS-181 Mutahhar Ahmad FA16-BCS-122 Usman Sajid FA16-BCS-288
- 2. Introduction and Problem Statement • Strive for Industry 4.0 • These perspectives are not just ideas! • Current situation of Pakistani Industry • What are the problems we focus on? 1. Govern Access 2. Monitor Activities in Real-time 3. Alert to Environmental Risks
- 3. Honeywell’s Video Analytics Microsoft’s Amazing Works! Amazon Works! “Real-world Anomaly Detection in Surveillance Videos” by Waqas Sultani, Chen Chen and Mubarak Shah “A Review of the Applications of Computer Vision to Construction Safety” by Brian H.W Guo, Yang Zou and Long Chen
- 4. On-site 3D Vision Tracking of Construction Personnel by Francisco Cordava and Joannis Brilakis MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications by Google Deep Face Recognition by University of Oxford OpenPose: Realtime Mutli-Person 2D Pose Estimation using Part Affinity Fields Simple Online and Realtime Tracking With Deep Association Metric by Queensland University
- 5. • Problem of Deploying the Application? • Why two databases? • Importance of API Layer • System logins? Why?
- 7. DATABASE API DESIGN FACIAL RECOGNITION UNIFORM DETECTION POSE RECOGNITION UI INTEGRATION COMMUNICATI ON MODULE
- 9. • Initial Database • Not Normalized • Repeated Information • Not Any Centralized Table • Duty Rostrum Missing
- 10. • Normalized • Duty Rostrum • Centralized Information
- 12. API and Database • Apache Thrift Services • Command: thrift --gen <language> <.Thrift File> • Reusable Interfaces • Server based on C# with Thrift • Client based on Python with Thrift • TBinary Protocol Layer • TSocket Transport Layer • MSSQL Server DB • DJango Admin Panel
- 13. • Thrift Architecture • API Interfaces • C# Thrift Server • Attaining an Admin Panel with Django and Database
- 16. Face Recognition Face Recognition Pipeline • Locate and Extract faces • Identify Facial Features • Represent Face As Measurement • Compare Faces Our Works 1- Face Recognition Using VGG-Face (Transfer Learning) 2- DLIB based Facial Recognition
- 17. VGG-FACE? LABELED FACES IN THE WILD (LFW) TRANSFER LEARNING?
- 18. OUR DATASET? FACE DETECTION? MMOD FACE DETECTOR FACE EMBEDDINGS? REMOVING THE LAST ACTIVATION LAYER? OUR CLASSIFIER?
- 19. Accuracy loss: 0.0421 Accuracy: 0.9962 Validation Loss: 0.1581 Validation Accuracy: 0.9423
- 20. Our Results!
- 21. Problems? Scope? •Training time requires time. •Dataset too small. •Did augmentation help? According to requirement, no. Although better results. •Runtime training?
- 22. Functionalities • DLIB Face Detector • Face Locations • CNN based • HOG based* • Face Encodings (128 measurements) • Compare Faces
- 27. Multi-label Classification Problem! Workflow • Creating Dataset • Preprocessing Data • Train Our Model • Testing Our Model
- 28. Creating Our Dataset! • Firstly, we scraped images from Google. Problem? • Using Microsoft’s Bing Image Search API.
- 29. Preprocessing Data! • Extract Multi-class Labels • Labels list is a “lists of lists” • Scaling • Binarize the labels – MultiLabelBinarizer • transform? Two-hot encoding
- 30. • Our Model? SmallerVGGNet! • Image Augmentation Accuracy Loss: 0.0405, Accuracy: 0.9857 Validation Loss: 0.0429 Validation Accuracy: 0.9842
- 31. Testing!
- 34. What is OpenPose? Ildoo Kims Amazing Work Pipeline • Heatmaps and PAFs • NMS – get part candidates • Bipartite Graphs • Line Integral • Assignment • Merging!
- 35. Pretrained Model. Why not the pyopenpose library? Trained on Coco dataset with Thin MobileNet Depth-wise? Point to point wise convolutions.
- 37. EXTENSION OF SORT (SIMPLE REAL-TIME TRACKER) THE KALMAN FILTER THE ASSIGNMENT PROBLEM DISTANCE METRIC EFFICIENT ALGORITHM – HUNGARIAN ALGORITHM THE APPEARANCE FEATURE VECTOR
- 38. Pretrained Model on: • MARS Dataset (Motion Analysis and Re- identification Set)
- 39. DeepSort
- 40. Activity Recognition To get started with Activity Recognition: • Using OpenCV • Kinetics Dataset – Created on 2017 • 400 Activities (78.4 – 94.5% Accuracy) • “Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet”? - 2019
- 42. To understand pose recognition we need to help of human pose estimation Dataset! stand = 0, walk = 1, operate = 2, fall_down = 3
- 44. model = Sequential() model.add(Dense(units=128, activation='relu')) model.add(BatchNormalization()) model.add(Dense(units=64, activation='relu')) model.add(BatchNormalization()) model.add(Dense(units=16, activation='relu')) model.add(BatchNormalization()) model.add(Dense(units=4, activation='softmax'))
- 47. Integration Direct Approach • Using Models for Predictions Directly Integrated with UI • Cannot Use Parallel Approach for Recognitions • Can Only Be Used On a Single Machine At a Time Service Oriented Approach • Distributed Approach • Uses Django REST API services • All recognition applications are services • Parallel request handling for multiple requests • Uses POST request for data • JSON data for each service
- 49. • PyQt5 • Application UI • Output UI • Training UI
- 50. Conclusion • To be completed? • Our Integration. • Our Communication Module. • Scope? • Our goals in the future?
- 52. Our Dataset
- 53. Face Detection? Mmod_human-face_detector used! Cropped Images
- 54. • Vgg-Face to create embeddings! • (224,224) Target Image • (1,2262) Dimensional Tensor • Output layer • vgg_face=Model(inputs =model.layers[0].input ,outputs=model.layers [-2].output)
- 57. Before Discarding Output Layer After Discarding Output Layer
- 58. • Our Classifier! • Tuning parameters • Learning rate? • Epochs?
- 59. model.add(Conv2D(32, (3, 3), padding="same", input_shape=inputShape)) model.add(Activation("relu")) model.add(BatchNormalization(axis=chanDim)) model.add(MaxPooling2D(pool_size=(3, 3))) model.add(Dropout(0.25)) # (CONV => RELU) * 2 => POOL model.add(Conv2D(64, (3, 3), padding="same")) model.add(Activation("relu")) model.add(BatchNormalization(axis=chanDim)) model.add(Conv2D(64, (3, 3), padding="same")) model.add(Activation("relu")) model.add(BatchNormalization(axis=chanDim)) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Dropout(0.25)) # (CONV => RELU) * 2 => POOL model.add(Conv2D(128, (3, 3), padding="same")) model.add(Activation("relu")) model.add(BatchNormalization(axis=chanDim)) model.add(Conv2D(128, (3, 3), padding="same")) model.add(Activation("relu")) model.add(BatchNormalization(axis=chanDim)) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Dropout(0.25)) # first (and only) set of FC => RELU layers model.add(Flatten()) model.add(Dense(1024)) model.add(Activation("relu")) model.add(BatchNormalization()) model.add(Dropout(0.5)) # softmax classifier model.add(Dense(classes)) model.add(Activation(finalAct))
- 61. • Parts and Pairs? • Heatmaps? Marks Confidence for a Part. • Part Affinity Fields? Position and Orientation of Pairs.
- 62. • Non Maximum Suppression? Transform Confidence into Certainty. • Extract the Local Maximums • Compare and Suppress!
- 63. • Bipartite Graph? Connect to Form Pairs. • Assignment Problem! Edge Should Have Weights.
- 64. • Line Integral! Measure the effect of a field along a connection. • Assignment!
- 65. • Merging! Keeping in mind the same index. • Results!
- 68. Object Tracking • Detecting Objects • Analyzing Temporal Information • Challenges! 1. Occlusion 2. Variation in View Points 3. Non-stationary Camera 4. Annotating Training Data
- 69. Traditional Methods • Centroid Tracking • Meanshift • Optical Flow • Kalman Filter Deep Learning based approaches • ROLO-Recurrent YOLO
- 70. Centroid Tracking Step 1 – Accept bounding box coordinates and compute centroids
- 71. Centroid Tracking Step 2 – Compute Euclidean distance between new bounding boxes and existing objects
- 72. Centroid Tracking Step 3 – Update (x,y) coordinates of existing objects. Lonely objects? Associate centroid with minimum distances between subsequent frames.
- 73. Centroid Tracking Step 4 – Register new objects. Step 5 – Deregister old objects. Object lost? Disappeared? Left the field of view.
- 74. Object Tracking Limitations of Centroid Tracking 1 – Using computationally expensive object detector, frame detection pipeline will slow down tremendously. 2 - Underlying assumption of centroid tracking? Overlapping? Object ID switching. 3 – Just Euclidean distance? Need more heuristics.
- 75. Meanshift or Mode seeking • Used in clustering and unsupervised problems • Replaces centroid technique of calculating clusters with a weighted average • Gives importance to points closer to mean • Find modes in the given data distribution • Extract certain features • Tracks new largest mode in each frame
- 76. Object Tracking Optical Flow • Uses spatio-temporal image brightness variations at a pixel level • Focus on displacement vector • Assumptions 1 – Brightness Consistency 2 – Spatial Coherence 3 – Temporal Persistence 4 – Limited Motion • Lucas-Kanade method to obtain equation for the velocity of certain points to be tracked.
- 77. DeepSort The Kalman Filter • Core idea? Use available detections and previous predictions. • Errors? • Constant Velocity Model. • Noise component? Process Noise? Measurement Noise? • Recursive Nature. • Why Kalman works? Gaussian Realm.
- 78. The Kalman Filter • Our state contains 8 variables (u,v,a,h,u’,v’,a’,h’) • Assumptions? Variables have absolute positions and velocity factors. • Kalman? Good fit for bounding boxes. • For every detection, create a track. • Track, delete tracks and eliminate tracks.
- 79. DeepSort The Assignment Problem - How to associate new detections with new predictions? We need two things: 1 – A Distance Metric 2 – An Efficient Algorithm
- 80. The Distance Metric Squared Mahalanobis distance to incorporate the uncertainties from the Kalman filter. The Efficient Algorithm Hungarian Algorithm for simple data association problem. The question is “WHERE IS DEEP LEARNING IN ALL THIS?” New distance metric on the basis of “appearance” of the object. (D = Lambda * D_k + (1 - Lambda) * D_a)