Human Pose Estimation by Deep Learning

Editor's Notes

• #2: Good afternoon everyone. Welcome to the first IVP seminar of this term. I’m YANG Wei. In last two seminars, Xingyu and Chu Xiao gave us a comprehensive overview of object detection as well as traditional human pose estimation approaches. In this talk, I will continue the discussion on recent developments of human pose estimation based on the powerful deep learning methods. Hope you can benefit from these methods.
• #3: First, we will briefly review the problem of human pose estimation Meanwhile, we will go over the traditional approaches for pose estimation, which have been discussed in the seminar given by Chu Xiao. Then we will spend most of the time discussing several important approaches based on deep learning techniques, from both global view and local view.
• #4: According to Wikipedia, the goal of articulated pose estimation is to “recovers the joint positions of articulated limbs, as we show here for a man playing baseball.
• #5: There are lots of applications where being able to estimate human pose is useful. For example, pose estimation is helpful for recognizing action. It also helps to parse clothing in fashion photographs. Recently, pose estimation has been successful applied in human tracking and gaming systems.
• #6: However, In unconstrained images, human pose estimation can be a very hard problem because people can appear with a variety of poses, clothing, and body shape. In the slides, you can see some very interesting and unusual examples that demonstrate how flexible the human pose is.
• #7: Traditional approaches for human pose estimation model the human as a set of parts, such as a head, torso, arm, and leg part. In 3D, these parts can be modeled as cylinders. Pictorial structures use 2D part models, where geometric relations between parts are encoded by springs. However, capturing the whole range of appearances using pictorial structures is still quite difficult. A big problem is that even projections of a simple cylinder into 2D yields many different appearances. So one usually has to explicitly evaluate many different possible in-plane orientations and foreshortenings in order to find a good match for a part template. Yang propose mini parts to approximate these transformations. in this case the mini-parts are tuned to represent near-vertical and near horizontal limbs.
• #8: As the fast development of DL, in recent two years, several pose estimation methods based on deep learning technich have been proposed. Some based on holistic view (global view), e.g., directly regress body joints location. Some based on local appearance. Some combine global view and local view in a unified framework, and achieve state-of-the-art methods. Finaly, we will also discuss some pioneer works on pose estimation in videos.
• #10: For example, in the left image. We can guess the location of the right arm only because we see the rest of the pose and anticipate the activity of the person. Similarly, in the right image, the left half body of the person is not visible at all. Since Deep Neural Networks can model very complex relationships, the authors believe that DNN can provide a holistic reasoning.
• #11: The initial stage of DeepPose is quite straight forward. It trains a DNN to regress the locations of all the body joints given an input image. DeepPose adopts AlexNet as the basic network structure. This structure was proposed in 2012. It won the imagenet competition on a large margin, and is the first time that deep model is shown to be effective on large scale computer vision task.
• #12: This is the visualized results on LSP dataset. We can see that this method has limitations in high precision regions, such as lower arms and lower legs. It is worth to mention that this method is very fast, since predictions can be get by batch forward propagation.
• #13: The pose estimation results from the initial stage are very coarse especially in high precision regions: One possible reason is that the input size is fixed as 220 by 220, the network has limited capacity to look at details. To refine coarse regression results, the authors further train cascade of pose regressors for more precise joint localization
• #14: Given the predicted joint locations from the last stage. We first crop image patches centering at the predicted location. And then train a DNN-based regressor to refine the respected locations. This process can be repeated several times. It is helpful to refine the coarse predictions because the network can see higher resolution regions.
• #15: The ground truths are in green and predicted poses are in red. We can see that the initial stage is usually successful at estimating roughly correct pose. However, the results are not precise enough. After one stage of refinement, the results are much more accurate.
• #18: We observe that local image patches are not only able to capture part presence, but also able to reason pairwise spatial relationships. For example, consider the patch centered at wrist can predict the relative position of elbow; the patch centered at elbow can reliably predict position of shoulder and wrist. We use mixture model to define different types of spatial relationships. The right panel shows typical spatial relationships the wrist can have with its neighbor elbow. The left panel shows the typical spatial relationships the elbow can have with its two neighbors, say shoulder and wrist.
• #19: Based on this observation, we can define human pose as a tree structure graph, where each node denotes the position of each part, and the edges denote the pairwise spatial relationships.
• #20: We define the score function of part locations p and pairwise relation types t. It is computed by summing the Unary appearance term and the pairwise relationship term. The unary term is the part presence map indicating the probability that part I appears at each location of the image. Pairwise term consists of two part. The first part is the pairwise relationship map, and the second part is the deformation cost. Theta are parameters which are learned by CNN. Inference is to find the positions and mixture types to maximize this score. As the relational graph is tree structure, it can be efficiently solved by dynamic programming.
• #21: Here we talk about how to learn theta. Given an image, we want produce a score map to indicate its probability of a specific type. This is done by learn a multi class classifier on local image patches. First we need to derive type label for each patch.
• #22: Then we use two convolutional layers with 1 by 1 kernels to replace the original fully connected layers. Then the network becomes a fully convolutional network, and can perform convolutions on input image with arbitrary size, and the output is the scoremap for each type, as we want. Then we can easily compute the part presence map and pairwise relationship maps as this figure illustrated. For example, to compute part presence map of elbow, we just add all the score maps associated with elbow to shoulder, and elbow to wrist together. To compute pairwise relationship maps, we need to perform marginalization.
• #23: Here are
• #24: As we discussed before, both global and local methods have merits and drawbacks for human estimation. Hence in this years CVPR, a paper combining both local appearance and holistic view is proposed.
• #25: In this paper, the authors train a network by dual-sources. Which is to say that each input is an image pair. One image is the body patch, which incorporate local appearance information. One image is the full image, which incorporate the global context information. The authors hope that this combination would result in more accurate human pose estimation.
• #26: The authors first use the objectiveness methods to propose a lot of category-independent object proposals, as shown in the boxes in the image. Then the part patches are selected by some restrictions. First the region cannot be too small, it must contain a whole body part. Second, the proposed region cannot be too big either. Because all patches will be warped to the same size as the input of the network, too large regions lacks sufficient resolution. Moreover, for efficient training, all classes of joints are covered by similar number of part patches During testing, part patches are selected from multi-scale sliding windows.
• #27: Body patches are also selected from region proposals. The region must cover all the body parts. In testing stage, these regions can be generated by human detection. The binary mask is concatenated with the body patch as an additional alpha channel.
• #28: During training, both part patch and body patch are fed into a two branch CNN. The local part branch is to predict the label of the part patch. This is a classification problem, and is trained by using softmax loss function. The global branch is to predict the x, y coordinate given the body patch and the corresponding part mask. This is a regression problem and is trained by using the Euclidean loss function. Note that the structures of the two branch are the same, hence the weights are shared except for the last layer.
• #29: In test stage, a heap map is generated for each part. The heap map has the same size of the input image. First, the part patches are obtained by sliding window method. Then use the trained network to predict the probability of a each label for each part. The pixels within the patch have the same probability. Finally, sum and average over all pixels to get the final heat map.
• #30: While the heat map provides a rough estimation of the joint location, it is insufficient to accurately localize the body joints. Remember that the global branch predicts the accurate joint location within a given patch. Hence for a specific part, we select part patches with high probability. And compute the weighted sum of the predicted joint locations to get the final joint location.
• #31: Here is the PCP value on LSP dataset. We can see that this method improves the performance on a large margin.
• #32: OK. After discuss methods from local and global view. Lets discuss some applications of pose estimation in videos.
• #33: We all know that motion is a powerful…. This figure illustrates the optical flow. The left side is the average of two adjacent frames. The right side is the estimated optical flow. We can see that the background can be greatly suppressed by the motion feature. Which would be a great help for pose estimation.
• #34: Here, a method called modeep try to incorporate motion features to improve human pose estimation. This method extended the FLIC dataset with additional motion features, as shown is the figure.
• #35: Then it trains a multi-resolution convolutional network to predict the heat maps for each body parts with the additional motion features as the input.
• #36: Since FLIC is a dataset with multiple people within an image， but only one person is annotated. In testing stage, the tors box can be used to help determine which pose to be estimated. This method compute a spatial mask of each part with respect to the torso box. This mask is helpful for suppressing false positives.
• #37: Here are some experiment results. The first line are the estimated pose without motion feature and the second lines are with motion feature. We can see that motion can greatly improve the results in occlusion, cluttered background, and the motion blur situation.
• #38: A very similar work also use optical flow to track human pose in videos. This work has published in this years ICCV. It first use a CNN to predict heatmaps for each body parts for each frame, then for the t’th frame, it computes the optical flow of t-n to t+n frame with respect to the t-th frame. The heatmaps are then warped to the t-th frame according to the optical flow. Finally, the authors use a 1by 1 convolutional layer to combine all the heat maps together.
• #39: Here is an illustration of the network producing part heat maps. The authors discussed why….
• #42: Finally. I wanna give brief introduction of my ongoing project. As we have discussed before, most of the pose estimation frameworks are not end-to-end. They often learn pose features first, and then fixed the feature to optimize a relationship model. In my work, I design an end-to-end pose estimation framework. It can be viewed as the feature extraction part plus a deformable part model. However, we plug the deformable part model into the network. And the parameter of both parts can be learned jointly.
• #43: Here is an illustration of the deformable part model. Since the relation graphs of human pose are often in tree structure. We can use message passing method for efficient inference. This is very similar to traditional recurrent neural network. Here each time step denote a part. The message is passed from the leaves to the root. The deformation weights are shared across different parts. To learn the parameters, we can use backpropagation to learn the deformation weights and the weights of convolution layer and fully connected layers jointly.
• #44: Preliminary experiments shows that the proposed method out performs most of the traditional approaches. However, it still not better can recent deep learning methods. In in the future, we plan to learn the pose relational graph from the dataset. Meanwhile, pose estimation may benefit from related tasks such as human detection and human segmentation. Finally, we need to figure out how to combine global information into this framework.