[AAAI2018] Multispectral Transfer Network: Unsupervised Depth Estimation for All-day Vision

Editor's Notes

• #3: Visual perception such as object detection, semantic segmentation, and tracking is one of essential techniques to recognize objects around us. For the advanced self-driving stage which is fully automatic decision making, the depth is crucial when it comes to understand the world by using objects we found. If you would like to know where you are, where you want to go, and how far away are, rhe depth estimation in all-day conditions is the most important technical issue for autonomous systems.
• #4: The following natural question is “how do we get high-quality depth information from the world? In a general way, we have a stereo camera and 2D/3D-LiDAR.The stereo camera works well at day-time, but as you can see, it totally fails at night-time because RGB camera has a dependency to illumination changes. LiDAR sensors have some advantages at night-time, but the depth measurement is very sparse and sensitive to weather conditions such as raining and snowing.
• #5: From these perspectives, we argue that an alternative sensor is needed to overcome the limitations. Therefore, we proposed a thermal camera as alternative sensor beyond RGB.
• #6: We can divide the single-image depth estimation into three folds as follows: (Click) Given the input image and the corresponding depth image, we can train the model to predict the depth directly. However, for the outdoor scenario, it is very difficult to obtain the high-quality and dense depth image for the training purpose. (Click) To overcome this fact, unsupervised meth od has been proposed recently. Given the rectified stereo images, the model is trained to learn the geometric relation between two images, called the inverse depth as disparity. Without the high-quality depth ground truth, this approach has a big advantage to only using the calibrated stereo. // (구체적인 방법론 스킵) When we optimize the model, we first warp the input image by using the predicted disparity to make the synthetic right image, and then we minimize the pixel-wise difference between the output right image and target right image. (Click) Last approach is a semi-supervised method. This model is trained to minimize an unsupervised- and a supervised loss from the additional supervision as LiDAR. From these perspectives, our model is based on the unsupervised framework to predict the depth image.
• #7: Now, we explain the key concept of all-day depth estimation. As we mentioned above, RGB-based unsupervised approaches operates on the well-lit conditions, while it performs poor on ill-lit conditions such as night, dawn, sunrise, and sunset.
• #8: Compared to RGB images, the thermal image has a great advantage to capture the contents regardless of amount of lighting, Because a thermal sensor measures a long wavelength radiation emitted by subjects. Therefore, various thermal sensors have been increasingly used in modern robotics and computer vision research on all-day recognition.
• #9: Let’s suppose that we have a depth prediction model trained by RGB images in day-time. And if we have an aligned thermal image to the input RGB image, we can make a pair of the thermal and the corresponding depth image in day-time.
• #10: So, if we can train the model by using the thermal and depth corresponding pair in day-time conditions, it is possible to “adapt” this model to night conditions, since a thermal image is less affected by illumination conditions. To sum up, we can train the model by using thermal input images and RGB stereo pair to estimate the depth in all-day conditions. This is a key concept of the proposed multispectral transfer network.
• #11: For our purpose, there are two main requirements when estimating all-day depths from thermal images. The first requirement is large-scale multispectral datasets. So, we designed the multispectral dataset for depth estimations. Our dataset is captured with calibrated RGB stereo pair, a thermal image co-aligned with left-view RGB images and 3D measurements. As shown in the example, we focus on real-world driving conditions and we captured all scenarios in well-lit and ill-lit conditions.
• #12: Another requirement is the modeling the framework with multispectral pairs to predict depth information. Now, I will explain the proposed multispectral transfer network.
• #13: Based on the unsupervised framework and our multispectral dataset, we propose MTN method for single thermal based depth estimation. Instead of two separate models including RGB-based unsupervised model and thermal-based supervised model, we combined two functions into a single model.
• #15: Now, I’ll introduce our four contributions in this work. -- Our four contributions are as follows: First) (we propose) Unsupervised(Efficient) Multi-task learning with firstly adjusting color information to depth estimation Second) (we propose) A new type of multi-tasking architecture to prevent adverse effect of general multi-task topologies. Third) (we propose) Photometric correction to augment thermal image with respect to cyclic illumination for all-day adaptation. Forth) (we propose) Adaptive activation function for stable training. In Today’s talk, I will present second contribution due to the presentation time.
• #17: In general, surface normal, segmentation mask, and poses have been used as an additional tasks for multi-tasking. In the outdoor environment, however, it is not easy to gather this kind of heavy supervision as auxiliary data. Therefore, we choose “chromaticity” as an auxiliary task of multi-task learning as following three reasons. First, obtaining color information doesn’t need human-intensive annotation. Second, color information has been used in many depth-related works in terms of keeping geometric structure. Third, it has been proved that color information includes a contextual information so that it is useful for the network to learn a local structure.
• #18: The second contribution is to propose a new module called “Interleaver” for the better multi-task learning. Skip-connected Encoder-decoder models have been widely used in previous pixel-level prediction tasks. Based on this model, we place the interLeaver at the middle of skip-connection feature maps.
• #19: Our goal is to increase the representation power for depth estimations / by explicitly focusing on important features of the additional task as color regression / and suppressing unnecessary ones of it. Since the skip-connected input has mostly spatial informative features, we adopt our module to blend the meaningful features from the color regression tasks together. We assume that both depth and color-related tasks have different point of views in receptive field, so that we first pass into global/un-pooling mechanism similar to [ParseNet] and then feed it into gated convolution to control the degree of the effectiveness of an additional tasks. We place Interleaver in every skip-connected paths.
• #20: Compared to previous multi-tasking topologies, our gating mechanism of Interleaver encourages the model to find the optimal split position and reduce the adverse effects which commonly happed to the conventional methods. Therefore, we argue that our interleaver can be a generalized topologies of skip-connection of encoder-decoder architectures. In each skip-connected layers, our interleaver automatically learn the control parameters. Moreover, unlike the connected architectures, the Interleaver can be removed in the inference step for depth estimations.
• #21: The third contribution is about data augmentation for all-day adaptation. The thermal images are invariant to the illumination, but variant to the temperature. This makes the contrast of thermal image varies over time. To relieve this effect, we proposed the data-driven correction function which tunes up the thermal image for temperature-variant augmentation in training phase.
• #22: The final key contribution is to propose an adaptive activation function. For disparity estimation, the network is easily prone to predict larger disparity than ground truth, so it causes divergence of network training when we used in bilinear sampler. So our trick is to increase the maximum disparity iteratively with the scaled sigmoid activation.
• #24: Here are our results at day time. (Pointing) This is the thermal input, (Pointing) corresponding color image for visualization, and (Pointing) ground truth. To verify our contributions, we show several results. (Pointing) You can see this black and white error map to compare these results easily. As you can see in the binary error map, our final MTN model shows most accurate result. In the table, our final MTN model also achieves best performance in every metrics. All the multi-task models, LsMTN/DsMTN/ and MTN, outperforms single task model, so our multi-task learning is effective. And the LsMTN and DsMTN are outperformed by our final model which uses interleaver modules. So empirically say, our novel module works well for multi-task learning. Finally, we can show that our photometric correction is another important factor, too.
• #25: These are the results at night time. As you can see here, thermal image shows good visibility even the color image is totally dark. In this situation, it’s so hard to get dense ground truth. So, we proposed a metric which measures the performance on the locations from LiDAR ground truth. For the reasonable comparison, our metric measures not only the ordinal relations between points, but also measures the depth accuracy simultaneously With a new metric, our final model still shows good performance.
• #26: This is a video result for our method. Colors are mapped just for visualization. Even our method takes a single thermal image for inference, it achieves reasonable accuracy.
• #27: In conclusion, we employ multi-task learning for depth estimation. And we propose a novel architecture for effective multi-task learning. Our photometric correction and adaptive sigmoid function are usefull for training. Also we show that our interleaver helps the receptive field of the network wider and valid.
• #28: That’s it. Thank you. Please refer to the following website for the dataset and related codes used in this paper.