Recovering Inner Slices of Translucent Objects by Multi-frequency Illumination, CVPR 2015

Editor's Notes

• #2: Thank you for introduction, and good morning everyone. I’m Kenichiro Tanaka from Osaka University in Japan.
• #3: The goal of this work is recovering the appearance of inner layers of translucent objects, such as oil paints. This movie shows the process of my drawing, and the goal is to recover the original black tree hidden under the oil paint. One may consider using a near infrared photography for observing the inner layer; however, it results in a photograph like this (emphasize the middle bottom figure). The goal of our method is to recover a clear texture of inner layer like this (emphasize the right bottom figure).
• #4: There are many methods for recovering clear images from degraded observations. We categorize them in two aspects. One is that whether the scene is degraded by scattering or not, and the other is whether the target object is single surface or layered. A scene of single surface without scattering is already clear, and typical computer vision methods assume this type of observations. For the scene with scattering, there are descattering methods to obtain a clear image, such as the works of Narasimhan and Nayar (mention names). For layered scenes, such as reflection on showcase windows, there are some layer decomposition methods, such as the works of Szeliski et al and Li et al. Related to these works, our goal is to separate translucent layers for recovering clear inner appearances.
• #5: Here is an overview of our method. We use a projector camera system to achieve our goal. We obtain direct reflections by projecting high-frequency pattern based on Nayar’s method. The unique point of our work is that we use multiple pitches of patterns and obtain different types of direct components. Then, we recover clear inner layers via optimization. To begin with, I explain how translucent layered objects are observed.
• #6: Translucent layered objects have complex appearance. It is a superposition of all layers and each layer is blurred by subsurface scattering. It can be modeled using a convolution with depth-dependent PSF. Usually, deeper layer is more blurry, so the PSF of inner layer is more spread. To recover the original inner layer, we have to think of two problems, descattering and layer separation.
• #7: Our method is built upon High frequency illumination. High frequency illumination method can separate into direct reflection and scattering. When the projector projects high frequency patterns, direct reflection keeps the pattern but scattering lose the pattern. So they can be separated. Direct component is a clear image because scattering lights are removed.
• #8: So What is direct component? Especially, how can it be explained in a PSF imaging model? Let me explain our interpretation.
• #9: The point spread function of normal observation expresses intensities of both direct reflection and scattering. Intensity of direct reflection is the center of the PSF, and scattering intensity is the skirt of the PSF. On the other hand, PSF of direct component is sharply pointed because direct component does not have scattering lights. So we can say high frequency illumination method extracts a center part of original PSF. Here, the extracted PSF has some size in a small scale view because projection pattern have some width. This size depends on the pitch of projection pattern. When the pitch of the pattern become larger, this size become also larger, and it makes direct component brighter. Increase of the size of the PSF is occurred in a very small scale, so the image of direct component is still clear and only brightness is changed.
• #10: There is another example. If the target object scarcely has scattering component, direct image is very similar to the normal observation. In this case, the change of the pattern pitch hardly affects to the brightness. The difference of these brightness curves depends on how spread the PSF is. We found this difference can be used for layer separation.
• #11: So what happened when we apply high frequency illumination method to a layered scene? We can get a direct image, which is a superposition of all layers’ direct components. As I explained, each layer have different PSFs. When the pitch of the pattern become larger, each layer’s direct component become brighter depending on their PSF. As inner layer’s PSF is more spread, the increase of the brightness of inner layer is larger than that of upper layer. This relationship between pattern pitch changes and direct component brightness is a key observation of our method. We use multiple pitches of patterns for recovering inner layer, and we call this multi-frequency illumination.
• #12: We can obtain different types of direct components by multi-frequency illumination. They are superposition of all layer’s direct components at different brightness. This relationship can be rewritten using brightness terms alpha and original textures. And then we can rewrite in matrix form. When we estimate this matrix via optimization, we can recover each layer component.
• #13: We can obtain different types of direct components by multi-frequency illumination. They are superposition of all layer’s direct components at different brightness. This relationship can be rewritten using brightness terms alpha and original textures. And then we can rewrite in matrix form. When we estimate this matrix via optimization, we can recover each layer component.
• #14: We can obtain different types of direct components by multi-frequency illumination. They are superposition of all layer’s direct components at different brightness. This relationship can be rewritten using brightness terms alpha and original textures. And then we can rewrite in matrix form. When we estimate this matrix via optimization, we can recover each layer component.
• #15: Here, I explain the concept of our optimization. Since we do not know how deep the inner layer is, or how many layers the scene has, we have to estimate them. To find informative layers, we assume that informative layers exist sparsely along depth direction. Here, D is a vector of direct components of multi-frequency illumination, A is a matrix containing brightness of candidate depths, and R is a vector of what we want to recover. We solve this system using sparsity prior. The solution of this optimization is that informative layer have some values while others become zero. As a result we can recover important inner layer.
• #16: This is our experimental environment. We use a coaxial projector-camera system. The projector can project near infrared light pattern, and the camera can take high quality infrared picture. Pixel correspondence of projector and camera is kept at any depth in a coaxial setup, so direct component of layered scene can be easily separated.
• #17: Now, we show the results. The first target object is an oil paint. The scene has two layers, and they are blurred and superposed in normal observation under infrared light. This is the result of this target. Textures of round-shape tree and inner spiny tree are clearly separated.
• #18: This is another part of oil paints, and we can separate brush stroke texture and the painter’s signature.
• #19: This method can also use with visible lights. In this case, we apply each color channel separately, and compose a color image. This is a result of a mural paint and we can see the original texture and original color.
• #20: Our method can be applied to Arts and History field such as observing hidden oil paints, mural paints, and very old documents. For example, there is a hidden paint in this Pablo Picasso’s paint and it can be slightly seen using near infrared light. With our method, we expect that a clearer observation can be obtained. And also we expect our method can be applied for forensic purpose such as recovering hidden secret, and Medical field such as skin diagnosis.
• #21: In summary, we use multiple pitches of patterns and separate different brightness direct components. From such observations via optimization, we recover inner layer of translucent layered object. That’s all, thank you.