mihara_iccp16_presentation

Editor's Notes

• #2: Thank you for introduction, and good morning everyone. I’m Hajime Mihara from Nara Institute of Science and Technology in Japan. Today, I’d like to talk about our work, 4D light field segmentation with spatial and angular consistencies.
• #3: First, as you know, image editing is popular. Usually, after take the scene with digital camera, people edit the image data using editing software. Photoshop or GIMP is most famous editing software. <<play movie>> This movie is example of efficient image editing. This example is color manipulating of the red air plane. These software allow us efficient image editing like this movie.
• #4: In recent days, light field imaging going to be popular by spreading of light field camera such as Lytro. Light fields provide a very rich representation of a scene. For example, we can refocus or render free view point image after taking the scene using light field. All of you know that light field imaging is cool.
• #5: Here, let’s consider about light field editing. After taking the scene with light field camera, can we edit the light field data with an existing software? There is no light field editing software like Photoshop or GIMP yet.
• #6: So in this study, we aim to establish efficient light field editing method. Today, I show the first step toward this goal. We will present a supervised light field segmentation method for editing use. Let’s suppose that this light field is given, and change the color of this Angry bird.
• #7: User give the input as brush stroke to !!only!! one image. We call these input as seeds. The red strokes correspond to the target object of editing. The blue strokes correspond to otherwise.
• #8: Our segmentation method can extract the target region from the entire light field semi automatically.
• #9: For example, I adjust the hue value of this region to orange.
• #10: This is the color manipulated region.
• #11: After that, the color of the Angry Bird is changed. Our segmentation method contributes the efficient light field editing via semi automatic region selection.
• #12: Prior work about interface of light field editing by Jarabo, states that “Editing light fields are challenging task”. They introduced three difficulties. Firstly, there are no interfaces for 4D editing. Secondly, software must use depth to minimize redundancy for user interaction. Thirdly, the local edit on a light field needs to preserve the coherency in the entire light field. So, we solve these difficulties. I will explain how to solve them.
• #13: The first problem is about interface. We solve by asking user to input seeds on only central 2D image.
• #14: Although we can usually use only 2D input device like mouse or pen tablet, light field data inherently has 4 dimensional data. Interfaces that can edit 4 dimensional data can not to be realized easily. So we built this method to require user input to only one 2D image from the light field. Inputs of our method are 4D light field and seeds for only one image. From only a few interaction, the segmented light field can be obtained. I will mention that how to use the seeds for the segmentation.
• #15: Next difficulty is that Software must use depth to minimize redundancy for user. We propose propagating seeds to other images based on color and estimated depth.
• #16: 2D image segmentation, such as GrabCut, use the color information for the segmentation. On the other hand, in 4D light field, depth information can be estimated. Depth information is helpful to the robust segmentation. In this research, we define “objectness”, which represents likelihood that the ray will have the label such as foreground or background. Objectness is calculated using feature vector by color and depth. For calculating objectness, we use machine learning method. I explain about how to get objectness using example below. <<laser pointer>> In this example, let’s consider to get objectness of the butterfly. Here are given seeds by user for one image. Firstly, set the feature vector in brush stroked pixel on butterfly as positive samples. Secondly, set the feature vector in other brush stroked pixels as negative samples. We adopt SVM to get objectness. After training, objectness is predicted by inputting the feature vector of all rays.
• #17: For multi label segmentation, objectnesses can be obtained by using multiclass SVMs. We get objectnesses for each labels by using one-vs-rest SVMs.
• #18: The final difficulty is The local edit on a light field needs to preserve the coherency. Our solution for this problem is Introducing smoothness constrants for Spatial and angular neighbors.
• #19: To preserve coherency, we bring an assumption, when neighboring rays have similar color, they will have the same label. In our method, we define two types of neighbor rays. First, we define spatial neighbor. Please look at the left part and pay your attention to around this blue ray. The spatial neighbor of blue ray are like these orange rays. Spatial neighbor rays are slightly positioned from the blue ray. This idea is similar with 2D segmentation method. Next, we define angular neighbor rays. It is necessary to preserve coherency through 4D light field. Please see the right part and pay your attention to around this blue ray again. The angular neighbor ray of this blue ray is like these green rays. We define that angular neighbor rays are that directions are slightly different with the blue ray. These rays have the same label when the colors are similar. This idea is unique in 4D light field segmentation. This constraint can help to preserve the segmentation coherency.
• #20: I have explained the three problems and solutions. Now, I will talk that how to combine these solutions. We solve light field segmentation as energy minimization problem.
• #21: our minimization strategy is similar to 2D image segmentation. The energy function is built by two terms, data term and smoothness term. Data term represents that when the objectness is higher, the ray tend to assigned the label with highest obejctness. Smoothness term represents the assumption that neighboring rays have same label when the colors are similar. This energy function can be minimized graph cut algorithm. After minimization, the optimal labels that satisfy the energy function can be obtained like this.
• #22: Next, I would like to present some experimental results. We apply our method for four public datasets by Wanner. These datasets are synthesized by Blender. Datasets contain brush strokes and ground truth of segmentation. We compare accuracy with prior work by Wanner. Their method can segment 2D image using 4D light field based on random forest for utilizing color and depth. This method give smoothness by minimizing total variation.
• #23: This is the result. Our method can segment the entire light field from only few user input. Comparing with Wanner etal, segmentation quality of ours looks good than Wanner etal.
• #24: This is the another result. In this dataset, our method works well too.
• #25: The table shows result of numerical evaluation for the entire light fields. This result suggests that our segmentation method works well.
• #26: the right part of the table shows image segmentation accuracy by Wanner and Our segmentation accuracy of center image from light field. From this result, accuracy of our method is as high as Wanner’s method.
• #27: So we can say the result shows that our method can segment entire of light fields in high accuracy.
• #28: From now, I show examples of efficient light field editing.
• #29: It’s same with light field editing which I show previous. This is the input light field.
• #30: User input few seeds to only one image.
• #31: Our segmentation method extracts rays from light field.
• #32: This is the color manipulated region.
• #33: After that, we can get edited light field.
• #34: This is an other example. The light field data is taken by Lytro.
• #35: User only give seeds to portion of the image.
• #36: We can extract the rays like this.
• #37: In this example, I changed the color red to purple.
• #38: These are input and output light field. Our segmentation method contributes the efficient light field editing.
• #39: Ok, I summarize this presentation. We have developed supervised 4D light field segmentation method for editing use. We solve three difficulties in light field editing by Asking user to input seeds on central 2D image, Propagatiing seeds to other images based on color and estimated depth, Introducing smoothness constraints for spatial and angular neighbors. We formulate segmentation problem as energy minimization. Also we show that our method can segment the entire light field with much high accuracy. Finally, editing result suggest that our method helps efficient light field editing
• #40: Thank you for your attention.