A Closed-form Solution to Photorealistic Image Stylization
- 1. A Closed-form Solution to Photorealistic Image Stylization Presenter: Sheroz Jumaboev
- 2. Abstract 2 Photorealistic image stylization concerns transferring style of a reference photo to a content photo with the constraint that the stylized photo should remain photorealistic. While several photorealistic image stylization methods exist, they tend to generate spatially inconsistent stylizations with noticeable artifacts. (a) Style (c) Ouput Image (b) Content The stylization step transfers the style of the reference photo to the content photo, the smoothing step ensures spatially consistent stylizations.
- 3. Given a style photo (a) and a content photo (b), photorealistic image stylization aims at transferring style of the style photo to the content photo as shown in (c), (d) and (e). Comparing with existing methods, the output photos computed by our method are stylized more consistently and with fewer artifacts. Moreover, our method runs an order of magnitude faster. Introduction (a) Style (b) Content (c) Gatys et al. (d) Luan et al. (e) Ours
- 4. Introduction Photorealistic image stylization aims at changing style of a photo to that of a reference photo. For a faithful stylization, content of the photo should remain the same.The output photo should look like a real photo as it were captured by a camera Classical photorealistic stylization methods are mostly based on color/tone matching and are often limited to specific scenarios (e.g., seasons and headshot portraits).
- 5. Introduction (a) Style (b) Content (c) Gatys et al. (d) Luan et al. (e) Ours Gatys et al. shows that the correlations between deep features encode the visual style of an image and propose an optimization-based method, the neural style transfer algorithm, for image stylization.
- 6. Introduction (a) Style (b) Content (c) Gatys et al. (d) Luan et al. (e) Ours Luan et al. propose adding a regularization term to the optimization objective function of the neural style transfer algorithm for avoiding distortions in the stylization output. However, this often results in inconsistent stylizations in semantically uniform regions
- 7. Introduction The stylization step is based on the whitening and coloring transform (WCT), which stylizes images via feature projections. The WCT was designed for artistic stylization. Similar to the neural style transfer algorithm, it suffers from structural artifacts when applied to photorealistic image stylization. Stylization step Smoothing step Output Image Our WCT-based stylization step resolves the issue by utilizing a novel network design for feature transform.
- 8. Photorealistic Image Stylization The 1st step is a stylization transform F1 called PhotoWCT. Given a style photo IS, F1 transfer the style of IS to the content photo Ic while minimizing structural artifacts in the output image. Although F1 can faithfully stylize Ic, it often generates inconsistent stylizations in semantically similar regions. Therefore, we use a photorealistic smoothing function F2, to eliminate these artifacts Algorithm:
- 9. PhotoWCT The PhotoWCT and WCT share the same encoder architecture and projection steps. In the PhotoWCT, we replace the upsampling layers (pink) with unpooling layers (green). Note that the unpooling layer is used together with the pooling mask (yellow) which records where carries the maximum over each max pooling region in the corresponding pooling layer
- 10. PhotoWCT This figure illustrates the network architecture difference between the WCT and the proposed PhotoWCT The PhotoWCT function is formulated as: PhotoWCT design is motivated by the observation that the max-pooling operation in the WCT reduces spatial information in feature maps. Simply upsampling feature maps in the decoder fails to recover detailed structures of the input image. The PhotoWCT replaces the upsampling layers in theWCT with unpooling layers.
- 11. Photorealistic Smoothing 1 2 Pixels with similar content in a local neighborhood should be stylized similarly. The output should not deviate significantly from the PhotoWCT result in order to maintain the global stylization effects We 1st represent all pixels as nodes in a graph and define an affinity matrix: (N is the number of pixels) to describe pixel similarities. 2 Goals in Smoothing Step
- 12. Photorealistic Smoothing The stylization output generated by the PhotoWCT better preserves local structures in the content images, which is important for the image smoothing step as shown in (e) and (f).
- 13. Experiments We use the layers from conv1 1 to conv4 1 in the VGG-19 network for the encoder E. The encoder weights are given by ImageNet-pretrained weights. The decoder D is the inverse of the encoder. We train the decoder by minimizing the sum of the L2 reconstruction loss and perceptual loss using the Microsoft COCO dataset. We adopt the multi-level stylization strategy proposed in the WCT where we apply the PhotoWCT to VGG features in different layers. Implementation details: Visual Comparison:
- 14. Visual comparisons with photorealistic stylization methods. In addition to color transfer, our method also synthesizes patterns in the style photos (e.g., the dark cloud in the top example, the snow at the bottom example). Visual Comparison:
- 15. Visual Comparison: Visual comparison with artistic stylization algorithms. Note the structural distortions on object boundaries (e.g., building) and detailed edges (e.g., sea, cloud) generated by the competing stylization methods.
- 16. λ controls the balance of the smoothness term and a fitting term. A smaller renders smoother results, while a larger renders results that are more faithful to the queries (the PhotoWCT result) Sensitivity analysis on λ Figure shows results of using different values. In general, decreasing λ helps remove artifacts and hence improves photorealism. However, if λ is too small, the output image tends to be over-smoothed.
- 17. Run-time Compare the run-time of the proposed algorithm to that of the state-of-the-art. We note that while our algorithm has a closed-form solution, Luan et al. rely on non-convex optimization. To stylize a photo, Luan et al. solve two non-convex optimization problems sequentially where a solution to the 1st optimization problem is used as an initial solution to solve the second optimization problem. We report the total run-time requiredfor obtaining the final stylization results.
- 18. Failure case Both the proposed and other photorealistic stylization algorithms fail to transfer the flower patterns to the pot. Content/Style Reinhard et al. Pitie et al. Luan et al. Ours
- 19. Conclusion This is a novel fast photorealistic image stylization method. It consists of a stylization step and a photorealistic smoothing step. Both steps have efficient closed-form solutions. Experimental results show that PhotoWCT algorithm generates stylization outputs that are much more preferred by human subject as compared to those by the state-of-the-art, while running much faster.
- 20. Thank you