Adaptive Spectral Projection

Editor's Notes

• #2: Testing.
• #3: Three of the most important qualities of a display
• #4: Resolutions have substantially improved, from VGA to HD to 4K UHD and beyond, where for many displays, the pixels are no longer visible to the human eye.
• #5: Contrast has also been increasing, both through improvements in the black levels of displays, as well as, through new High dynamic range displays.
• #6: Color, on the other hand, remains an unsolved problem. Large gamuts have yet to be fully realized for general consumption. ( The newRec20:20 specification for UHD calls for a very large gamut, but this gamut, as well as even larger color gamuts have yet to be fully realized for general consumption. )
• #7: We focus on this problem in the work that I will present today.
• #8: The problem of color reproduction is highlighted by the fact that most displays can only produce a limited set of colors.
• #9: The gray horseshoe shape represents all colors the standard human observer can see, whereas the green triangle represents the subset of colors that conventional displays (shown here with the common sRGB gamut) are able to produce.
• #10: The issue is that limited gamuts prevent accurate color reproduction.
• #11: This was highlighted when just recently, I was in kyoto, and I visited the ginkakuji silver temple. The garden there is covered with different mosses in every concievable shade of green. I wanted to capture and display these images so that I could share them with my friends, but no matter how hard I tried, my iPhone camera and its display could not achieve the colors that I wanted. There are, of course, two issues at hand: the camera, and the display.
• #12: There are multiple approaches for overcoming the camera limitation: first, computer generated images can possess any desired gamut, and second, there has recently been much commercial development of cheap and accessible multi-spectral cameras. These cameras are capable of capturing all desired color content. It remains a challenge, however, to display such content.
• #13: To solve the display problem, we advocate the use a flexible gamut, one whose color primaries depend on the content to be displayed. This enables one to efficiently use the available hardware capabilities. We developed a hardware prototype that enables flexible gamut selection. And we developed an optimization algorithm that chooses the best primaries for a given image.
• #14: The result is that we can obtain a much better match between the target image and the image reproduced by the projector, shown in this example. As our input image, we use a multispectral image, whose sRGB representation is visualized here. To demonstrate the performance of our algorithm, we compare the results of our and legacy algorithms based on the error between the perceived image color and the target image color. As our input image, we use a multispectral image, whose sRGB representation is visualized here (obviously, the projector in this room is not capable of showing the full multispectral
• #15: As our perceptual error metric, we use the euclidean distance between the target and the result in CIELAB-76 color space.
• #16: With a standard fixed gamut and gamut mapping algorithm, there is a large error – represented by the warm colors in this error plot.
• #17: Using a flexible gamut algorithm that was proposed recently but that does not optimize for smallest perceptual error, we see better performance but still some error.
• #18: Finally, with our perceptually-driven algorithm, we see nearly zero error between the target and the output, as indicated by the blue images in the bottom row.
• #20: This new design is more robust, easier to align, cheaper, and more manufacturable. And is the design that we used throughout the rest of the paper.
• #21: Finally, we found that with our algorithm, human users were better able to see and distinguish colors that were indistinguishable when using legacy techniques.
• #22: Okay, so what actually limits the gamut of conventional projectors.
• #23: Many projectors temporally multiplex red, green, and blue images that are then averaged together by your brain. When designing the RGB filters, tradeoffs have to be made between the brightness of the projector and color purity of the primaries.
• #24: Because of the limited speed of current displays, for each full color image, it is only possible to show 3 or 4 primaries, such as red green and blue.
• #25: Fortunately, as it turns out, most natural images can be well represented by just three primaries. We did an experiment here to validate that previously discovered claim.
• #26: Importantly, however, the 3 primaries are different for each image. This insight inspired our ‘flexible gamut’ approach, where we are able to choose different primaries for different images.
• #27: Before describing our work, it is worth discussing previous related work. The gamut mapping problem of squeezing image content into the gamut of a particular display has received a lot of attention, and these days every company has their own secret sauce.
• #28: Gamut selection is also essential to the design of current displays, where the gamut must be chosen and permanently fixed during the manufacturing process.
• #29: There has also been work to extend color gamuts by overlaying multiple projectors that each have different gamuts. This is a strong approach, but requires careful calibration and is not the route we decided to pursue.
• #30: Instead, we chose to solve the problem of joint gamut mapping and primary selection: For each image, we simultaneously determine both the best gamut and gamut mapping that are possible within the constraints of our hardware. The small amount of previous work on this problem of joint selection has been limited to using non-negative matrix factorization in a non-perceptually accurate color space. I will explain more specfiically what that means over the next few slides.
• #31: It is easiest to describe our algorithm in the context of the actual hardware implementation that we used. In particular, we had an LED light engine with, in our case, six LEDs, coupled into a DMD based grayscale projector.
• #32: Gamut selection consisted of choosing, for each time point, the linear combination of LED intensities yielding a color primary, which corresponds to one corner of the gamut triangle.
• #33: The gamut mapping problem is then equivalent to choosing the grayscale image to display on the DMD corresponding to each color primary.
• #34: Thus, for each time point, we have a vector of 6 LED coefficients, and the grayscale value of each pixel. Because the three time points are averaged together by the observer, the image formation can be represented as a matrix multiplication.
• #35: The joint gamut mapping and gamut selection problem then consists of choosing the entries of these matrices, which we call G and H.
• #36: The matrix product of G and H yields for each pixel a 6-component color vector representing the averaged color of that pixel.
• #37: We project this six-component color vector onto the spectral sensitivity responses of the retinal cone cells. The result is a three-coordinate color in CIEXYZ color space for each pixel. Now, given a target multispectral image, which can also be represented in XYZ color space, again,our goal is to find G and H that results in the best match to the target image.
• #38: This optimization can be represented by a non-negative matrix factorization problem, non-negative because we can only have positive pixel and LED values. The error metric here is the euclidean difference between the target and result image in XYZ color space.
• #39: However, the previous description assumes that color differences are linear in XYZ space. Unfortunately, human color perception is in fact not linear in this space.
• #40: This is represented by the fact that in XYZ color space, the size of the Just Noticeable Difference region is not constant.
• #41: By switching to CIELAB76, we can obtain something that is a much better approximation to human perception. While not a perfect representation, locally it is nearly linear, and our assumption is that it is more perceptually relevant. We note that there are even more modern color spaces, some of which we were also able to include in our optimization. See the paper for details.
• #42: So, how do we incorporate the CIELAB color space into our optimization?
• #43: We do so by including the nonlinear function that converts from XYZ to LAB.
• #45: The addition of this term makes the optimization problem significantly harder. Whereas before, the problem was bi-convex, the addition of this nonlinear term makes the problem nonconvex.
• #46: We initially tried levengerb-marquardt update rules (i.e. Alternating least squares), but it was far too slow and offered no promise of substantial speedup. Note – you cannot just apply the inverse of phi to Ilab, because then your metric is still in linear space…
• #47: Instead, to efficiently solve this nonconvex problem, we reformulated it in a way that allowed us to split up the easy and hard parts of the problem.
• #48: Instead, to efficiently solve this nonconvex problem, we reformulated it in a way that allowed us to split up the easy and hard parts of the problem.
• #49: This formulation then fits directly into the Standard ADMM update rules.
• #50: The key is that rather than solving a nonlinear levenberg marquardt across the whole image simultaneoulsy, we instead solve a bunch of small, pixelwise problems in parallel.
• #51: Then, we solve the standard NMF problem; it has been shown previously that this can be implemented on a GPU in real time.
• #52: The implementation is Efficient and also displays smooth convergence, across all of the images we tested. Although the run time for our implementation was very slow, about 2 hours per image, but it was entirely implemented in matlab and no effort was made to speed it up by porting to a GPU.
• #53: What does the output of our algorithm look like? Here, represented in false color, we see the output pixel values corresponding to the three primaries for an example multispectral dataset.
• #54: The gamuts themselves looked like this.
• #55: Note that the algorithm made a tradeoff in the full 3D color space between color coverage and maximum image brightness.
• #56: We can observe the performance of the algorithm by looking at the final residual.
• #57: If we compare this to the legacy adaptive gamut algorithm, however, we see a stark improvement.
• #58: These results held across many tested images. Our algorithm is the bottom row. Further, our algorithm outperformed both the legacy flexible gamut algorithm and the fixed-gamut gamut mapping. These results held across many tested images. Our algorithm is the bottom row.
• #59: Pay attention to the bottom row, and the lack of error in the blue images.
• #61: This performance improvement can be summarized across all tested multispectral images.
• #62: Additionally, even though we optimized in LAB94 color space, our results were high performing in additional, more complicated and more perceptually accurate color spaces such as sCIELAB and LAB2000
• #63: Finally, I want to briefly discuss our hardware implementation. Given an optimization algorithm that is doing its job to select the best color gamuts for a given image, how to do we actually display the image using this gamut?
• #64: We used a Lumencor light engine as the illumination source for a modified Texas Instruments LightCrafter DMD projector.
• #65: We synchronized the DMD with the light engine using a National Instruments Data Acquisition board.
• #66: We calibrated the projector by measuring the spectrum of each LED in the light engine.
• #67: We could then calculate the gamut of the projector, the solid black line, which could cover nearly the entire 2D chromaticity diagram. Thus, we have a lot of room to work around in and play with when determining the optimal 3-primary gamut for an image.
• #68: Our projector was able to successfully display vivid color images, as shown in this photograph of the setup.
• #69: While I can’t show you the results using the projector in this room, because the color gamut of this projector is not as wide as that of our projector, I can at least show you that our projector and the entire optimization pipeline was capable of producing artifact free images. You will have to rely primarily on the error plots of the previous few slides as a measure of the performance of our algorithm. We also performed a user study that attempted to support the fact that our algorithm is better at maintaining color differences.
• #70: We designed this user study to corroborate the effectiveness of our approach, while holding off for now on a full user study that would rely on subjective evaluation of image quality. We chose three pairs of colors, and tested whether unknowing users could distinguish those pairs by asking them how many circles they saw displayed in an image, interleaving the images generated by the three different algorithms: fixed gamut, non-perceptual flexible gamut, and our perceptual flexible gamut. The results highlight that our algorithm better preserves perceptual color differences than both a fixed gamut as well as the legacy adaptive gamut algorithm. Thus, in conclusion we have developed an algorithm and hardware implementation that ensures that the colors displayed by your projector, match the colors you perceive in the world. 1) A flexible gamut has the ability to adapt more readily in situations which cause a fixed gamut to fail; 2) PNMF can maintain perceptual color differences better than NMF; 3) Our hardware prototype is capable of conveying the results of the flexible gamut algorithm.
• #71: Thank you.