02 Fall09 Lecture Sept18web

Editor's Notes

• #5: Like wristwatches?
• #27: http://www.flickr.com/photos/pgoyette/107849943/in/photostream/
• #40: New techniques are trying decrease this distance using a folded optics approach. The origami lens uses multiple total internal reflection to propagate the bundle of rays.
• #42: CPUs and computers don’t mimic the human brain. And robots don’t mimic human activities. Should the hardware for visual computing which is cameras and capture devices, mimic the human eye? Even if we decide to use a successful biological vision system as basis, we have a range of choices. For single chambered to compounds eyes, shadow-based to refractive to reflective optics. So the goal of my group at Media Lab is to explore new designs and develop software algorithms that exploit these designs.
• #43: Current explosion in information technology has been derived from our ability to control the flow of electrons in a semiconductor in the most intricate ways. Photonic crystals promise to give us similar control over photons - with even greater flexibility because we have far more control over the properties of photonic crystals than we do over the electronic properties of semiconductors.
• #44: Changes in the index of refraction of air are made visible by Schlieren Optics. This special optics technique is extremely sensitive to deviations of any kind that cause the light to travel a different path. Clearest results are obtained from flows which are largely two-dimensional and not volumetric. In schlieren photography, the collimated light is focused with a lens, and a knife-edge is placed at the focal point, positioned to block about half the light. In flow of uniform density this will simply make the photograph half as bright. However in flow with density variations the distorted beam focuses imperfectly, and parts which have focussed in an area covered by the knife-edge are blocked. The result is a set of lighter and darker patches corresponding to positive and negative fluid density gradients in the direction normal to the knife-edge.
• #45: Full-Scale Schlieren Image Reveals The Heat Coming off of a Space Heater, Lamp and Person
• #49: 4 blocks : light, optics, sensors, processing, (display: light sensitive display)
• #50: 4 blocks : light, optics, sensors, processing, (display: light sensitive display)
• #62: But what if the user is not wearing the special clothing. Can we still understand the gestures using a simple camera? The problem is that in a cluttered scene, it is often difficult to do image processing.
• #84: But what if the user is not wearing the special clothing. Can we still understand the gestures using a simple camera? The problem is that in a cluttered scene, it is often difficult to do image processing.
• #92: Good afternoon and thank you for attending our talk entitled “Dual Photography”.
• #95: I will start off by giving you a quick overview of our technique. Suppose you had the scene shown which is being imaged by a camera on the left and is illuminated by a projector on the right. If you took a picture with the camera, here’s what it would look like. You can see the scene is being illuminated from the right, from the position of the projector located off camera. Dual photography allows us to virtually exchange the positions of the camera and the projector, generating this image. This image is synthesized by our technique. We never had a camera in this position. You can see that the technique has captured shadows, refraction, reflection and other global illumination effects.
• #96: In this talk I will start off by discussing how dual photography works. I will give a motivation For dual photography by applying it to the problem of scene relighting, and show that it can be Used to greatly accelerate the acquisition of the data needed. I will then talk about an algorithm We developed to accelerate the acquisition of the light transport needed to perform dual photography And I will end with some conclusions.
• #97: Dual photography is based on the principle of Helmholtz reciprocity. Suppose we have a ray leaving the light with intensity I and scattering off the scene towards the eye with a certain attenuation. Let’s call this the primal configuration. In the dual configuration, the positions of the eye and the light are interchanged. Helmholtz reciprocity says that the scattering is symmetric, thus the same ray in the opposite direction will have the same attenuation.
• #102: This photocell configuration might remind us of imaging techniques. For example, in the early days of television a similar method was use to create one of the first TV cameras. Known as a “flying-spot” camera, a beam of extremely bright light would scan the scene and a bank of photosensors would measure the reflected light. The value measured at these sensors would be immediately sent out via closed-circuit to television sets whose electron beam was synchronized with the beam of light. Thus they drew out the image as it was being measured by the Photosensors. This allowed for the creation of a television system that did not have to have a means to store “frames” of video. Another related applications. Finally scanning electron microscopes (and other scanned beam systems for that matter) can be thought of as employing the principle of dual photography. Thus while some of these applications might be new, what is new is the framework that establishes dual photography in this manner and gives us insights to possible applications such as relighting as we shall see in a moment.
• #103: Suppose we had the scene shown and we illuminated it with a projector from the left and imaged it with a camera on the right. Now the pixels of the projector and the camera form solid angles in space whose size depends on the resolution of each. Lets assume a resolution of pxq for the projector and mxn for the camera. What dual photography does is transform the camera into a projector and the projector into a camera. Note that the properties of the new projector (such as position, field-of-view, and resolution) are the same as that of the old camera, and vice versa. We call this the dual configuration. In this work we shall see that it is possible to attain this dual configuration by making measurements only in the original primal configuration. We will do this by measuring the light transport between individual pixels of the projector to individual pixels of the camera. Because there are 2D pixels in the projector and 2D pixels in the camera, this light transport is a 4D function. Now lets see how we can represent this system with mathematical notation.
• #104: Fortunately, the superposition of light makes this a linear system and so we can represent this setup with a simple linear equation. We can represent the projected pattern as a column vector of resolution pq x 1 and likewise we can represent the camera as a column vector of resolution mn x 1. This means that the 4-D light transport function that specifies the transport of light from a pixel in the projector to a pixel in the camera can be represented as a matrix of resolution mn x pq.
• #105: If we put these elements together, we can see that it forms a simple linear equation. Here we apply the projected pattern at vector P to the transport matrix, which we call the “T” matrix in the paper and we get a result at vector C, which is our resulting camera image for that projector pattern. I must mention that if T is properly measured, it will contain all illumination paths from the projector to the camera, including multiple bounces, subsurface scattering, and other global illumination contributions which are often desireable. At this point, let’s gain an intuition as to the composition of T. What does this matrix look like?
• #106: We can gain insight into this by illuminating patterns at the projector that have only a single pixel turned on as shown here. We can see if we apply this vector at P, it will address a single column of T which will be output at C.
• #107: This is true for each vector P with a single pixel turned on; they will extract a different column of T.
• #108: Thus we can see that the columns of T are composed of the images we would take at C with a different pixel turned on at the projector.
• #109: So this is the way light flows in our primal configuration…
• #110: We’re going to put primes above the P and C vectors to indicate that they are in the primal space. So what will happen when we go to the dual space and interchange the roles of the camera and the projector? Now light will be emitted by the camera and photographed by the projector. So this gives us the linear equation shown at the bottom. It is obviously still a linear system, except that now light leaves the camera is transformed by this new transport matrix (let’s call it T2) and results in an image at the projector. Note that the dimensions of the camera and the projector stay the same. So dimensional analysis indicates that the dimension of T2 is now pq x mn. The key observation of dual photography is that the new matrix T2 is related to the original T. We can see that if we look at the transport for a particular pixel of the projector to a particular pixel of the camera. Let’s look at the transport from pixel “j” of the projector to pixel “i” of the camera. The transport between this pair is specified by a single element in the T matrix, in this case element Tij. Now let’s look at the same pixels in at the dual configuration with the camera emitting light and the projector capturing it. The pixel Of interest in the camera is still pixel I and the pixel of interest in the projector is still j. In this case the element that relates These two is T2ji. Dual Photography is made possible by Helmholtz reciprocity which can be shown to indicate that the pixel To pixel transport is symmetric, that is the transport will be the same whether the light is leaving the projector pixel and going to The camera pixel, or going from the camera pixel to the projector pixel. Thus we can write that T2ji = Tij. This means that T2 is simply the transpose the original T.
• #111: Thus we can define “Dual Photography” as the process of transposing this transport matrix to generate pictures from the point of view of the projector as illuminated by the camera. To create a dual image, we must first capture the transport matrix T between the projector and camera in the primary configuration. As I indicated earlier, lighting up individual pixels of the projector extract single columns of the T matrix, and if we do that for all the columns T can be acquired in that manner. We shall talk about an acceleration technique later in the talk. Again, dual photography is based only on the fact that the pixel-to-pixel transport is symmetric. We formally prove this in the Appendix of the paper.
• #112: Before we continue, let’s take a look at some initial results taken by our system. Here we show the primal image of a set of famous graphics objects. Here the projector is to the right. If we take a look at the dual image, we can see that we are now looking at these objects face on and the illumination is coming in from where the camera used to be. Note that the shading on all the objects is correct.
• #113: In this next example, we have a few objects viewed from above by the camera. The projector is in front of them and forms a fairly grazing angle with the floor So it is gray. If we look at the dual image, we can see the objects from in front being lit from above. Note that the floor is now brighter because the new light source (which was the original camera) is viewing it from a more perpendicular direction. Also see for example that the shadow on the horse in the dual image corresponds To the portion of the horse that the pillar is occluding. So in some ways, what we have here is a real-life shadow map, where the primal is the shadow map fro the dual. One thing I really like about this image is that you can see detail in the dual that is not visible in the primal. Take a look at the concentric rings in the detail at the base Of the pillar. This detail is simply not visible in the primal because of the angle but is very clear in the dual. Also the detail of the lions heads is more clear in the dual than in the primal.
• #114: We observe that since we have the complete pixel-to-pixel transport, we can relight either the primal or dual images with a new 2D projector pattern.
• #115: As far as the equations are concerned, what the photosensor is doing is integrating all of the values of the C vector into a single scalar value. Assume that this integration is being done uniformly across the field-of-view of the photosensor. So this is our new primal equation. Since the T matrix is no longer relating a vector to a vector, it collapses into a row vector of dimensions pq x 1 as shown here. We can measure this T vector in the same manner, by illuminating single pixels at the projector to extract the elements of T. If we transpose this vector into a column vector, we get the dual configuration, meaning the photograph taken by the projector and illuminated by the photocell. Here the incident illumination provided by c cannot be spatially varying since C is a scalar. This means that our light is a Uniform scaling of T. The picture shown here is an image that we acquired using a photocell shown and a projector.
• #116: I will now show some videos that show the projector patterns animating.
• #135: As far as the equations are concerned, what the photosensor is doing is integrating all of the values of the C vector into a single scalar value. Assume that this integration is being done uniformly across the field-of-view of the photosensor. So this is our new primal equation. Since the T matrix is no longer relating a vector to a vector, it collapses into a row vector of dimensions pq x 1 as shown here. We can measure this T vector in the same manner, by illuminating single pixels at the projector to extract the elements of T. If we transpose this vector into a column vector, we get the dual configuration, meaning the photograph taken by the projector and illuminated by the photocell. Here the incident illumination provided by c cannot be spatially varying since C is a scalar. This means that our light is a Uniform scaling of T. The picture shown here is an image that we acquired using a photocell shown and a projector.
• #202: http://web.media.mit.edu/~raskar/NPAR04/