![2.1 Image mode
As mentioned above, in the image mode users can select the four points of the
image which represent the screens corners. This is done by clicking with the left
mouse button at the right position. In case of mistake, pressing ’c’ will clear
all the inserted points. When the four points have been correctly marked, by
pressing ’s’ the projection starts.
2.2 Video mode
Differently from the image mode, in the video mode the four corners are identified
automatically as the four best features to track3
in the first video frame. To do
so, the screen was constructed as a white area with a black solid border in
order to increase the contrast on the extremities. The frame is then converted to
the grayscale color space, and subsequently binary thresholded to simplify the
identification between screen, border and the rest of the environment. Since the
border is quite thick, each corner would be identified twice: the binary image is
therefore substituted by its skeleton. Then, the identification of the corners is
immediate; when this last operation is completed, the projection can start.
3
see [1]](https://image.slidesharecdn.com/virtualprojector-161213103353/85/Virtual-projector-3-320.jpg)
![3 Projecting the movie
Projecting the movie as in an augmented reality scenario is just a matter of
warping every frame of the movie using a perspective transformation in order
to fit in the actual screen position. While the perspective transformation in the
image mode has to be computed only at the beginning - since the screen does
not move, and so its corners do not change - in the video mode, where the screen
position changes from frame to frame, the corners have to be tracked and the
projection has to be recomputed every time. Tracking is performed through the
OpenCV function calcOpticalFlowPyrLK which calculates an optical flow for a
sparse feature set using the iterative Lucas-Kanade method with pyramids4
.
The warped image is computed using the OpenCV function
getPerspectiveTransform which, given in input the four absolute corners of the
movie and the four corresponding corners of the screen5
calculates the 3 × 3
matrix M of a perspective transform such that:
tixi
tiyi
ti
= M ·
xi
yi
1
where
(xi, yi) is the ith
corner of the movie
(xi, yi) is the ith
corner of the screen
for i = 0...3
We recall that the matrix has to be computed only once in the image mode,
but has to be recalculated at every frame, after identifying the new positions of
the screen’s corners through the optical flow computation, in the video mode.
The projection matrix M is used for every frame of the movie in the OpenCV
function warpPerspective which creates a new image from the source movie frame
where dst(x, y) = src
M11x + M12y + M13
M31x + M32y + M33
,
M21x + M22y + M23
M31x + M32y + M33
.
The warped movie frame is then copied on the screen, removing all the black
pixels left from the transformation which lie outside its border.
4
see [2]
5
corners both of the movie and of the screen were ordered clockwise starting from the
top-left one](https://image.slidesharecdn.com/virtualprojector-161213103353/85/Virtual-projector-4-320.jpg)
![4 Conclusion
We have implemented a virtual projector which suits an augmented reality sce-
nario and which is very stable and generates a fluid and realistic video. The
program can also save the execution to a video file6
, so the output can be used
for other applications.
To sum up, we present here a pseudo-code of the algorithm, which recaps in
short the main parts of the implementation:
// media definition
Mat frame
if IMAGE MODE then
frame ← screenImage
else if VIDEO MODE then
frame ← video.at(0)
end
// corners’ initialization
Point[] corners
if IMAGE MODE then
corners ← frame.getFourPointsFromUser()
else if VIDEO MODE then
frame.convert(gray)
frame.threshold(binary)
frame.opening()
frame.thin()
corners ← goodFeaturesToTrack(frame, 4)
end
int w ← movie.cols - 1
int h ← movie.rows - 1
Point[] movieCorners ← {(0,0), (w,0), (w,h), (0,h)}
Mat projectionMatrix
if IMAGE MODE then
// order the cornes as movieCorners, so they can be correcly mapped
corners.orderClockwiseFromTopLeft()
projectionMatrix ← getPerspectiveTransform(corners, movieCorners)
end
// continues on next page...
6
as described in the second paragraph of the Introduction](https://image.slidesharecdn.com/virtualprojector-161213103353/85/Virtual-projector-5-320.jpg)
Virtual projector
- 1. The Virtual Projector Alessandro Florio - 161704 University of Trento Department of Information Engineering and Computer Science Master of Science in Computer Science Abstract. In this report we are going to explain how a movie is virtually projected into an image, or into another video, replacing a white screen, in order to reproduce an augmented reality scenario. In the case it is projected on a video, the corners of the ”screen” are tracked through each frame to update the area onto which the movie will be shown.
- 2. Table of Contents 1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2 Screen corners initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2.1 Image mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.2 Video mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3 Projecting the movie . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1 Introduction The program can project a movie both on a still image which contains a screen, and on a video in which the screen moves due to rotations and translations of the camera. The code has been implemented in a way that, after compiling the source, if the generated executable is called without arguments1 it runs by default the video mode. If instead it is executed with an argument2 , whatever this may be, it runs the image mode. This has to be kept in mind when running the executable file with the ”-r” option which saves the execution to a video file called output.avi: in fact, adding this option toggles the execution mode for the rule described above (see also the relative footnotes)! In the following sections we are going to describe in detail how the algorithm has been implemented, always explaining the differences and similarities between the image and video mode. 2 Screen corners initialization The procedure of initializing the screen corners follows two different approaches in the image and video modes: for the first one it is manual, so that it can be easier to change the image with the screen on which the movie is projected, while for the second one it is automatic, but in order to be so it needs to do some image cleaning which was studied for the provided video and which may not adapt to all situations. 1 or with an even number of arguments 2 or with an odd number of arguments
- 3. 2.1 Image mode As mentioned above, in the image mode users can select the four points of the image which represent the screens corners. This is done by clicking with the left mouse button at the right position. In case of mistake, pressing ’c’ will clear all the inserted points. When the four points have been correctly marked, by pressing ’s’ the projection starts. 2.2 Video mode Differently from the image mode, in the video mode the four corners are identified automatically as the four best features to track3 in the first video frame. To do so, the screen was constructed as a white area with a black solid border in order to increase the contrast on the extremities. The frame is then converted to the grayscale color space, and subsequently binary thresholded to simplify the identification between screen, border and the rest of the environment. Since the border is quite thick, each corner would be identified twice: the binary image is therefore substituted by its skeleton. Then, the identification of the corners is immediate; when this last operation is completed, the projection can start. 3 see [1]
- 4. 3 Projecting the movie Projecting the movie as in an augmented reality scenario is just a matter of warping every frame of the movie using a perspective transformation in order to fit in the actual screen position. While the perspective transformation in the image mode has to be computed only at the beginning - since the screen does not move, and so its corners do not change - in the video mode, where the screen position changes from frame to frame, the corners have to be tracked and the projection has to be recomputed every time. Tracking is performed through the OpenCV function calcOpticalFlowPyrLK which calculates an optical flow for a sparse feature set using the iterative Lucas-Kanade method with pyramids4 . The warped image is computed using the OpenCV function getPerspectiveTransform which, given in input the four absolute corners of the movie and the four corresponding corners of the screen5 calculates the 3 × 3 matrix M of a perspective transform such that: tixi tiyi ti = M · xi yi 1 where (xi, yi) is the ith corner of the movie (xi, yi) is the ith corner of the screen for i = 0...3 We recall that the matrix has to be computed only once in the image mode, but has to be recalculated at every frame, after identifying the new positions of the screen’s corners through the optical flow computation, in the video mode. The projection matrix M is used for every frame of the movie in the OpenCV function warpPerspective which creates a new image from the source movie frame where dst(x, y) = src M11x + M12y + M13 M31x + M32y + M33 , M21x + M22y + M23 M31x + M32y + M33 . The warped movie frame is then copied on the screen, removing all the black pixels left from the transformation which lie outside its border. 4 see [2] 5 corners both of the movie and of the screen were ordered clockwise starting from the top-left one
- 5. 4 Conclusion We have implemented a virtual projector which suits an augmented reality sce- nario and which is very stable and generates a fluid and realistic video. The program can also save the execution to a video file6 , so the output can be used for other applications. To sum up, we present here a pseudo-code of the algorithm, which recaps in short the main parts of the implementation: // media definition Mat frame if IMAGE MODE then frame ← screenImage else if VIDEO MODE then frame ← video.at(0) end // corners’ initialization Point[] corners if IMAGE MODE then corners ← frame.getFourPointsFromUser() else if VIDEO MODE then frame.convert(gray) frame.threshold(binary) frame.opening() frame.thin() corners ← goodFeaturesToTrack(frame, 4) end int w ← movie.cols - 1 int h ← movie.rows - 1 Point[] movieCorners ← {(0,0), (w,0), (w,h), (0,h)} Mat projectionMatrix if IMAGE MODE then // order the cornes as movieCorners, so they can be correcly mapped corners.orderClockwiseFromTopLeft() projectionMatrix ← getPerspectiveTransform(corners, movieCorners) end // continues on next page... 6 as described in the second paragraph of the Introduction
- 6. // play the movie forall movieFrame do if VIDEO MODE then /* compute the corner’s actual position based on the optical flow * between the previous and the current frame, and the previous * corners’ positions */ corners ← calcOpticalFlowPyrLK(prevFrame, frame, corners) // order the cornes as movieCorners, so they can be correcly mapped corners.orderClockwiseFromTopLeft() // re-compute the projection matrix for this frame projectionMatrix ← getPerspectiveTransform(corners, movieCorners) end Mat projection ← warpPerspective(movieFrame, projectionMatrix) projection.copyNonZeroPixelsTo(frame) show(frame) end References 1. Jianbo Shi and Carlo Tomasi, ”Good features to track” 2. Jean-Yves Bouguet, ”Pyramidal Implementation of the Lucas Kanade Feature Tracker Description of the algorithm” 3. http : //docs.opencv.org/modules/video/doc/motion analysis and object tracking.html 4. http : //docs.opencv.org/modules/imgproc/doc/geometric transformations.html