SlideShare a Scribd company logo

transformations2 and fitting on image processing.pptx

Download as PPTX, PDF
Indra Hermawan, profile picture
Indra Hermawan

transformations2 and fitting on image processing.pptx

Devices & Hardware
1 of 80
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
Transformation s and Fitting EECS 442 – David Fouhey Winter 2023, University of Michigan https://web.eecs.umich.edu/~fouhey/teaching/EECS442_W23/
Administrivia • Discussion this week = office hours
So Far 1. How do we find distinctive / easy to locate features? (Harris/Laplacian of Gaussian) 2. How do we describe the regions around them? (histogram of gradients) 3. How do we match features? (L2 distance) 4. How do we handle outliers? (RANSAC)
Today As promised: warping one image to another
Why Mosaic? • Compact Camera FOV = 50 x 35° Slide credit: Brown & Lowe
Why Mosaic? • Compact Camera FOV = 50 x 35° • Human FOV = 200 x 135° Slide credit: Brown & Lowe
Why Mosaic? • Compact Camera FOV = 50 x 35° • Human FOV = 200 x 135° • Panoramic Mosaic = 360 x 180° Slide credit: Brown & Lowe
Why Bother With This Math? Slide credit: A. Efros
Homework 1 Style Translation only via alignment Slide credit: A. Efros
Result Slide credit: A. Efros
Image Transformations f x T g x f x T g x Image filtering: change range of image 𝑔( 𝑥)=𝑇 ( 𝑓 ( 𝑥)) 𝑔 ( 𝑥 )= 𝑓 ¿ Image warping: change domain of image Slide credit: A. Efros
Image Transformations T T Image filtering: change range of image 𝑔(𝑥 , 𝑦)=𝑇 ( 𝑓 (𝑥, 𝑦 )) 𝑔 ( 𝑥 , 𝑦 )= 𝑓 ¿ Image warping: change domain of image f g f g Slide credit: A. Efros
Parametric (Global) warping translation rotation aspect affine perspective cylindrical Examples of parametric warps Slide credit: A. Efros
Parametric (Global) Warping T p’ = (x’,y’) T is a coordinate changing machine p = (x,y) Note: T is the same for all points, has relatively few parameters, and does not depend on image content 𝒑′ =𝑇 (𝒑) Slide credit: A. Efros
Parametric (Global) Warping T p’ = (x’,y’) p = (x,y) Today we’ll deal with linear warps 𝒑′ ≡𝑻𝒑 T: matrix; p, p’: 2D points. Start with normal points and =, then do homogeneous cords and ≡ Slide credit: A. Efros
Scaling  2 Scaling multiplies each component (x,y) by a scalar. Uniform scaling is the same for all components. Note the corner goes from (1,1) to (2,2) Slide credit: A. Efros
Scaling Non-uniform scaling multiplies each component by a different scalar. X  2, Y  0.5 Slide credit: A. Efros
Scaling What does T look like? 𝑥′ =𝑎𝑥 𝑦 ′ =𝑏𝑦 Let’s convert to a matrix: [𝑥 ′ 𝑦 ′ ]= [𝑎 0 0 𝑏][𝑥 𝑦 ] scaling matrix S What’s the inverse of S? Slide credit: A. Efros
2D Rotation Rotation Matrix But wait! Aren’t sin/cos non-linear? x’ is a linear combination/function of x, y x’ is not a linear function of θ What’s the inverse of Rθ? 𝑰 =𝑹𝜽 𝑇 𝑹𝜽 [𝑥 ′ 𝑦 ′ ]= [c os ⁡(𝜃) − sin (𝜃 ) sin ( 𝜃) cos(𝜃 ) ][𝑥 𝑦 ] Slide credit: A. Efros
Things You Can Do With 2x2 Identity / No Transformation Shear [𝑥 ′ 𝑦 ′ ]= [ 1 𝑠 h𝑥 𝑠h𝑦 1 ][𝑥 𝑦 ] [𝑥 ′ 𝑦 ′ ]= [1 0 0 1 ][𝑥 𝑦 ] Slide credit: A. Efros
Things You Can Do With 2x2 2D Mirror About Y-Axis [𝑥 ′ 𝑦 ′ ]= [−1 0 0 1 ][𝑥 𝑦 ] Before After 2D Mirror About X,Y [𝑥 ′ 𝑦 ′ ]= [−1 0 0 −1][𝑥 𝑦 ] Before After Slide credit: A. Efros
What’s Preserved? Projections of parallel 3D lines are not necessarily parallel, so not parallelism 3D lines project to 2D lines so lines are preserved Distant objects are smaller so size is not preserved
What’s Preserved With a 2x2 [𝑥 ′ 𝑦 ′ ]= [𝑎 𝑏 𝑐 𝑑][𝑥 𝑦 ]=𝑇 [𝑥 𝑦] After multiplication by T (irrespective of T) • Origin is origin: 0 = T0 • Lines are lines • Parallel lines are parallel
Things You Can’t Do With 2x2 What about translation? x’ = x + tx, y’ = y+ty +(2,2) How do we make it linear?
Homogeneous Coordinates Again What about translation? x’ = x + tx, y’ = y+ty +(2,2) [ 𝑥+𝑡𝑥 𝑦 +𝑡 𝑦 1 ]≡ [ 𝑥′ 𝑦 ′ 1 ]≡ [ 1 0 𝑡𝑥 0 1 𝑡𝑦 0 0 1 ][ 𝑥 𝑦 1 ] Slide credit: A. Efros
Representing 2D Transformations How do we represent a 2D transformation? Let’s pick scaling [ 𝑥′ 𝑦 ′ 1 ]≡ [ 𝑠𝑥 0 𝑎 0 𝑠𝑦 𝑏 𝑑 𝑒 𝑓 ][ 𝑥 𝑦 1 ] a b d e f 0 0 0 0 1 What’s
Affine Transformations Affine: linear transformation plus translation In general (without homogeneous coordinates) 𝒙 ′= 𝑨𝒙+𝒃 Will the last coordinate w’ always be 1? [ 𝑥 ′ 𝑦 ′ 𝑤 ′]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 0 0 1 ][ 𝑥 𝑦 1 ] t
Matrix Composition [ 𝑥′ 𝑦 ′ 𝑤 ′]≡ [ 1 0 𝑡𝑥 0 1 𝑡 𝑦 0 0 1 ][ cos (𝜃 ) −sin ( 𝜃) 0 sin (𝜃) cos ( 𝜃) 0 0 0 1][ 𝑠𝑥 0 0 0 𝑠 𝑦 0 0 0 1 ][ 𝑥 𝑦 𝑤] 𝑇 (𝑡𝑥 ,𝑡𝑦 ) 𝑅( 𝜃) 𝑆(𝑠𝑥 ,𝑠𝑦 ) We can combine transformations via matrix multiplication. Does order matter? Slide credit: A. Efros
What’s Preserved With Affine After multiplication by T (irrespective of T) • Origin is origin: 0 = T0 • Lines are lines • Parallel lines are parallel [ 𝑥′ 𝑦 ′ 1 ]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 0 0 1 ][ 𝑥 𝑦 1 ]≡ 𝑻 [ 𝑥 𝑦 1 ]
Homogeneous Equivalence z x y [x,y,w] λ[x,y,w] Two homogeneous coordinates are equivalent if they are proportional to each other. Not = ! [ 𝑢 𝑣 𝑤]≡ [ 𝑢′ 𝑣′ 𝑤′ ]↔ [ 𝑢 𝑣 𝑤]= 𝜆 [ 𝑢′ 𝑣′ 𝑤′ ] 𝜆≠ 0 Triple / Equivalent Double / Equals
Perspective Transformations Set bottom row to not [0,0,1] Called a perspective/projective transformation or a homography [ 𝑥′ 𝑦 ′ 𝑤 ′]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥 𝑦 𝑤] Can compute [x’,y’,w’] via matrix multiplication. How do we get a 2D point? (x’/w’, y’/w’)
Perspective Transformations Set bottom row to not [0,0,1] Called a perspective/projective transformation or a homography [ 𝑥′ 𝑦 ′ 𝑤 ′]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥 𝑦 𝑤] How many degrees of freedom?
How Many Degrees of Freedom? Can always scale coordinate by non-zero value [ 𝑥 ′ 𝑦 ′ 𝑤 ′]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥 𝑦 𝑤] Perspective [ 𝑥 ′ 𝑦 ′ 𝑤 ′]≡ 1 𝑖 [ 𝑥′ 𝑦 ′ 𝑤′ ] Homography can always be re-scaled by λ≠0 Typically pick it so last entry is 1. ≡ 1 𝑖 [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥 𝑦 𝑤 ]
What’s Preserved With Perspective After multiplication by T (irrespective of T) • Origin is origin: 0 = T0 • Lines are lines • Parallel lines are parallel • Ratios between distances [ 𝑥′ 𝑦 ′ 1 ]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥 𝑦 1 ]≡ 𝑻 [ 𝑥 𝑦 1 ]
Transformation Families In general: transformations are a nested set of groups Diagram credit: R. Szeliski
What Can Homographies Do? Homography example 1: any two views of a planar surface Figure Credit: S. Lazebnik
What Can Homographies Do? Homography example 2: any images from two cameras sharing a camera center Figure Credit: S. Lazebnik
What Can Homographies Do? Homography sort of example “3”: far away scene that can be approximated by a plane Figure credit: Brown & Lowe
Fun With Homographies Original image St. Petersburg photo by A. Tikhonov Virtual camera rotations Slide Credit: A. Efros
Analyzing Patterns Homography Automatically rectified floor The floor (enlarged) Slide from A. Criminisi
Analyzing Patterns Slide from A. Criminisi Automatic rectification From Martin Kemp The Science of Art (manual reconstruction)
Fitting Transformations Setup: have pairs of correspondences (𝑥𝑖 , 𝑦𝑖) (𝑥 ′𝑖 , 𝑦 ′𝑖) M,t [𝑥𝑖 ′ 𝑦𝑖 ′ ]=𝑴 [𝑥𝑖 𝑦𝑖 ]+𝒕 Slide Credit: S. Lazebnik
Fitting Transformation Data: (xi,yi,x’i,y’i) for i=1,…,k Model: [x’i,y’i] = M[xi,yi]+t Objective function: ||[x’i,y’i] – (M[xi,yi]+t)||2 M,t Affine Transformation: M,t
Fitting Transformations [ ⋮ 𝑥𝑖 ′ 𝑦𝑖 ′ ⋮ ]= [ ⋯ ¿ ¿ 𝑥𝑖 𝑦 𝑖 0 0 0 0 𝑥𝑖 𝑦𝑖 1 0 0 1 ¿ ⋯ ][ 𝑚1 𝑚2 𝑚3 𝑚4 𝑡𝑥 𝑡𝑦 ] [𝑥𝑖 ′ 𝑦𝑖 ′ ]= [𝑚1 𝑚2 𝑚3 𝑚4 ][𝑥𝑖 𝑦𝑖 ]+ [𝑡𝑥 𝑡𝑦 ] Given correspondences: [x’i,y’i] ↔ [xi,yi] Set up two equations per point
Fitting Transformations [ ⋮ 𝑥𝑖 ′ 𝑦𝑖 ′ ⋮ ]= [ ⋯ ¿ ¿ 𝑥𝑖 𝑦 𝑖 0 0 0 0 𝑥𝑖 𝑦𝑖 1 0 0 1 ¿ ⋯ ][ 𝑚1 𝑚2 𝑚3 𝑚4 𝑡𝑥 𝑡𝑦 ] 2 equations per point, 6 unknowns How many points do we need to properly constrain the problem? 2k 6
Fitting Transformations [ ⋮ 𝑥𝑖 ′ 𝑦𝑖 ′ ⋮ ]= [ ⋯ ¿ ¿ 𝑥𝑖 𝑦 𝑖 0 0 0 0 𝑥𝑖 𝑦𝑖 1 0 0 1 ¿ ⋯ ][ 𝑚1 𝑚2 𝑚3 𝑚4 𝑡𝑥 𝑡𝑦 ] Want: b = Ax (x contains all parameters) Overconstrained, so solve How? 2k 6 b A x
Fitting Transformation Data: (xi,yi,x’i,y’i) for i=1,…,k Model: [x’i,y’i,1] ≡ H[xi,yi,1] Objective function: It’s complicated H Homography: H
9 k points → 2k 𝑨𝒉=𝟎 What do we use from last time? h ∗ =arg min ‖h‖=1 ‖ h 𝐴 ‖ 2 Eigenvector of AT A with smallest eigenvalue 𝒑 𝒊= [ 𝑥𝑖 𝑦𝑖 1 ] Fitting Transformation [ 𝟎 𝑇 − 𝒑1 𝑇 𝑦1 ′ 𝒑1 𝑇 𝒑1 𝑇 𝟎 𝑇 − 𝑥1 ′ 𝒑1 𝑇 ⋮ 𝟎𝑇 − 𝒑𝑛 𝑇 𝑦𝑛 ′ 𝒑𝑛 𝑇 𝒑𝑛 𝑇 𝟎 𝑇 − 𝑥𝑛 ′ 𝒑𝑛 𝑇 ][ 𝒉𝟏 𝒉𝟐 𝒉𝟑 ]=𝟎 Row 1 of H
In Practice [ 𝟎 𝑇 − 𝒑1 𝑇 𝑦1 ′ 𝒑1 𝑇 𝒑1 𝑇 𝟎 𝑇 − 𝑥1 ′ 𝒑1 𝑇 ⋮ 𝟎𝑇 − 𝒑𝑛 𝑇 𝑦𝑛 ′ 𝒑𝑛 𝑇 𝒑𝑛 𝑇 𝟎 𝑇 − 𝑥𝑛 ′ 𝒑𝑛 𝑇 ][ 𝒉𝟏 𝒉𝟐 𝒉𝟑 ]=𝟎 9 k points → 2k 𝑨𝒉=𝟎 Should consist of lots of {x,y,x’,y’,0, and 1}. If it fails, assume you mistyped. Re-type differently and compare all entries. Debug first with transformations you know. 𝒑 𝒊= [ 𝑥𝑖 𝑦𝑖 1 ] Row 1 of H
Small Nagging Detail ||Ah||2 doesn’t measure model fit (it’s an algebraic error that’s mainly just convenient to minimize) Also, there’s a least-squares setup that’s wrong but often works. ∑ 𝑖=1 𝑘 ‖[𝑥𝑖 ′ , 𝑦𝑖 ′ ]−𝑇 ([𝑥𝑖 , 𝑦𝑖])‖ 2 +‖[𝑥𝑖 , 𝑦𝑖]−𝑇 −1 ([𝑥𝑖 ′ , 𝑦𝑖 ′ ])‖ 2 Really want geometric error:
Small Nagging Detail In RANSAC, we always take just enough points to fit. Why might this not make a big difference when fitting a model with RANSAC? Solution: initialize with algebraic (min ||Ah||), optimize with geometric using standard non-linear optimizer
Image Warping x y x y f(x,y) g(x,y) T(x,y) Given a coordinate transform (x’,y’) = T(x,y) and a source image f(x,y), how do we compute a transformed image g(x’,y’) = f(T(x,y))? Slide Credit: A. Efros
Forward Warping x y x' y' f(x,y) g(x’,y’) T(x,y) Send the value at each pixel (x,y) to the new pixel (x’,y’) = T([x,y]) Slide Credit: A. Efros
Forward Warping x y f(x,y) x-1 x x+1 y-1 y y+1 x'-1 x' x'+1 y'-1 y' y'+1 x' y’ g(x’,y’) If you don’t hit an exact pixel, give the value to each of the neighboring pixels (“splatting”). T(x,y)
Forward Warping Suppose T(x,y) scales by a factor of 3. Hmmmm.
Inverse Warping x y x' y' f(x,y) g(x’,y’) T-1 (x,y) Find out where each pixel g(x’,y’) should get its value from, and steal it. Note: requires ability to invert T Slide Credit: A. Efros
Inverse Warping x'-1 x' x'+1 y'-1 y' y'+1 x' y’ g(x’,y’) x y f(x,y) x-1 x x+1 y-1 y y+1 If you don’t hit an exact pixel, figure out how to take it from the neighbors. T-1 (x,y)
Mosaicing Warped Input 1 I1 Warped Input 2 I2 Image Credit: A. Efros Can warp an image. Pixels that don’t have a corresponding pixel in the image are set to a chosen value (often 0)
Mosaicing Warped Input 1 I1 α Warped Input 2 I2 αI1 + (1-α)I2 Image Credit: A. Efros
Mosaicing Warped Input 1 I1 α Warped Input 2 I2 αI1 + (1-α)I2 Slide Credit: A. Efros Can also warp an image containing 1s. Pixels that don’t have a corresponding pixel in the image are set to a chosen value (often 0)
Putting it Together How do you make a panorama? Step 1: Find “features” to match Step 2: Describe Features Step 3: Match by Nearest Neighbor Step 4: Fit H via RANSAC Step 5: Blend Images
Putting It Together 1 • (Multi-scale) Harris; or • Laplacian of Gaussian Find corners/blobs
Putting It Together 2 Describe Regions Near Features Build histogram of gradient orientations (SIFT) (But in practice use opencv) 𝑥𝑞 ∈ 𝑅128
Putting It Together 3 Match Features Based On Region 𝑥1 ∈ 𝑅128 𝑥2 ∈ 𝑅128 𝑥𝑞 ∈ 𝑅128 𝑥𝑞 Sort by distance to: ‖𝑥𝑞 − 𝑥1‖<‖𝑥𝑞 − 𝑥2 ‖<‖𝑥𝑞 − 𝑥3‖ Accept match if: ‖𝑥𝑞 − 𝑥1‖/‖𝑥𝑞 − 𝑥2‖ Nearest neighbor is far closer than 2nd nearest neighbor
Putting It Together 4 Fit transformation H via RANSAC for trial in range(Ntrials): Pick sample Fit model Check if more inliers Re-fit model with most inliers arg min ‖𝒉‖=1 ‖𝑨𝒉‖ 2
Putting It Together 5 Warp images together Resample images with inverse warping and blend (but in practice, just call opencv for inverse warping)
transformations2 and fitting on image processing.pptx
Backup
A pencil of rays contains all views real camera synthetic camera Can generate any synthetic camera view as long as it has the same center of projection! Slide Credit: A. Efros
Bonus Art
Automatically rectified floor St. Lucy Altarpiece, D. Veneziano Analyzing Patterns What is the (complicated) shape of the floor pattern? Slide from A. Criminisi
From Martin Kemp, The Science of Art (manual reconstruction) Automatic rectification Analyzing Patterns Slide from A. Criminisi
Homography Derivation • This has gotten cut in favor of showing more of the setup. • The key to the set-up is to try to move towards a setup where you can pull [h1,h2,h3] out, or where each row is a linear equation in [h1,h2,h3]
[ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥𝑖 𝑦 𝑖 𝑤𝑖 ]≡ [ 𝑥𝑖 ′ 𝑦𝑖 ′ 𝑤𝑖 ′ ]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥𝑖 𝑦𝑖 𝑤𝑖 ] Want: Recall: 𝒂≡𝒃 𝒂=𝜆𝒃 𝒂×𝒃=𝟎 In turn 𝒑 𝒊= [ 𝑥𝑖 𝑦𝑖 1 ] 𝑯 𝒑𝒊≡ [ 𝒉𝟏 𝑻 𝒉𝟐 𝑻 𝒉𝟑 𝑻 ]𝒑𝒊 ≡ [ 𝒉𝟏 𝑻 𝒑𝒊 𝒉𝟐 𝑻 𝒑𝒊 𝒉𝟑 𝑻 𝒑𝒊 ] [ 𝑥𝑖 ′ 𝑦𝑖 ′ 𝑤𝑖 ′ ]× [ 𝒉𝟏 𝑻 𝒑𝒊 𝒉𝟐 𝑻 𝒑𝒊 𝒉𝟑 𝑻 𝒑𝒊 ]=𝟎 In the end want: Why Cross products? Cross products have explicit forms
Fitting Transformation [ 𝑥𝑖 ′ 𝑦𝑖 ′ 𝑤𝑖 ′ ]× [ 𝒉𝟏 𝑻 𝒑𝒊 𝒉𝟐 𝑻 𝒑𝒊 𝒉𝟑 𝑻 𝒑𝒊 ]=𝟎 Want: [ 𝑦𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 −𝑤𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊 𝑤𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 − 𝑥𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 𝑥𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊 − 𝑦𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 ]=𝟎 Cross- product [ 𝒉𝟏 𝑻 𝟎−𝑤𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊+ 𝑦𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 𝑤𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 +𝒉𝟐 𝑻 𝟎− 𝑥𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 − 𝑦𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 +𝑥𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊 +𝒉𝟑 𝑻 𝟎]=𝟎 Re-arrange and put 0s in Note: calculate this explicitly. It looks ugly, but do it by doing [a,b,c] x [a’,b’,c’] then re-substituting. You want to be able to right- multiply by [h1,h2,h3]
Fitting Transformation [ 𝒉𝟏 𝑻 𝟎−𝑤𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊+ 𝑦𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 𝑤𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 +𝒉𝟐 𝑻 𝟎− 𝑥𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 − 𝑦𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 +𝑥𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊 +𝒉𝟑 𝑻 𝟎]=𝟎 Equation Pull out h [ 𝟎 𝑻 −𝑤 ′ 𝑖 𝒑𝒊 𝑻 𝑦 ′𝑖 𝒑𝒊 𝑻 𝑤𝑖 ′ 𝒑𝒊 𝑻 𝟎 𝑻 − 𝑥𝑖 ′ 𝒑𝒊 𝑻 − 𝑦𝑖 ′ 𝒑𝒊 𝑻 𝑥𝑖 ′ 𝒑𝒊 𝑻 𝟎 𝑻 ][ 𝒉𝟏 𝒉𝟐 𝒉𝟑 ]=𝟎 Only two linearly independent equations Yank out h once you have all the coefficients. If you’re head-scratching about the two equations, it’s not obvious to me at first glance that the three equations aren’t linearly independent either.
Simplification: Two-band Blending • Brown & Lowe, 2003 • Only use two bands: high freq. and low freq. • Blend low freq. smoothly • Blend high freq. with no smoothing: binary alpha Figure Credit: Brown & Lowe
Low frequency (l > 2 pixels) High frequency (l < 2 pixels) 2-band “Laplacian Stack” Blending
Linear Blending
2-band Blending

More Related Content

PPTX
3 d scaling and translation in homogeneous coordinates
KRIPA SHNAKAR TIWARI
 
PPTX
lec07_transformations.pptx
AneesAbbasi14
 
PPTX
Computer Vision transformations
Wael Badawy
 
PPTX
2D Transformation.pptx
MuhammadNaqi7
 
PDF
Lecture 06 geometric transformations and image registration
obertksg
 
PPT
GeometricTransformations.ppt
Debjit Doira
 
PPT
2d transformation
Sarkunavathi Aribal
 
PPT
Computer graphics
Bala Murali
 
3 d scaling and translation in homogeneous coordinates
KRIPA SHNAKAR TIWARI
 
lec07_transformations.pptx
AneesAbbasi14
 
Computer Vision transformations
Wael Badawy
 
2D Transformation.pptx
MuhammadNaqi7
 
Lecture 06 geometric transformations and image registration
obertksg
 
GeometricTransformations.ppt
Debjit Doira
 
2d transformation
Sarkunavathi Aribal
 
Computer graphics
Bala Murali
 

Similar to transformations2 and fitting on image processing.pptx (20)

PDF
2d Transformation.pdf
Mattupallipardhu
 
PPT
Modeling Transformations
Tarun Gehlot
 
PPTX
2 d transformations and homogeneous coordinates
Tarun Gehlot
 
PPT
2D-Transformations-Transformations are the operations applied to geometrical ...
BINJAD1
 
PPT
Geometric transformation cg
harinipriya1994
 
PPTX
Transformations.pptx
mekelle university(EiT-M)
 
PPTX
Transformations computer graphics
Vikram Halder
 
PPT
september4.ppt
CharlesMatu2
 
PPT
Matrix 2 d
xyz120
 
PPT
2d/3D transformations in computer graphics(Computer graphics Tutorials)
Daroko blog(www.professionalbloggertricks.com)
 
PPT
Transforms UNIt 2
sandeep kumbhkar
 
PPTX
Computer Graphics - transformations in 2d
Hisham Al Kurdi, EAVA, DMC-D-4K, HCCA-P, HCAA-D
 
PPTX
Computer Graphic - Transformations in 2D
2013901097
 
PPT
affine transformation for computer graphics
DrSUGANYADEVIK
 
PPTX
Two dimensionaltransformations
Nareek
 
PDF
Structure and Motion - 3D Reconstruction of Cameras and Structure
Giovanni Murru
 
PPT
06.Transformation.ppt
RobinAhmedSaikat
 
PPT
2D_transformatiomcomputer graphics 2d translation, rotation and scaling trans...
renuka b
 
PPTX
Computer Vision alignment
Wael Badawy
 
PPT
Transformations
Syed Zaid Irshad
 
2d Transformation.pdf
Mattupallipardhu
 
Modeling Transformations
Tarun Gehlot
 
2 d transformations and homogeneous coordinates
Tarun Gehlot
 
2D-Transformations-Transformations are the operations applied to geometrical ...
BINJAD1
 
Geometric transformation cg
harinipriya1994
 
Transformations.pptx
mekelle university(EiT-M)
 
Transformations computer graphics
Vikram Halder
 
september4.ppt
CharlesMatu2
 
Matrix 2 d
xyz120
 
2d/3D transformations in computer graphics(Computer graphics Tutorials)
Daroko blog(www.professionalbloggertricks.com)
 
Transforms UNIt 2
sandeep kumbhkar
 
Computer Graphics - transformations in 2d
Hisham Al Kurdi, EAVA, DMC-D-4K, HCCA-P, HCAA-D
 
Computer Graphic - Transformations in 2D
2013901097
 
affine transformation for computer graphics
DrSUGANYADEVIK
 
Two dimensionaltransformations
Nareek
 
Structure and Motion - 3D Reconstruction of Cameras and Structure
Giovanni Murru
 
06.Transformation.ppt
RobinAhmedSaikat
 
2D_transformatiomcomputer graphics 2d translation, rotation and scaling trans...
renuka b
 
Computer Vision alignment
Wael Badawy
 
Transformations
Syed Zaid Irshad
 
Ad

More from Indra Hermawan (20)

PDF
L6 Visual Output LED_7SEGMEN_LEDMATRIX upate.pdf
Indra Hermawan
 
PDF
Organic Light Emitting Diode (OLED) for embedded system
Indra Hermawan
 
PDF
Organic Light Emitting Diode (OLED) in embedded system
Indra Hermawan
 
PDF
L6 Visual Output LED_7SEGMEN_LEDMATRIX upate.pdf
Indra Hermawan
 
PPTX
over or under fitting in neural network.pptx
Indra Hermawan
 
PPT
zigbee networks using xbee modules for wsn
Indra Hermawan
 
PDF
pengenalan mengnai ai dan machine learning
Indra Hermawan
 
PPTX
Market_Behavior_Analytics_AIoT TMJ 6A.pptx
Indra Hermawan
 
PPTX
7-1-Artificial Neural Network dan multilayer perceptron.pptx
Indra Hermawan
 
PPTX
ZigBee_Presentation sensor and cotrol network
Indra Hermawan
 
PPTX
ZigBee_Comprehensive_Presentation and implementaiton
Indra Hermawan
 
PPTX
Numerical Linear Algebra in digital image processing
Indra Hermawan
 
PPTX
Filtering in digital signal and image processing
Indra Hermawan
 
PPTX
Sistem Pengomposan Cerdas Berbasis IoT untuk Pengelolaan Sampah Organik Berke...
Indra Hermawan
 
PDF
introduction to camera, the operation of camera
Indra Hermawan
 
PDF
lab_linear_regression_hy539 (1)_221109_035050.pdf
Indra Hermawan
 
PPTX
perancangan program kerja KBK perancangan program kerja KBK perancangan progr...
Indra Hermawan
 
PPT
zigbee networks using xbee modules zigbee networks using xbee modules
Indra Hermawan
 
PPTX
lecture_8-wsn.pptx
Indra Hermawan
 
PPT
Trik Pembuatan Surat Lamaran Pekerjaan.ppt
Indra Hermawan
 
L6 Visual Output LED_7SEGMEN_LEDMATRIX upate.pdf
Indra Hermawan
 
Organic Light Emitting Diode (OLED) for embedded system
Indra Hermawan
 
Organic Light Emitting Diode (OLED) in embedded system
Indra Hermawan
 
L6 Visual Output LED_7SEGMEN_LEDMATRIX upate.pdf
Indra Hermawan
 
over or under fitting in neural network.pptx
Indra Hermawan
 
zigbee networks using xbee modules for wsn
Indra Hermawan
 
pengenalan mengnai ai dan machine learning
Indra Hermawan
 
Market_Behavior_Analytics_AIoT TMJ 6A.pptx
Indra Hermawan
 
7-1-Artificial Neural Network dan multilayer perceptron.pptx
Indra Hermawan
 
ZigBee_Presentation sensor and cotrol network
Indra Hermawan
 
ZigBee_Comprehensive_Presentation and implementaiton
Indra Hermawan
 
Numerical Linear Algebra in digital image processing
Indra Hermawan
 
Filtering in digital signal and image processing
Indra Hermawan
 
Sistem Pengomposan Cerdas Berbasis IoT untuk Pengelolaan Sampah Organik Berke...
Indra Hermawan
 
introduction to camera, the operation of camera
Indra Hermawan
 
lab_linear_regression_hy539 (1)_221109_035050.pdf
Indra Hermawan
 
perancangan program kerja KBK perancangan program kerja KBK perancangan progr...
Indra Hermawan
 
zigbee networks using xbee modules zigbee networks using xbee modules
Indra Hermawan
 
lecture_8-wsn.pptx
Indra Hermawan
 
Trik Pembuatan Surat Lamaran Pekerjaan.ppt
Indra Hermawan
 
Ad

Recently uploaded (20)

PDF
20A LG INR18650HJ2 3.6V 2900mAh Battery cells for Power Tools Vacuum Cleaner
PERMA Tech Co., Ltd.
 
PPTX
ELETRONIC-PRODUCTS-ASSEMBLY-AND-SERVICING-NC-II-WEEK-1-Copy.pptx
criszellehidden
 
PPTX
New professional education PROF-ED-7_103359.pptx
JAYZHELARTIZA
 
PDF
SAHIL PROdhdjejss yo yo pdf TOCOL PPT.pdf
akansharohilla2002
 
PPTX
Subordinate_Clauses_BlueGradient_Optimized.pptx
AngadSingh427299
 
PPTX
Clauses_Part1.hshshpjzjxnznxnxnndndndndndndndnndptx
AngadSingh427299
 
PDF
Presented by ATHUL KRISHNA.S_20250813_191657_0000.pdf
athulkrishnas966
 
PDF
Dozuki_Solution-hardware minimalization.
kurtaku1
 
PPTX
vortex flow measurement in instrumentation
vijaykumarnarayanasw
 
PPTX
Entre CHtzyshshshshshshshzhhzzhhz 4MSt.pptx
Eyob262486
 
PDF
2- Physical Layer (06).pdfgshshshbsbshshshhs
sinhasumit200508shri
 
DOCX
Copy-OT LIST 12.8.25.docxjdjfufufufufuuffuf
somusachin100
 
PPTX
RTS MASTER DECK_Household Convergence Scorecards. Use this file copy.pptx
6xdj6pmy9m
 
PPT
System Unit Components and its Functions
JdAnadon
 
PPTX
PLC ANALOGUE DONE BY KISMEC KULIM TD 5 .0
MahadzirMatRabi1
 
PPTX
Grade 10 System Servicing for Hardware and Software
CaloyMasion
 
PPTX
Growth Capital Investment - Espresso Capital.pptx
EspressoCapital
 
PDF
Tcl Scripting for EDA.pdf
Ahmed Abdelazeem
 
PDF
GENERATOR AND IMPROVED COIL THEREFOR HAVINGELECTRODYNAMIC PROPERTIES
AhmadSubiyanto
 
PDF
Printing Presentation to show beginners.
John306149
 
20A LG INR18650HJ2 3.6V 2900mAh Battery cells for Power Tools Vacuum Cleaner
PERMA Tech Co., Ltd.
 
ELETRONIC-PRODUCTS-ASSEMBLY-AND-SERVICING-NC-II-WEEK-1-Copy.pptx
criszellehidden
 
New professional education PROF-ED-7_103359.pptx
JAYZHELARTIZA
 
SAHIL PROdhdjejss yo yo pdf TOCOL PPT.pdf
akansharohilla2002
 
Subordinate_Clauses_BlueGradient_Optimized.pptx
AngadSingh427299
 
Clauses_Part1.hshshpjzjxnznxnxnndndndndndndndnndptx
AngadSingh427299
 
Presented by ATHUL KRISHNA.S_20250813_191657_0000.pdf
athulkrishnas966
 
Dozuki_Solution-hardware minimalization.
kurtaku1
 
vortex flow measurement in instrumentation
vijaykumarnarayanasw
 
Entre CHtzyshshshshshshshzhhzzhhz 4MSt.pptx
Eyob262486
 
2- Physical Layer (06).pdfgshshshbsbshshshhs
sinhasumit200508shri
 
Copy-OT LIST 12.8.25.docxjdjfufufufufuuffuf
somusachin100
 
RTS MASTER DECK_Household Convergence Scorecards. Use this file copy.pptx
6xdj6pmy9m
 
System Unit Components and its Functions
JdAnadon
 
PLC ANALOGUE DONE BY KISMEC KULIM TD 5 .0
MahadzirMatRabi1
 
Grade 10 System Servicing for Hardware and Software
CaloyMasion
 
Growth Capital Investment - Espresso Capital.pptx
EspressoCapital
 
Tcl Scripting for EDA.pdf
Ahmed Abdelazeem
 
GENERATOR AND IMPROVED COIL THEREFOR HAVINGELECTRODYNAMIC PROPERTIES
AhmadSubiyanto
 
Printing Presentation to show beginners.
John306149
 

transformations2 and fitting on image processing.pptx

  • 1. Transformation s and Fitting EECS 442 – David Fouhey Winter 2023, University of Michigan https://web.eecs.umich.edu/~fouhey/teaching/EECS442_W23/
  • 2. Administrivia • Discussion this week = office hours
  • 3. So Far 1. How do we find distinctive / easy to locate features? (Harris/Laplacian of Gaussian) 2. How do we describe the regions around them? (histogram of gradients) 3. How do we match features? (L2 distance) 4. How do we handle outliers? (RANSAC)
  • 4. Today As promised: warping one image to another
  • 5. Why Mosaic? • Compact Camera FOV = 50 x 35° Slide credit: Brown & Lowe
  • 6. Why Mosaic? • Compact Camera FOV = 50 x 35° • Human FOV = 200 x 135° Slide credit: Brown & Lowe
  • 7. Why Mosaic? • Compact Camera FOV = 50 x 35° • Human FOV = 200 x 135° • Panoramic Mosaic = 360 x 180° Slide credit: Brown & Lowe
  • 8. Why Bother With This Math? Slide credit: A. Efros
  • 9. Homework 1 Style Translation only via alignment Slide credit: A. Efros
  • 11. Image Transformations f x T g x f x T g x Image filtering: change range of image 𝑔( 𝑥)=𝑇 ( 𝑓 ( 𝑥)) 𝑔 ( 𝑥 )= 𝑓 ¿ Image warping: change domain of image Slide credit: A. Efros
  • 12. Image Transformations T T Image filtering: change range of image 𝑔(𝑥 , 𝑦)=𝑇 ( 𝑓 (𝑥, 𝑦 )) 𝑔 ( 𝑥 , 𝑦 )= 𝑓 ¿ Image warping: change domain of image f g f g Slide credit: A. Efros
  • 13. Parametric (Global) warping translation rotation aspect affine perspective cylindrical Examples of parametric warps Slide credit: A. Efros
  • 14. Parametric (Global) Warping T p’ = (x’,y’) T is a coordinate changing machine p = (x,y) Note: T is the same for all points, has relatively few parameters, and does not depend on image content 𝒑′ =𝑇 (𝒑) Slide credit: A. Efros
  • 15. Parametric (Global) Warping T p’ = (x’,y’) p = (x,y) Today we’ll deal with linear warps 𝒑′ ≡𝑻𝒑 T: matrix; p, p’: 2D points. Start with normal points and =, then do homogeneous cords and ≡ Slide credit: A. Efros
  • 16. Scaling  2 Scaling multiplies each component (x,y) by a scalar. Uniform scaling is the same for all components. Note the corner goes from (1,1) to (2,2) Slide credit: A. Efros
  • 17. Scaling Non-uniform scaling multiplies each component by a different scalar. X  2, Y  0.5 Slide credit: A. Efros
  • 18. Scaling What does T look like? 𝑥′ =𝑎𝑥 𝑦 ′ =𝑏𝑦 Let’s convert to a matrix: [𝑥 ′ 𝑦 ′ ]= [𝑎 0 0 𝑏][𝑥 𝑦 ] scaling matrix S What’s the inverse of S? Slide credit: A. Efros
  • 19. 2D Rotation Rotation Matrix But wait! Aren’t sin/cos non-linear? x’ is a linear combination/function of x, y x’ is not a linear function of θ What’s the inverse of Rθ? 𝑰 =𝑹𝜽 𝑇 𝑹𝜽 [𝑥 ′ 𝑦 ′ ]= [c os ⁡(𝜃) − sin (𝜃 ) sin ( 𝜃) cos(𝜃 ) ][𝑥 𝑦 ] Slide credit: A. Efros
  • 20. Things You Can Do With 2x2 Identity / No Transformation Shear [𝑥 ′ 𝑦 ′ ]= [ 1 𝑠 h𝑥 𝑠h𝑦 1 ][𝑥 𝑦 ] [𝑥 ′ 𝑦 ′ ]= [1 0 0 1 ][𝑥 𝑦 ] Slide credit: A. Efros
  • 21. Things You Can Do With 2x2 2D Mirror About Y-Axis [𝑥 ′ 𝑦 ′ ]= [−1 0 0 1 ][𝑥 𝑦 ] Before After 2D Mirror About X,Y [𝑥 ′ 𝑦 ′ ]= [−1 0 0 −1][𝑥 𝑦 ] Before After Slide credit: A. Efros
  • 22. What’s Preserved? Projections of parallel 3D lines are not necessarily parallel, so not parallelism 3D lines project to 2D lines so lines are preserved Distant objects are smaller so size is not preserved
  • 23. What’s Preserved With a 2x2 [𝑥 ′ 𝑦 ′ ]= [𝑎 𝑏 𝑐 𝑑][𝑥 𝑦 ]=𝑇 [𝑥 𝑦] After multiplication by T (irrespective of T) • Origin is origin: 0 = T0 • Lines are lines • Parallel lines are parallel
  • 24. Things You Can’t Do With 2x2 What about translation? x’ = x + tx, y’ = y+ty +(2,2) How do we make it linear?
  • 25. Homogeneous Coordinates Again What about translation? x’ = x + tx, y’ = y+ty +(2,2) [ 𝑥+𝑡𝑥 𝑦 +𝑡 𝑦 1 ]≡ [ 𝑥′ 𝑦 ′ 1 ]≡ [ 1 0 𝑡𝑥 0 1 𝑡𝑦 0 0 1 ][ 𝑥 𝑦 1 ] Slide credit: A. Efros
  • 26. Representing 2D Transformations How do we represent a 2D transformation? Let’s pick scaling [ 𝑥′ 𝑦 ′ 1 ]≡ [ 𝑠𝑥 0 𝑎 0 𝑠𝑦 𝑏 𝑑 𝑒 𝑓 ][ 𝑥 𝑦 1 ] a b d e f 0 0 0 0 1 What’s
  • 27. Affine Transformations Affine: linear transformation plus translation In general (without homogeneous coordinates) 𝒙 ′= 𝑨𝒙+𝒃 Will the last coordinate w’ always be 1? [ 𝑥 ′ 𝑦 ′ 𝑤 ′]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 0 0 1 ][ 𝑥 𝑦 1 ] t
  • 28. Matrix Composition [ 𝑥′ 𝑦 ′ 𝑤 ′]≡ [ 1 0 𝑡𝑥 0 1 𝑡 𝑦 0 0 1 ][ cos (𝜃 ) −sin ( 𝜃) 0 sin (𝜃) cos ( 𝜃) 0 0 0 1][ 𝑠𝑥 0 0 0 𝑠 𝑦 0 0 0 1 ][ 𝑥 𝑦 𝑤] 𝑇 (𝑡𝑥 ,𝑡𝑦 ) 𝑅( 𝜃) 𝑆(𝑠𝑥 ,𝑠𝑦 ) We can combine transformations via matrix multiplication. Does order matter? Slide credit: A. Efros
  • 29. What’s Preserved With Affine After multiplication by T (irrespective of T) • Origin is origin: 0 = T0 • Lines are lines • Parallel lines are parallel [ 𝑥′ 𝑦 ′ 1 ]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 0 0 1 ][ 𝑥 𝑦 1 ]≡ 𝑻 [ 𝑥 𝑦 1 ]
  • 30. Homogeneous Equivalence z x y [x,y,w] λ[x,y,w] Two homogeneous coordinates are equivalent if they are proportional to each other. Not = ! [ 𝑢 𝑣 𝑤]≡ [ 𝑢′ 𝑣′ 𝑤′ ]↔ [ 𝑢 𝑣 𝑤]= 𝜆 [ 𝑢′ 𝑣′ 𝑤′ ] 𝜆≠ 0 Triple / Equivalent Double / Equals
  • 31. Perspective Transformations Set bottom row to not [0,0,1] Called a perspective/projective transformation or a homography [ 𝑥′ 𝑦 ′ 𝑤 ′]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥 𝑦 𝑤] Can compute [x’,y’,w’] via matrix multiplication. How do we get a 2D point? (x’/w’, y’/w’)
  • 32. Perspective Transformations Set bottom row to not [0,0,1] Called a perspective/projective transformation or a homography [ 𝑥′ 𝑦 ′ 𝑤 ′]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥 𝑦 𝑤] How many degrees of freedom?
  • 33. How Many Degrees of Freedom? Can always scale coordinate by non-zero value [ 𝑥 ′ 𝑦 ′ 𝑤 ′]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥 𝑦 𝑤] Perspective [ 𝑥 ′ 𝑦 ′ 𝑤 ′]≡ 1 𝑖 [ 𝑥′ 𝑦 ′ 𝑤′ ] Homography can always be re-scaled by λ≠0 Typically pick it so last entry is 1. ≡ 1 𝑖 [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥 𝑦 𝑤 ]
  • 34. What’s Preserved With Perspective After multiplication by T (irrespective of T) • Origin is origin: 0 = T0 • Lines are lines • Parallel lines are parallel • Ratios between distances [ 𝑥′ 𝑦 ′ 1 ]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥 𝑦 1 ]≡ 𝑻 [ 𝑥 𝑦 1 ]
  • 35. Transformation Families In general: transformations are a nested set of groups Diagram credit: R. Szeliski
  • 36. What Can Homographies Do? Homography example 1: any two views of a planar surface Figure Credit: S. Lazebnik
  • 37. What Can Homographies Do? Homography example 2: any images from two cameras sharing a camera center Figure Credit: S. Lazebnik
  • 38. What Can Homographies Do? Homography sort of example “3”: far away scene that can be approximated by a plane Figure credit: Brown & Lowe
  • 39. Fun With Homographies Original image St. Petersburg photo by A. Tikhonov Virtual camera rotations Slide Credit: A. Efros
  • 40. Analyzing Patterns Homography Automatically rectified floor The floor (enlarged) Slide from A. Criminisi
  • 41. Analyzing Patterns Slide from A. Criminisi Automatic rectification From Martin Kemp The Science of Art (manual reconstruction)
  • 42. Fitting Transformations Setup: have pairs of correspondences (𝑥𝑖 , 𝑦𝑖) (𝑥 ′𝑖 , 𝑦 ′𝑖) M,t [𝑥𝑖 ′ 𝑦𝑖 ′ ]=𝑴 [𝑥𝑖 𝑦𝑖 ]+𝒕 Slide Credit: S. Lazebnik
  • 43. Fitting Transformation Data: (xi,yi,x’i,y’i) for i=1,…,k Model: [x’i,y’i] = M[xi,yi]+t Objective function: ||[x’i,y’i] – (M[xi,yi]+t)||2 M,t Affine Transformation: M,t
  • 44. Fitting Transformations [ ⋮ 𝑥𝑖 ′ 𝑦𝑖 ′ ⋮ ]= [ ⋯ ¿ ¿ 𝑥𝑖 𝑦 𝑖 0 0 0 0 𝑥𝑖 𝑦𝑖 1 0 0 1 ¿ ⋯ ][ 𝑚1 𝑚2 𝑚3 𝑚4 𝑡𝑥 𝑡𝑦 ] [𝑥𝑖 ′ 𝑦𝑖 ′ ]= [𝑚1 𝑚2 𝑚3 𝑚4 ][𝑥𝑖 𝑦𝑖 ]+ [𝑡𝑥 𝑡𝑦 ] Given correspondences: [x’i,y’i] ↔ [xi,yi] Set up two equations per point
  • 45. Fitting Transformations [ ⋮ 𝑥𝑖 ′ 𝑦𝑖 ′ ⋮ ]= [ ⋯ ¿ ¿ 𝑥𝑖 𝑦 𝑖 0 0 0 0 𝑥𝑖 𝑦𝑖 1 0 0 1 ¿ ⋯ ][ 𝑚1 𝑚2 𝑚3 𝑚4 𝑡𝑥 𝑡𝑦 ] 2 equations per point, 6 unknowns How many points do we need to properly constrain the problem? 2k 6
  • 46. Fitting Transformations [ ⋮ 𝑥𝑖 ′ 𝑦𝑖 ′ ⋮ ]= [ ⋯ ¿ ¿ 𝑥𝑖 𝑦 𝑖 0 0 0 0 𝑥𝑖 𝑦𝑖 1 0 0 1 ¿ ⋯ ][ 𝑚1 𝑚2 𝑚3 𝑚4 𝑡𝑥 𝑡𝑦 ] Want: b = Ax (x contains all parameters) Overconstrained, so solve How? 2k 6 b A x
  • 47. Fitting Transformation Data: (xi,yi,x’i,y’i) for i=1,…,k Model: [x’i,y’i,1] ≡ H[xi,yi,1] Objective function: It’s complicated H Homography: H
  • 48. 9 k points → 2k 𝑨𝒉=𝟎 What do we use from last time? h ∗ =arg min ‖h‖=1 ‖ h 𝐴 ‖ 2 Eigenvector of AT A with smallest eigenvalue 𝒑 𝒊= [ 𝑥𝑖 𝑦𝑖 1 ] Fitting Transformation [ 𝟎 𝑇 − 𝒑1 𝑇 𝑦1 ′ 𝒑1 𝑇 𝒑1 𝑇 𝟎 𝑇 − 𝑥1 ′ 𝒑1 𝑇 ⋮ 𝟎𝑇 − 𝒑𝑛 𝑇 𝑦𝑛 ′ 𝒑𝑛 𝑇 𝒑𝑛 𝑇 𝟎 𝑇 − 𝑥𝑛 ′ 𝒑𝑛 𝑇 ][ 𝒉𝟏 𝒉𝟐 𝒉𝟑 ]=𝟎 Row 1 of H
  • 49. In Practice [ 𝟎 𝑇 − 𝒑1 𝑇 𝑦1 ′ 𝒑1 𝑇 𝒑1 𝑇 𝟎 𝑇 − 𝑥1 ′ 𝒑1 𝑇 ⋮ 𝟎𝑇 − 𝒑𝑛 𝑇 𝑦𝑛 ′ 𝒑𝑛 𝑇 𝒑𝑛 𝑇 𝟎 𝑇 − 𝑥𝑛 ′ 𝒑𝑛 𝑇 ][ 𝒉𝟏 𝒉𝟐 𝒉𝟑 ]=𝟎 9 k points → 2k 𝑨𝒉=𝟎 Should consist of lots of {x,y,x’,y’,0, and 1}. If it fails, assume you mistyped. Re-type differently and compare all entries. Debug first with transformations you know. 𝒑 𝒊= [ 𝑥𝑖 𝑦𝑖 1 ] Row 1 of H
  • 50. Small Nagging Detail ||Ah||2 doesn’t measure model fit (it’s an algebraic error that’s mainly just convenient to minimize) Also, there’s a least-squares setup that’s wrong but often works. ∑ 𝑖=1 𝑘 ‖[𝑥𝑖 ′ , 𝑦𝑖 ′ ]−𝑇 ([𝑥𝑖 , 𝑦𝑖])‖ 2 +‖[𝑥𝑖 , 𝑦𝑖]−𝑇 −1 ([𝑥𝑖 ′ , 𝑦𝑖 ′ ])‖ 2 Really want geometric error:
  • 51. Small Nagging Detail In RANSAC, we always take just enough points to fit. Why might this not make a big difference when fitting a model with RANSAC? Solution: initialize with algebraic (min ||Ah||), optimize with geometric using standard non-linear optimizer
  • 52. Image Warping x y x y f(x,y) g(x,y) T(x,y) Given a coordinate transform (x’,y’) = T(x,y) and a source image f(x,y), how do we compute a transformed image g(x’,y’) = f(T(x,y))? Slide Credit: A. Efros
  • 53. Forward Warping x y x' y' f(x,y) g(x’,y’) T(x,y) Send the value at each pixel (x,y) to the new pixel (x’,y’) = T([x,y]) Slide Credit: A. Efros
  • 54. Forward Warping x y f(x,y) x-1 x x+1 y-1 y y+1 x'-1 x' x'+1 y'-1 y' y'+1 x' y’ g(x’,y’) If you don’t hit an exact pixel, give the value to each of the neighboring pixels (“splatting”). T(x,y)
  • 55. Forward Warping Suppose T(x,y) scales by a factor of 3. Hmmmm.
  • 56. Inverse Warping x y x' y' f(x,y) g(x’,y’) T-1 (x,y) Find out where each pixel g(x’,y’) should get its value from, and steal it. Note: requires ability to invert T Slide Credit: A. Efros
  • 57. Inverse Warping x'-1 x' x'+1 y'-1 y' y'+1 x' y’ g(x’,y’) x y f(x,y) x-1 x x+1 y-1 y y+1 If you don’t hit an exact pixel, figure out how to take it from the neighbors. T-1 (x,y)
  • 58. Mosaicing Warped Input 1 I1 Warped Input 2 I2 Image Credit: A. Efros Can warp an image. Pixels that don’t have a corresponding pixel in the image are set to a chosen value (often 0)
  • 59. Mosaicing Warped Input 1 I1 α Warped Input 2 I2 αI1 + (1-α)I2 Image Credit: A. Efros
  • 60. Mosaicing Warped Input 1 I1 α Warped Input 2 I2 αI1 + (1-α)I2 Slide Credit: A. Efros Can also warp an image containing 1s. Pixels that don’t have a corresponding pixel in the image are set to a chosen value (often 0)
  • 61. Putting it Together How do you make a panorama? Step 1: Find “features” to match Step 2: Describe Features Step 3: Match by Nearest Neighbor Step 4: Fit H via RANSAC Step 5: Blend Images
  • 62. Putting It Together 1 • (Multi-scale) Harris; or • Laplacian of Gaussian Find corners/blobs
  • 63. Putting It Together 2 Describe Regions Near Features Build histogram of gradient orientations (SIFT) (But in practice use opencv) 𝑥𝑞 ∈ 𝑅128
  • 64. Putting It Together 3 Match Features Based On Region 𝑥1 ∈ 𝑅128 𝑥2 ∈ 𝑅128 𝑥𝑞 ∈ 𝑅128 𝑥𝑞 Sort by distance to: ‖𝑥𝑞 − 𝑥1‖<‖𝑥𝑞 − 𝑥2 ‖<‖𝑥𝑞 − 𝑥3‖ Accept match if: ‖𝑥𝑞 − 𝑥1‖/‖𝑥𝑞 − 𝑥2‖ Nearest neighbor is far closer than 2nd nearest neighbor
  • 65. Putting It Together 4 Fit transformation H via RANSAC for trial in range(Ntrials): Pick sample Fit model Check if more inliers Re-fit model with most inliers arg min ‖𝒉‖=1 ‖𝑨𝒉‖ 2
  • 66. Putting It Together 5 Warp images together Resample images with inverse warping and blend (but in practice, just call opencv for inverse warping)
  • 69. A pencil of rays contains all views real camera synthetic camera Can generate any synthetic camera view as long as it has the same center of projection! Slide Credit: A. Efros
  • 71. Automatically rectified floor St. Lucy Altarpiece, D. Veneziano Analyzing Patterns What is the (complicated) shape of the floor pattern? Slide from A. Criminisi
  • 72. From Martin Kemp, The Science of Art (manual reconstruction) Automatic rectification Analyzing Patterns Slide from A. Criminisi
  • 73. Homography Derivation • This has gotten cut in favor of showing more of the setup. • The key to the set-up is to try to move towards a setup where you can pull [h1,h2,h3] out, or where each row is a linear equation in [h1,h2,h3]
  • 74. [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥𝑖 𝑦 𝑖 𝑤𝑖 ]≡ [ 𝑥𝑖 ′ 𝑦𝑖 ′ 𝑤𝑖 ′ ]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥𝑖 𝑦𝑖 𝑤𝑖 ] Want: Recall: 𝒂≡𝒃 𝒂=𝜆𝒃 𝒂×𝒃=𝟎 In turn 𝒑 𝒊= [ 𝑥𝑖 𝑦𝑖 1 ] 𝑯 𝒑𝒊≡ [ 𝒉𝟏 𝑻 𝒉𝟐 𝑻 𝒉𝟑 𝑻 ]𝒑𝒊 ≡ [ 𝒉𝟏 𝑻 𝒑𝒊 𝒉𝟐 𝑻 𝒑𝒊 𝒉𝟑 𝑻 𝒑𝒊 ] [ 𝑥𝑖 ′ 𝑦𝑖 ′ 𝑤𝑖 ′ ]× [ 𝒉𝟏 𝑻 𝒑𝒊 𝒉𝟐 𝑻 𝒑𝒊 𝒉𝟑 𝑻 𝒑𝒊 ]=𝟎 In the end want: Why Cross products? Cross products have explicit forms
  • 75. Fitting Transformation [ 𝑥𝑖 ′ 𝑦𝑖 ′ 𝑤𝑖 ′ ]× [ 𝒉𝟏 𝑻 𝒑𝒊 𝒉𝟐 𝑻 𝒑𝒊 𝒉𝟑 𝑻 𝒑𝒊 ]=𝟎 Want: [ 𝑦𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 −𝑤𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊 𝑤𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 − 𝑥𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 𝑥𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊 − 𝑦𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 ]=𝟎 Cross- product [ 𝒉𝟏 𝑻 𝟎−𝑤𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊+ 𝑦𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 𝑤𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 +𝒉𝟐 𝑻 𝟎− 𝑥𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 − 𝑦𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 +𝑥𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊 +𝒉𝟑 𝑻 𝟎]=𝟎 Re-arrange and put 0s in Note: calculate this explicitly. It looks ugly, but do it by doing [a,b,c] x [a’,b’,c’] then re-substituting. You want to be able to right- multiply by [h1,h2,h3]
  • 76. Fitting Transformation [ 𝒉𝟏 𝑻 𝟎−𝑤𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊+ 𝑦𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 𝑤𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 +𝒉𝟐 𝑻 𝟎− 𝑥𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 − 𝑦𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 +𝑥𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊 +𝒉𝟑 𝑻 𝟎]=𝟎 Equation Pull out h [ 𝟎 𝑻 −𝑤 ′ 𝑖 𝒑𝒊 𝑻 𝑦 ′𝑖 𝒑𝒊 𝑻 𝑤𝑖 ′ 𝒑𝒊 𝑻 𝟎 𝑻 − 𝑥𝑖 ′ 𝒑𝒊 𝑻 − 𝑦𝑖 ′ 𝒑𝒊 𝑻 𝑥𝑖 ′ 𝒑𝒊 𝑻 𝟎 𝑻 ][ 𝒉𝟏 𝒉𝟐 𝒉𝟑 ]=𝟎 Only two linearly independent equations Yank out h once you have all the coefficients. If you’re head-scratching about the two equations, it’s not obvious to me at first glance that the three equations aren’t linearly independent either.
  • 77. Simplification: Two-band Blending • Brown & Lowe, 2003 • Only use two bands: high freq. and low freq. • Blend low freq. smoothly • Blend high freq. with no smoothing: binary alpha Figure Credit: Brown & Lowe
  • 78. Low frequency (l > 2 pixels) High frequency (l < 2 pixels) 2-band “Laplacian Stack” Blending