![Dynamic Programming on Trees
n v2
E(L) = ∑ m (l ) + ∑ d (l ,l )
i i ij i j
i=1 (vi,vj) ∈ E v1
• For each l1 find best l2:
- Best (l ) = min [m (l ) + d
2 1
l2
2 2 12(l1,l2) ]
• “Delete” v2 and solve problem with smaller model
• Keep removing leafs until there is a single part left](https://image.slidesharecdn.com/deformable-120208064150-phpapp01/85/Object-Recognition-with-Deformable-Models-9-320.jpg)
![Min-Convolution Speedup
v2
Best2(l1) = min [m2(l2) + d12(l1,l2)] v1
l2
• Brute force: O(k2) --- k is number of locations
• Suppose d12(l1,l2) = g(l1-l2):
- Best (l ) = min [m (l ) + g(l -l )]
2 1
l2
2 2 1 2
• Min-convolution: O(k) if g is convex](https://image.slidesharecdn.com/deformable-120208064150-phpapp01/85/Object-Recognition-with-Deformable-Models-10-320.jpg)
![Matching
h
f c d i
e g c | a,b
a
b w p
e | a,c d | c,b
r
v f | a,e g | e,c h | c,d i | d,b
q
u
model curve
Match(v, [p,q]) = w1
Match(u, [q,r]) = w2
Match(w, [p,r]) = w1 + w2 + dif((e|a,c), (q|p,r))
similar to parsing with the CKY algorithm](https://image.slidesharecdn.com/deformable-120208064150-phpapp01/85/Object-Recognition-with-Deformable-Models-20-320.jpg)
Object Recognition with Deformable Models
- 1. Object Recognition with Deformable Models Pedro F. Felzenszwalb Department of Computer Science University of Chicago Joint work with: Dan Huttenlocher, Joshua Schwartz, David McAllester, Deva Ramanan.
- 2. Example Problems Detecting rigid objects PASCAL challenge Medical image Detecting non-rigid objects analysis Segmenting cells
- 3. Deformable Models • Significant challenge: - Handling variation in appearance within object classes - Non-rigid objects, generic categories, etc. • Deformable models approach: - Consider each object as a deformed version of a template - Compact representation - Leads to interesting modeling and algorithmic problems
- 4. Overview • Part I: Pictorial Structures - Deformable part models - Highly efficient matching algorithms • Part II: Deformable Shapes - Triangulated polygons - Hierarchical models • Part III: The PASCAL Challenge - Recognizing 20 object categories in realistic scenes - Discriminatively trained, multiscale, deformable part models
- 5. Part I: Pictorial Structures • Introduced by Fischler and Elschlager in 1973 • Part-based models: - Each part represents local visual properties - “Springs” capture spatial relationships Matching model to image involves joint optimization of part locations “stretch and fit”
- 6. Local Evidence + Global Decision • Parts have a match quality at each image location • Local evidence is noisy - Parts are detected in the context of the whole model part test image match quality
- 7. Matching Problem • Model is represented by a graph G = (V, E) - V = {v ,...,v } are the parts 1 n - (v ,v ) ∈ E indicates a connection between parts i j • mi(li) is a cost for placing part i at location li • dij(li,lj) is a deformation cost • Optimal configuration for the object is L = (l1,...,ln) minimizing n E(L) = ∑ m (l ) + ∑ d (l ,l ) i i ij i j i=1 (vi,vj) ∈ E
- 8. Matching Problem n E(L) = ∑ m (l ) + ∑ d (l ,l ) i i ij i j i=1 (vi,vj) ∈ E • Assume n parts, k possible locations for each part - There are k n configurations L • If graph is a tree we can use dynamic programming - O(nk ) algorithm 2 • If dij(li,lj) = g(li-lj) we can use min-convolutions - O(nk) algorithm - As fast as matching each part separately!
- 9. Dynamic Programming on Trees n v2 E(L) = ∑ m (l ) + ∑ d (l ,l ) i i ij i j i=1 (vi,vj) ∈ E v1 • For each l1 find best l2: - Best (l ) = min [m (l ) + d 2 1 l2 2 2 12(l1,l2) ] • “Delete” v2 and solve problem with smaller model • Keep removing leafs until there is a single part left
- 10. Min-Convolution Speedup v2 Best2(l1) = min [m2(l2) + d12(l1,l2)] v1 l2 • Brute force: O(k2) --- k is number of locations • Suppose d12(l1,l2) = g(l1-l2): - Best (l ) = min [m (l ) + g(l -l )] 2 1 l2 2 2 1 2 • Min-convolution: O(k) if g is convex
- 11. Finding Motorbikes Model with 6 parts: 2 wheels 2 headlights front & back of seat
- 13. Human Tracking Ramanan, Forsyth, Zisserman, Tracking People by Learning their Appearance IEEE Pattern Analysis and Machine Intelligence (PAMI). Jan 2007
- 14. Part II: Deformable Shapes • Shape is a fundamental cue for recognizing objects • Many objects have no well defined parts - We can capture their outlines using deformable models
- 15. Triangulated Polygons • Polygonal templates • Delauney triangulation gives natural decomposition of an object • Consider deforming each triangle “independently” Rabbit ear can be bent by changing shape of a single triangle
- 16. Structure of Triangulated Polygons There are 2 graphs associated with a triangulated polygon If the polygon is simple (no holes): Dual graph is a tree Graphical structure of triangulation is a 2-tree
- 17. Deformable Matching Consider piecewise affine maps from model to image (taking triangles to triangles) Find globally optimal deformation using Model dynamic programming over 2-tree Matching to MRI data
- 18. Hierarchical Shape Model • Shape-tree of curve from a to b: - Select midpoint c, store relative location c | a,b. - Left child is a shape-tree of sub-curve from a to c. - Right child is a shape-tree of sub-curve from c to b. h f c d i e g c | a,b b a e | a,c d | c,b f | a,e g | e,c h | c,d i | d,b
- 19. Deformations • Independently perturb relative locations stored in a shape-tree - Local and global properties are preserved - Reconstructed curve is perceptually similar to original
- 20. Matching h f c d i e g c | a,b a b w p e | a,c d | c,b r v f | a,e g | e,c h | c,d i | d,b q u model curve Match(v, [p,q]) = w1 Match(u, [q,r]) = w2 Match(w, [p,r]) = w1 + w2 + dif((e|a,c), (q|p,r)) similar to parsing with the CKY algorithm
- 21. Recognizing Leafs Nearest neighbor classification 15 species Shape-tree 96.28 75 examples per species Inner distance 94.13 (25 training, 50 test) Shape context 88.12
- 22. Part III: PASCAL Challenge • ~10,000 images, with ~25,000 target objects - Objects from 20 categories (person, car, bicycle, cow, table...) - Objects are annotated with labeled bounding boxes
- 24. Model Overview detection root filter part filters deformation models Model has a root filter plus deformable parts
- 25. Histogram of Gradient (HOG) Features • Image is partitioned into 8x8 pixel blocks • In each block we compute a histogram of gradient orientations - Invariant to changes in lighting, small deformations, etc. • We compute features at different resolutions (pyramid)
- 26. Filters • Filters are rectangular templates defining weights for features • Score is dot product of filter and subwindow of HOG pyramid H W Score of H at this location is H ⋅ W HOG pyramid
- 27. Object Hypothesis Score is sum of filter scores plus deformation scores Image pyramid HOG feature pyramid Multiscale model captures features at two-resolutions
- 28. Training • Training data consists of images with labeled bounding boxes • Need to learn the model structure, filters and deformation costs Training
- 29. Connection With Linear Classifiers • Score of model is sum of filter scores plus deformation scores - Bounding box in training data specifies that score should be high for some placement in a range w is a model x is a detection window z are filter placements concatenation of filters and concatenation of features deformation parameters and part displacements
- 30. Latent SVMs Linear in w if z is fixed Regularization Hinge loss
- 31. Learned Models Bicycle Sofa Car Bottle
- 32. Example Results
- 33. More Results
- 34. Overall Results • 9 systems competed in the 2007 challenge • Out of 20 classes we get: - First place in 10 classes - Second place in 6 classes • Some statistics: - It takes ~2 seconds to evaluate a model in one image - It takes ~3 hours to train a model - MUCH faster than most systems
- 35. Component Analysis PASCAL2006 Person 1 0.9 Root (0.18) Root+Latent (0.24) 0.8 Parts+Latent (0.29) 0.7 Root+Parts+Latent (0.34) 0.6 precision 0.5 0.4 0.3 0.2 0.1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 recall
- 36. Summary • Deformable models provide an elegant framework for object detection and recognition - Efficient algorithms for matching models to images - Applications: pose estimation, medical image analysis, object recognition, etc. • We can learn models from partially labeled data - Generalized standard ideas from machine learning - Leads to state-of-the-art results in PASCAL challenge • Future work: hierarchical models, grammars, 3D objects