SlideShare a Scribd company logo

filipe-costa-thesis-presentation-static-ilovepdf-compressed

F
Filipe Costa

This document discusses reconstructing phylogenies from images and videos. It outlines contributions in image phylogeny, including approaches to reconstructing phylogeny forests, new dissimilarity measures, and video phylogeny for misaligned and compressed videos. For image phylogeny, the author proposes techniques like Automatic Optimum Branching and Extended Automatic Optimum Branching to exactly reconstruct phylogeny forests from dissimilarity matrices. A fusion approach is also introduced to combine results from different reconstruction algorithms to improve forest quality. Experiments are conducted on datasets containing thousands of test cases to evaluate the techniques.

1 of 97
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
Image and Video Phylogeny Reconstruction Filipe de Oliveira Costa Advisor: Prof. Anderson Rocha Co-advisor: Prof. Zanoni Dias
Outline • Introduction • Image Phylogeny • Video Phylogeny • Conclusions 2
Introduction 3
Multimedia document Modified versions Source: Google Images Dark side of the moon internet 4
Near-duplicate detection 5 Near-duplicate detection
Multimedia Phylogeny 6
Multimedia Phylogeny 7
8 Multimedia Phylogeny Phylogeny tree Phylogeny forest
Main Objective Design and develop solutions that allow us to identify the structure (phylogeny tree / forest) that represents the generation process overtime of images and videos 9
Contributions • Image phylogeny • Approaches to image phylogeny forest reconstruction • New dissimilarity measures for image phylogeny • Video Phylogeny • Phylogeny reconstruction for misaligned and compressed video sequences 10
Image Phylogeny
Image Phylogeny • Divided into two basic steps • Dissimilarity calculation • Phylogeny Reconstruction 12
Dissimilarity Calculation Itgt d(Isrc, Itgt) 13 d(Isrc, Itgt) = min T!2T |Itgt T!(Isrc)| point-wise comparison L. Isrc
Dissimilarity Calculation Geometric matching 14 Itgt Isrc
Dissimilarity Calculation Color and compression matching Isrc Itgt R G B Compression 15
Dissimilarity Calculation Point-wise comparison (Mean Squared Error) I0 src L 16 Itgt
Phylogeny Reconstruction A B C D E A - 2 1 3 4 B 4 - 1 2 6 C 3 4 - 1 2 D 4 7 4 - 4 E 5 7 8 3 - Dissimilarity matrix 17 A C E D B Phylogeny tree 2 2 1 1
Phylogeny Reconstruction 18 • Oriented Kruskal (OK) [Dias et al. (2010)] • Based on the classic Kruskal's Minimum Spanning Tree algorithm • Adapted for oriented graphs • Optimum Branching (OB) [Edmonds (1957)] • Always returns the optimum tree
• Image Phylogeny Forests Reconstruction F. O. Costa, M. Oikawa, Z. Dias, S. Goldenstein, and A. Rocha. Image phylogeny forests reconstruction. IEEE Transactions on Information Forensics and Security (TIFS), vol.9, no. 10, pp. 1533–1546, 2014 • New dissimilarity measures F. O. Costa, A. Oliveira, P. Ferrara, Z. Dias, S. Goldenstein, and A. Rocha. New dissimilarity measures for image phylogeny reconstruction. Submitted to Springer Pattern Analysis and Applications, 2016 Image Phylogeny Contributions 19
Image Phylogeny Forests Reconstruction 20
21 Image Phylogeny Forests Reconstruction Phylogeny tree Phylogeny forest
Image Phylogeny Forests Reconstruction Dias et al.(2013b): Automatic Oriented Kruskal (AOK) • Adaptation of the OK algorithm for forests • Select the edges in crescent order • Stops selecting edges according to a threshold 22
Hypothesis #1 Using exact approaches for phylogeny forests reconstruction problem is at least as effective than using heuristic-based approaches 23
Image Phylogeny Forests Reconstruction We designed and developed an algorithm for forests reconstruction • Automatic Optimum Branching (AOB) • Adaptation of the OB algorithm 24
Automatic Optimum Branching (AOB) Optimum Branching 25 Dissimilarity matrix M 0 1 2 3 4 5 6 7 8 9 0 - 52 29 57 43 55 24 52 49 26 1 50 - 45 35 60 68 40 42 63 53 2 42 48 - 56 42 54 28 52 50 29 3 56 23 55 - 65 67 50 22 66 58 4 45 61 35 67 - 29 42 63 43 52 5 25 69 65 70 27 - 48 68 45 65 6 26 50 30 52 39 50 - 50 48 48 7 51 32 49 30 61 67 49 - 62 55 8 52 65 55 69 24 30 49 64 - 26 9 25 54 26 59 50 61 30 56 30 - 17 3 6 8 4 5 2 9 0 40 2322 26 26 26 30 24 29
Step X Y d(X,Y) Diff 1 3 7 22 - - 2 3 1 23 1 - 3 8 4 24 1 1,41 4 9 2 26 2 2,00 5 6 0 26 0 3,42 6 0 9 26 0 3,58 7 4 5 29 3 3,52 8 9 8 30 1 4,68 9 1 6 40 10 5,53 Automatic Optimum Branching (AOB) aob = 2.026 17 3 6 8 4 5 2 9 0 40 2322 26 26 26 30 24 29 aob ⇥
Automatic Optimum Branching (AOB) aob = 2.027 17 3 6 8 4 5 2 9 0 2322 26 26 26 30 24 29 Forest: [6, 3, 9, 3, 8, 4, 6, 3, 9, 0] Cost: 206 Dissimilarity matrix M 0 1 2 3 4 5 6 7 8 9 0 - 52 29 57 43 55 24 52 49 26 1 50 - 45 35 60 68 40 42 63 53 2 42 48 - 56 42 54 28 52 50 29 3 56 23 55 - 65 67 50 22 66 58 4 45 61 35 67 - 29 42 63 43 52 5 25 69 65 70 27 - 48 68 45 65 6 26 50 30 52 39 50 - 50 48 48 7 51 32 49 30 61 67 49 - 62 55 8 52 65 55 69 24 30 49 64 - 26 9 25 54 26 59 50 61 30 56 30 -
M 1 3 7 0 2 4 5 6 8 9 1 - 35 42 - - - - - - - 3 23 - 22 - - - - - - - 7 32 30 - - - - - - - - 0 - - - - 29 43 55 24 49 26 2 - - - 42 - 42 54 28 50 29 4 - - - 45 35 - 29 42 43 52 5 - - - 25 65 27 - 48 45 65 6 - - - 26 30 39 50 - 48 48 8 - - - 52 55 24 30 49 - 26 9 - - - 25 26 50 61 30 30 - Extended Automatic Optimum Branching (E-AOB) 28 17 3 6 8 4 5 2 9 0 Dissimilarity matrix
Extended Automatic Optimum Branching (E-AOB) Forest: [5, 3, 9, 3, 8, 4, 0, 3, 9, 0] Cost: 204 29 17 3 6 8 4 5 2 9 0 2322 26 26 26 30 24 29 M 1 3 7 0 2 4 5 6 8 9 1 - 35 42 - - - - - - - 3 23 - 22 - - - - - - - 7 32 30 - - - - - - - - 0 - - - - 29 43 55 24 49 26 2 - - - 42 - 42 54 28 50 29 4 - - - 45 35 - 29 42 43 52 5 - - - 25 65 27 - 48 45 65 6 - - - 26 30 39 50 - 48 48 8 - - - 52 55 24 30 49 - 26 9 - - - 25 26 50 61 30 30 - Dissimilarity matrix
Image Phylogeny Forests Reconstruction • Main approaches • AOK: Greedy approach • AOB/E-AOB: Exact approach 30
Hypothesis #2 It is possible to lead to complementary properties of different forest reconstruction algorithm and combine them for improving the quality of the phylogeny forest reconstruction 31
Fusion approach 32 M Additive noise 100 variations
Fusion approach Tree count V = [1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5] Median(V) = (2 + 2)/2 = 2 100 variations … AOK 0 35 1 4 7 8 2 9 6 100 variations … AOB 0 3 5 1 4 7 8 2 9 6 100 variations … E-AOB 0 35 1 4 7 8 2 9 6 33
Fusion approach 100 variations … AOK 0 35 1 4 7 8 2 9 6 100 variations … AOB 0 3 5 1 4 7 8 2 9 6 100 variations … E-AOB 0 35 1 4 7 8 2 9 6 Sum of votes of each node [260, 250, 90, 30, 35, 24, 0, 0, 0, 0] Roots {0, 1} 34
0 351 4 7 8 2 9 6 Fusion approach 100 variations … AOK 0 35 1 4 7 8 2 9 6 100 variations … AOB 0 3 5 1 4 7 8 2 9 6 M 0 1 2 3 4 5 6 7 8 9 0 - - 0 270 0 286 0 0 0 0 1 - - 0 0 255 0 0 280 0 0 2 - - - 0 0 0 130 0 270 300 3 - - 280 - 0 0 0 0 30 0 4 - - 0 0 - 0 0 20 0 0 5 - - 20 0 0 - 0 0 0 0 6 - - 0 0 0 0 - 0 0 0 7 - - 0 0 10 0 0 - 0 0 8 - - 0 0 0 0 0 0 - 0 9 - - 0 0 0 0 170 0 0 - 100 variations … E-AOB 0 35 1 4 7 8 2 9 6 Final forest [0, 1, 3, 0, 1, 0, 2, 1, 2, 2] 35
Experiments 36
Datasets 37 • 36,000 test cases • [2..10] trees • Ground truth is available • One Camera and Multiple Cameras
Datasets 38 • Transformations • Crop • Resize • Brightness, contrast
 and gamma correction • Rotation • JPEG Compression
10 6 7 9 8 25 3 11 7 Evaluation 39 2 8 9 103 65 4 39 4 Ground truth Reconstructed forest Roots
3 10 6 71 2 8 95 1 7 Evaluation 40 2 8 9 103 65 4 40 4 Ground truth Reconstructed forest Edges
10 6 71 2 9 3 1 7 Evaluation 41 2 8 9 103 65 4 41 8 5 4 Ground truth Reconstructed forest Leaves
7 10 6 1 83 25 1 7 Evaluation Ancestry 42 2 8 9 103 65 4 42 9 4 Ground truth Reconstructed forest
Results - One Camera 43 2 3 4 5 6 7 8 9 10 # of trees 0.65 0.7 0.75 0.8 0.85 0.9 0.95 Score Root - One Camera AOK AOB E-AOB FUSION 2 3 4 5 6 7 8 9 10 # of trees 0.65 0.7 0.75 0.8 0.85 0.9 0.95 Score Ancestry - One Camera AOK AOB E-AOB FUSION Error Reduction (E-AOB vs. AOK): 19% Error Reduction (E-AOB vs. AOK): 18% Error Reduction (FUSION vs. AOK): 17% Error Reduction (Fusion vs. AOK): 17%
Results - Multiple Cameras 44 2 3 4 5 6 7 8 9 10 # of trees 0.65 0.7 0.75 0.8 0.85 0.9 0.95 Score Root - Multiple Cameras AOK AOB E-AOB FUSION 2 3 4 5 6 7 8 9 10 # of trees 0.65 0.7 0.75 0.8 0.85 0.9 0.95 Score Ancestry - Multiple Cameras AOK AOB E-AOB FUSION Error Reduction (E-AOB vs. AOK): 20% Error Reduction (E-AOB vs. AOK): 19% Error Reduction (FUSION vs. AOK): 28% Error Reduction (Fusion vs. AOK): 21%
New dissimilarity measures for image phylogeny reconstruction 45
New dissimilarity measures for image phylogeny reconstruction • Previous work: Mean Squared Error (MSE) • Problems • Few information about the content of the images • Small misaligned caused on mapping can affect the results 46
Hypothesis #3 The comparison of the distributions of the pixel values of two near-duplicate images is more effective for the dissimilarity calculation than their point-wise comparison 47
Gradient point-wise comparison MSE Gradient Computation I0 src Itgt 48 Gradient Computation
Mutual Information comparison 49 MI(I0 src, Itgt) I0 src Itgt MI(I0 src, Itgt) = H(Itgt) H(Itgt|I0 src) H(X) = Ex[log(P(X))]
Comparing gradients with mutual information Mutual
 Information I0 src Itgt MSE 50 X Gradient Computation Gradient Computation
Color matching • Previous work • Normalization based on mean and standard deviation of another image • Problem • Unexpected Artifacts • New color matching approach • Histogram matching [Gonzales (2007)] 51
Histogram color matching 52 0 20 40 60 80 100 120 140 160 180 200 220 240 255 i 0 0.025 0.05 0.075 0.1 0.125 0.15 p(i) Histogram - SRC 0 20 40 60 80 100 120 140 160 180 200 220 240 255 i 0 0.025 0.05 0.075 0.1 0.125 0.15 p(i) Histogram - TGT Histogram
0 20 40 60 80 100 120 140 160 180 200 220 240 255 i 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CDF(i) CDF - SRC 0 20 40 60 80 100 120 140 160 180 200 220 240 255 i 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CDF(i) CDF - TGT 99 114 Histogram color matching 53 200 238 20 22 Cumulative Distribution Function
Histogram color matching 54 0 20 40 60 80 100 120 140 160 180 200 220 240 255 i 0 0.025 0.05 0.075 0.1 0.125 0.15 p(i) Histogram - TGT 0 20 40 60 80 100 120 140 160 180 200 220 240 255 i 0 0.025 0.05 0.075 0.1 0.125 0.15 p(i) Result of historgram matching Resultant histogram
Color matching approaches Result: Mean/STD Result: Histogram Isrc Itgt Color Matching Result: Mean/STD maching Result: Histogram maching 55
Experiments 56
Experimental setup • Reconstruction • Extended AOB • Evaluation • Roots, edges, leaves and ancestry 57
Roots - One Camera Accuracy 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Number of trees 1 2 3 4 5 6 7 8 9 10 MSE GRAD MINF GRMI HGMI Results 58 Roots - Multiple Cameras Accuracy 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Number of trees 1 2 3 4 5 6 7 8 9 10 MSE GRAD MINF GRMI HGMI Error Reduction (HGMI vs. MSE): 53% Error Reduction (HGMI vs. MSE): 47%
Results 59 Ancestry - One Camera Accuracy 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Number of trees 1 2 3 4 5 6 7 8 9 10 MSE GRAD MINF GRMI HGMI Error Reduction (HGMI vs. MSE): 60% Error Reduction (HGMI vs. MSE): 58% Ancestry - Multiple Cameras Accuracy 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Number of trees 1 2 3 4 5 6 7 8 9 10 MSE GRAD MINF GRMI HGMI
Qualitative results Ellen DeGeneres’ Selfie 60 Group bGroup a Group c Group d Group e
a09 a00 a11 a01 a02 a04 a05 a06 a07 a08 a03a10 a12 b03 b01 b06 b04 b02 c07 b05 c01 c08 c02 c05 c09 c10 c06 c03 c04 d08 d01 d07 d03 d02 d06 d05 d04 e01 e02 e03 e04 e05 e06 e07 61 Group b Group a Group c Group d Group e Qualitative results Ellen DeGeneres’ Selfie
a09 a00 a11 a01 a02 a04 a05 a06 a07 a08 a03a10 a12 b03 b01 b06 b04 b02 c07 b05 c01 c08 c02 c05 c09 c10 c06 c03 c04 d08 d01 d07 d03 d02 d06 d05 d04 e01 e02 e03 e04 e05 e06 e07 62 Group b Group a Group c Group d Group e Qualitative results Ellen DeGeneres’ Selfie
a09 a00 a11 a01 a02 a04 a05 a06 a07 a08 a03a10 a12 b03 b01 b06 b04 b02 c07 b05 c01 c08 c02 c05 c09 c10 c06 c03 c04 d08 d01 d07 d03 d02 d06 d05 d04 e01 e02 e03 e04 e05 e06 e07 63 Group b Group a Group c Group d Group e Qualitative results Ellen DeGeneres’ Selfie
a09 a00 a11 a01 a02 a04 a05 a06 a07 a08 a03a10 a12 b03 b01 b06 b04 b02 c07 b05 c01 c08 c02 c05 c09 c10 c06 c03 c04 d08 d01 d07 d03 d02 d06 d05 d04 e01 e02 e03 e04 e05 e06 e07 64 Group b Group a Group c Group d Group e Qualitative results Ellen DeGeneres’ Selfie
Video Phylogeny
Video Phylogeny • More challenging problem than Image Phylogeny • Now we have a new dimension: time • Videos can be misaligned • Videos can have different compression parameters 66
Hypothesis #4 Temporal alignment is paramount for a proper video phylogeny reconstruction process 67
• Phylogeny reconstruction for misaligned and compressed video sequences F. O. Costa, S. Lameri, P. Bestagini, Z. Dias, A. Rocha, M. Tagliassachi, S. Tubaro. Phylogeny reconstruction for misaligned and compressed video sequences. IEEE International Conference on Image Processing (ICIP), 2015 • Hashing-based Temporal Alignment for Video Phylogeny Reconstruction F. O. Costa, P. Bestagini, S. Tubaro, Z. Dias, A. Rocha. Hashing-based Temporal Alignment for Video Phylogeny Reconstruction. Submitted to IEEE WIFS, 2016 Video Phylogeny Contributions 68
Phylogeny reconstruction for misaligned and compressed video sequences 69
Video dissimilarity calculation • Temporal alignment • Frame registration • Color matching • Coding matching (controlled) • Comparison (MSE) 70
Temporal alignment 71 V 1 2 3 4 5 6 7 8 9 10 … n-2 n-1 n y1 y2 y3 y4 y5 y6 y7 y8 y9 y10 yn-2 yn-1 yn yi = Average of luminance of V(i) LDV = [y2 - y1, y3 - y2, … , yn-1 - yn-2, yn - yn-1 ] Difference of luminance
50 100 150 200 250 0 i L 50 100 150 200 250 0 5 10 i LDV(i) V = Vsrc V = Vtgt −200 −100 0 100 200 0 0.2 0.4 0.6 i phase-corr. Video Alignment 72 Phase-correlation: ˆi = arg maxi F 1 h F[LDVsrc ]·F[LDVtgt ]⇤ |F[LDVsrc ]·F[LDVtgt ]⇤| i (i)
Comparison • We perform two kinds of comparison • Average of MSE for the correspondent frames of two videos • Parenthood matrix 73
Parenthood matrix 74 Frequency of the edges Parenthood Matrix Optimum branching (MAX) Final phylogeny tree …M1 M2 M3 M4 M5 M6 Mn-1 Mn V3 … V1 … V2 … V4 … V5 …
Experiments 75
Dataset 76 Available at https://media.xiph.org/video/derf/ •200 trees with 10 videos •100 - No clip •100 - Clip at the Beginning/end of the stream •Probability of clip: 50% •CODEC: MPEG2, MPEG4, H264 •Crop, resize, bright, contrast, coding and temporal clipping
Experimental Setup • Phylogeny Reconstruction • OB algorithm • Evaluation metrics • Root, edges, leaves, ancestry and depth 77 1 2 3 5 4 4 23 5 1 Reconstructed treeGround truth Depth = 1
Results #1 78 Verifying alignment and coding matching Root Depth Edges Leaves Ancestry 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Score Results - Dataset without temporal clipping Without temporal alignment With temporal alignment With temporal alignment and compression matching Root Depth Edges Leaves Ancestry 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Score Results - Dataset with temporal clipping Without alignment With alignment With alignment and compression matching
Root Depth Edges Leaves Ancestry 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Score Results - Dataset with temporal clipping A - NO COD P - NO COD A - COD P - COD Root Depth Edges Leaves Ancestry Results - Dataset without temporal clipping Results #2 79 Comparison: Average of MSE (A) x Parenthood matrix (P) Root Depth Edges Leaves A 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Score Results - Dataset with temporal clipping A - NO COD P - NO COD A - COD P - COD Root Depth Edges Leaves Ancestry 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Score Results - Dataset without temporal clipping A - NO COMPRESSION MATCHING P - NO COMPRESSION MATCHING A - WITH COMPRESSION MATCHING P - WITH COMPRESSION MATCHING
Hashing-based temporal alignment for video phylogeny 80
• Near-duplicate matching • Near-duplicate extraction • Near-duplicate alignment Hashing-based temporal alignment 81
Step 1 Near-duplicate matching • Groups of 64 frames • All frames are resized to 32 x 32 pixel 82 1 frame Temporal axis …… 1 2 3 4 5 6 7 8 9 10 59 60 61 62 63 64 65 66 67 68
Step 1 Near-duplicate matching 83 64 coefficients 32 coefficients 32coefficients DCT coefficients 4 x 4 x 4 DCT coefficients … 3D Discrete Cosine Transform
Step 1 Near-duplicate matching • Binary hash • According to the median of DCT • Distance matrix • Hamming distance of two binary hashes of different videos 84
Step 2 Near-duplicate extraction 85 0 30 60 90 120 150 180 210 236 0 20 40 60 80 100 120 140 160 180 200 0 30 60 90 0 20 40 60 80 100 120 140 160 180 200 20 150 180 210 236 0 30 60 90 120 150 180 210 236 0 20 40 60 80 100 120 140 160 180 200 0 30 60 90 0 20 40 60 80 100 120 140 160 180 200 20 150 180 210 236 0 30 60 90 120 150 180 210 236 0 20 40 60 80 100 120 140 160 180 200 0 30 60 90 0 20 40 60 80 100 120 140 160 180 200 20 150 180 210 236 0 30 60 90 120 150 180 210 236 0 20 40 60 80 100 120 140 160 180 200 BinarizationOpeningExtraction
V1 V2 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Sequences without temporal alignment 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Alignment by phase correlation 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Hashing-based temporal alignment 86 Step 3 Near-duplicate alignment
V1 V2 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Sequences without temporal alignment 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Alignment by phase correlation 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Hashing-based temporal alignment V1 V2 0 20 40 60 80 100 120 140 -3 -2 -1 0 1 2 3 Diff.ofluminance Sequences witho 3 Alignment by 87 Step 3 Near-duplicate alignment
V1 V2 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Sequences without temporal alignment 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Alignment by phase correlation 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Hashing-based temporal alignment 88 Step 3 Near-duplicate alignment0 20 40 60 80 100 120 140 -3 -2 -1 Diff. 0 20 40 60 80 100 120 140 -3 -2 -1 0 1 2 3 Diff.ofluminance Alignment by
V1 V2 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Sequences without temporal alignment 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Alignment by phase correlation 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Hashing-based temporal alignment 89 Step 3 Near-duplicate alignment 0 20 40 60 80 100 120 140 -3 -2 -1 0 Diff.oflu -3 -2 -1 0 1 2 3 Diff.ofluminance Hashing-based
Experiments 90
Experiments •Dataset •300 trees with 10 videos •100 - No clip •100 - Clip at the Beginning/end of the stream •100 - Clip anywhere 91
Results 92 Diff. of luminance vs. Hashing-based alignment Difference of luminance Hashing-based Root Depth Edges Leaves Ancestry Root Depth Edges Leaves Ancestry No coding matching No clip 0,630 0,500 0,772 0,832 0,674 0,650 0,440 0,777 0,828 0,702 Clip border 0,710 0,35 0,788 0,826 0,724 0,710 0,330 0,807 0,846 0,740 Clip any 0,620 0,560 0,724 0,758 0,628 0,670 0,450 0,755 0,789 0,683 With coding matching No clip 0,870 0,140 0,839 0,859 0,812 0,690 0,400 0,778 0,831 0,711 Clip border 0,840 0,170 0,863 0,901 0,834 0,660 0,400 0,780 0,824 0,722 Clip any 0,720 0,390 0,801 0,828 0,696 0,660 0,470 0,765 0,810 0,696
Conclusion 93
Conclusion • New solutions for multimedia phylogeny • Robust methods for phylogeny forest reconstruction • More robust dissimilarity measures • Gradient + mutual information • Good results for video phylogeny when dealing with misaligned and compressed videos 94
Future work • Other transformations • Blurring, sharpness, content insertion… • Videos with different frame rates • Video dissimilarity calculation using time information • Robust methods to reliably estimate the compression parameters directly from the video 95
Acknowledgements • Examiners • Prof. Anderson Rocha and Prof. Zanoni Dias • RECOD - IC/Unicamp • ISPG - Politecnico di Milano • FAPESP • CAPES 96
Thanks! 97

More Related Content

PDF
Chapter 1 ex-1.3
emadMaths
 
PPTX
Data visualization with Python and SVG
Sukjun Kim
 
DOCX
Kelompok 17 metal mas junda
GhalibAbyan1
 
PDF
Capítulo 02 considerações estatísticas
Jhayson Carvalho
 
PDF
Digital signal processing (2nd ed) (mitra) solution manual
Ramesh Sundar
 
PDF
10th Maths
singarls19
 
PDF
CAESAR II:The Combination of Direct Geometry Method and CAESAR Algorithm for ...
BIOVIA
 
PPT
Algae
Rituparna Kalita
 
Chapter 1 ex-1.3
emadMaths
 
Data visualization with Python and SVG
Sukjun Kim
 
Kelompok 17 metal mas junda
GhalibAbyan1
 
Capítulo 02 considerações estatísticas
Jhayson Carvalho
 
Digital signal processing (2nd ed) (mitra) solution manual
Ramesh Sundar
 
10th Maths
singarls19
 
CAESAR II:The Combination of Direct Geometry Method and CAESAR Algorithm for ...
BIOVIA
 
Algae
Rituparna Kalita
 

Similar to filipe-costa-thesis-presentation-static-ilovepdf-compressed (20)

PDF
Microsoft power point fowt2018-boscolofrancesco_v2.pptx
TECNALIA Marine Energy
 
PPTX
Nist project final report
alikhosravani
 
PPTX
Nist project final report
alikhosravani
 
PPTX
Nist project final report
alikhosravani
 
PPTX
Nist project final report
alikhosravani
 
PDF
Analysis update for GENEVA meeting 2011
USC
 
PDF
Introduction to machine learning algorithms
bigdata trunk
 
DOCX
Đề Thi HK2 Toán 6 - THCS Ngô Thời Nhiệm
Trung Tâm Gia Sư Việt Trí
 
PDF
Lecture-3-Filtering-Part-I.pdf
TechEvents1
 
PPT
jpg image processing nagham salim_as.ppt
naghamallella
 
PDF
Ae abc general_grenoble_29_juin_25_30_min
Michael Blum
 
PDF
Comparing Machine Learning Algorithms in Text Mining
Andrea Gigli
 
PDF
HOW 2019: Machine Learning for the Primary Vertex Reconstruction
Henry Schreiner
 
PDF
06 ch ken black solution
Krunal Shah
 
PDF
Tugas mtk 1.01
arehansanada
 
PDF
Elementary Linear Algebra 8th Edition Larson Solutions Manual.pdf
solutionsmanual3
 
PPTX
Adaptive Aperture Commissioning Presentation at 57th PTCOG
Minglei Kang
 
PDF
Model Compression
DarshanG13
 
PDF
Implicit two step adam moulton hybrid block method with two off step points f...
Alexander Decker
 
PPTX
Vibration study of a OCDC bracket
Russell Varvel
 
Microsoft power point fowt2018-boscolofrancesco_v2.pptx
TECNALIA Marine Energy
 
Nist project final report
alikhosravani
 
Nist project final report
alikhosravani
 
Nist project final report
alikhosravani
 
Nist project final report
alikhosravani
 
Analysis update for GENEVA meeting 2011
USC
 
Introduction to machine learning algorithms
bigdata trunk
 
Đề Thi HK2 Toán 6 - THCS Ngô Thời Nhiệm
Trung Tâm Gia Sư Việt Trí
 
Lecture-3-Filtering-Part-I.pdf
TechEvents1
 
jpg image processing nagham salim_as.ppt
naghamallella
 
Ae abc general_grenoble_29_juin_25_30_min
Michael Blum
 
Comparing Machine Learning Algorithms in Text Mining
Andrea Gigli
 
HOW 2019: Machine Learning for the Primary Vertex Reconstruction
Henry Schreiner
 
06 ch ken black solution
Krunal Shah
 
Tugas mtk 1.01
arehansanada
 
Elementary Linear Algebra 8th Edition Larson Solutions Manual.pdf
solutionsmanual3
 
Adaptive Aperture Commissioning Presentation at 57th PTCOG
Minglei Kang
 
Model Compression
DarshanG13
 
Implicit two step adam moulton hybrid block method with two off step points f...
Alexander Decker
 
Vibration study of a OCDC bracket
Russell Varvel
 
Ad

filipe-costa-thesis-presentation-static-ilovepdf-compressed

  • 1. Image and Video Phylogeny Reconstruction Filipe de Oliveira Costa Advisor: Prof. Anderson Rocha Co-advisor: Prof. Zanoni Dias
  • 2. Outline • Introduction • Image Phylogeny • Video Phylogeny • Conclusions 2
  • 4. Multimedia document Modified versions Source: Google Images Dark side of the moon internet 4
  • 9. Main Objective Design and develop solutions that allow us to identify the structure (phylogeny tree / forest) that represents the generation process overtime of images and videos 9
  • 10. Contributions • Image phylogeny • Approaches to image phylogeny forest reconstruction • New dissimilarity measures for image phylogeny • Video Phylogeny • Phylogeny reconstruction for misaligned and compressed video sequences 10
  • 12. Image Phylogeny • Divided into two basic steps • Dissimilarity calculation • Phylogeny Reconstruction 12
  • 13. Dissimilarity Calculation Itgt d(Isrc, Itgt) 13 d(Isrc, Itgt) = min T!2T |Itgt T!(Isrc)| point-wise comparison L. Isrc
  • 15. Dissimilarity Calculation Color and compression matching Isrc Itgt R G B Compression 15
  • 16. Dissimilarity Calculation Point-wise comparison (Mean Squared Error) I0 src L 16 Itgt
  • 17. Phylogeny Reconstruction A B C D E A - 2 1 3 4 B 4 - 1 2 6 C 3 4 - 1 2 D 4 7 4 - 4 E 5 7 8 3 - Dissimilarity matrix 17 A C E D B Phylogeny tree 2 2 1 1
  • 18. Phylogeny Reconstruction 18 • Oriented Kruskal (OK) [Dias et al. (2010)] • Based on the classic Kruskal's Minimum Spanning Tree algorithm • Adapted for oriented graphs • Optimum Branching (OB) [Edmonds (1957)] • Always returns the optimum tree
  • 19. • Image Phylogeny Forests Reconstruction F. O. Costa, M. Oikawa, Z. Dias, S. Goldenstein, and A. Rocha. Image phylogeny forests reconstruction. IEEE Transactions on Information Forensics and Security (TIFS), vol.9, no. 10, pp. 1533–1546, 2014 • New dissimilarity measures F. O. Costa, A. Oliveira, P. Ferrara, Z. Dias, S. Goldenstein, and A. Rocha. New dissimilarity measures for image phylogeny reconstruction. Submitted to Springer Pattern Analysis and Applications, 2016 Image Phylogeny Contributions 19
  • 22. Image Phylogeny Forests Reconstruction Dias et al.(2013b): Automatic Oriented Kruskal (AOK) • Adaptation of the OK algorithm for forests • Select the edges in crescent order • Stops selecting edges according to a threshold 22
  • 23. Hypothesis #1 Using exact approaches for phylogeny forests reconstruction problem is at least as effective than using heuristic-based approaches 23
  • 24. Image Phylogeny Forests Reconstruction We designed and developed an algorithm for forests reconstruction • Automatic Optimum Branching (AOB) • Adaptation of the OB algorithm 24
  • 25. Automatic Optimum Branching (AOB) Optimum Branching 25 Dissimilarity matrix M 0 1 2 3 4 5 6 7 8 9 0 - 52 29 57 43 55 24 52 49 26 1 50 - 45 35 60 68 40 42 63 53 2 42 48 - 56 42 54 28 52 50 29 3 56 23 55 - 65 67 50 22 66 58 4 45 61 35 67 - 29 42 63 43 52 5 25 69 65 70 27 - 48 68 45 65 6 26 50 30 52 39 50 - 50 48 48 7 51 32 49 30 61 67 49 - 62 55 8 52 65 55 69 24 30 49 64 - 26 9 25 54 26 59 50 61 30 56 30 - 17 3 6 8 4 5 2 9 0 40 2322 26 26 26 30 24 29
  • 26. Step X Y d(X,Y) Diff 1 3 7 22 - - 2 3 1 23 1 - 3 8 4 24 1 1,41 4 9 2 26 2 2,00 5 6 0 26 0 3,42 6 0 9 26 0 3,58 7 4 5 29 3 3,52 8 9 8 30 1 4,68 9 1 6 40 10 5,53 Automatic Optimum Branching (AOB) aob = 2.026 17 3 6 8 4 5 2 9 0 40 2322 26 26 26 30 24 29 aob ⇥
  • 27. Automatic Optimum Branching (AOB) aob = 2.027 17 3 6 8 4 5 2 9 0 2322 26 26 26 30 24 29 Forest: [6, 3, 9, 3, 8, 4, 6, 3, 9, 0] Cost: 206 Dissimilarity matrix M 0 1 2 3 4 5 6 7 8 9 0 - 52 29 57 43 55 24 52 49 26 1 50 - 45 35 60 68 40 42 63 53 2 42 48 - 56 42 54 28 52 50 29 3 56 23 55 - 65 67 50 22 66 58 4 45 61 35 67 - 29 42 63 43 52 5 25 69 65 70 27 - 48 68 45 65 6 26 50 30 52 39 50 - 50 48 48 7 51 32 49 30 61 67 49 - 62 55 8 52 65 55 69 24 30 49 64 - 26 9 25 54 26 59 50 61 30 56 30 -
  • 28. M 1 3 7 0 2 4 5 6 8 9 1 - 35 42 - - - - - - - 3 23 - 22 - - - - - - - 7 32 30 - - - - - - - - 0 - - - - 29 43 55 24 49 26 2 - - - 42 - 42 54 28 50 29 4 - - - 45 35 - 29 42 43 52 5 - - - 25 65 27 - 48 45 65 6 - - - 26 30 39 50 - 48 48 8 - - - 52 55 24 30 49 - 26 9 - - - 25 26 50 61 30 30 - Extended Automatic Optimum Branching (E-AOB) 28 17 3 6 8 4 5 2 9 0 Dissimilarity matrix
  • 29. Extended Automatic Optimum Branching (E-AOB) Forest: [5, 3, 9, 3, 8, 4, 0, 3, 9, 0] Cost: 204 29 17 3 6 8 4 5 2 9 0 2322 26 26 26 30 24 29 M 1 3 7 0 2 4 5 6 8 9 1 - 35 42 - - - - - - - 3 23 - 22 - - - - - - - 7 32 30 - - - - - - - - 0 - - - - 29 43 55 24 49 26 2 - - - 42 - 42 54 28 50 29 4 - - - 45 35 - 29 42 43 52 5 - - - 25 65 27 - 48 45 65 6 - - - 26 30 39 50 - 48 48 8 - - - 52 55 24 30 49 - 26 9 - - - 25 26 50 61 30 30 - Dissimilarity matrix
  • 30. Image Phylogeny Forests Reconstruction • Main approaches • AOK: Greedy approach • AOB/E-AOB: Exact approach 30
  • 31. Hypothesis #2 It is possible to lead to complementary properties of different forest reconstruction algorithm and combine them for improving the quality of the phylogeny forest reconstruction 31
  • 33. Fusion approach Tree count V = [1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5] Median(V) = (2 + 2)/2 = 2 100 variations … AOK 0 35 1 4 7 8 2 9 6 100 variations … AOB 0 3 5 1 4 7 8 2 9 6 100 variations … E-AOB 0 35 1 4 7 8 2 9 6 33
  • 34. Fusion approach 100 variations … AOK 0 35 1 4 7 8 2 9 6 100 variations … AOB 0 3 5 1 4 7 8 2 9 6 100 variations … E-AOB 0 35 1 4 7 8 2 9 6 Sum of votes of each node [260, 250, 90, 30, 35, 24, 0, 0, 0, 0] Roots {0, 1} 34
  • 35. 0 351 4 7 8 2 9 6 Fusion approach 100 variations … AOK 0 35 1 4 7 8 2 9 6 100 variations … AOB 0 3 5 1 4 7 8 2 9 6 M 0 1 2 3 4 5 6 7 8 9 0 - - 0 270 0 286 0 0 0 0 1 - - 0 0 255 0 0 280 0 0 2 - - - 0 0 0 130 0 270 300 3 - - 280 - 0 0 0 0 30 0 4 - - 0 0 - 0 0 20 0 0 5 - - 20 0 0 - 0 0 0 0 6 - - 0 0 0 0 - 0 0 0 7 - - 0 0 10 0 0 - 0 0 8 - - 0 0 0 0 0 0 - 0 9 - - 0 0 0 0 170 0 0 - 100 variations … E-AOB 0 35 1 4 7 8 2 9 6 Final forest [0, 1, 3, 0, 1, 0, 2, 1, 2, 2] 35
  • 37. Datasets 37 • 36,000 test cases • [2..10] trees • Ground truth is available • One Camera and Multiple Cameras
  • 38. Datasets 38 • Transformations • Crop • Resize • Brightness, contrast
 and gamma correction • Rotation • JPEG Compression
  • 39. 10 6 7 9 8 25 3 11 7 Evaluation 39 2 8 9 103 65 4 39 4 Ground truth Reconstructed forest Roots
  • 40. 3 10 6 71 2 8 95 1 7 Evaluation 40 2 8 9 103 65 4 40 4 Ground truth Reconstructed forest Edges
  • 41. 10 6 71 2 9 3 1 7 Evaluation 41 2 8 9 103 65 4 41 8 5 4 Ground truth Reconstructed forest Leaves
  • 42. 7 10 6 1 83 25 1 7 Evaluation Ancestry 42 2 8 9 103 65 4 42 9 4 Ground truth Reconstructed forest
  • 43. Results - One Camera 43 2 3 4 5 6 7 8 9 10 # of trees 0.65 0.7 0.75 0.8 0.85 0.9 0.95 Score Root - One Camera AOK AOB E-AOB FUSION 2 3 4 5 6 7 8 9 10 # of trees 0.65 0.7 0.75 0.8 0.85 0.9 0.95 Score Ancestry - One Camera AOK AOB E-AOB FUSION Error Reduction (E-AOB vs. AOK): 19% Error Reduction (E-AOB vs. AOK): 18% Error Reduction (FUSION vs. AOK): 17% Error Reduction (Fusion vs. AOK): 17%
  • 44. Results - Multiple Cameras 44 2 3 4 5 6 7 8 9 10 # of trees 0.65 0.7 0.75 0.8 0.85 0.9 0.95 Score Root - Multiple Cameras AOK AOB E-AOB FUSION 2 3 4 5 6 7 8 9 10 # of trees 0.65 0.7 0.75 0.8 0.85 0.9 0.95 Score Ancestry - Multiple Cameras AOK AOB E-AOB FUSION Error Reduction (E-AOB vs. AOK): 20% Error Reduction (E-AOB vs. AOK): 19% Error Reduction (FUSION vs. AOK): 28% Error Reduction (Fusion vs. AOK): 21%
  • 45. New dissimilarity measures for image phylogeny reconstruction 45
  • 46. New dissimilarity measures for image phylogeny reconstruction • Previous work: Mean Squared Error (MSE) • Problems • Few information about the content of the images • Small misaligned caused on mapping can affect the results 46
  • 47. Hypothesis #3 The comparison of the distributions of the pixel values of two near-duplicate images is more effective for the dissimilarity calculation than their point-wise comparison 47
  • 49. Mutual Information comparison 49 MI(I0 src, Itgt) I0 src Itgt MI(I0 src, Itgt) = H(Itgt) H(Itgt|I0 src) H(X) = Ex[log(P(X))]
  • 50. Comparing gradients with mutual information Mutual
 Information I0 src Itgt MSE 50 X Gradient Computation Gradient Computation
  • 51. Color matching • Previous work • Normalization based on mean and standard deviation of another image • Problem • Unexpected Artifacts • New color matching approach • Histogram matching [Gonzales (2007)] 51
  • 52. Histogram color matching 52 0 20 40 60 80 100 120 140 160 180 200 220 240 255 i 0 0.025 0.05 0.075 0.1 0.125 0.15 p(i) Histogram - SRC 0 20 40 60 80 100 120 140 160 180 200 220 240 255 i 0 0.025 0.05 0.075 0.1 0.125 0.15 p(i) Histogram - TGT Histogram
  • 53. 0 20 40 60 80 100 120 140 160 180 200 220 240 255 i 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CDF(i) CDF - SRC 0 20 40 60 80 100 120 140 160 180 200 220 240 255 i 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CDF(i) CDF - TGT 99 114 Histogram color matching 53 200 238 20 22 Cumulative Distribution Function
  • 54. Histogram color matching 54 0 20 40 60 80 100 120 140 160 180 200 220 240 255 i 0 0.025 0.05 0.075 0.1 0.125 0.15 p(i) Histogram - TGT 0 20 40 60 80 100 120 140 160 180 200 220 240 255 i 0 0.025 0.05 0.075 0.1 0.125 0.15 p(i) Result of historgram matching Resultant histogram
  • 55. Color matching approaches Result: Mean/STD Result: Histogram Isrc Itgt Color Matching Result: Mean/STD maching Result: Histogram maching 55
  • 57. Experimental setup • Reconstruction • Extended AOB • Evaluation • Roots, edges, leaves and ancestry 57
  • 58. Roots - One Camera Accuracy 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Number of trees 1 2 3 4 5 6 7 8 9 10 MSE GRAD MINF GRMI HGMI Results 58 Roots - Multiple Cameras Accuracy 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Number of trees 1 2 3 4 5 6 7 8 9 10 MSE GRAD MINF GRMI HGMI Error Reduction (HGMI vs. MSE): 53% Error Reduction (HGMI vs. MSE): 47%
  • 59. Results 59 Ancestry - One Camera Accuracy 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Number of trees 1 2 3 4 5 6 7 8 9 10 MSE GRAD MINF GRMI HGMI Error Reduction (HGMI vs. MSE): 60% Error Reduction (HGMI vs. MSE): 58% Ancestry - Multiple Cameras Accuracy 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Number of trees 1 2 3 4 5 6 7 8 9 10 MSE GRAD MINF GRMI HGMI
  • 60. Qualitative results Ellen DeGeneres’ Selfie 60 Group bGroup a Group c Group d Group e
  • 61. a09 a00 a11 a01 a02 a04 a05 a06 a07 a08 a03a10 a12 b03 b01 b06 b04 b02 c07 b05 c01 c08 c02 c05 c09 c10 c06 c03 c04 d08 d01 d07 d03 d02 d06 d05 d04 e01 e02 e03 e04 e05 e06 e07 61 Group b Group a Group c Group d Group e Qualitative results Ellen DeGeneres’ Selfie
  • 62. a09 a00 a11 a01 a02 a04 a05 a06 a07 a08 a03a10 a12 b03 b01 b06 b04 b02 c07 b05 c01 c08 c02 c05 c09 c10 c06 c03 c04 d08 d01 d07 d03 d02 d06 d05 d04 e01 e02 e03 e04 e05 e06 e07 62 Group b Group a Group c Group d Group e Qualitative results Ellen DeGeneres’ Selfie
  • 63. a09 a00 a11 a01 a02 a04 a05 a06 a07 a08 a03a10 a12 b03 b01 b06 b04 b02 c07 b05 c01 c08 c02 c05 c09 c10 c06 c03 c04 d08 d01 d07 d03 d02 d06 d05 d04 e01 e02 e03 e04 e05 e06 e07 63 Group b Group a Group c Group d Group e Qualitative results Ellen DeGeneres’ Selfie
  • 64. a09 a00 a11 a01 a02 a04 a05 a06 a07 a08 a03a10 a12 b03 b01 b06 b04 b02 c07 b05 c01 c08 c02 c05 c09 c10 c06 c03 c04 d08 d01 d07 d03 d02 d06 d05 d04 e01 e02 e03 e04 e05 e06 e07 64 Group b Group a Group c Group d Group e Qualitative results Ellen DeGeneres’ Selfie
  • 66. Video Phylogeny • More challenging problem than Image Phylogeny • Now we have a new dimension: time • Videos can be misaligned • Videos can have different compression parameters 66
  • 67. Hypothesis #4 Temporal alignment is paramount for a proper video phylogeny reconstruction process 67
  • 68. • Phylogeny reconstruction for misaligned and compressed video sequences F. O. Costa, S. Lameri, P. Bestagini, Z. Dias, A. Rocha, M. Tagliassachi, S. Tubaro. Phylogeny reconstruction for misaligned and compressed video sequences. IEEE International Conference on Image Processing (ICIP), 2015 • Hashing-based Temporal Alignment for Video Phylogeny Reconstruction F. O. Costa, P. Bestagini, S. Tubaro, Z. Dias, A. Rocha. Hashing-based Temporal Alignment for Video Phylogeny Reconstruction. Submitted to IEEE WIFS, 2016 Video Phylogeny Contributions 68
  • 69. Phylogeny reconstruction for misaligned and compressed video sequences 69
  • 70. Video dissimilarity calculation • Temporal alignment • Frame registration • Color matching • Coding matching (controlled) • Comparison (MSE) 70
  • 71. Temporal alignment 71 V 1 2 3 4 5 6 7 8 9 10 … n-2 n-1 n y1 y2 y3 y4 y5 y6 y7 y8 y9 y10 yn-2 yn-1 yn yi = Average of luminance of V(i) LDV = [y2 - y1, y3 - y2, … , yn-1 - yn-2, yn - yn-1 ] Difference of luminance
  • 72. 50 100 150 200 250 0 i L 50 100 150 200 250 0 5 10 i LDV(i) V = Vsrc V = Vtgt −200 −100 0 100 200 0 0.2 0.4 0.6 i phase-corr. Video Alignment 72 Phase-correlation: ˆi = arg maxi F 1 h F[LDVsrc ]·F[LDVtgt ]⇤ |F[LDVsrc ]·F[LDVtgt ]⇤| i (i)
  • 73. Comparison • We perform two kinds of comparison • Average of MSE for the correspondent frames of two videos • Parenthood matrix 73
  • 74. Parenthood matrix 74 Frequency of the edges Parenthood Matrix Optimum branching (MAX) Final phylogeny tree …M1 M2 M3 M4 M5 M6 Mn-1 Mn V3 … V1 … V2 … V4 … V5 …
  • 76. Dataset 76 Available at https://media.xiph.org/video/derf/ •200 trees with 10 videos •100 - No clip •100 - Clip at the Beginning/end of the stream •Probability of clip: 50% •CODEC: MPEG2, MPEG4, H264 •Crop, resize, bright, contrast, coding and temporal clipping
  • 77. Experimental Setup • Phylogeny Reconstruction • OB algorithm • Evaluation metrics • Root, edges, leaves, ancestry and depth 77 1 2 3 5 4 4 23 5 1 Reconstructed treeGround truth Depth = 1
  • 78. Results #1 78 Verifying alignment and coding matching Root Depth Edges Leaves Ancestry 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Score Results - Dataset without temporal clipping Without temporal alignment With temporal alignment With temporal alignment and compression matching Root Depth Edges Leaves Ancestry 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Score Results - Dataset with temporal clipping Without alignment With alignment With alignment and compression matching
  • 79. Root Depth Edges Leaves Ancestry 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Score Results - Dataset with temporal clipping A - NO COD P - NO COD A - COD P - COD Root Depth Edges Leaves Ancestry Results - Dataset without temporal clipping Results #2 79 Comparison: Average of MSE (A) x Parenthood matrix (P) Root Depth Edges Leaves A 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Score Results - Dataset with temporal clipping A - NO COD P - NO COD A - COD P - COD Root Depth Edges Leaves Ancestry 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Score Results - Dataset without temporal clipping A - NO COMPRESSION MATCHING P - NO COMPRESSION MATCHING A - WITH COMPRESSION MATCHING P - WITH COMPRESSION MATCHING
  • 81. • Near-duplicate matching • Near-duplicate extraction • Near-duplicate alignment Hashing-based temporal alignment 81
  • 82. Step 1 Near-duplicate matching • Groups of 64 frames • All frames are resized to 32 x 32 pixel 82 1 frame Temporal axis …… 1 2 3 4 5 6 7 8 9 10 59 60 61 62 63 64 65 66 67 68
  • 83. Step 1 Near-duplicate matching 83 64 coefficients 32 coefficients 32coefficients DCT coefficients 4 x 4 x 4 DCT coefficients … 3D Discrete Cosine Transform
  • 84. Step 1 Near-duplicate matching • Binary hash • According to the median of DCT • Distance matrix • Hamming distance of two binary hashes of different videos 84
  • 85. Step 2 Near-duplicate extraction 85 0 30 60 90 120 150 180 210 236 0 20 40 60 80 100 120 140 160 180 200 0 30 60 90 0 20 40 60 80 100 120 140 160 180 200 20 150 180 210 236 0 30 60 90 120 150 180 210 236 0 20 40 60 80 100 120 140 160 180 200 0 30 60 90 0 20 40 60 80 100 120 140 160 180 200 20 150 180 210 236 0 30 60 90 120 150 180 210 236 0 20 40 60 80 100 120 140 160 180 200 0 30 60 90 0 20 40 60 80 100 120 140 160 180 200 20 150 180 210 236 0 30 60 90 120 150 180 210 236 0 20 40 60 80 100 120 140 160 180 200 BinarizationOpeningExtraction
  • 86. V1 V2 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Sequences without temporal alignment 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Alignment by phase correlation 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Hashing-based temporal alignment 86 Step 3 Near-duplicate alignment
  • 87. V1 V2 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Sequences without temporal alignment 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Alignment by phase correlation 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Hashing-based temporal alignment V1 V2 0 20 40 60 80 100 120 140 -3 -2 -1 0 1 2 3 Diff.ofluminance Sequences witho 3 Alignment by 87 Step 3 Near-duplicate alignment
  • 88. V1 V2 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Sequences without temporal alignment 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Alignment by phase correlation 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Hashing-based temporal alignment 88 Step 3 Near-duplicate alignment0 20 40 60 80 100 120 140 -3 -2 -1 Diff. 0 20 40 60 80 100 120 140 -3 -2 -1 0 1 2 3 Diff.ofluminance Alignment by
  • 89. V1 V2 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Sequences without temporal alignment 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Alignment by phase correlation 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 -3 -2 -1 0 1 2 3 Diff.ofluminance Hashing-based temporal alignment 89 Step 3 Near-duplicate alignment 0 20 40 60 80 100 120 140 -3 -2 -1 0 Diff.oflu -3 -2 -1 0 1 2 3 Diff.ofluminance Hashing-based
  • 91. Experiments •Dataset •300 trees with 10 videos •100 - No clip •100 - Clip at the Beginning/end of the stream •100 - Clip anywhere 91
  • 92. Results 92 Diff. of luminance vs. Hashing-based alignment Difference of luminance Hashing-based Root Depth Edges Leaves Ancestry Root Depth Edges Leaves Ancestry No coding matching No clip 0,630 0,500 0,772 0,832 0,674 0,650 0,440 0,777 0,828 0,702 Clip border 0,710 0,35 0,788 0,826 0,724 0,710 0,330 0,807 0,846 0,740 Clip any 0,620 0,560 0,724 0,758 0,628 0,670 0,450 0,755 0,789 0,683 With coding matching No clip 0,870 0,140 0,839 0,859 0,812 0,690 0,400 0,778 0,831 0,711 Clip border 0,840 0,170 0,863 0,901 0,834 0,660 0,400 0,780 0,824 0,722 Clip any 0,720 0,390 0,801 0,828 0,696 0,660 0,470 0,765 0,810 0,696
  • 94. Conclusion • New solutions for multimedia phylogeny • Robust methods for phylogeny forest reconstruction • More robust dissimilarity measures • Gradient + mutual information • Good results for video phylogeny when dealing with misaligned and compressed videos 94
  • 95. Future work • Other transformations • Blurring, sharpness, content insertion… • Videos with different frame rates • Video dissimilarity calculation using time information • Robust methods to reliably estimate the compression parameters directly from the video 95
  • 96. Acknowledgements • Examiners • Prof. Anderson Rocha and Prof. Zanoni Dias • RECOD - IC/Unicamp • ISPG - Politecnico di Milano • FAPESP • CAPES 96