Template-Based Paper Reconstruction from a Single Image is Well Posed when the Rullings are Parallel

Template-based Paper Reconstruction from a Single Image is Well Posed when the Rulings are Parallel Pierluigi Taddei , Politecnico di Milano, Milano, Italy [email_address] Adrien Bartoli, LASMEA (CNRS / UBP), Clermont-Ferrand, France [email_address]

The problem at a glance We aim to reconstruct the pose of a piece of paper which is subject to a subset of possible isometries perspective image paper template internal camera parameters set of point correspondences 3D pose of the piece of paper

The problem at a glance We show that for particular isometries this is a well posed problem perspective image paper template internal camera parameters set of point correspondences 3D pose of the piece of paper

Related works Template-based monocular deformable surface registration may be performed using general models Generic deformable surfaces using triangular mesh grids (Julien Pilet, Vincent Lepetit, Pascal Fua) Monocular deformable surface reconstruction is possible if some priors are known 3D Morphable Models for face reconstruction ( Volker Blanz and Thomas Vetter )  Great works but use either empirical models or learnt models to describe real deformations

Related works We address the case of developable surface to model material such as paper Useful for augmentation Paper reconstruction may be performed using shape-from-contour mainly for document digitization  requires the full knowledge of the contours Not useful in the case of occlusion

In order to perform a full 3D reconstruction we assume: Set of point correspondences Internal camera parameters known S Metric size of the template ( W, H) Physical model, developabel surfaces Deformations are isometries , thus distances are mantained Vanishing gaussian curvatur Assumpions H W

Is the problem well posed? The general case of isometric deformations is ill-posed We consider a subset of the possible isometries The rulings of the developable surface are constrained to be parallel, i.e. the surface is a generalized cylinder Intuitively this is what happens when book pages are deformed by keeping the binding and the opposite edge parallel. Generalized cylinder, well posed Generic isometry, ill posed

Parametrization In the case of a generalized cylinder the surface is parameterized as follows: A generatrix plane  which is perpendicular to all rulings and contains the lower border of the surface A transformation T which maps the XY plane to  , the origin to the bottom left corner, the X axis to the corner segment A mapping  which maps u coordinates to a 2D curve on 

Reduction to a 2D reconstruction using 1D cameras (1) By considering the projection equation we can derive that:

Reduction to a 2D reconstruction using 1D cameras (2) The problem is equivalent to the reconstruction of 2D points given a pair of 1D cameras for each surface slice u varies the point position over the 2D curve v varies the two cameras internal parameters

Formulating the problem (1) Isometries preserves gaussian curvature : the gaussian curvature is, thus, vanishing everywhere since the parameterization is given by a developable surface this constraint is enforced by construction Isometries preserves the metric : By construction distances are preserved along the rulings Since we are assuming a generalized cylinder, if the metric is preserved on  section then it is preserved everywhere   must be a 1D isometry

Metric constraints: Moreover, we aim to minimize: the reprojection error of the point correspondences a smoothing term Formulating the problem (2)

The problem is expressed as a functional optimization: This problem depends on the free variable u , function  and its first and second derivatives The problem possess natural boundary condition (i.e. the boundary are not fixed) The functional E is given by the weighted sum of: : data term, which describe the reprojection error : smoothing term : metric term Formulating the problem (3)

This functional optimization is solved by applying the Euler-Lagrange equations this gives a system of PDEs depending up to the fourth derivatives of  and a set of PDEs related to the natural boundary condition The PDE system is solved using numerical methods: The domain is sampled at N nodes Derivatives are replaced by finite differences approximation Solving the problem

Generatrix plane recovery (1) To exploit the exposed parametrization the generatrix plane transormation is needed This can be done by exploiting the template dimensions at least two pair of points on the same ruling, for instance the corners of the largest visible rectangle Using the template dimension the points distances are easily calculated We know the camera internal parameter The problem can be solved using an optimization procedure

Generatrix plane recovery (1) Done exploiting the template dimensions and at least two pairs of points on the same ruling Using the template dimension the inter point distances are easily calculated The problem can be solved using an optimization procedure

Generatrix plane recovery (2) Given the four points c 1, c 2 and c 3, c 4 , the two segments length d and the camera internal parameters The unknowns are the four perspective depths  1  2  3 and  4 These may be recovered by enfocing the constriants : C 1 C 2 is parallel to C 3 C 4 , C 1 C 2 is orthogonal to C 1 C 4 , C 1 C 2 has length d , C 3 C 4 has length d

Generatrix plane recovery (3) The generatrix plane  is orthogonal to the plane containing the detected rectangle In particular we consider a transformation T which brings The plane XY to ,  The axis X parallel to C 1 C 4 , The axis Z parallel to C 1 C 2

Experimental results (texture replacement)

Experimental results (augmentation)

Experimental results (handling occlusion)

Conclusion and future works Template based reconstruction of a generalized cylinder is well posed The reconstruction is probably well posed also in the generalized cone case Even if more general, this case is more difficult to reproduce, and the generalized cone parameters are more difficult to recover

Template-Based Paper Reconstruction from a Single Image is Well Posed when the Rullings are Parallel

More Related Content

What's hot (20)

Similar to Template-Based Paper Reconstruction from a Single Image is Well Posed when the Rullings are Parallel (20)

Recently uploaded (20)

Template-Based Paper Reconstruction from a Single Image is Well Posed when the Rullings are Parallel