Cvpr2010 open source vision software, intro and training part vii point cloud procesing in ros. point cloud library. - rusu - unknown - 2010

Introduction Motivation Acquisition Data representation Storage PCL

PointCloud(2) procesing in ROS.
Point Cloud Library.
Radu Bogdan RUSU
June 14, 2010

[Introduction] Motivation Acquisition Data representation Storage PCL

Outline

1. Introduction

2. Motivation

3. Acquisition

4. Data representation

5. Storage

6. PCL

Radu Bogdan RUSU Point Cloud Library


Introduction (1/6)
What is ROS?
ROS is a comprehensive meta operating system for robotics
organized via nodes, packages, stacks, repositories
collection of building blocks
community driven effort - all free/open source (!)
stop reinventing the wheel, bring the best ouf of robotics
replicating research results



Introduction (2/6)
What are Point Clouds?

*
Point Cloud = a ”cloud“ (i.e., collection) of nD points
(usually n = 3)
p i = {xi , yi , zi } −→ P = {p 1 , p 2 , . . . , p i , . . . , p n }
used to represent 3D information about the world



Introduction (3/6)

besides XYZ data, each point p can hold additional
information
examples include: RGB colors, intensity values, distances,
segmentation results, etc



Introduction (4-6/6)


Introduction [Motivation] Acquisition Data representation Storage PCL

Outline

1. Introduction

2. Motivation

3. Acquisition


5. Storage

6. PCL



Motivation (1/4)
Why are Point Clouds important?

Point Clouds are important for a lot of reasons (!). Besides
representing geometry, they can complement and supersede
images when data has a high dimensionality.



Motivation (2/4)

Concrete example 1: get the cup from the drawer.

⇐⇒
Handle Drawer

*



Motivation (3/4)

Concrete example 2: ﬁnd the door and its handle, and open it.

Door

⇐⇒ Handle

*



Motivation (4/4)

Concrete example 3: safe motion planning/manipulation.

⇒
*


Introduction Motivation [Acquisition] Data representation Storage PCL

Outline

1. Introduction

2. Motivation

3. Acquisition


5. Storage

6. PCL



Acquisition (1/3)
How are Point Clouds acquired? Where do they come from?

There are many different sensors that can generate 3D
information. Examples:
laser/lidar sensors (2D/3D)
stereo cameras
time-of-ﬂight (TOF) cameras
etc...

Radu Bogdan RUSU
* Point Cloud Library


Acquisition (2/3)

The PR2 sensor head:

*
two pairs of stereo cameras (narrow + wide)
tilting laser sensor



Acquisition (3/3)

Simulation (!):

raytracing + stereo imagery fed into the same algorithmic
modules that are used to process real data


Introduction Motivation Acquisition [Data representation] Storage PCL

Outline

1. Introduction

2. Motivation

3. Acquisition


5. Storage

6. PCL



Data representation (1/7)
Representing Point Clouds

As previously presented:
a point p is represented as an n-tuple, e.g.,
p i = {xi , yi , zi , ri , gi , bi , disti , · · · }
a Point Cloud P is represented as a collection of points p i ,
e.g., P = {p 1 , p 2 , · · · , p i , · · · , p n }



Point Cloud Data structures

In terms of data structures:
an XYZ point can be represented as:
float32 x
float32 y
float32 z
a n-dimensional point can be represented as:
float32[] point
which is nothing else but a:
std::vector<float32> point
in C++
potential problem: everything is represented as ﬂoats (!)




In terms of data structures:
therefore a point cloud P is:
Point[] points
or:
std::vector<Point> points
in C++, where Point is the structure/data type representing
a single point p




Because Point Clouds are big:
operations on them are typically slower (more data, more
computations)
they are expensive to store, especially if all data is
represented as ﬂoats/doubles
Solutions:




Because Point Clouds are big:
operations on them are typically slower (more data, more
computations)
they are expensive to store, especially if all data is
represented as ﬂoats/doubles
Solutions:
store each dimension data in different (the most
appropriate) formats, e.g., rgb - 24bits, instead of 3 × 4
(sizeof ﬂoat)
group data together, and try to keep it aligned (e.g., 16bit
for SSE) to speed up computations



ROS representations for Point Cloud Data

The ROS PointCloud(2) data format
(sensor_msgs/PointCloud2.msg):
# This message holds a collection of nD points , as a binary blob .
Header header
# 2 D structure of the point cloud . If the cloud is unordered ,
# height is 1 and width is the length of the point cloud .
uint32 height
uint32 width
# Describes the channels and their layout in the binary data blob
PointField [] fields
bool is_bigendian # Is this data bigendian ?
uint32 point_step # Length of a point in bytes
uint32 row_step # Length of a row in bytes
uint8 [] data # Actual point data , size is ( row_step * height )
bool is_dense # True if there are no invalid points




where PointField (sensor_msgs/PointField.msg) is:
# This message holds the description of one point entry in the # PointCloud2 message format .
uint8 INT8 = 1
uint8 UINT8 = 2
uint8 INT16 = 3
uint8 UINT16 = 4
uint8 INT32 = 5
uint8 UINT32 = 6
uint8 FLOAT32 = 7
uint8 FLOAT64 = 8
string name # Name of field
uint32 offset # Offset from start of point struct
uint8 datatype # Datatype enumeration see above
uint32 count # How many elements in field

PointField examples:
"x", , 7, 1
"y", 4, 7, 1
"z", 8, 7, 1
" rgba " , 12 , 6, 1
" normal_x " , 16 , 8, 1
" normal_y " , 2 , 8, 1
" normal_z " , 24 , 8, 1
" fpfh " , 32 , 7 , 33



binary blobs are hard to work with
we provide a custom converter, Publisher/Subscriber,
transport tools, ﬁlters, etc, similar to images
templated types: PointCloud2 −→ PointCloud<PointT>
examples of PointT:
struct PointXYZ
{
float x ;
float y ;
float z ;
}
struct Normal
{
float normal [3];
float curvature ;
}


Introduction Motivation Acquisition Data representation [Storage] PCL

Outline

1. Introduction

2. Motivation

3. Acquisition


5. Storage

6. PCL



Point Cloud Data storage (1/2)
ROS input/output

PointCloud2.msg and PointField.msg are ROS messages
they can be published on the network, saved/loaded
to/from BAG ﬁles (ROS message logs)
usage example:
$ rostopic find sensor_msgs/PointCloud2 | xargs rosrecord -F foo
[ INFO ] [1271297447.6564145 2]: Recording to foo . bag .
^C
[ INFO ] [127129745 .7235 4983]: Closing foo . bag .
$ rosplay -c foo.bag
bag : foo . bag
version : 1.2
start_time : 127129744797428 542
end_time : 1271297449983577462
length : 2 929692
topics :
- name : / narrow_stereo_textured / points2
count : 3
datatype : sensor_msgs / PointCloud2
md5sum : 1158 d486dd51d683ce2f1be655c3c181



Point Cloud Data storage (2/2)
PCD (Point Cloud Data) ﬁle format

In addition, point clouds can be stored to disk as ﬁles, into the
PCD format.
# Point Cloud Data ( PCD ) file format v .5
FIELDS x y z rgba
SIZE 4 4 4 4
TYPE F F F U
WIDTH 3 72
HEIGHT 1
POINTS 3 72
DATA binary
...

DATA can be either ascii or binary. If ascii, then
...
DATA ascii
. 54216 .11349 . 4 749
- . 17447 .11425 . 41273
- . 1 661 .11338 . 4 916
. 26422 .11499 . 32623
...


Introduction Motivation Acquisition Data representation Storage [PCL]

Outline

1. Introduction

2. Motivation

3. Acquisition


5. Storage

6. PCL



Point Cloud Library (1/11)

http://www.ros.org/wiki/pcl


What is PCL (Point Cloud Library)?

PCL is:
fully templated modern C++ library for 3D point cloud
processing
uses SSE optimizations (Eigen backend) for fast
computations on modern CPUs
uses OpenMP and Intel TBB for parallelization
passes data between modules (e.g., algorithms) using
Boost shared pointers
OpenCV’s 3D little sister
PCL deprecates older ROS packages such as
point_cloud_mapping and replaces
sensor_msgs/PointCloud.msg with the modern
sensor_msgs/PointCloud2.msg format (!)


PCL (Point Cloud Library) structure

collection of smaller, modular C++ libraries:
libpcl_features: many 3D features (e.g., normals and curvatures,
boundary points, moment invariants, principal curvatures, Point Feature
Histograms (PFH), Fast PFH, ...)
libpcl_surface: surface reconstruction techniques (e.g.,
meshing, convex hulls, Moving Least Squares, ...)
libpcl_filters: point cloud data filters (e.g., downsampling, outlier
removal, indices extraction, projections, ...)
libpcl_io: I/O operations (e.g., writing to/reading from PCD (Point
Cloud Data) and BAG files)
libpcl_segmentation: segmentation operations (e.g.,cluster
extraction, Sample Consensus model fitting, polygonal prism extraction, ...)
libpcl_registration: point cloud registration methods (e.g.,
Iterative Closest Point (ICP), non linear optimizations, ...)

unit tests, examples, tutorials (some are work in progress)
C++ classes are templated building blocks (nodelets!)


PPG: Perception Processing Graphs

Philosophy: write once, parameterize everywhere




Why PPG?
Algorithmically:
door detection = table detection = wall detection = ...
the only thing that changes is: parameters (constraints)!



More on architecture

Inheritance simpliﬁes development and testing:

pcl :: Feature < PointT > feat ;
feat = pcl :: Normal < PointT > ( input );
feat = pcl :: FPFH < PointT > ( input );
feat = pcl :: BoundaryPoint < PointT > ( input );
...
feat . compute (& output );
...



PCL 0.1.x statistics

Misc, stats:
over 100 classes
over 25k lines of code
external dependencies (for now) on eigen, cminpack, ANN,
FLANN, TBB
internal dependencies (excluding the obvious) on
dynamic_reconﬁgure, message_ﬁlters
tf_pcl package for TF integration



Nodelets

write once, parameterize everywhere =⇒ modular code
ideally, each algorithm is a “building block” that consumes
input(s) and produces some output(s)
in ROS, this is what we call a node. inter-process data
passing however is inefﬁcient. ideally we need shared
memory.
Solution:
nodelets = “nodes in nodes” = single-process, multi-threading



Nodelets

write once, parameterize everywhere =⇒ modular code
ideally, each algorithm is a “building block” that consumes
input(s) and produces some output(s)
in ROS, this is what we call a node. inter-process data
passing however is inefficient. ideally we need shared
memory.
Solution:
nodelets = “nodes in nodes” = single-process, multi-threading
same ROS API as nodes (subscribe, advertise, publish)
dynamically (un)loadable
optimizations for zero-copy Boost shared_ptr passing
PCL nodelets use dynamic_reconfigure for on-the-fly
parameter setting


Nodelets

Performance hints:
29 msg/s (inter process) vs 4664 mgs/s (intra process)
fast (un)loading for algorithms (one service request)
natively multithreaded
(boost::thread::hardware_concurrency ())
threading scheduler and granularity controllable in
OpenMP/TBB
Goals:
GUI for managing nodelets => 3D “Simulink” (!)
get to a point where fast prototyping is easy



Downsample and ﬁltering example with nodelets

< launch >
< node pkg = " nodelet " type = " standalone_nodelet "
name = " pcl_manager " output = " screen " / >

< node pkg = " nodelet " type = " nodelet " name = " foo "
args = " voxel_grid VoxelGrid pcl_manager " >
< remap from " / voxel_grid / input "
to = " / narrow_stereo_textured / points " / >
< rosparam >
# -[ Mandatory parameters
leaf_size: [ . 15 , . 15 , . 15]
# -[ O p t i o n a l p a r a m e t e r s
# field containing distance values ( for filtering )
filter_field_name: " z "
# filtering points outside of < .8 ,5. >
filter_limit_min: .8
filter_limit_max: 5.
use_indices: false # false by default
</ rosparam >
</ node >
...
</ launch >



Normal estimation example with nodelets

< launch >

< node pkg = " nodelet " type = " nodelet " name = " foo "
args = " normal_estimation NormalEstimation pcl_manager " >
< remap from = " / normal_estimation / input "
to = " / voxel_grid / output " / >
< remap from = " / normal_estimation / surface "
to = " / narrow_stereo_textured / points " / >
< rosparam >
# -[ M a n d a t o r y p a r a m e t e r s
# Set either ’ k_search ’ or ’ radius_search ’
k_search:
radius_search: .1
# Set the spatial locator . Possible values are:
# ( ANN ) , 1 ( FLANN ) , 2 ( organized )
spatial_locator:
</ rosparam >
</ node >
...
</ launch >



PCL - Table Object Detector
How to extract a table plane and the objects lying on it



PCL - Web tutorials (1/3)




Demo
http://www.ros.org/wiki/pcl/Tutorials



Questions?
Thanks!

http://www.ros.org/wiki/pcl

Cvpr2010 open source vision software, intro and training part vii point cloud procesing in ros. point cloud library. - rusu - unknown - 2010

More Related Content

More from zukun (20)

Cvpr2010 open source vision software, intro and training part vii point cloud procesing in ros. point cloud library. - rusu - unknown - 2010