Challenges for Perception in Navigation and Manipulation

DariusBurschka–PerceptionChallenges
http://www6.in.tum.de/burschka/
Perception and Manipulation,
Closing the Gap
Darius Burschka
Machine Vision and Perception Group 
Department of Computer Science
Technische Universität München

Where
do
we
need
perception?
“known” clutter “unknown” clutter

Role of perception in an unknown
environment
labeling motion parameters
• Acquisition of the scene
• Clustering of raw sensor
data
• Identification of
independent motion
components
• Estimation of action
• Understanding actions

What
is
in
the
scene?

(sensor
data
abstraction)

Feature Extraction
• For all pairs of surflets at 
distance d insert the triple 
 
 
 
plus a pointer to its model in a 
hash-table.
• Do this for all models using the 
same hash-table.

• For each model surflet pair 
in the hash-table cell: 
 
Compute the rigid 
transform T that 
best aligns 
 
Online Recognition Phase
model hash-table
IJRR 2012 Special Issue, Papazov et al.

IJRR 2012 Special Issue, Papazov et al.

What
happens
if
an
object
is
similar
to

one
in
the
database?
Indexing to the Atlas database needs
to be extended to object classes
-> deformable shape registration
needed
Atlas information Observed object

Deformable Registration from  
generic models (special issue SGP'11 Papazov et al.)
Matching of a detailed shape
to a primitive prior
The manipulation “heat map” from
the generic model gets propagated

Alignment of non-rigid
shapes for inspection and
localization
Similarity estimates for
categorization of objects
Visual identification of
kinematic chains

The proposed hierarchical MRFs consists of two layers:
segmentation and correspondence layer.
Segmentation layer is designed for the dynamic foreground
extraction.
Correspondence layer is designed to maintain the
correspondences of extracted foreground data across
consecutive frames

What do we try to extract from the
environment?
labeling motion parameters

Hybrid Model of the Environment  
(JC Ramirez)

World model saves additional info,  
like texture, motion, etc
VISAPP 2013 J.Ramirez et al.)

Fusion
of
sensor
readings
-‐
Construction
of
3D
models

Pose Estimation
➢Two motion prediction
concepts
• 2D feature propagation by
motion derivatives
• IMU-based feature
prediction
➢Combination of both:
• translation propagation by
feature velocity (2D)
• rotation propagation by
gyroscopes
no feature propagation
Strobl, Mair, Bodenmüller, Kielhofer, Sepp, Suppa, Burschka, Hirzinger
IROS, IEEE/RSJ, 2009, Best Paper Finalist
Mair, Strobl, Bodenmüller, Suppa, Burschka 
KI, Springer Journal, 2010
MachineVisionandPerceptionGroup@TUMMVP

KI, Springer Journal, 2010
Feature
Propagation
concepts
• 2D feature propagation by
motion derivatives
prediction
• translation propagation by
feature velocity (2D)
gyroscopes
linear feature propagation

Feature
Propagation
(Data
fusion)
concepts
• 2D feature propagation
by motion derivatives
prediction
• translation propagation
by feature velocity (2D)
gyroscopes
KI, Springer Journal, 2010linear + gyros based
prop.

Local
Feature
Tracking
Algorithms
•Image-gradient based à Extended KLT (ExtKLT)
• patch-based implementation
• feature propagation
• corner-binding
+ sub-pixel accuracy
• algorithm scales bad with number  
of features
•Tracking-By-Matching à AGAST tracker
• AGAST corner detector
• efficient descriptor
• high frame-rates (hundrets of  
features in a few milliseconds)
+ algorithm scales well with number 
of features
• pixel-accuracy
8

Real
Time
Pose
Tracking

Perception-based loop closure
Obstacle avoidance
Navigation
system
with
accuracy

estimation

Navigation example
Estimation of 3 rotational
angles
Estimation of a translation
vector

Collaborative Reconstruction with  
Self-Localization

Asynchronous stereo for  
dynamic scenes
NTP
VISAPP 2014

Estimation
of
unobservable
properties
Mapping of
Knowledge

Physical
and
Geometric
Properties

of
an
Object
(ICRA
2012
Petsch
et
al.)

Functional
Properties
of
an
Object

stored
in
Functionality
Map

Each tool used in the
procedure has its own
container describing its
shape, handling properties
etc.
Knowledge
Representation
Functionality map for a specific
procedure describes the way
how the tool was used during
the procedure while moved
between points in the world(Petsch/Burschka IROS2011)

Estimation of non-observable
Scene properties (robot plays)
• Estimation
of
the
Center
of
mass
• Estimation
of
Stiffness

• Estimation
of
Mass
and
Friction
Force
• Estimation
of
Mass
Distribution
simulator
predict act perceive

Sensory Substitution
Flood of sensors required on modern service
robots tocover varying measurement ranges
and operating spaces
• Video Cameras
• Laser-Range Finders
• Accelerometers
• Force Sensors
• Artifical Skin
• Chemical Sensors
…..
The sensor configuration does change also
with the operating conditions, e.g., baseline
dependency of a binocular stereo setup
➔Implementation of missing sensing

accelerometer
camera(s)
gyro
Bag of
Transformations
(physical laws)
Transformation
Graph
physical sensors Sensing modalities
Force
Acceleration
Force distribution
3D Structure
Substitution Framework

Force Sensing without
Force Sensor?

Surface Response for
different types of Objects
hard
soft
deformable

Accelerometer always optimal for
acceleration measurement?
Graph instrumentation

Challenges
for
Perception

• Perception is essential to interact with an unknown and changing
environments
• Perception allows to compensate own uncertainties of the robot
by providing robust reference to the environment
• Level of abstraction of the raw data reduces the amount of
information needed to represent a scene at the cost of increased
processing – an optimum for a given system needs to be found
• System needs to be able to cope not only with static scenes but
also with dynamic changes (actions, failures) in the environment
• Observation of other agents (human other robots) can provide
essential data to fill in the unobservable parameters of the scene

Challenges for Perception in Navigation and Manipulation

More Related Content

Similar to Challenges for Perception in Navigation and Manipulation (20)

Recently uploaded (20)

Challenges for Perception in Navigation and Manipulation