SlideShare a Scribd company logo
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 1
Humanoid Robot Control of
Complex Postural
Tasks based on Learning from
Demonstration
Miguel González-Fierro Palacios
Advisers:
Prof. Carlos Balaguer
Universidad Carlos III
Dr. Thrishantha Nanayakkara
King’s College London
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 2
Behavior transfer & postural control
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 3
High level order
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 4
Introduction
Objectives
1. Postural planning and
control of a single skill
2. Postural planning and
control of consecutive
skills
3. Execution of high level
order
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 5
Introduction
Outline
HIGH LEVEL
ORDER
Environment analysis
Postural planning
Postural control
SINGLE
SKILL
BASIC
CONCEPTS
SEQUENTIAL
SKILLS
POSTURAL
CONTROL
2
3
4
5
6Sequential imitation
Sequential innovation
Behavior selection
Model selection
Fractional control
PID vs
Imitation learning
Skill innovation
Behavior representation
Humanoid models
Gait
Dance mimicking
 
PI D
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 6
2. BASIC REPRESENTATIONS FOR
POSTURAL CONTROL IN
HUMANOIDS
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 7
Basic Representations
Humanoid robot models
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 8
Basic Representations
Zero Moment Point
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 9
Basic Representations
3D Linear Inverted Pendulum Model
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 10
Basic Representations
Cart Table Model
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 11
Basic Representations
Cart Table Model
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 12
Basic Representations
Gait Simulation
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 13
Basic Representations
Humanoid Gait
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 14
Basic Representations
Robot Mimicking
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 15
Basic Representations
Dance Routine
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 16
Basic Representations
Video
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 17
3. IMITATION LEARNING AND SKILL
INNOVATION IN HUMANOIDS
THROUGH REWARD TEMPLATES
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 18
Postural Planning of a Single Skill
Learning in Humans
How do they learn?
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 19
Postural Planning of a Single Skill
Imitation in Robots
1. Determine what
to imitate
2. Establish a
metrics for
imitation
3. Map between
dissimilar bodies
4. Compute the
control
commands
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 20
Postural Planning of a Single Skill
Imitation and Innovation of a Skill
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 21
Postural Planning of a Single Skill
Data Acquisition
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 22
Postural Planning of a Single Skill
ZMP and Torque Data
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 23
Postural Planning of a Single Skill
Reward Function
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 24
Postural Planning of a Single Skill
Markov Chains and Transition Matrix
Markov property:
Markov transition matrix:
Polynomial reward function
Gaussian reward function
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 25
Postural Planning of a Single Skill
Behavior Strategy Asumption
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 26
Postural Planning of a Single Skill
Imitation and Innovation Reward
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 27
Postural Planning of a Single Skill
Robot ZMP
ZMP imitation ZMP innovation
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 28
Postural Planning of a Single Skill
Robot Torque
Torque imitation Torque innovation
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 29
Postural Planning of a Single Skill
Experiments
Imitation Innovation
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 30
Postural Planning of a Single Skill
Video
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 31
4. LEARNING AND IMPROVING A
SEQUENCE OF GOAL DIRECTED
SKILLS
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 32
Postural Planning of Sequential Skills
Planning Sequential Skills
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 33
Postural Planning of Sequential Skills
Overview of Sequential Imitation
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 34
Postural Planning of Sequential Skills
Overview of Sequential Innovation
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 35
Postural Planning of Sequential Skills
Policy Definition
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 36
Postural Planning of Sequential Skills
Encoding Demonstrations
GMM GMRData
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 37
Postural Planning of Sequential Skills
Behavior Definition
B1: Approaching the door B2: Grasping the knob
B3: Pulling the door B4: Releasing the knob
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 38
Postural Planning of Sequential Skills
Behavior Selector
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 39
Postural Planning of Sequential Skills
Contex-Based Reward Functions
Basis Reward Function:
Behavior based Reward Function:
Reward Profiles
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 40
Postural Planning of Sequential Skills
Humanoid Opening a Door
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 41
Postural Planning of Sequential Skills
Video
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 42
5. ROBUST CONTROL OF HUMANOID
MODELS THROUGH FRACTIONAL
CALCULUS
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 43
Postural Control
Humanoid Models Discussion
VS
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 44
Postural Control
Our Approach
Reduced model
+
Fractional Order Controller
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 45
Postural Control
Fractional Order Controller
Cope with mismatches in the model
Robust against disturbances
Approximation as a rational function
+1Kg
+1Kg
+1Kg
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 46
Postural Control
Model Identification
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 47
Postural Control
State Space Equations
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 48
Postural Control
Control System
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 49
Postural Control
PID vs Fractional PID
Performance comparison between nominal and overloaded system (+3Kg)
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 50
6. CONTROL OF HUMANOID ROBOTS
EXECUTING COMPLEX TASKS
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 51
High Level Architecture
Architecture Overview
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 52
High Level Architecture
Floor and Obstacles Estimation
Normals Estimation Plane Estimation Obstacle Clustering
Covariance matrix
Eigenvalues/Eigenvectors
Normals direction
RANSAC
Inliers: 70% (Plane)
Outliers: 30% (Obstacles)
Euclidean Clustering using
K-mean and KD-trees
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 53
High Level Architecture
Environment Analysis
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 54
High Level Architecture
Environment Model
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 55
High Level Architecture
Video
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 56
High Level Architecture
Global Path Planning
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 57
High Level Architecture
Video
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 58
High Level Architecture
Postural Skill Generator
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 59
High Level Architecture
Postural Motion Planning
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 60
High Level Architecture
Whole Body Postural Control
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 61
High Level Architecture
Correction of Disturbances
distF
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 62
7. CONCLUSIONS AND
FUTURE WORKS
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 63
Conclusions
Summary
HIGH LEVEL
ORDER
Environment analysis
Postural planning
Postural control
SINGLE
SKILL
BASIC
CONCEPTS
SEQUENTIAL
SKILLS
POSTURAL
CONTROL
2
3
4
5
6Sequential imitation
Sequential innovation
Behavior selection
Model selection
Fractional control
PID vs
Imitation learning
Skill innovation
Behavior representation
Humanoid models
Gait
Dance mimicking
 
PI D
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 64
Conclusions
Key Contributions
Postural Motion Planning and Control
Reward trajectory as behavior metrics
Reward Transition Matrix as behavior representation
Postural planning and control architecture
Learning from Demonstration
Single skill transference through multimodal reward profile
Skill transference through Reward Transition Matrix
Imitation extended to a set of sequential skills
Skill innovation
Maximization of positive difference of imitation reward
Application to a set of sequential skills
Modelling and Control
Reduced model of the humanoid controlled by fractional controller
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 65
Conclusions
Future Works
New approaches on goal emulation and intention understanding
Selection of reward profile through machine learning techniques
Behavior transference between robot-robot or human-human
Implementation of fractional controller in the real humanoid
Decision module to autonomously generate the postural path
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 66
Conclusions
List of Publications
Journal papers
- González-Fierro, M., Balaguer, C., Swann, N., and Nanayakkara, T. (2014). Full-Body Postural Control
of a Humanoid Robot with Both Imitation Learning and Skill Innovation. International Journal of
Humanoid Robotics, 11(2):1450012.
- Monje, C.A., Pierro, P., Ramos, T., González-Fierro, M., Balaguer, C. (2013). Modelling and Simulation
of the Humanoid Robot HOAP-3 in the OpenHRP3 Platform Cybernetics and Systems, 44(8):663-680.
- Bueno, J. G., González-Fierro, M., Moreno, L., and Balaguer, C. (2013). Facial Emotion Recognition
and Adaptative Postural Reaction by a Humanoid based on Neural Evolution International Journal of
Advanced Computer Science, 3(10):481-493.
- González-Fierro, M., Hernández, D., Nanayakkara, T., and Balaguer, C. (2014).Behavior Sequencing
Based on Demonstrations-a Case of a Humanoid Opening a Door while Walking. submitted to Advanced
Robotics.
- González-Fierro, M., Monje, C. A., and Balaguer, C. (2014). Fractional Control of a Humanoid Robot
Reduced Model with Model Disturbances. submitted to Cybernetics and Systems
Conference/Symposium papers
- González-Fierro, M., Balaguer, C., Swann, N., and Nanayakkara, T. (2013). A Humanoid Robot
Standing Up Through Learning from Demonstration Using a Multimodal Reward Function. In IEEERAS
International Conference on Humanoid Robots, 2013. Humanoids 2013.
- González-Fierro, M., Bueno, J., Balaguer, C., and Moreno, L. (2013). A Complete 3D Perception and
Path Planning Architecture for a Humanoid. Robocity2030 11th Workshop: Robots Sociales, pages 167-184.
- González-Fierro, M., Maldonado, M. A., Víctores, J. G., Morante, S., and Balaguer, C. (2013).
Object Tagging for Human-Robot Interaction by Recolorization using Gaussian Mixture Models. In
Proceedings of Robocity2030 12th Workshop: Robótica Cognitiva, pages 67-76.
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 67
Conclusions
List of Publications
- González-Fierro, M., Monje, C., González, V., and Balaguer, C. (2013). Evolutionary Fractional Order
Control of a Humanoid Robot Modeled as a Triple Inverted Pendulum. In Proceedings of Robocity2030
11th Workshop: Robots Sociales, pages 245-263.
- González-Fierro, M., Monje, C. A., and Balaguer, C. (2013). Robust Control of a Reduced Humanoid
Robot Model using Genetic Algorithms and Fractional Calculus. In Mathematical Methods in Engineering
International Conference MME2013, pages 183-194.
- Bueno, J., Martín, A., González-Fierro, M., Moreno, L., and Balaguer, C. (2013). Distinguishing
between Similar Objects based on Geometrical Features in 3D Perception. In Proceedings of Robocity2030
12th Workshop: Robótica Cognitiva, pages 77-92.
- Víctores, J. G., Morante, S., González-Fierro, M., and Balaguer, C. (2013). Augmented Reality and
Social Interaction platform through Multirobot Design. In Proceedings of Robocity2030 11th Workshop:
Robots Sociales, pages 131-143.
- González-Fierro, M., Hernández, D., Pierro, P., and Balaguer, C. (2012). Dynamic Modelling of
Humanoid Robots Using Spatial Algebra. In XXXIII Jornadas de Automática. CEA.
- Pierro, P., Hernández, D., Herrero, D., González-Fierro, M., and Balaguer, C. (2012). Perception
System for Working with Humanoid Robots in Unstructured Collaborative Scenarios. In Proceedings of
the 2012 International IEEE Intelligent Vehicles Symposium. Workshops V Perception in Robotics. IEEE.
- Bueno, J. G., González-Fierro, M., L.Moreno, and C.Balaguer (2012). Facial Gesture Recognition using
Active Appearance Models based on Neural Evolution. In 2012 International Conference on Human-Robot
Interaction (HRI 2012), pages 133-134. IEEE.
- Monje, C. A., Pierro, P., Ramos, T., González-Fierro, M., and Balaguer, C. (2011). Modelling and
Simulation of the Humanoid Robot HOAP-3 in the OpenHRP3 Platform. In Proceedings of Robot 2011.
Workshop Robots Humanoides.
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 68
Conclusions
List of Publications
- Bueno, J. G., González-Fierro, M., Moreno, L., and Balaguer, C. (2011). Facial Gesture Recognition
and Postural Interaction using Neural Evolution Algorithm and Active Appearance Models. In Proceedings
of Robocity2030 9th Workshop: Robots colaborativos e interacción humano-robot, pages 145-159.
- González-Fierro, M., Jardón, A., Martínez de la Casa, S., Stoelen, M. F., Víctores, J. G., Balaguer, C.
(2010). Educational Initiatives Related with the CEABOT Contest. Proceedings of SIMPAR, pp 649-658.
- Mateo, A. P., González-Fierro, M., Hernández, D., Pierro, P., and Balaguer, C. (2010). Robust Real
Time Stabilization: Estabilización de la Imagen con Aplicación en el Robot Humanoide HOAP-3. In
Proceedings of Robocity2030 7th Workshop: Visión en Robótica.
- Peña, A., Hernández, D., González-Fierro, M., Pierro, P., and Balaguer, C. (2010). Sistema de
Visión del Humanoide HOAP-3 para la Detección e Identificación de Objetos Mediante Librerías
OpenCV. In Proceedings of Robocity2030 7th Workshop: Visión en Robótica.
- Pierro, P., González-Fierro, M., and Balaguer, C. (2009). El Proyecto Europeo ROBOT@CWE:
Advanced Robotic Systems in Future Collaborative Working Environments. In Proceedings of Robot 2009.
II Workshop de Robotica (ROBOT 2009).
- Pierro, P., Hernández, D., González-Fierro, M., Blasi, L., Milani, A., and Balaguer, C. (2009). A
Human-Humanoid Interface for Collaborative Tasks. In Proceedings on the Second workshop for young
Researchers on Human-friendly robotics, Sestri Levante, Italy.
- Pierro, P., Hernández, D., González-Fierro, M., Blasi, L., Milani, A., and Balaguer, C. (2009).
Humanoid Teleoperation System for Space Environments. In Advanced Robotics, 2009. ICAR 2009.
International Conference on, pages 1-6. IEEE.
- González-Fierro, M., Pierro, P., Jardón, A., Herrero, D., and Balaguer, C. (2009). Realización de Tareas
Colaborativas entre Robots Humanoides. Experimentación con dos Robots Robonova. In Proceedings
of Robocity2030 5th Workshop: Cooperación en Robótica.
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 69
One more thing…
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 70
Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 71
Humanoid Robot Control of
Complex Postural
Tasks based on Learning from
Demonstration
Miguel González-Fierro Palacios
Advisers:
Prof. Carlos Balaguer
Universidad Carlos III
Dr. Thrishantha Nanayakkara
King’s College London

More Related Content

PPTX
Los retos de la inteligencia artificial en la sociedad actual
PDF
Knowledge Graph Recommendation Systems For COVID-19
PPTX
Best practices in coding for beginners
PDF
Distributed training of Deep Learning Models
PPTX
Running Intelligent Applications inside a Database: Deep Learning with Python...
PPTX
Deep Learning for Sales Professionals
PPTX
Deep Learning for Lung Cancer Detection
PPTX
Mastering Computer Vision Problems with State-of-the-art Deep Learning
Los retos de la inteligencia artificial en la sociedad actual
Knowledge Graph Recommendation Systems For COVID-19
Best practices in coding for beginners
Distributed training of Deep Learning Models
Running Intelligent Applications inside a Database: Deep Learning with Python...
Deep Learning for Sales Professionals
Deep Learning for Lung Cancer Detection
Mastering Computer Vision Problems with State-of-the-art Deep Learning

Recently uploaded (20)

PPTX
microscope-Lecturecjchchchchcuvuvhc.pptx
PPTX
cpcsea ppt.pptxssssssssssssssjjdjdndndddd
PPTX
Microbiology with diagram medical studies .pptx
PPTX
2. Earth - The Living Planet earth and life
PPT
The World of Physical Science, • Labs: Safety Simulation, Measurement Practice
PPTX
neck nodes and dissection types and lymph nodes levels
PPTX
DRUG THERAPY FOR SHOCK gjjjgfhhhhh.pptx.
PPTX
Derivatives of integument scales, beaks, horns,.pptx
PPTX
EPIDURAL ANESTHESIA ANATOMY AND PHYSIOLOGY.pptx
PDF
Formation of Supersonic Turbulence in the Primordial Star-forming Cloud
PPTX
Comparative Structure of Integument in Vertebrates.pptx
PDF
ELS_Q1_Module-11_Formation-of-Rock-Layers_v2.pdf
PDF
SEHH2274 Organic Chemistry Notes 1 Structure and Bonding.pdf
PPT
POSITIONING IN OPERATION THEATRE ROOM.ppt
PPTX
famous lake in india and its disturibution and importance
PDF
Sciences of Europe No 170 (2025)
PPTX
G5Q1W8 PPT SCIENCE.pptx 2025-2026 GRADE 5
PPTX
2. Earth - The Living Planet Module 2ELS
PPTX
Protein & Amino Acid Structures Levels of protein structure (primary, seconda...
PPTX
TOTAL hIP ARTHROPLASTY Presentation.pptx
microscope-Lecturecjchchchchcuvuvhc.pptx
cpcsea ppt.pptxssssssssssssssjjdjdndndddd
Microbiology with diagram medical studies .pptx
2. Earth - The Living Planet earth and life
The World of Physical Science, • Labs: Safety Simulation, Measurement Practice
neck nodes and dissection types and lymph nodes levels
DRUG THERAPY FOR SHOCK gjjjgfhhhhh.pptx.
Derivatives of integument scales, beaks, horns,.pptx
EPIDURAL ANESTHESIA ANATOMY AND PHYSIOLOGY.pptx
Formation of Supersonic Turbulence in the Primordial Star-forming Cloud
Comparative Structure of Integument in Vertebrates.pptx
ELS_Q1_Module-11_Formation-of-Rock-Layers_v2.pdf
SEHH2274 Organic Chemistry Notes 1 Structure and Bonding.pdf
POSITIONING IN OPERATION THEATRE ROOM.ppt
famous lake in india and its disturibution and importance
Sciences of Europe No 170 (2025)
G5Q1W8 PPT SCIENCE.pptx 2025-2026 GRADE 5
2. Earth - The Living Planet Module 2ELS
Protein & Amino Acid Structures Levels of protein structure (primary, seconda...
TOTAL hIP ARTHROPLASTY Presentation.pptx

Thesis dissertation: Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration

  • 1. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 1 Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration Miguel González-Fierro Palacios Advisers: Prof. Carlos Balaguer Universidad Carlos III Dr. Thrishantha Nanayakkara King’s College London
  • 2. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 2 Behavior transfer & postural control
  • 3. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 3 High level order
  • 4. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 4 Introduction Objectives 1. Postural planning and control of a single skill 2. Postural planning and control of consecutive skills 3. Execution of high level order
  • 5. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 5 Introduction Outline HIGH LEVEL ORDER Environment analysis Postural planning Postural control SINGLE SKILL BASIC CONCEPTS SEQUENTIAL SKILLS POSTURAL CONTROL 2 3 4 5 6Sequential imitation Sequential innovation Behavior selection Model selection Fractional control PID vs Imitation learning Skill innovation Behavior representation Humanoid models Gait Dance mimicking   PI D
  • 6. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 6 2. BASIC REPRESENTATIONS FOR POSTURAL CONTROL IN HUMANOIDS
  • 7. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 7 Basic Representations Humanoid robot models
  • 8. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 8 Basic Representations Zero Moment Point
  • 9. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 9 Basic Representations 3D Linear Inverted Pendulum Model
  • 10. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 10 Basic Representations Cart Table Model
  • 11. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 11 Basic Representations Cart Table Model
  • 12. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 12 Basic Representations Gait Simulation
  • 13. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 13 Basic Representations Humanoid Gait
  • 14. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 14 Basic Representations Robot Mimicking
  • 15. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 15 Basic Representations Dance Routine
  • 16. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 16 Basic Representations Video
  • 17. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 17 3. IMITATION LEARNING AND SKILL INNOVATION IN HUMANOIDS THROUGH REWARD TEMPLATES
  • 18. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 18 Postural Planning of a Single Skill Learning in Humans How do they learn?
  • 19. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 19 Postural Planning of a Single Skill Imitation in Robots 1. Determine what to imitate 2. Establish a metrics for imitation 3. Map between dissimilar bodies 4. Compute the control commands
  • 20. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 20 Postural Planning of a Single Skill Imitation and Innovation of a Skill
  • 21. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 21 Postural Planning of a Single Skill Data Acquisition
  • 22. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 22 Postural Planning of a Single Skill ZMP and Torque Data
  • 23. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 23 Postural Planning of a Single Skill Reward Function
  • 24. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 24 Postural Planning of a Single Skill Markov Chains and Transition Matrix Markov property: Markov transition matrix: Polynomial reward function Gaussian reward function
  • 25. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 25 Postural Planning of a Single Skill Behavior Strategy Asumption
  • 26. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 26 Postural Planning of a Single Skill Imitation and Innovation Reward
  • 27. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 27 Postural Planning of a Single Skill Robot ZMP ZMP imitation ZMP innovation
  • 28. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 28 Postural Planning of a Single Skill Robot Torque Torque imitation Torque innovation
  • 29. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 29 Postural Planning of a Single Skill Experiments Imitation Innovation
  • 30. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 30 Postural Planning of a Single Skill Video
  • 31. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 31 4. LEARNING AND IMPROVING A SEQUENCE OF GOAL DIRECTED SKILLS
  • 32. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 32 Postural Planning of Sequential Skills Planning Sequential Skills
  • 33. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 33 Postural Planning of Sequential Skills Overview of Sequential Imitation
  • 34. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 34 Postural Planning of Sequential Skills Overview of Sequential Innovation
  • 35. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 35 Postural Planning of Sequential Skills Policy Definition
  • 36. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 36 Postural Planning of Sequential Skills Encoding Demonstrations GMM GMRData
  • 37. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 37 Postural Planning of Sequential Skills Behavior Definition B1: Approaching the door B2: Grasping the knob B3: Pulling the door B4: Releasing the knob
  • 38. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 38 Postural Planning of Sequential Skills Behavior Selector
  • 39. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 39 Postural Planning of Sequential Skills Contex-Based Reward Functions Basis Reward Function: Behavior based Reward Function: Reward Profiles
  • 40. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 40 Postural Planning of Sequential Skills Humanoid Opening a Door
  • 41. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 41 Postural Planning of Sequential Skills Video
  • 42. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 42 5. ROBUST CONTROL OF HUMANOID MODELS THROUGH FRACTIONAL CALCULUS
  • 43. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 43 Postural Control Humanoid Models Discussion VS
  • 44. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 44 Postural Control Our Approach Reduced model + Fractional Order Controller
  • 45. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 45 Postural Control Fractional Order Controller Cope with mismatches in the model Robust against disturbances Approximation as a rational function +1Kg +1Kg +1Kg
  • 46. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 46 Postural Control Model Identification
  • 47. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 47 Postural Control State Space Equations
  • 48. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 48 Postural Control Control System
  • 49. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 49 Postural Control PID vs Fractional PID Performance comparison between nominal and overloaded system (+3Kg)
  • 50. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 50 6. CONTROL OF HUMANOID ROBOTS EXECUTING COMPLEX TASKS
  • 51. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 51 High Level Architecture Architecture Overview
  • 52. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 52 High Level Architecture Floor and Obstacles Estimation Normals Estimation Plane Estimation Obstacle Clustering Covariance matrix Eigenvalues/Eigenvectors Normals direction RANSAC Inliers: 70% (Plane) Outliers: 30% (Obstacles) Euclidean Clustering using K-mean and KD-trees
  • 53. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 53 High Level Architecture Environment Analysis
  • 54. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 54 High Level Architecture Environment Model
  • 55. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 55 High Level Architecture Video
  • 56. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 56 High Level Architecture Global Path Planning
  • 57. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 57 High Level Architecture Video
  • 58. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 58 High Level Architecture Postural Skill Generator
  • 59. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 59 High Level Architecture Postural Motion Planning
  • 60. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 60 High Level Architecture Whole Body Postural Control
  • 61. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 61 High Level Architecture Correction of Disturbances distF
  • 62. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 62 7. CONCLUSIONS AND FUTURE WORKS
  • 63. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 63 Conclusions Summary HIGH LEVEL ORDER Environment analysis Postural planning Postural control SINGLE SKILL BASIC CONCEPTS SEQUENTIAL SKILLS POSTURAL CONTROL 2 3 4 5 6Sequential imitation Sequential innovation Behavior selection Model selection Fractional control PID vs Imitation learning Skill innovation Behavior representation Humanoid models Gait Dance mimicking   PI D
  • 64. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 64 Conclusions Key Contributions Postural Motion Planning and Control Reward trajectory as behavior metrics Reward Transition Matrix as behavior representation Postural planning and control architecture Learning from Demonstration Single skill transference through multimodal reward profile Skill transference through Reward Transition Matrix Imitation extended to a set of sequential skills Skill innovation Maximization of positive difference of imitation reward Application to a set of sequential skills Modelling and Control Reduced model of the humanoid controlled by fractional controller
  • 65. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 65 Conclusions Future Works New approaches on goal emulation and intention understanding Selection of reward profile through machine learning techniques Behavior transference between robot-robot or human-human Implementation of fractional controller in the real humanoid Decision module to autonomously generate the postural path
  • 66. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 66 Conclusions List of Publications Journal papers - González-Fierro, M., Balaguer, C., Swann, N., and Nanayakkara, T. (2014). Full-Body Postural Control of a Humanoid Robot with Both Imitation Learning and Skill Innovation. International Journal of Humanoid Robotics, 11(2):1450012. - Monje, C.A., Pierro, P., Ramos, T., González-Fierro, M., Balaguer, C. (2013). Modelling and Simulation of the Humanoid Robot HOAP-3 in the OpenHRP3 Platform Cybernetics and Systems, 44(8):663-680. - Bueno, J. G., González-Fierro, M., Moreno, L., and Balaguer, C. (2013). Facial Emotion Recognition and Adaptative Postural Reaction by a Humanoid based on Neural Evolution International Journal of Advanced Computer Science, 3(10):481-493. - González-Fierro, M., Hernández, D., Nanayakkara, T., and Balaguer, C. (2014).Behavior Sequencing Based on Demonstrations-a Case of a Humanoid Opening a Door while Walking. submitted to Advanced Robotics. - González-Fierro, M., Monje, C. A., and Balaguer, C. (2014). Fractional Control of a Humanoid Robot Reduced Model with Model Disturbances. submitted to Cybernetics and Systems Conference/Symposium papers - González-Fierro, M., Balaguer, C., Swann, N., and Nanayakkara, T. (2013). A Humanoid Robot Standing Up Through Learning from Demonstration Using a Multimodal Reward Function. In IEEERAS International Conference on Humanoid Robots, 2013. Humanoids 2013. - González-Fierro, M., Bueno, J., Balaguer, C., and Moreno, L. (2013). A Complete 3D Perception and Path Planning Architecture for a Humanoid. Robocity2030 11th Workshop: Robots Sociales, pages 167-184. - González-Fierro, M., Maldonado, M. A., Víctores, J. G., Morante, S., and Balaguer, C. (2013). Object Tagging for Human-Robot Interaction by Recolorization using Gaussian Mixture Models. In Proceedings of Robocity2030 12th Workshop: Robótica Cognitiva, pages 67-76.
  • 67. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 67 Conclusions List of Publications - González-Fierro, M., Monje, C., González, V., and Balaguer, C. (2013). Evolutionary Fractional Order Control of a Humanoid Robot Modeled as a Triple Inverted Pendulum. In Proceedings of Robocity2030 11th Workshop: Robots Sociales, pages 245-263. - González-Fierro, M., Monje, C. A., and Balaguer, C. (2013). Robust Control of a Reduced Humanoid Robot Model using Genetic Algorithms and Fractional Calculus. In Mathematical Methods in Engineering International Conference MME2013, pages 183-194. - Bueno, J., Martín, A., González-Fierro, M., Moreno, L., and Balaguer, C. (2013). Distinguishing between Similar Objects based on Geometrical Features in 3D Perception. In Proceedings of Robocity2030 12th Workshop: Robótica Cognitiva, pages 77-92. - Víctores, J. G., Morante, S., González-Fierro, M., and Balaguer, C. (2013). Augmented Reality and Social Interaction platform through Multirobot Design. In Proceedings of Robocity2030 11th Workshop: Robots Sociales, pages 131-143. - González-Fierro, M., Hernández, D., Pierro, P., and Balaguer, C. (2012). Dynamic Modelling of Humanoid Robots Using Spatial Algebra. In XXXIII Jornadas de Automática. CEA. - Pierro, P., Hernández, D., Herrero, D., González-Fierro, M., and Balaguer, C. (2012). Perception System for Working with Humanoid Robots in Unstructured Collaborative Scenarios. In Proceedings of the 2012 International IEEE Intelligent Vehicles Symposium. Workshops V Perception in Robotics. IEEE. - Bueno, J. G., González-Fierro, M., L.Moreno, and C.Balaguer (2012). Facial Gesture Recognition using Active Appearance Models based on Neural Evolution. In 2012 International Conference on Human-Robot Interaction (HRI 2012), pages 133-134. IEEE. - Monje, C. A., Pierro, P., Ramos, T., González-Fierro, M., and Balaguer, C. (2011). Modelling and Simulation of the Humanoid Robot HOAP-3 in the OpenHRP3 Platform. In Proceedings of Robot 2011. Workshop Robots Humanoides.
  • 68. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 68 Conclusions List of Publications - Bueno, J. G., González-Fierro, M., Moreno, L., and Balaguer, C. (2011). Facial Gesture Recognition and Postural Interaction using Neural Evolution Algorithm and Active Appearance Models. In Proceedings of Robocity2030 9th Workshop: Robots colaborativos e interacción humano-robot, pages 145-159. - González-Fierro, M., Jardón, A., Martínez de la Casa, S., Stoelen, M. F., Víctores, J. G., Balaguer, C. (2010). Educational Initiatives Related with the CEABOT Contest. Proceedings of SIMPAR, pp 649-658. - Mateo, A. P., González-Fierro, M., Hernández, D., Pierro, P., and Balaguer, C. (2010). Robust Real Time Stabilization: Estabilización de la Imagen con Aplicación en el Robot Humanoide HOAP-3. In Proceedings of Robocity2030 7th Workshop: Visión en Robótica. - Peña, A., Hernández, D., González-Fierro, M., Pierro, P., and Balaguer, C. (2010). Sistema de Visión del Humanoide HOAP-3 para la Detección e Identificación de Objetos Mediante Librerías OpenCV. In Proceedings of Robocity2030 7th Workshop: Visión en Robótica. - Pierro, P., González-Fierro, M., and Balaguer, C. (2009). El Proyecto Europeo ROBOT@CWE: Advanced Robotic Systems in Future Collaborative Working Environments. In Proceedings of Robot 2009. II Workshop de Robotica (ROBOT 2009). - Pierro, P., Hernández, D., González-Fierro, M., Blasi, L., Milani, A., and Balaguer, C. (2009). A Human-Humanoid Interface for Collaborative Tasks. In Proceedings on the Second workshop for young Researchers on Human-friendly robotics, Sestri Levante, Italy. - Pierro, P., Hernández, D., González-Fierro, M., Blasi, L., Milani, A., and Balaguer, C. (2009). Humanoid Teleoperation System for Space Environments. In Advanced Robotics, 2009. ICAR 2009. International Conference on, pages 1-6. IEEE. - González-Fierro, M., Pierro, P., Jardón, A., Herrero, D., and Balaguer, C. (2009). Realización de Tareas Colaborativas entre Robots Humanoides. Experimentación con dos Robots Robonova. In Proceedings of Robocity2030 5th Workshop: Cooperación en Robótica.
  • 69. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 69 One more thing…
  • 70. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 70
  • 71. Miguel González-Fierro | Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration 71 Humanoid Robot Control of Complex Postural Tasks based on Learning from Demonstration Miguel González-Fierro Palacios Advisers: Prof. Carlos Balaguer Universidad Carlos III Dr. Thrishantha Nanayakkara King’s College London