From Vision to Actions - Towards Adaptive & Autonomous Humanoid Robots [PhD Defense]
2 likes1,971 views
This document summarizes the work of Jürgen Leitner on developing approaches for visual perception, spatial perception, and object manipulation in humanoid robots. The key points are: 1) Developing a novel approach called CGP-IP for object segmentation using Cartesian genetic programming. 2) Using learning approaches to teach spatial perception to robots from vision input. 3) Integrating approaches for action and perception to enable reactive grasping and improve object detection through manipulation.
![!
icImage* BlueCupFilter::runFilter() {
icImage* node43 = InputImages[4];
icImage* node49 = node43->LocalAvg(15);
!
icImage* out = node49->threshold(81.532f);
return out;
}
icVision
framework](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-11-320.jpg)
![learning
approach
[Matthews, 1975]
TP… true positive, TN .. true negative!
FP… false positive, FN .. false negative](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-14-320.jpg)
![!
icImage GreenTeaBoxDetector::runFilter() {
icImage node0 = InputImages[6];
icImage node1 = InputImages[1];
icImage node2 = node0.absdiff(node1);
icImage node5 = node2.SmoothBilateral(11);
icImage node12 = InputImages[0];
icImage node16 = node12.Sqrt();
icImage node33 = node16.erode(6);
icImage node34 = node33.log();
icImage node36 = node34.min(node5);
icImage node49 = node36.Normalize();
!
//cleanup
...
icImage out = node49.threshold(230.7218f);
return out;
}
detection](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-16-320.jpg)
![cgp
approach
[Leitner et al, iSAIRAS 2012]](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-20-320.jpg)
![[Leitner et al, CEC 2013]
visual
hand detection](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-21-320.jpg)
![detecting
hands
[Leitner et al, CEC 2013]](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-22-320.jpg)
![Autonomous Approach
feature
clustering
saliency
map
[Leitner et al, ICDL/EpiRob 2012]](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-24-320.jpg)
![comparing
results
features
[Leitner et al, ICDL/EpiRob 2012]
CGP-IP](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-26-320.jpg)
![current
[P2,1] * F * [P1,1]' = 0
localisation approaches](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-28-320.jpg)
![[Leitner et al, IROS 2012]
!
transferring
spatial perception](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-31-320.jpg)
![600
550
500
450
400
350
300
250
200
150
-200
-300
-400
-500
-600
Z (mm) Sample Index
-700
-800
400
300
200
100
0
-100
-200
-300
-400
-500
-600
-900
results
-1000
-1100
ANN -1200
-1300
0 200 400 600 800 1000
-700
0 200 400 600 800 1000
Y (mm)
Sample Index
Predicted
Expected
100
0 200 400 600 800 1000
X (mm)
Sample Index
-200
-300
-400
-500
-600
-700
-800
(mm)
Z -900
-1000
-1100
-1200
-1300
0 [Leitner et al, IJCNN 2013]](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-35-320.jpg)
![simplified case
on the table
x"="17.81"−"0.0191"v1"+"0.1527"v4"+"0.1378"v7"+"0.0111"v10
""""""−"0.0296"v11"−"0.1207"v12"
!
y"="1.1242"+"0.1296"v10"+"0.1156"v8"+"0.0170"v0
v0 … left camera x coordinate, GP
results
[Leitner et al, IJARS 2012]
v1 … left camera y coordinate,
v2 … right camera x coordinate,
v3 … right camera y coordinate,
v4-6 … neck
v7-9 … eyes v10-12 … torso](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-36-320.jpg)
![MoBeE
[Frank, Leitner et al., ICINCO, 2012]](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-38-320.jpg)
![MoBeE
framework [Frank, Leitner et al., ICINCO, 2012]](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-39-320.jpg)
![[Frank, Leitner et al., ICINCO, 2012]](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-40-320.jpg)
![robot
motion
[Stollenga, Leitner et al, IROS 2013]](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-41-320.jpg)
![MoBeEv2
[Frank, Leitner et al. 2012, 2013]](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-43-320.jpg)
![op-space forcing
hand/arm
CSWorld
CSHand
CSR/CSL
[Leitner et al, in prep]
-](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-44-320.jpg)
![action
repertoire
[Leitner et al, in prep]](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-45-320.jpg)
![op-space forcing
hand/arm
CSWorld
CSHand
CSR/CSL
[Leitner et al, in prep]
- CSHand’](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-48-320.jpg)
![!
icImage* BlueCupFilter::runFilter() {
icImage* node43 = InputImages[4];
icImage* node49 = node43->LocalAvg(15);
!
icImage* out = node49->threshold(81.532f);
return out;
}
detection](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-57-320.jpg)
![!
icImage* BlueCupFilter::runFilter() {
icImage* node0 = InputImages[4].Exp();
icImage* node5 = InputImages[0];
icImage* node16 = node0->Gabor(-8,14,1,13);
icImage* node17 = InputImages[4]->LocalAvg(6);
icImage* node18 = node16->Laplace(5);
icImage* node19 = node5->Sobel(13,9);
icImage* node24 = node17->Erode(5);
icImage* node28 = node19->Min(node18);
icImage* node29 = node28->Min(node24);
icImage* node41 = node29->LocalAvg(7);
icImage* node49 = node41->LocalMax(7);
!
icImage* out = node49->threshold(68.032f);
return out;
}
resulting
detection](https://image.slidesharecdn.com/leitner-140925030207-phpapp02/85/From-Vision-to-Actions-Towards-Adaptive-Autonomous-Humanoid-Robots-PhD-Defense-58-320.jpg)
From Vision to Actions - Towards Adaptive & Autonomous Humanoid Robots [PhD Defense]
- 1. Jürgen ‘Juxi’ Leitner i'stituto dalle molle di studi sull’intelligenza artificiale idsia / supsi / usi From Vision to Actions towards adaptive & autonomous robots
- 5. overview a novel way of object segmentation (CGPIP) learning and teaching spatial perception integration action-perception side reactive reaching/grasping improving perception with (inter-)actions
- 6. visual ! perception thanks to G. Metta and IIT for this picture
- 7. towards learning object manipulation http://robotics.idsia.ch/
- 9. detecting ! objects Harding, Leitner, Schmidhuber, 2013 Leitner et al., ICDL 2012, IJARS 2012, BICA 2012, CEC 2013
- 10. + dilate INP INP thresh + min dilate avg INP blur + min dil ate avg normali + min di late INP INP diff using ze min dilate avg input OpenCV building blocks
- 11. ! icImage* BlueCupFilter::runFilter() { icImage* node43 = InputImages[4]; icImage* node49 = node43->LocalAvg(15); ! icImage* out = node49->threshold(81.532f); return out; } icVision framework
- 12. + min di late avg OpenCV functions full images embedding domain knowledge
- 13. + min dilate avg INP INP INP cartesian genetic programming Harding, Leitner, Schmidhuber, 2013 Leitner et al., ICDL 2012, IJARS 2012, BICA 2012, CEC 2013
- 14. learning approach [Matthews, 1975] TP… true positive, TN .. true negative! FP… false positive, FN .. false negative
- 16. ! icImage GreenTeaBoxDetector::runFilter() { icImage node0 = InputImages[6]; icImage node1 = InputImages[1]; icImage node2 = node0.absdiff(node1); icImage node5 = node2.SmoothBilateral(11); icImage node12 = InputImages[0]; icImage node16 = node12.Sqrt(); icImage node33 = node16.erode(6); icImage node34 = node33.log(); icImage node36 = node34.min(node5); icImage node49 = node36.Normalize(); ! //cleanup ... icImage out = node49.threshold(230.7218f); return out; } detection
- 17. detect
- 18. detect
- 19. detection
- 20. cgp approach [Leitner et al, iSAIRAS 2012]
- 21. [Leitner et al, CEC 2013] visual hand detection
- 22. detecting hands [Leitner et al, CEC 2013]
- 23. supervised learning approach BUT
- 24. Autonomous Approach feature clustering saliency map [Leitner et al, ICDL/EpiRob 2012]
- 25. pre segmentation
- 26. comparing results features [Leitner et al, ICDL/EpiRob 2012] CGP-IP
- 27. towards learning object manipulation http://robotics.idsia.ch/
- 28. current [P2,1] * F * [P1,1]' = 0 localisation approaches
- 29. ! learning spatial perception
- 30. 9DOF! iCub trainingset frame! the position in per eye 2/6 Cartesian! Coordinates
- 31. [Leitner et al, IROS 2012] ! transferring spatial perception
- 32. learning setup
- 33. 9DOF! iCub trainingset frame! the position in per eye 2/6 Cartesian! Coordinates . ~1000 . .
- 34. spatial perception neural network ! fully connected! ..!! . 9DOF! iCub position in the image! 2/6 per eye ! fully connected! Cartesian! Coordinates ..!! .
- 35. 600 550 500 450 400 350 300 250 200 150 -200 -300 -400 -500 -600 Z (mm) Sample Index -700 -800 400 300 200 100 0 -100 -200 -300 -400 -500 -600 -900 results -1000 -1100 ANN -1200 -1300 0 200 400 600 800 1000 -700 0 200 400 600 800 1000 Y (mm) Sample Index Predicted Expected 100 0 200 400 600 800 1000 X (mm) Sample Index -200 -300 -400 -500 -600 -700 -800 (mm) Z -900 -1000 -1100 -1200 -1300 0 [Leitner et al, IJCNN 2013]
- 36. simplified case on the table x"="17.81"−"0.0191"v1"+"0.1527"v4"+"0.1378"v7"+"0.0111"v10 """"""−"0.0296"v11"−"0.1207"v12" ! y"="1.1242"+"0.1296"v10"+"0.1156"v8"+"0.0170"v0 v0 … left camera x coordinate, GP results [Leitner et al, IJARS 2012] v1 … left camera y coordinate, v2 … right camera x coordinate, v3 … right camera y coordinate, v4-6 … neck v7-9 … eyes v10-12 … torso
- 37. towards learning object manipulation http://robotics.idsia.ch/
- 38. MoBeE [Frank, Leitner et al., ICINCO, 2012]
- 39. MoBeE framework [Frank, Leitner et al., ICINCO, 2012]
- 40. [Frank, Leitner et al., ICINCO, 2012]
- 41. robot motion [Stollenga, Leitner et al, IROS 2013]
- 42. Shakey 2013 Winner gmeonteiorantion
- 43. MoBeEv2 [Frank, Leitner et al. 2012, 2013]
- 44. op-space forcing hand/arm CSWorld CSHand CSR/CSL [Leitner et al, in prep] -
- 45. action repertoire [Leitner et al, in prep]
- 46. towards learning object manipulation http://robotics.idsia.ch/
- 47. tele operation
- 48. op-space forcing hand/arm CSWorld CSHand CSR/CSL [Leitner et al, in prep] - CSHand’
- 49. tele operation
- 51. model
- 53. manipulation for improved perception
- 57. ! icImage* BlueCupFilter::runFilter() { icImage* node43 = InputImages[4]; icImage* node49 = node43->LocalAvg(15); ! icImage* out = node49->threshold(81.532f); return out; } detection
- 58. ! icImage* BlueCupFilter::runFilter() { icImage* node0 = InputImages[4].Exp(); icImage* node5 = InputImages[0]; icImage* node16 = node0->Gabor(-8,14,1,13); icImage* node17 = InputImages[4]->LocalAvg(6); icImage* node18 = node16->Laplace(5); icImage* node19 = node5->Sobel(13,9); icImage* node24 = node17->Erode(5); icImage* node28 = node19->Min(node18); icImage* node29 = node28->Min(node24); icImage* node41 = node29->LocalAvg(7); icImage* node49 = node41->LocalMax(7); ! icImage* out = node49->threshold(68.032f); return out; } resulting detection
- 59. conclusions a novel way of object segmentation learning and teaching spatial perception integration action-perception side reactive reaching/grasping improving perception with (inter-)actions
- 60. S. Harding! M. Frank! A. Förster! M. Stollenga! L. Pape! http://dilbert.com/strips/comic/2013-10-24/ thanks to
- 61. for listening thank you juxi@idsia.ch ! http://Juxi.net/projects
- 62. references 1/2 http://Juxi.net/papers Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub. Jürgen Leitner, Mikhail Frank, Alexander Förster, Jürgen Schmidhuber. ICINCO 2014. ! Improving Robot Vision Models for Object Detection Through Interaction. Jürgen Leitner, Alexander Förster, Jürgen Schmidhuber. WCCI - IJCNN 2014. ! Teleoperation of a 7 DOF Humanoid Robot Arm Using Human Arm Accelerations and EMG Signals.. J Leitner, M Luciw, A Förster, J Schmidhuber. iSAIRAS 2014. ! Curiosity driven reinforcement learning for motion planning on humanoids. Mikhail Frank, Jürgen Leitner, Alexander Förster, Jürgen Schmidhuber. Frontiers in Neurorobotics, 7:25, 2014. ! Task-Relevant Roadmaps: A Framework for Humanoid Motion Planning. M Stollenga, L Pape, M Frank, J Leitner, A Förster, J Schmidhuber. IROS 2013. ! ALife in Humanoids: Developing a Framework to Employ Artificial Life Techniques for High-Level Perception and Cognition Tasks on Humanoid Robots. Jürgen Leitner, Simon Harding, Mikhail Frank, Alexander Förster, Jürgen Schmidhuber. Alife WS @ ECAL 2013. ! Artificial Neural Networks For Spatial Perception: Towards Visual Object Localisation in Humanoid Robots. Jürgen Leitner, Simon Harding, Mikhail Frank, Alexander Förster, Jürgen Schmidhuber. IJCNN 2013. ! Learning Visual Object Detection and Localisation Using icVision. Jürgen Leitner, Simon Harding, Pramod Chandrashekhariah, Mikhail Frank, Alexander Förster, Jochen Triesch, Jürgen Schmidhuber. Biologically Inspired Cognitive Architectures, Vol. 5, 2013. ! Humanoid Learns to Detect Its Own Hands. Jürgen Leitner, Simon Harding, Mikhail Frank, Alexander Förster, Jürgen Schmidhuber. CEC 2013. ! Cartesian Genetic Programming for Image Processing (CGP-IP). Simon Harding, Jürgen Leitner, Jürgen Schmidhuber. In Genetic Programming Theory and Practice X, pp 31-44. ISBN: 978-1-4614-6845-5. Springer, 2013.
- 63. references 2/2 http://Juxi.net/papers Cartesian Genetic Programming for Image Processing and Robot Vision. J. Leitner, S. Harding, A. Förster, J. Schmidhuber. IEEE RAS Summer School on "Robot Vision and Applications". Santiago, Chile. December 2012. ! Learning Spatial Object Localization from Vision on a Humanoid Robot. Jürgen Leitner, Simon Harding, Mikhail Frank, Alexander Förster, Jürgen Schmidhuber. International Journal of Advanced Robotic Systems, Vol. 9, 2012. ! Autonomous Learning Of Robust Visual Object Detection And Identification On A Humanoid. J. Leitner, P. Chandrashekhariah, S. Harding, M. Frank, G. Spina, A. Förster, J. Triesch, J. Schmidhuber. ICDL/EpiRob 2012. Paper of Excellence Award ! An Integrated, Modular Framework for Computer Vision and Cognitive Robotics Research (icVision). Jürgen Leitner, Simon Harding, Mikhail Frank, Alexander Förster, Jürgen Schmidhuber. BICA 2012. ! Transferring Spatial Perception Between Robots Operating In A Shared Workspace. J Leitner, S Harding, M Frank, A Förster, J Schmidhuber. IROS 2012. ! Towards Spatial Perception: Learning to Locate Objects From Vision. J Leitner, S Harding, M Frank, A Förster, J Schmidhuber. RobotDoC-PhD 2012. ! Mars Terrain Image Classification using Cartesian Genetic Programming. J Leitner, S Harding, A Förster, J Schmidhuber. i-SAIRAS 2012. ! The Modular Behavioral Environment for Humanoids & other Robots (MoBeE). Mikhail Frank, Jürgen Leitner, Marijn Stollenga, Gregor Kaufmann, Simon Harding, Alexander Förster, Jürgen Schmidhuber. ICINCO 2012. ! MT-CGP: Mixed Type Cartesian Genetic Programming. S. Harding, V. Graziano, J. Leitner, Jürgen Schmidhuber. GECCO 2012. ! icVision: A Modular Vision System for Cognitive Robotics Research. J. Leitner, S. Harding, M. Frank, A. Förster, J. Schmidhuber. CogSys 2012. ! Artificial Curiosity for Autonomous Space Exploration. Vincent Graziano, Tobias Glasmachers, Tom Schaul, Leo Pape, Giuseppe Cuccu, Jürgen Leitner and Jürgen Schmidhuber. Acta Futura, 4, pp. 41-52, 2011.