Robotic Vision - Vision for Robotics #IEEE #QLD #CIS #Colloquium
3 likes913 views
The document discusses advancements in robotic vision, focusing on processes such as object recognition, motion detection, and the evolutionary history of vertebrate eyes. It highlights various projects and methodologies for enhancing robotic perception and action, emphasizing the integration of learning approaches and frameworks. The content is presented by Jürgen 'Juxi' Leitner from Queensland University of Technology, outlining both theoretical aspects and practical applications in robotics.
![controlling
spacecraft with
ANNs
[Leitner et al, iSAIRAS 2010]](https://image.slidesharecdn.com/cis-colloqium-ieee-qld-151102020012-lva1-app6891/85/Robotic-Vision-Vision-for-Robotics-IEEE-QLD-CIS-Colloquium-11-320.jpg)
![position, velocity, orientation
thruster
[Leitner et al, iSAIRAS 2010]](https://image.slidesharecdn.com/cis-colloqium-ieee-qld-151102020012-lva1-app6891/85/Robotic-Vision-Vision-for-Robotics-IEEE-QLD-CIS-Colloquium-12-320.jpg)
![trajectory
robustness
[Leitner et al, iSAIRAS 2010]](https://image.slidesharecdn.com/cis-colloqium-ieee-qld-151102020012-lva1-app6891/85/Robotic-Vision-Vision-for-Robotics-IEEE-QLD-CIS-Colloquium-13-320.jpg)
![icVision
icImage* BlueCupFilter::runFilter() {
icImage* node43 = InputImages[4];
icImage* node49 = node43->LocalAvg(15);
icImage* out = node49->threshold(81.532f);
return out;
}
framework](https://image.slidesharecdn.com/cis-colloqium-ieee-qld-151102020012-lva1-app6891/85/Robotic-Vision-Vision-for-Robotics-IEEE-QLD-CIS-Colloquium-22-320.jpg)
![detection
icImage GreenTeaBoxDetector::runFilter() {
icImage node0 = InputImages[6];
icImage node1 = InputImages[1];
icImage node2 = node0.absdiff(node1);
icImage node5 = node2.SmoothBilateral(11);
icImage node12 = InputImages[0];
icImage node16 = node12.Sqrt();
icImage node33 = node16.erode(6);
icImage node34 = node33.log();
icImage node36 = node34.min(node5);
icImage node49 = node36.Normalize();
//cleanup
...
icImage out = node49.threshold(230.7218f);
return out;
}](https://image.slidesharecdn.com/cis-colloqium-ieee-qld-151102020012-lva1-app6891/85/Robotic-Vision-Vision-for-Robotics-IEEE-QLD-CIS-Colloquium-25-320.jpg)
![approach
cgp
[Leitner et al, iSAIRAS 2012]](https://image.slidesharecdn.com/cis-colloqium-ieee-qld-151102020012-lva1-app6891/85/Robotic-Vision-Vision-for-Robotics-IEEE-QLD-CIS-Colloquium-29-320.jpg)
![visual
hand detection
[Leitner et al, CEC 2013]](https://image.slidesharecdn.com/cis-colloqium-ieee-qld-151102020012-lva1-app6891/85/Robotic-Vision-Vision-for-Robotics-IEEE-QLD-CIS-Colloquium-30-320.jpg)
![hands
detecting
[Leitner et al, CEC 2013]](https://image.slidesharecdn.com/cis-colloqium-ieee-qld-151102020012-lva1-app6891/85/Robotic-Vision-Vision-for-Robotics-IEEE-QLD-CIS-Colloquium-31-320.jpg)
![clustering
feature
saliency
map
Autonomous Approach
[Leitner et al, ICDL/EpiRob 2012]](https://image.slidesharecdn.com/cis-colloqium-ieee-qld-151102020012-lva1-app6891/85/Robotic-Vision-Vision-for-Robotics-IEEE-QLD-CIS-Colloquium-33-320.jpg)
![results
comparing
[Leitner et al, ICDL/EpiRob 2012]
features
CGP-‐IP](https://image.slidesharecdn.com/cis-colloqium-ieee-qld-151102020012-lva1-app6891/85/Robotic-Vision-Vision-for-Robotics-IEEE-QLD-CIS-Colloquium-35-320.jpg)
![transferring
spatial perception
[Leitner et al, IROS 2012]](https://image.slidesharecdn.com/cis-colloqium-ieee-qld-151102020012-lva1-app6891/85/Robotic-Vision-Vision-for-Robotics-IEEE-QLD-CIS-Colloquium-39-320.jpg)
![results
ANN-1300
-1200
-1100
-1000
-900
-800
-700
-600
-500
-400
-300
-200
0 200 400 600 800 1000
Z(mm)
Sample Index
-700
-600
-500
-400
-300
-200
-100
0
100
200
300
400
0 200 400 600 800 1000
Y(mm)
Sample Index
Predicted
Expected
100
150
200
250
300
350
400
450
500
550
600
0 200 400 600 800 1000
X(mm)
Sample Index
-1300
-1200
-1100
-1000
-900
-800
-700
-600
-500
-400
-300
-200
0
Z(mm)
[Leitner et al, IJCNN 2013]](https://image.slidesharecdn.com/cis-colloqium-ieee-qld-151102020012-lva1-app6891/85/Robotic-Vision-Vision-for-Robotics-IEEE-QLD-CIS-Colloquium-43-320.jpg)
![MoBeEv2
[Frank, Leitner et al. 2012, 2013]](https://image.slidesharecdn.com/cis-colloqium-ieee-qld-151102020012-lva1-app6891/85/Robotic-Vision-Vision-for-Robotics-IEEE-QLD-CIS-Colloquium-45-320.jpg)
![[Frank, Leitner et al., ICINCO, 2012]](https://image.slidesharecdn.com/cis-colloqium-ieee-qld-151102020012-lva1-app6891/85/Robotic-Vision-Vision-for-Robotics-IEEE-QLD-CIS-Colloquium-46-320.jpg)
![MoBeE
framework [Frank, Leitner et al., ICINCO, 2012]](https://image.slidesharecdn.com/cis-colloqium-ieee-qld-151102020012-lva1-app6891/85/Robotic-Vision-Vision-for-Robotics-IEEE-QLD-CIS-Colloquium-47-320.jpg)
![hand/arm
op-space forcing
CSWorld
CSHand
CSR/CSL
[Leitner et al, in prep]](https://image.slidesharecdn.com/cis-colloqium-ieee-qld-151102020012-lva1-app6891/85/Robotic-Vision-Vision-for-Robotics-IEEE-QLD-CIS-Colloquium-48-320.jpg)
![motion
robot
[Stollenga, Leitner et al, IROS 2013]](https://image.slidesharecdn.com/cis-colloqium-ieee-qld-151102020012-lva1-app6891/85/Robotic-Vision-Vision-for-Robotics-IEEE-QLD-CIS-Colloquium-50-320.jpg)
![http://roboticvision.org/59
deep learning visual control
[Zhang et al, ACRA 2015/ICRA2016]](https://image.slidesharecdn.com/cis-colloqium-ieee-qld-151102020012-lva1-app6891/85/Robotic-Vision-Vision-for-Robotics-IEEE-QLD-CIS-Colloquium-59-320.jpg)
![http://roboticvision.org/60
deep learning visual control
[Zhang et al, ACRA 2015/ICRA2016]](https://image.slidesharecdn.com/cis-colloqium-ieee-qld-151102020012-lva1-app6891/85/Robotic-Vision-Vision-for-Robotics-IEEE-QLD-CIS-Colloquium-60-320.jpg)
Robotic Vision - Vision for Robotics #IEEE #QLD #CIS #Colloquium
- 1. Robotic Vision Jürgen ‘Juxi’ Leitner arc centre of excellence for robotic vision queensland university of technology a vision for robotics j.leitner@qut.edu.au - http://Juxi.net
- 3. http://roboticvision.org/3 recognising objects & stuff recognising places detec4ng mo4on move to see see to move context for seeing seeing for context seeing creates memories memory helps seeing paying a;en4on recognizing humans, their ac4vi4es and intent Seeing
- 4. http://roboticvision.org/4 10 96%eye designs animals haveeyes Nature Reviews | Neuroscience Protostomes Bilateria Ecdysozoa Lophotrochozoa ~580 Mya ~550 Mya ~530 Mya ~500 Mya ~430 Mya Deuterostomes Chordates* Craniates* Vertebrates* Gnathostomes* Arthropods Annelids Molluscs Hemichordates Echinoderms Cephalochordates* Tunicates* Myxiniformes* Petromyzoniformes* Last fossil jawless fish Stages of interest in vertebrate eye evolution Cambrian Mya 600 550 500 450 400 0 21 3 4 5 6 Lampreys Jawed vertebrates Hagfish Lancelets Sea squirts Ocellus Eye patch Figure1|The origin of vertebrates. Theevolutionofjawedvertebratesisillustratedagainstanapproximatetime-scaleof millionsofyearsago(Mya).ThetaxaconsideredinthisReviewareindicatedwithanasteriskandareaccompaniedby schematicsanddiagramsofthe‘eye’region.Theearliestchordates,representedbyextantcephalochordatesandtunicates, arethoughttohaveappearedaround550Mya.Jawlesscraniates(agnathans)werepresentintheearlyCambrian,by525Mya, andatimeof530Myahasbeenindicatedfortheirpresumedfirstappearance.AselaboratedoninBOX 1,thereis considerablecontroversyastowhethermyxiniformes(solelyrepresentedbyextanthagfish)divergedbeforeorafterthe separationoflampreysfromjawedvertebrates(shownasdashedblackandgreylines).Numerouslinesofjawlessfishevolved between500and430Myaago,althoughnonehavesurvivedtothepresentday.Thefirstjawedvertebratearosearound430 Mya,andthislineisrepresentedtodaybycartilagenousfish,bonyfishandtetrapods.Six‘stagesofinterest’invertebrateeye evolutioncorrespondtothetimeintervalsbetweenthedivergenceofimportantsurvivingtaxa.Thisdiagramdoesnot includetheevolutionarychangesthathaveoccurredinthelast400millionyears.Thepresentedtimelineisbasedprimarily onevidencefromthefossilrecord;seeREFS 2,13,15,17,18,144,160–163.Theschematicsaremodified,withpermission,from REF. 11 (1996)OxfordUniversityPress(lancelet,seasquirt,hagfishandlamprey)andREF. 164 (2004)AcademicPress (jawedvertebrate).Theeyeimagesarereproduced,withpermission,fromthefollowingreferences:lancelet,REF. 165 BIODIDAC(1996)UniversityofCaliforniaMuseumofPaleontology;seasquirt,REF. 63 (2006)BlackwellPublishing;hagfish, REF. 166 (2006)AustralianMuseum.LampreyandjawedvertebrateeyeimagesarecourtesyofG.WesthoffandS.P.Collin). CIENCE VOLUME 8 | DECEMBER 2007 | 961
- 10. http://roboticvision.org/ Dalle Molle Institute for AI (IDSIA) 10 Work Juxi Leitner PhD Informatics / Intelligent Systems MSc Space Robotics & Automation BSc Information & Software Engineering Intelligent (Space) Robots European Space Agency (ESA) Erasmus Intelligent Systems Work (Humanoid) Robot Vision Instituto Superior Técnico (IST) Mobility Intelligent Space Systems Laboratory About Me Work Robotic Vision and Actions h"p://Juxi.net Queensland University of Technology (QUT)
- 11. controlling spacecraft with ANNs [Leitner et al, iSAIRAS 2010]
- 12. position, velocity, orientation thruster [Leitner et al, iSAIRAS 2010]
- 13. trajectory robustness [Leitner et al, iSAIRAS 2010]
- 16. perception visual thanks to G. Metta and IIT for this picture
- 19. objects detecting Harding, Leitner, Schmidhuber, 2013 Leitner et al., ICDL 2012, IJARS 2012, BICA 2012, CEC 2013
- 20. embedding domain knowledge + mindilate avg OpenCV func5ons full images
- 21. + mindilateINP INP diff using building blocksOpenCV + dilateINP INP thresh+ mindilate avgINP blur+ mindilate avg normali ze mindilate avginput
- 22. icVision icImage* BlueCupFilter::runFilter() { icImage* node43 = InputImages[4]; icImage* node49 = node43->LocalAvg(15); icImage* out = node49->threshold(81.532f); return out; } framework
- 23. cartesian genetic programming + mindilate avgINP INP INP Harding, Leitner, Schmidhuber, 2013 Leitner et al., ICDL 2012, IJARS 2012, BICA 2012, CEC 2013
- 25. detection icImage GreenTeaBoxDetector::runFilter() { icImage node0 = InputImages[6]; icImage node1 = InputImages[1]; icImage node2 = node0.absdiff(node1); icImage node5 = node2.SmoothBilateral(11); icImage node12 = InputImages[0]; icImage node16 = node12.Sqrt(); icImage node33 = node16.erode(6); icImage node34 = node33.log(); icImage node36 = node34.min(node5); icImage node49 = node36.Normalize(); //cleanup ... icImage out = node49.threshold(230.7218f); return out; }
- 26. detect
- 27. detect
- 28. detection
- 29. approach cgp [Leitner et al, iSAIRAS 2012]
- 30. visual hand detection [Leitner et al, CEC 2013]
- 31. hands detecting [Leitner et al, CEC 2013]
- 33. clustering feature saliency map Autonomous Approach [Leitner et al, ICDL/EpiRob 2012]
- 34. pre segmentation
- 35. results comparing [Leitner et al, ICDL/EpiRob 2012] features CGP-‐IP
- 38. trainingset 9DOF iCub position in the frame 2/6 per eye Cartesian Coordinates
- 39. transferring spatial perception [Leitner et al, IROS 2012]
- 40. setup learning
- 41. trainingset 9DOF iCub position in the frame 2/6 per eye Cartesian Coordinates . . . ~1000
- 43. results ANN-1300 -1200 -1100 -1000 -900 -800 -700 -600 -500 -400 -300 -200 0 200 400 600 800 1000 Z(mm) Sample Index -700 -600 -500 -400 -300 -200 -100 0 100 200 300 400 0 200 400 600 800 1000 Y(mm) Sample Index Predicted Expected 100 150 200 250 300 350 400 450 500 550 600 0 200 400 600 800 1000 X(mm) Sample Index -1300 -1200 -1100 -1000 -900 -800 -700 -600 -500 -400 -300 -200 0 Z(mm) [Leitner et al, IJCNN 2013]
- 45. MoBeEv2 [Frank, Leitner et al. 2012, 2013]
- 46. [Frank, Leitner et al., ICINCO, 2012]
- 47. MoBeE framework [Frank, Leitner et al., ICINCO, 2012]
- 48. hand/arm op-space forcing CSWorld CSHand CSR/CSL [Leitner et al, in prep]
- 49. tele operation
- 50. motion robot [Stollenga, Leitner et al, IROS 2013]
- 54. model
- 57. http://roboticvision.org/57 case #1learning from supervised to robotic-assisted unsupervised learning Autonomous Learning Of Robust Visual Object Detection And Identification On A Humanoid. J. Leitner, P. Chandrashekhariah, S. Harding, M. Frank, G. Spina, A. Förster, J. Triesch, J. Schmidhuber. ICDL/EpiRob 2012.
- 58. http://roboticvision.org/58 case : poking segmentation before & after action
- 59. http://roboticvision.org/59 deep learning visual control [Zhang et al, ACRA 2015/ICRA2016]
- 60. http://roboticvision.org/60 deep learning visual control [Zhang et al, ACRA 2015/ICRA2016]
- 61. http://roboticvision.org/61 conclusions a novel way of object segmentation learning and teaching perception integration action-perception side reactive reaching/grasping improving perception with (inter-)actions learning neuro-controllers
- 63. http://roboticvision.org/ Australian Deep Learning WS 63 http://Juxi.net/workshop/deeplearning-applications-vision-robotics-2015/