How to test when robots become part of your process? Workshop robotesting agile testing days 2017
1 like134 views
The document outlines the agenda for a robotesting workshop held on November 15, 2017, during Agile Testing Days. It covers topics such as robotics, machine intelligence, and cognitive computing, with a focus on hands-on exploration of testing chatbots and robots using collaborative techniques. The workshop aims to impart knowledge on quality attributes and ethical considerations related to robot testing.
How to test when robots become part of your process? Workshop robotesting agile testing days 2017
- 1. Robotesting Workshop Agile Testing Days 15 November 2017
- 2. How to test when robots become part of your process? Agile Testing Days 15 November 2017
- 3. | Agenda 1. Welcome & introductions 10 minutes 2. Intro: What is “robotics”, “machine intelligence”, “cognitive computing” etc.? 10 minutes 3. Workshop: 2 sprints 10 minutes intro and 2 x 20 minutes execution 4. Wrap-up: debriefing of your results and what you learned on robotesting 20 minutes; 1 minute per team and 10 minutes overall © Sogeti 2017 4
- 4. | Welcome and introductions Workshop Robotesting Rik Marselis, Sogeti Nederland 5© Sogeti 2017
- 5. | Who has a robot? © Sogeti 2017 6 Who has a robot? In 10 years all of you will !!
- 6. | This is R0B3 (pronounced as: ar-null-bee-three) How can you stop the robot, without touching it? © Sogeti 2017 7
- 7. 1980 Rik Marselis@rikmarselis 2007 2012 20122008 Workgroup -member 20142009
- 8. | 2. What is “robotics”, “machine intelligence”, “cognitive computing” etc.? Workshop Robotesting Rik Marselis, Sogeti Nederland 9© Sogeti 2017
- 9. | Terminology © Sogeti 2017 10 Natural language processing
- 10. | What is a “Robot”? It’s a machine that gathers information about its environment by input of sensors and based on this input changes its behavior. Combined with machine learning and machine intelligence the robot’s reactions over time get more and more adequate. The use of Internet of Things, Big Data Analytics and Cloud technology make a robot versatile. A Robot can come in many different shapes and forms. It’s not just the metallic man. It may just as well be a smart algorithm on social media, an autonomous vacuum cleaner or a self-driving car. This definition was assembled by Rik Marselis from various sources. © Sogeti 2017 11
- 11. | What is a “Cobot”? Autonomous, intelligent, and self-learning: A new generation of robots is taking over factories around the world. They work with their human colleagues more closely and safely than ever before. Sensors and image recognition technology make it possible. A collaborative robot Cobot !! © Sogeti 2017 12
- 12. | What is a “bot” or “chatbot”? The following definition appears workable: ‘A bot (short for robot) is an algorithm acting on behalf of an individual, business or program, that can mimic human conversation’. Also known as “digital assistant”. Source: Paper “The Bot Effect: ‘Friending Your Brand’ on Machine Intelligence”by SogetiLabs 2016 © Sogeti 2017 13
- 13. | Machine learning (in the film “Short circuit” 1986) © Sogeti 2017 14 <<< vervangen door filmclip >>>
- 14. |© Sogeti 2017 15
- 15. | Testing machine learning: it’s about the input!! © Sogeti 2017 16 The tester as “psychologist” of the intelligent machine
- 16. | Cognitive The use of cognitive computing to assist quality assurance & testing. Cognitive = “knowing and perceiving” Source of picture: Film “Wargames” 1983 © Sogeti 2017 17
- 17. | Image recognition (how hard can that be?) © Sogeti 2017 18
- 18. | Robot automatically performs regression tests © Sogeti 19
- 19. | Robot automatically performs regression tests Uses a € 300,- robot-arm © Sogeti 2017 20 Low-cost robot-arm tests mobile devices. Connected to test manage- ment tool. Test cases check if the quality goals are met and risks are covered.
- 20. | Setting the stage Testing OF robots Testing WITH robots © Sogeti 2017 21 Todays focus The next level in test automation
- 21. | For today we use “Robotics” as overarching term © Sogeti 2017 22
- 22. | How to limit social impact of robots? The laws of Isaac Asimov: 0. A robot may not harm humanity, or, by inaction, allow humanity to come to harm. 1. A robot may not injure a human being or, through inaction, allow a human being to come to harm. 2. A robot must obey the orders given it by human beings except where such orders would conflict with the First Law. 3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Laws. Others created some more laws: 4. A robot must establish its identity as a robot in all cases. 5. A robot must know it is a robot. 6. A robot must reproduce. As long as such reproduction does not interfere with the First or Second or Third Law. source: Wikipedia © Sogeti 2017 23
- 23. | Different angles of quality for robotics © Sogeti 2017 24 Mechanical Electrical Information Processing Machine Intelligence Business Impact Social Impact new new new
- 24. | Quality attributes © ISO 25010 2011 25 ISO25010 product quality Functionality Completeness Correctness Appropriate- ness Performance Time behavior Resource utilisation Capacity Compatibility Co-existence Interoperability Usability Operability Learnability User error protection User interface aesthetics Accessibility Reliability Maturity Availibility Fault tolerance Recoverability Security Confidentiality Integrity Non- repudiation Accountability Authenticity Maintainability Modularity Reusability Analysability Modifiability Testability Portability Adaptability Installability Replaceability ISO 25010 consists of 2 parts: - Product Quality (this slide) - Quality in use (next slide)
- 25. | Quality attributes © ISO 25010 2011 26 ISO25010 Quality in use Effectiveness Effectiveness Efficiency Efficiency Satisfaction Usefulness Trust Pleasure Comfort Freedom from risk Economic risk mitigation Health and Safety risk mitigation Environmental risk mitigation Context coverage Context completeness Flexibility So ISO25010 is extensive, but is it enough for the new AI & Robotics world? What about: • Ethics • Embodiment • Predictability • Natural interaction • Working in a team • Transparency • Generalizability • and more (work in progress) © Sogeti 2017 • Privacy • Empathy • Engagement • Human friendliness (Asimov’s laws) • Charisma (wow-factor) • Happiness
- 26. | Let’s get started! We’ll use Exploratory Testing (the proper way!!) Workshop Robotesting Rik Marselis, Sogeti Nederland 27© Sogeti 2017
- 27. | E.T. has many flavours, this is our flavour: Source: www.tmap.net Source for term “mob-testing”: Maaret Pyhäjärvi © Sogeti 2017 28 • Focus on confidence (risk-based) • Structured (charter, log, debriefing) • Session-based & Timeboxed (not too short, not too long) • Tandem-approach (today we’ll do “mob-testing”) • Combine experience-based and coverage-based testing • Simultaneous test design, test execution and learning • Flexible (fit for Agile and DevOps) • Prepared (test ideas, testing tours) • Tools (heuristics, checklists, test design tooling) • Serious Fun !!
- 28. | Our pragmatic charter and log template Exploratory Testing © Sogeti 2017 29 Charter Log De- briefing Test ideas: e.g. testing tours, test data, etc.
- 29. | Even with E.T.: always think of expected output Exploratory Testing steps: • Before start of the session think of “Test ideas” and write on the charter (useful thoughts that help in determining what next test to do) • During the test session iterate the next steps: • Think of what you want to learn • Think of what test you can do to learn this • Think of the result you expect from the test • Log your test and expected result • Execute the test (“test case” if you like) • Log the actual result • What did you learn? • What is the next test you now want to do? © Sogeti 2017 30
- 30. | Let’s get going! 2 challenges for every team Workshop Robotesting Rik Marselis, Sogeti Nederland 31© Sogeti 2017
- 31. | Challenge 1: Test the chatbot We’ll use the chatbot “Kayak” on facebook messenger. One person of your group logs in to facebook messenger Contact KAYAK Find a return flight from your airport to an airport on another continent, next find a hotel in that city and two things to do. © Sogeti 2017 32 You’ll get a charter
- 32. | Workshop item 1: chatbot https://www.facebook.com/messages/t/kayak © Sogeti 2017 33 Short demo
- 33. | Challenge 2: Test the robot We’ll use the Lego Mindstorms robots. The robots have a color-scanning capability that makes them react to certain combinations of colors. Determine whether the robot works as designed and is fit for purpose. © Sogeti 2017 34 You’ll get a charter
- 34. | User-stories for color-recognizing robot © Sogeti 2017 35 • As a robot I scan 3 colors during the session so that I can show how that makes me feel • As a robot I will wink when I see white so that everyone knows I’m not that serious • As a robot I will show loving eyes when I see black so that everyone knows I’m into gothic • As a robot I will cheer when I see red two times so that everyone can see I’m happy • As a robot I will shout “boo” when I see blue twice so that everyone knows I don’t like blue • As a robot I stop moving when I hit something so that I don’t hurt any human (as per Asimov’s law) • As a robot I’ll drive backwards when I’m ready scanning AND the remote control is operated so that I will return to my starting position and be ready for the next try. Short demo
- 35. | Let’s get started! We’ll make 8 teams (of equal size). Workshop Robotesting Rik Marselis, Sogeti Nederland 36© Sogeti 2017
- 36. | The challenge We’ll make 8 teams. Every team gets: • Exploratory testing charter (with room for debriefing info on the back) • Log form to register your tests (and, very important, expected results!!) © Sogeti 2017 37
- 37. | Workshop: Round 1 20 minutes 38© Sogeti 2017
- 38. | Team Asimov Team Johnny 5 Team Robot Team R2D2 Test the robot Team Grassbot Team Vacubot Team Cobot Team C3P0 Test the chatbot Who does what challenge in round 1? © Sogeti 2017 39
- 39. | Workshop: Round 2 20 minutes 40© Sogeti 2017
- 40. | Team Grassbot Team Vacubot Team Cobot Team C3P0 Test the robot Team Asimov Team Johnny 5 Team Robot Team R2D2 Test the chatbot Who does what challenge in round 2? © Sogeti 2017 41
- 41. | Debriefing: your results and what you learned on Robotesting Workshop Robotesting Rik Marselis, Sogeti Nederland 42© Sogeti 2017
- 42. | Your key-learnings in 1 minute per team © Sogeti 2017 43 What did you learn about testing of Robots and Chatbots?
- 43. | What did you learn about quality attributes? © Sogeti 2017 44
- 44. | Digital happiness Exponential growth of technology, Ever-increasing speed of digitalization, Security-breaches, Fake-news, Cyberwar, and Privacy-violations… People wonder: ‘Does all this technology make me happier?’ The conclusion for now is that: • Happiness is becoming humanity’s explicit goal (instead of only GDP for instance) • Technology will help measuring happiness (and maybe increase our happiness obsession?) • Companies will be reviewed through this happiness perspective: “Does your business make me happier?” © Sogeti 2017 45
- 46. Robotesting The end Thank you! Agile Testing Days 15 November 2017