Software Testing Attacks for Mobile and Embedded Devices
Download as PPTX, PDF3 likes2,595 views
The document provides an overview of software testing strategies specifically for mobile and embedded devices, highlighting essential concepts, risks, and testing methodologies. Jon Hagar, an expert with over 30 years of experience, discusses types of mobile-embedded applications, common errors, and various testing techniques, emphasizing exploratory risk-based testing. He underscores the necessity for testers to adapt their approaches to the unique challenges of mobile and embedded systems while maintaining a comprehensive understanding of both hardware and software interactions.
Software Testing Attacks for Mobile and Embedded Devices
- 1. Software Testing for Mobile and Embedded Devices Jon Hagar hosted by XBOSoft
- 2. XBOSoft info • Founded in 2006 • Dedicated to software quality • Software QA consulting • Software testing services • Offices in San Francisco, Beijing, Oslo and Amsterdam
- 3. On the XBO side: Amy Jan
- 4. Housekeeping • Everyone except the speakers is muted • Questions via the gotowebinar control on the right side of your screen • Questions can be asked throughout the webinar, we’ll try to fit them in when appropriate • General Q & A at the end of the webinar • You will receive info on recording after the webinar
- 5. About Jon Hagar Jon…. • has over 30 years in software testing/verification and validation • is the owner of Grand Software Testing, a company specializing in software test consulting and training for mobile and embedded systems. • author of “Software Test Attacks to Break Mobile and Emedded Devices”
- 6. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 6 SOFTWARE TESTING CONCEPTS FOR MOBILE AND EMBEDDED DEVICES Jon Hagar embedded@ecentral.com jon.d.hagar@gmail.com breakingembeddedsoftware.wordpress.com
- 7. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 7How to Attack Embedded Software 7 AGENDA FOR TODAY • Introductions • Definitions • Risk based concepts • Exploratory approaches • Attacking the scenario(s) • Mobile and embedded tester skills • Wrap up and references
- 8. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 8How to Attack Embedded Software 8 MOBILE, SMART, AND HANDHELD • As the names imply, these are devices—small, held in the hand, often connected to communication networks, including • Cell and smart phones – apps • Tablets • Medical devices • Typically have: • Many of the problems of classic “embedded” systems • The power of PCs/IT • More user interface (UI) than classic embedded systems • Fast updates • Are getting more power, memory, and features (software e.g., apps) • The “hot” area of computers/software • Testing rules are “evolving”
- 9. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 9How to Attack Embedded Software 9 CLOSE COUSINS: Embedded Software Systems . . . • Interacts with unique hardware/systems to solve specialized problems in the “real world” • IT software runs with largely “generic” hardware • Users are barely aware the device uses or has software • Usually have significant hardware interface issues and concerns • Initialization, noise, power up/down, timers, sensors, etc. • Often are resource constrained • RAM, ROM, stack, power, speed, time, etc. • Typically has a restricted or no Human User/Computer Interface (HCI) but is evolving rapidly • Often no way (or only a risky way) to update and/or change the software • Involves risks, hazards, safety, and/or some specialized domain knowledge and logic/algorithms usually controlling hardware
- 10. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 10How to Attack Embedded Software 10 WHAT DO THESE LOOK LIKE? Examples – Avionics systems: planes, cars, rockets, military,….. – Telecom: switch, routers, phones, cell devices,…. – Transportation: traffic control, railroad, trucking, …. – Industrial control: lighting, machines, HVAC, nuclear/power,… – Medical: pacemaker, dispensers, ……. – Home and office systems: control, entertainment (TV box), … – And the list goes on • Now starting to include PDA’s and other items that “blur” the lines
- 11. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 11How to Attack Embedded Software 11 TYPES OF MOBILE- EMBEDDED APPS • Native Applications • Local to device • Hybrid Applications • Local to device but interacts w/internet • Web Applications • Not local to device All interactions on internet And JeanAnn Haarrison
- 12. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 12How to Attack Embedded Software 12 THE “WORLD” OF MOBILE- SMART/EMBEDDED SOFTWARE • Inputs and outputs involve hardware, software, and humans • Time dependent • NOTE: most software has “time” (performance) issues but here things are often “hard real time” • In Mobile/Embedded real-time may be a requirement Software Stimulus-Inputs Response-Outputs Expected Unexpected Wanted UnwantedHardware
- 13. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 13How to Attack Embedded Software 13 MOBILE AND EMBEDDED SOFTWARE TESTING: THE SAME BUT DIFFERENT • Same patterns in programming errors => same test patterns to be applied • But there are Differences • The development lifecycle • Mobile and embedded error patterns • “Embedded” unique sensor (input and output) • Mobility • Network connections • Resources
- 14. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 14How to Attack Embedded Software 14 Hardware Change Hardware New Product EXAMPLE HIGH LEVEL MOBILE-EMBEDDED LIFECYCLE DIFFERENCE System Creation Hardware Build Software Build Software Build Software Build Software Build Hw Issue Feature Results: Software is “late” Software Build Months, Days, Hours
- 15. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 15How to Attack Embedded Software 15 THE ERROR (BUG) PATTERN: SIMILAR AND DIFFERENT • Handheld/Embedded software has similar defects to other software • Requirements & Design • Logic & Math • Control Flow • Data • Initialization & Mode changes • Interfaces • Security • Gaming • etc. . . Mobile/Embedded additions • Software and hardware development cycles done in parallel, where aspects of the hardware may be unknown to the software development effort • Limited resources • Control/use of unique hardware (sensors) • Hardware problems which are often fixed with software late in the project • (a big one) Very tight real-time (often in milli- or micro-second ranges) or load performance issues
- 16. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 16How to Attack Embedded Software 16 MORE SAMPLE MOBILE-EMBEDDED PRODUCT ERRORS TESTERS SHOULD CONSIDER • Mobility - • Environment and device susceptible to outside influence: noise, weather, heat, cold, • Network connect: wifi, data, changing signal, etc • Battery factors: Heat, charge level, charge duration, device conflicts • Output —outputs to devices and electronics that are susceptible to noise influences, calibrations, changes to hardware, and time • Inputs – Sensors, speed, timing, and interrupts all can impact devices/software • Complexity—the size of the system or some aspect of the system makes missed bugs likely • Device Resources - • Screen size • Memory • CPU speed
- 17. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 17How to Attack Embedded Software 17 THUS TESTERS SHOULD THINK: RISKS • You cannot test everything • Risk-based testing helps bound the test scope problem • Testing is about providing information and understanding • Test exploration gets you started with whatever you have (or don’t have)
- 18. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 18How to Attack Embedded Software 18 IN MOBILE AND EMBEDDED: EXPLORATORY TESTING • Rapid feedback • Learning • Upfront rapid learning • Attacking • Address risk • Independent assessment • Target a defect • Prototyping • Need information • Test beyond the requirements Yes, we must test requirements (necessary, but not sufficient) So include exploratory risk-based testing:
- 19. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 19How to Attack Embedded Software 19 MUCH INFORMATION EXISTS ON EXPLORATORY RISK-BASED TESTING • Exploratory Testing: Based in the scientific method, where we plan-design, execute, learn, and change the test effort with limited “scripting” • Check out works by: • Kaner • Bach • Whittaker • ISO 29119 • I like to recommend and do exploratory attack-based testing
- 20. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 20How to Attack Embedded Software 20 WHY ATTACK? • Attacking your software is in part, the process of attempting to demonstrate a system (hardware, firmware, software and operations) does not meet requirements, functional and non-functional objectives. • Mobile/embedded software testing must include "the system" (hardware, software, operations, users) • Attack common modes of failure, especially where the application is engaged and visible by the user. Attack testing with approaches to include: Tools Levels Patterns Techniques
- 21. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 21How to Attack Embedded Software 21 SO, WHAT IS AN ATTACK? • Based on a common mode of failure seen over and over • Maybe seen as a negative, when it is really a positive • Goes after the “bugs” that may be in the software • Uses one or more classic test techniques and concepts • A Pattern (more than a process) which must be modified for the context at hand to do the testing • Many testers learn these for a domain after years and form a mental “pattern” model (most good testers attack)
- 22. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 22How to Attack Embedded Software 22 KINDS OF ATTACKS • Whittaker offers a good starting point for software attacks in general that can be applied to embedded: • User Interface Attacks • Data and Computation • File System Interface • Software/OS Interface • Whittaker’s “How to Break Software” lists 23 attacks • “Software Test Attacks to Break Mobile and Embedded Devices” lists 33 attacks and 8 sub attacks
- 23. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 23How to Attack Embedded Software 23 MOBILE AND EMBEDDED ATTACK CLASSIFICATION • Developer Attacks (unit/code testing) • Control System Attacks • Hardware-Software Attacks • Mobile and Embedded Software Domain Attacks • Time Attacks (Performance) • Human User Interface Attacks • Smart and/or Mobile Phone Functional App Attacks • Mobile/Embedded Security Attacks • Generic Attacks • Functional, mind mapping, and combinatorial tests
- 24. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 24 A DETAILED LIST OF ATTACKS • Attack 1: Static Code Analysis • Attack 2: Finding White–Box Data Computation Bugs • Attack 3: White–Box Structural Logic Flow Coverage • Attack 4: Finding Hardware–System Unhandled Uses in Software • Attack 5: Hw-Sw and Sw-Hw signal Interface Bugs • Attack 6: Long Duration Control Attack Runs • Attack 7: Breaking Software Logic and/or Control Laws • Attack 8: Forcing the Unusual Bug Cases • Attack 9 Breaking Software with Hardware and System Operations • 9.1 Sub–Attack: Breaking Battery Power • Attack 10: Finding Bugs in Hardware–Software Communications • Attack 11: Breaking Software Error Recovery • Attack 12: Interface and Integration Testing • 12.1 Sub–Attack: Configuration Integration Evaluation • Attack 13: Finding Problems in Software–System Fault Tolerance • Attack 14: Breaking Digital Software Communications • Attack 15: Finding Bugs in the Data • Attack 16: Bugs in System–Software Computation • Attack 17: Using Simulation and Stimulation to Drive Software Attacks • Attack 18: Bugs in Timing Interrupts and Priority Inversion • Attack 19: Finding Time Related Bugs • Attack 20: Time Related Scenarios, Stories and Tours • Attack 21: Performance Testing Introduction • Attack 22: Finding Supporting (User) Documentation Problems • Sub–Attack 22.1: Confirming Install–ability • Attack 23: Finding Missing or Wrong Alarms • Attack 24: Finding Bugs in Help Files • Attack 25: Finding Bugs in Apps • Attack 26: Testing Mobile and Embedded Games • Attack 27: Attacking App–Cloud Dependencies • Attack 28 Penetration Attack Test • Attack 28.1 Penetration Sub–Attacks: Authentication — Password Attack • Attack 28.2 Sub–Attack Fuzz Test • Attack 29: Information Theft—Stealing Device Data • Attack 29.1 Sub Attack –Identity Social Engineering • Attack 30: Spoofing Attacks • Attack 30.1 Location and/or User Profile Spoof Sub–Attack • Attack 30.2 GPS Spoof Sub–Attack • Attack 31: Attacking Viruses on the Run in Factories or PLCs • Attack 32: Using Combinatorial Tests • Attack 33: Attacking Functional Bugs
- 25. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 25How to Attack Embedded Software 25 AN EXAMPLE DETAILED TAXONOMY: A2D AND D2A BUG POSSIBILITIES Type Situation Impact Notes A2D A2D representation information is lost because measurement is not precise Software computation is based on incorrect data Number of bits used to store the analog converted data is not large enough or sampling rate to get bits is not correct. A2D A2D information is contaminated with noise Software computation use noise when it should not The noise term may not be known, accounted for, or misrepresented. A2D A2D information is calculated correctly Computation has unknown error Sources of error can come from: calibrations used on variables, variables lacking initialization, or calculations are not done with enough accuracy (single versus double floating point D2A D2A conversion losses “least significant bits” (LSB) in conversions, but bits are, in fact, important because computer word sizes are too small Output to analog device is wrong Number of bits stored from the digital world to the analog world do not have enough precision, so analog data is incorrect. D2A D2A information does not account for noise of the real world Software computation does not include a factor for noise The analog values are not correct given the noise of the real world (output data may be lost in the noise). D2A D2A information is calculated correctly because of internal factors Computation has unknown error Sources of error can come from: calibrations used on variables, variables lacking initialization, or calculations are not done with enough accuracy (single versus double floating point
- 26. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 26How to Attack Embedded Software 26 FINALLY, IN ADDITION TO TESTER SKILLS: SKILL FOR THE MOBILE/EMBEDDED TESTER • Tester skills: planning, design, execution, reporting, exploration, techniques, tools……………………. • Exposure or knowledge about products from the domain in which you are testing: aerospace, medical, automobile manufacturing, smart apps, airplanes, factory systems, robotics, regulated environments, etc. • Knowledge of hard sciences: math, physics, electronics, hardware, engineering, etc. for logical thought processes • Knowledge of Soft sciences: psychology, philosophy, sociology, human factors (human machine interface), gaming, arts… for creative & conceptual thought processes • Passion – follow your bliss
- 27. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 27How to Attack Embedded Software 27 FOR THE SOFTWARE YOU TEST ASK YOURSELF • Do you know how it fails? • Do you test for success or failure? • Both? • Where can you improve?
- 28. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 28How to Attack Embedded Software 28 SUMMARY: THANK YOU (IDEAS USED FROM) • James Whittaker (attacks) • Elisabeth Hendrickson (simulations) • Lee Copeland (techniques) • Brian Merrick (testing) • James Bach (exploratory & tours) • Cem Kaner (test thinking) • JeanAnn Harrison • Many teachers • Generations past and future • Books, references, etc.
- 29. Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 29How to Attack Embedded Software 29 BOOK LIST (MY FAVORITES) • “Software Test Attacks to Break Mobile and Embedded Devices” – Jon Hagar, • “How to Break Software” James Whittaker, 2003 • And his other “How To Break…” books • “Testing Embedded Software” Broeckman and Notenboom, 2003 • “A Practitioner’s Guide to Software Test Design” Copeland, 2004 • “A Practitioner’s Handbook for Real-Time Analysis” Klein et. al., 1993 • “Computer Related Risks”, Neumann, 1995 • “Safeware: System Safety and Computers”, Leveson, 1995 • Honorable mentions: • “Embedded System and Software Validation” Roychoudhury, 2009 • “Systems Testing with an Attitude” Petschenik 2005 • “Software System Testing and Quality Assurance” Beizer, 1987 • “Testing Computer Software” Kaner et. al., 1988 • “Systematic Software Testing” Craig & Jaskiel, 2001 • “Managing the Testing Process” Black, 2002
- 30. Questions & Answers Follow us: @XBOSoft Questions at: services@xbosoft.com
- 31. Thanks for joining us and hope to hear from you soon!
Editor's Notes
- #14: update