Mobile App Test Attacks to Efficiently Explore Software
Download as PPT, PDF0 likes582 views
The document outlines various software testing attacks aimed at mobile and embedded devices, detailing strategies to identify and address potential bugs. It highlights the necessity for improved testing methodologies as mobile technology evolves, including a mix of exploratory and risk-based testing approaches. Additionally, it discusses the importance of context in testing and provides a taxonomy of attack patterns to enhance software reliability.
Mobile App Test Attacks to Efficiently Explore Software
- 1. Mobile App Test Attacks to Efficiently Explore Software Jon D. Hagar, Consultant, Grand Software Testing embedded@ecentral.com Author: Software Test Attacks to Break Mobile and Embedded Devices Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices” 1
- 2. ∗ Gaming Testing Story ∗ It only takes a few minutes using an App before users like or hate it ∗ Worse than that. . . ∗ Many users will post a social media review of the app ∗ You don’t want to be a BAD Copyright 2015, Jon D. Hagar Mobile-Embedded Taxonomies from “Software Test Attacks to Break Mobile and Embedded Devices” 2 The Mobile Opportunity
- 3. ∗ Depth ∗ Passion ∗ Speed What Does it Take to be a Great Mobile App Tester? Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – Software Test Attacks to Break Mobile and Embedded Devices 3 Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – Software Test Attacks to Break Mobile and Embedded Devices 3
- 4. ∗ As the names imply, these are devices—small, held in the hand, connected to communication networks, including ∗ Cell and smart phones – apps ∗ Tablets ∗ Medical devices ∗ Typically have: ∗ Many of the problems of classic embedded systems ∗ The power of PCs/IT ∗ More user interface (UI) than classic embedded systems ∗ Fast and frequent updates ∗ However, mobile devices are “evolving” with more power, resources, apps, etc. ∗ Mobile is the “hot” area of computers/software ∗ Testing rules and concepts are still evolving ∗ Now starting to include IoT You know what they are right? Mobile? Copyright 2015, Jon D. Hagar Mobile-Embedded Taxonomies from “Software Test Attacks to Break Mobile and Embedded Devices”
- 5. ∗ Requirements verification checking ∗ Necessary but not sufficient ∗ Risk–based testing ∗ Tried and true in many contexts including mobile, but we need more We need to do more as testers We Need Better App Testing Copyright 2015, Jon D. Hagar Mobile-Embedded Taxonomies from “Software Test Attacks to Break Mobile and Embedded Devices”
- 6. ∗ Management directed “No testing” ∗ Dev-ops without enough “thinking” of context and risk to find the BUGS that “count” ∗ Stupid requirements verification checking without GOOD supporting test activities ∗ Testing without thinking of ∗ cost ∗ schedule ∗ users Current Situation in Mobile Projects Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices” 6
- 7. ∗ From Wikipedia: Taxonomy is the practice and science of classification. The word finds its roots in the Greek τάξις, taxis (meaning 'order', 'arrangement') and νόμος, nomos ('law' or 'science'). Taxonomy uses taxonomic units, known as taxa (singular taxon). In addition, the word is also used as a count noun: a taxonomy, or taxonomic scheme, is a particular classification ("the taxonomy of ..."), arranged in a hierarchical structure. ∗ The attacks of this session are based on a researched Taxonomy Copyright 2015, Jon D. Hagar Mobile-Embedded Taxonomies from “Software Test Attacks to Break Mobile and Embedded Devices” 7 Lets look for bugs, but where?
- 8. ∗ A pattern (of testing) based on a common mode of failure seen over and over ∗ Part of Exploratory Testing ∗ May be seen as a negative, when it really is a positive ∗ Goes after the “bugs” that may be in the software ∗ May include or use classic test techniques and test concepts ∗ Lee Copeland’s book on test design ∗ Many other good books ∗ A Pattern (more than a process) which must be modified for the context at hand to do the testing ∗ Testers learn mental attack patterns working over the years in a specific domain Apply Attack-based Testing What is an attack? Copyright 2015, Jon D. Hagar Mobile-Embedded Taxonomies from “Software Test Attacks to Break Mobile and Embedded Devices”
- 9. A Sampling of Attacks (from Software Test Attacks to Break Mobile and Embedded Devices) ∗ Attack 1: Static Code Analysis ∗ Attack 2: Finding White–Box Data Computation Bugs ∗ Attack 3: White–Box Structural Logic Flow Coverage ∗ Attack 4: Finding Hardware–System Unhandled Uses in Software ∗ Attack 5: Hw-Sw and Sw-Hw signal Interface Bugs ∗ Attack 6: Long Duration Control Attack Runs ∗ Attack 7: Breaking Software Logic and/or Control Laws ∗ Attack 8: Forcing the Unusual Bug Cases ∗ Attack 9 Breaking Software with Hardware and System Operations ∗ 9.1 Sub–Attack: Breaking Battery Power ∗ Attack 10: Finding Bugs in Hardware–Software Communications ∗ Attack 11: Breaking Software Error Recovery ∗ Attack 12: Interface and Integration Testing ∗ 12.1 Sub–Attack: Configuration Integration Evaluation ∗ Attack 13: Finding Problems in Software–System Fault Tolerance ∗ Attack 14: Breaking Digital Software Communications ∗ Attack 15: Finding Bugs in the Data ∗ Attack 16: Bugs in System–Software Computation ∗ Attack 17: Using Simulation and Stimulation to Drive Software Attacks ∗ Attack 18: Bugs in Timing Interrupts and Priority Inversion ∗ Attack 19: Finding Time Related Bugs ∗ Attack 20: Time Related Scenarios, Stories and Tours ∗ Attack 21: Performance Testing Introduction ∗ Attack 22: Finding Supporting (User) Documentation Problems ∗ Sub–Attack 22.1: Confirming Install–ability ∗ Attack 23: Finding Missing or Wrong Alarms ∗ Attack 24: Finding Bugs in Help Files ∗ Attack 25: Finding Bugs in Apps ∗ Attack 26: Testing Mobile and Embedded Games ∗ Attack 27: Attacking App–Cloud Dependencies ∗ Attack 28 Penetration Attack Test ∗ Attack 28.1 Penetration Sub–Attacks: Authentication — Password Attack ∗ Attack 28.2 Sub–Attack Fuzz Test ∗ Attack 29: Information Theft—Stealing Device Data ∗ Attack 29.1 Sub Attack –Identity Social Engineering ∗ Attack 30: Spoofing Attacks ∗ Attack 30.1 Location and/or User Profile Spoof Sub–Attack ∗ Attack 30.2 GPS Spoof Sub–Attack ∗ Attack 31: Attacking Viruses on the Run in Factories or PLCs ∗ Attack 32: Using Combinatorial Tests ∗ Attack 33: Attacking Functional Bugs Copyright 2015, Jon D. Hagar Mobile-Embedded Taxonomies from “Software Test Attacks to Break Mobile and Embedded Devices”
- 10. Attack 1: Static Code Analysis (testing) ∗ When to apply this attack? ∗ After/during coding ∗ What faults make this attack successful? ∗ Many ∗ Example: Issues with pointers ∗ Who conducts this attack? ∗ Developer, tester, independent party ∗ Where is this attack conducted? ∗ Tool/test lab ∗ How to determine if the attack exposes failures? ∗ Review warning messages and find true bugs ∗ How to conduct this attack? ∗ Obtain and run tool ∗ Find and eliminate false positive ∗ Identify and address real bugs ∗ Repeat as code evolves ∗ Single unit/object ∗ Class/Group ∗ Component ∗ Full system 10 Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedde
- 11. Attack 2: Finding White–Box Data Computation Bugs ∗ When to apply this attack? ∗ After/during coding ∗ What faults make this attack successful? ∗ Mistakes associated with data ∗ Example: Wrong value of Pi ∗ Who conducts this attack? ∗ Developer, tester, independent party ∗ Where is this attack conducted? ∗ Development Tool/test lab ∗ How to determine if the attack exposes failures? ∗ Structural-data test success criteria not met ∗ How to conduct this attack? ∗ Obtain tool ∗ Determine criteria and coverage ∗ Create test automation with specific values (really a programing problem) ∗ NOT NICE NUMBERS ∗ Run automated test cases ∗ Resolve failures ∗ Peer check test cases ∗ Repeat as code evolves Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedde
- 12. Attack 3: White–Box Structural Logic Flow Coverage ∗ When to apply this attack? ∗ After/during coding ∗ What faults make this attack successful? ∗ Many ∗ Example: Statement coverage ∗ Who conducts this attack? ∗ Developer, tester, independent ∗ Where is this attack conducted? ∗ Tool/test lab ∗ How to determine if the attack exposes failures? ∗ Coverage not met and/or success criteria fails ∗ How to conduct this attack? ∗ Obtain tool ∗ Determine criteria and coverage ∗ Create test automation with specific values to drive logic flow within code ∗ Run automated test cases ∗ Resolve failures ∗ Peer check test cases ∗ Repeat as code evolves Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedde
- 13. Attack 4: Finding Hardware–System Unhandled User Cases ∗ When to apply this attack? ∗ Starting at system-software analysis ∗ What faults make this attack successful? ∗ Lack of understand of the world ∗ Example: Car braking on ice ∗ Who conducts this attack? ∗ Developer, tester, analyst ∗ Where is this attack conducted? ∗ Environments, simulations, field ∗ How to determine if the attack exposes failures? ∗ An unhandled condition exist ∗ Note: data explosion problem ∗ How to conduct this attack? ∗ Knowledge ∗ Out-of-box thinking ∗ Operation Concepts ∗ Analysis ∗ Modeling ∗ Lab testing ∗ Field testing ∗ Feedback ∗ Repeat
- 14. Attack 22 and 24: Finding Supporting (User) Documentation and Help File Problems 14 ∗ When to apply this attack? ∗ As soon as user documents exist ∗ What faults make this attack successful? ∗ Incorrect information about how to “use” the app ∗ Who conducts this attack? ∗ Tester, independent party, stakeholders ∗ Where is this attack conducted? ∗ Conduct on the online or hardcopy documents ∗ How to determine if the attack exposes failures? ∗ Follow the instructions exactly and determine if system works ∗ How to conduct this attack? ∗ Access the documentation ∗ Use instructions to create a user story ∗ Play the role of different personas ∗ Consider giving the documentation to a independent party ∗ Repeat as document and systems change
- 15. Attack 22.1: Confirming Installability 15 ∗ When to apply this attack? ∗ When installation is available ∗ What faults make this attack successful? ∗ “Missing” part and/or incorrect configurations ∗ Configurations of hardware and software may not support the app (Device fragmentation) ∗ Who conducts this attack? ∗ Tester, independent party ∗ Where is this attack conducted? ∗ Tool/test lab, field ∗ How to determine if the attack exposes failures? ∗ System fails to install or run correctly after install ∗ How to conduct this attack? ∗ Obtain “clean” device/system (s) ∗ Identify load procedures ∗ Note: if doing device configuration operability test use of techniques such as combinatorial or market penetration identification may be needed ∗ Define test strategy and plan ∗ Define test design ∗ Automate if needed ∗ Execute test (follow load procedures) ∗ Confirm load and use configuration ∗ Repeat as needed Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedde
- 16. Attack 23: Finding Missing or Wrong Alarms ∗ When to apply this attack? ∗ Device has alarms or information notifications to drive user interaction ∗ What faults make this attack successful? ∗ Time or other interactions cause notification-alarm to be missed ∗ Who conducts this attack? ∗ Tester, independent party ∗ Where is this attack conducted? ∗ Tool/test lab, field ∗ How to determine if the attack exposes failures? ∗ Alarm is missed or wrong ∗ How to conduct this attack? ∗ Define alarms and conditions ∗ Define risks of alarms in usage and time ∗ Define strategy and test plan ∗ Define use cases ∗ Define test design within environments including time ∗ Run tests ∗ Review for missing/wrong alarms and cases to “force” ∗ Leap year Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedde
- 17. ∗ When to apply this attack? …all the time ∗ What faults make this attack successful? …apps can be quite complex ∗ Example: Games-Entertainment ( 40-60 % of downloads) ∗ Who conducts this attack? ∗ Test Team ∗ A-B “user” testing (crowd, Beta, early releases in continuous integration/Deployment, etc) ∗ Where is this attack conducted? …throughout lifecycle and in environments ∗ How to determine if the attack exposes failures? ∗ Unhappy “users” ∗ Bugs found ∗ See checklist Attack : Testing Usability Credit to Jean Ann Harison2013 Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
- 18. ∗ The developer(s)—see Attacks 1, 2, and 3. ∗ The app architect or director ∗ On-team tester(s) ∗ In-company “dog food” testers ∗ Independent test players ∗ Mass beta trials ∗ Not a tester—Finally, consider who should not be playing Note on roles: During the testing effort and as it progresses, don’t forget that there are many different user roles Exercise: WHAT ARE THE ROLES? Roles in Usability Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
- 19. ∗ Refine checklist to context scope ∗ Define a role ∗ Watch what is happening with this role ∗ Define a usage (many different user roles) ∗ Guided explorations or ad hoc ∗ Stress, unusual cases, explore options ∗ Capture understanding, risk, observations, etc. ∗ Checklist (watch for confusion) ∗ Run Exploratory Attack(s) ∗ Run A-B statistical Test with monitoring ∗ Learn ∗ Re-plan/design ∗ Watch for Bias ∗ Switch testers ∗ Repeat Usability Attack Pattern Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
- 20. ∗ Apply when the device is mobile and has ∗ Account numbers ∗ User-ids and passwords ∗ Location tags ∗ Restricted data ∗ Current authentication approaches in use on mobile devices ∗ Server-based ∗ Registry (user/password) ∗ Location or device-based ∗ Profile-based My Personal Pet Cause Security Attacks Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices”
- 21. ∗ Attack 28 Penetration Attack Test ∗ Attack 28.1 Penetration Sub–Attacks: Authentication — Password ∗ Attack 28.2 Sub–Attack Fuzz Test ∗ Attack 29: Information Theft—Stealing Device Data ∗ Attack 29.1 Sub Attack –Identity Social Engineering ∗ Attack 30: Spoofing Attacks ∗ Attack 30.1 Location and/or User Profile Spoof Sub–Attack ∗ Attack 30.2 GPS Spoof Sub–Attack Security Attacks (Con: only a starting point, a checklist of things to start with) Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices”
- 22. Security attacks must be done with the knowledge and approval of owners of the system and software Severe legal implications exist in this area Many of these attacks must be done in a lab (sandbox) In these attacks, I tell you conceptually how to “drive a car very fast (150 miles an hour) but there are places to do this with a car legally (a race track) and places where you will get a ticket (most public streets)” Be forewarned - Do not attack you favorite app on your phone or any connected server without the right permissions due to legal implications Warnings When Conducting Security Attacks Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices”
- 23. ∗ There will always be Good, Bad, and Ugly ∗ Work with the Good ∗ Work to over come the Bad ∗ Change the Ugly into good ∗ Understanding your local context and error patterns is important (one size does NOT fit all) ∗ Attacks are patterns…you must still THINK and tailor Wrap Up of this Session Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices”
- 24. ∗ James Whittaker (attacks) ∗ Elisabeth Hendrickson (simulations) ∗ Lee Copeland (techniques) ∗ Brian Merrick (testing) ∗ James Bach (exploratory and tours) ∗ Cem Kaner (test thinking) ∗ Jean Ann Harrison (her thinking and help) ∗ Many teachers ∗ Generations past and future ∗ Books, references, and so on Notes: Thank You (ideas used from) Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices”
- 25. ∗ “Software Test Attacks to Break Mobile and Embedded Devices” – Jon Hagar ∗ “How to Break Software” James Whittaker, 2003 ∗ And his other “How To Break…” books ∗ “A Practitioner’s Guide to Software Test Design” Copeland, 2004 ∗ “A Practitioner’s Handbook for Real-Time Analysis” Klein et. al., 1993 ∗ “Computer Related Risks”, Neumann, 1995 ∗ “Safeware: System Safety and Computers”, Leveson, 1995 ∗ Honorable mentions: ∗ “Systems Testing with an Attitude” Petschenik 2005 ∗ “Software System Testing and Quality Assurance” Beizer, 1987 ∗ “Testing Computer Software” Kaner et. al., 1988 ∗ “Systematic Software Testing” Craig & Jaskiel, 2001 ∗ “Managing the Testing Process” Black, 2002 Book/Notes List (my favorites) Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices”
- 26. • www.stickyminds.com – Collection of test info • www.embedded.com – info on attacks ∗ www.sqaforums.com - Mobile Devices, Mobile Apps - Embedded Systems Testing forum • Association of Software Testing – BBST Classes http://www.testingeducation.org/BBST/ • Your favorite search engine More Resources Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – Software Test Attacks to Break Mobile and Embedded Devices
Editor's Notes
- #3: Open with a story – game that I loaded, play (tested), and killed in 10 min You don’t want this for your app? I will give you some ideas to start to get to be a better tester and deliver apps that excite
- #4: Up - Pop up for each of these related to session with a pic Need to inspire - interest - intrigue - inform What is the Top line
- #6: Up - fly in
- #7: Tbd copyright