Testing Is How You Avoid Looking Stupid
1 like311 views
The document emphasizes the importance of testing in IoT systems to prevent failures that can lead to severe consequences such as injuries and financial losses. It advocates for a multi-dimensional testing approach, highlighting the need for commitment from all levels of an organization in ensuring products are safe and effective. Additionally, it discusses various testing methods and the associated costs, while warning against neglecting thorough testing processes.
Testing Is How You Avoid Looking Stupid
- 1. Testing Is How You Avoid Looking Stupid Steve Branam, SimpliSafe steve.branam@simplisafe.com IOT With The Best, October 14, 2017
- 2. Motivation
- 3. Customer Premises Equipment IOT Provider Premises Equipment IOT Triad • 3 sets of testable components • IOT devices • Backend • Servers • Monolithic • Microservices • Admin tools • Apps (frontend) • Web • Desktop • Mobile IOT Devices Backend Frontend May be deployed in cloud Inherent complexity means lots of opportunities all along the way for things to go wrong despite really good people working on it.
- 4. Third-Party Integration IOT Devices Backend Frontend 3rd Party Devices 3rd Party Frontend 3rd Party Backend Potential combinatorial explosion of complexity!
- 5. IOT In The Real World • The things we build have real effects on real people in the real world. • Not just fun gadgets: • Home/industrial automation controlling physical objects, including some with potential to cause injury or damage, or consume large amounts of resources • Implantable devices in the human body • Autonomous roving devices • Integration with personal financial resources (credit cards, electronic bank account access, autopayment, etc.) • We have a responsibility to build safe products that don't adversely affect our customers or those around them. • Never underestimate customers' ability to use, abuse, misuse, and confuse your products.
- 6. Good Testing Is Critical • Testing is the due diligence that closes the engineering loop to verify proper behavior. • A defensive mechanism to catch things going wrong in the midst of all that complexity. • Testing has a cost that is often perceived as pure overhead, so people try to minimize that cost or postpone it. • Not testing can incur unintended costs that far exceed any cost savings achieved. • Reality check: there is always a risk of problems slipping through no matter how rigorous and thorough the process is. • No single type of testing covers all bases. It requires a multi-dimensional approach. • You can't test high quality in, but you can verify it's there, before someone else discovers it's not.
- 7. Be Pessimistic • An IOT ecosystem is complex, with a lot of moving parts. • That means lots of opportunities for things to go wrong. • The goal of testing is to shake out all those things as early as possible. • Assume that untested code is broken code! • It will come back to bite you. • Be the “token pessimist”. • Trust nothing, and verify. • And have a nice day!
- 8. Commitment • It takes commitment all through the company, from individual developers up through the entire management chain. • Everyone has to be willing to take the time and invest the resources AND ACTUALLY RUN THE TESTS. • Lack of commitment is a sign you can expect trouble down the road. • This is an interview topic for both sides of the table. • For individual contributors, this is a factor in evaluating where you want to work and who you want working with you. • For managers, this is a factor in evaluating who you want working for you. • Cool companies, cool products, and 10x developers don't look so great when they start turning your life into misery due to lack of test discipline.
- 9. IOT Consequences Of Failure • Deaths • Injuries • Destruction of property • Destruction of personal finances • Destruction of lives • Unhappy customers • Unhappy investors: poor testing is a business liability
- 10. Recommended Reading• To cultivate the right mind-set and develop an appropriate level of paranoia. • Learn from others' mistakes so you can avoid making them yourself. • “All of this has happened before, and it will all happen again." • Risks Digest • Formal title: Forum on Risks to the Public in Computers and Related Systems, ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator • Includes 32-year archive of companies and organizations looking stupid. • Stories of actual death, injury, destruction, catastrophic financial failure. • https://catless.ncl.ac.uk/Risks • Computer-Related Risks, by Peter G. Neumann, 1995 • Based on first 10 year of Risks Digest. • Dozens of examples from aviation and transportation, space, defense, security, industrial control, nuclear and electrical power, medical devices, finance, communications. • Engineering A Safer World, by Nancy G. Leveson, 2011 • A bit farther from IOT, but focuses on end-to-end system thinking that's very appropriate to IOT. • Case studies: release of toxic chemicals in Bhopal, India; US friendly-fire helicopter shootdown in Iraq; loss of Milstar satellite launched from Cape Canaveral, Florida; contamination of public water supply in Ontario, Canada. • Free PDF download on the book’s MIT Press page, SO NO EXCUSE! • https://mitpress.mit.edu/books/engineering-safer-world
- 11. Recommended Reading • Working Effectively with Legacy Code, by Michael C. Feathers, 2004 • Practical approach to the issue of design for testability. • Better Embedded System Software, by Philip Koopman, 2010 • Among other useful information, has a nice chapter on testing. • Available half off retail at http://koopman.us/ • Security Engineering: A Guide To Building Dependable Distributed Systems, by Ross Anderson, 2nd ed. 2008. • Background knowledge for effective security testing. • The Practice of Programming, by Brian W. Kernighan and Rob Pike, 1999 • Nice chapters on testing and performance. • The Go Programming Language, by Alan A. A. Donovan and Brian W. Kernighan, 2015 • Go environment has a good emphasis on testing.
- 12. What Is "looking stupid”? • Externally: • Lawsuits • Fines • Government investigation • How many C-level executives want to spend time in the hot seat in front a Congressional investigation? • How many want to contemplate jail time for negligence? • Bad press • Appearance in "Risks Digest" • Loss of market share • Loss of reputation (loss of market opportunity) • Customer complaints
- 13. What Is "looking stupid”? • Internally: • Loss of reputation, increased friction • Between groups: engineering, QA, customer support • Between developers • Between management and developers • Delayed schedules • Delayed releases • Extra work to make schedule • Extra customer support load
- 14. Case Study: Toyota Unintended Acceleration (UA) • Not IOT. But, IOT is converging on similar capabilities, so this is a representative cautionary example illustrating the complexity of the overall problem and what’s at stake. • Multiple deaths attributed to UA. • 2013: Jury award of $3 million to plaintiffs in Oklahoma suit, settlement talks for hundreds of state and federal lawsuits, up to $2 billion in legal costs. (http://articles.latimes.com/2013/dec/12/business/la-fi-toyota-settlement-20131213) • 2014: $1.2 billion settlement in criminal case to US DOJ. (https://www.yahoo.com/news/toyota-pay-1-2b-settle-criminal-probe-193237658--finance.html) • Prof. Philip Koopman, Carnegie Mellon University, plaintiff expert witness, author of Better Embedded System Software • “A Case Study of Toyota Unintended Acceleration and Software Safety”, September 18, 2014 • https://users.ece.cmu.edu/~koopman/pubs/koopman14_toyota_ua_slides.pdf • betterembsw.blogspot.com • Michael Barr, plaintiff source code review team • https://embeddedgurus.com/barr-code/2013/10/an-update-on-toyota-and-unintended-acceleration/ (with updates through October 30, 2014) • Barr is a good source for a vast trove of very practical embedded system knowledge.
- 15. Cost Of Testing • Good testing is hard. It takes work and resources. • Achieving the right balance of investment is hard. • Test equipment • Test environment, real or virtual • Instances of each component in the IOT triad • Multiple separate environments for different types of testing • Monitoring and measurement equipment • Staff • A portion of product developer time • Dedicated test staff • Manual testers • Test automation developers • Multiple test groups, each with a different focus • Administrative • Bug tracking • Staff and equipment management
- 16. Process Feedback • Use testing as a way of improving your development process. • Use bug data to strengthen coding or review guidelines. • Immature process with high defect rate: • Helps to cut down on frequent bug types. • Mature process with low defect rate: • A bug indicates a gap in the process. • Fix the bugs, but also, fix the process. • Like a teenager, this is how things mature, from obnoxious 13-year-old to serious 18-year-old.
- 17. Test Coverage
- 18. Types of Testing • Functional: does it work? • Performance: does it work fast enough, and what resources does it consume? • Load/Scaling: how does it work as load increases? • Endurance/Longevity: does it work for long duration, and how do performance and resource consumption change? • Stress: how does it respond to stressful load conditions? • Security: is it secure against attack? • Regression: do things that were working before still work? • Smoke: quick testing to verify basic functionality
- 19. Test Cases • Use a variety of test cases that exercise as many code paths as possible. • Within the triad components themselves. • The interactions between them (including with third-party components). • Ideal world: 100% • Real world: 50-90%
- 20. Practical Test Coverage • At some point diminishing returns make it impractical to achieve additional coverage. • Pareto principle, 80/20 rule: 80% of the effects come from 20% of the causes (the vital few). • Lowell Arthur: "20% of the code has 80% of the errors. Find them, fix them!" (Roger S. Pressman's Software Engineering: A Practitioner's Approach) • Corollary: Effort expended on the remaining 80% of the code will only find 20% of the errors • But the maddening question is: which 20% should you focus on? • Microsoft: fixing top 20% of most-reported bugs would eliminate 80% of the related errors and crashes in a given system. • Pareto distribution representative of effort required to achieve higher cumulative coverage. • There is always a tension between the amount of resources to invest in testing vs. the return on that investment.
- 21. 0% 25% 50% 75% 100% $0x $1x $2x $3x $4x $5x $6x $7x $8x $9x $10x $20x $40x $80x $160x Cumulative Percentage Representative Pareto Distribution Potentially exponential cost growth to get last few percent!
- 22. Coverage Gaps • Untested code paths/scenarios represent a risk • Technical risk • Business risk • The cost to fix a bug increases the later it is found • Bugs found by customers in shipped product are the most expensive in direct and indirect costs. • Bug fixes are an opportunity to add coverage in an area that exhibited problems. • These are now regression tests to verify problems don't re-emerge.
- 23. The "Show Me" Method • Prior to code changes, capture test results that exhibit unsatisfactory characteristics. • Motivate effort to correct issue. • Donald Knuth: “Premature optimization is the root of all evil.” • After code changes, capture test results that exhibit satisfactory characteristics. • Demonstrate that corrective efforts have achieved desired result. • If you can't prove that you've done something to improve the results, how can you justify the effort? • Alternatively: how do you know when you're done?
- 24. Scope Of Testing
- 25. 3 Major Scopes • Unit • Integration • System
- 26. Unit Testing • Functional and performance testing of isolated portions of code. • Smallest executable scope. • As part of TDD (Test-Driven Development) or other methodology. • Incorporate unit testing early in the development process. • Make it work, keep it working as you continue development. • Backfilling unit tests later for existing code can be very difficult. • Technical: parts may have been implemented in ways that don’t lend themselves to unit testing (see Michael C. Feathers’ Working Effectively with Legacy Code). • Attitude: for managers and developers focused on go-go-go, it feels like you’re stalling momentum and forward progress. • Schedule: Oops, sorry, out of time, can’t test. Related to commitment and attitude.
- 27. What Comprises A Unit? • Break down a complete piece of software into component units. • Classes • Specific algorithms • Groups of related functions that operate on a specific data structure • Groups of functions related to providing a specific service • Each unit is characterized by its public interface and its internal implementation. • Client components use the unit via its public interface. • The internal implementation is free to change radically as long as it fulfills the stated functions of the public interface.
- 28. Building Unit Tests • Done by developer, the person who knows the code best. • Account for this in scheduling, it’s not magic zero-time consumption. • Wrap the unit in a test suite to test all functions of the public interface. • Setup the context of the test (the test case scenario), execute the code under test. • You can wrap code from any code base in unit tests that run on your development platform. • Including UI and embedded code intended for a completely different OS/hardware environment. • Not all problems can be shaken out (particularly concurrency or real-time scheduling issues), but you can prove out the components on their own. • Good for driving scenarios to step through in debugger. • Dependency on other components • Use mocks and fakes. • Dependency injection (design for testability). • Use unit test tools and frameworks to simplify the process.
- 29. Unit Test Cases • Positive test cases (happy paths) • Negative test cases (unhappy paths: failure, error paths) • Common paths • Uncommon paths (including "this can't possibly happen" paths) • Boundary cases • Bad input cases • Bad values (out of range, discontinuous, nonsensical) • Oversize inputs exceeding buffer sizes or scaling limits • Malformed messages, strings, or structures • Invalid combinations of inputs • Proper API sequence • Improper API sequence • Expected use cases: how you expect it to be used by client components • Unexpected use cases: you don't expect it to be used this way, but the interface allows it • Drive state machines through all possible transitions • Embedded systems, servers, and apps are all state-based reactive systems, some variation of MVC paradigm, or data, control, and management planes. • Mock event sources and controlled elements
- 30. Internal Knowledge • White box testing: you can see into the unit, with awareness of internal details such as code branches, FSM states and transitions. You can test to see that each branch/transition is exercised. • Black box testing: you can’t see into the unit, it’s an opaque black monolith. You can test to see that it meets requirements and intended behaviors. • Be careful about allowing tests to have too much knowledge of internals. • Avoid special functions or public data that allow tests to peer into internal data. • Results in brittle test cases that require work when internals change. • Stick with the public interface. • It's still worth instrumenting your code to allow monitoring at some level of abstraction. • That just becomes another part of the public interface. • Can itself be unit tested.
- 31. Unit Performance Tests • Allow measurement of specific structures and algorithms. • Analysis of algorithms predicts performance. • Testing verifies performance. • A/B testing of implementation performance. • Measure Worst-Case Execution Time (WCET) for real- time scheduling analysis. • Must be performed on target hardware.
- 32. Integration Testing • Functional, performance, load, and stress testing of isolated subsystems. • What happens when you integrate separate units into a larger component? • Within a single executable, between executables, across communication interfaces. • Recursively integrate small subsystems into larger subsystems, stepwise widening the scope. • Apply unit testing strategies to these larger components. • To some extent they're just bigger units. • A functioning subsystem tested against real and simulated components. • May be a fully-featured subsystem, or only partially implemented.
- 33. Simulators • Simulators are a huge boost. • An expansion of the mock and fake concept. • Many deployment options: • Run locally as separate processes on your development platform. • Run locally as separate processes on your test platform. • Spin up local or cloud VM's to run them. • Run on separate hardware, including some version of the IOT device. • Examples: • Real IOT device tested against simulated backend and apps. • Real backend server tested against simulated IOT devices and apps. • Real app tested against real backend server, simulated IOT devices.
- 34. Simulator Cost/Benefit • Benefits of simulators: • Decouple development: • The other components aren't ready yet, but you need to test your subsystem. • You’re in complete control of the simulated components, so they stay stable while the real ones are in flux. • You can make your simulators follow scriptable sequences, or drive unusual behavior. • Help to automate testing. • Spin up a VM and spawn off 1000 simulated IOT device processes to connect to your backend. • Spin up 1000 VM's and start a simulated IOT device or app on each one. • Costs of simulators: • Simulators need to be developed and maintained. • Inevitably, there will be some issues to iron out when you replace the simulators with real components.
- 35. Simulator Risks • Simulated components don't accurately model real components: • Message contents • Work flows • Performance • Scaling • Simulated components grow stale. • Simulators become a distraction, diverting resources from real shippable product.
- 36. System Testing • Functional, performance, load, endurance, stress, security testing of entire system. • Full stack • End-to-end • May be a fully-featured system, or only partially implemented. • Acceptance test is a form of system testing. • Is the current state of the system acceptable to ship as a usable version? • Multidimensional acceptance criteria. • Using real components, judicious simulator use. • Simulators can invalidate test results. • No matter how good they are, they aren’t the real thing. • You’ll need to repeat testing with real components. • Augment real components with simulators for load.
- 37. UI Testing • Have a sample of your actual user population use the system. • Be sure it is a representative demographic cross-section. • Functional and performance testing. • Watch for mistakes, mode confusion, and user frustration. Don’t just blame user error. • “Didn’t these idiots even try to use this before releasing it?” • If a system allows people to do stupid things, they will do stupid things, deliberately or accidentally.
- 39. Separate Environments • Production environment • Where you run with real customer resources: • Data • Devices • Connections • Test environment • Where you run with test resources. • Risky: mixed environment • Where you run tests with some mix of real and test resources.
- 40. Production Environment • The company jewels. • Portions connected to public Internet. • The barbarians are at the gates! • Security paramount, including privacy of customer resources. • Safety paramount, including safe operation of customer equipment. • You must protect it from external and internal threats. • Poor quality is an internal threat. • Therefore good testing is a defensive strategy.
- 41. Test Environments • Completely isolated from public Internet (physically or virtually). • Security provided by isolation and use of test resources: • The outside can't get in. • There’s no sensitive data in use. • At each scope: • Unit test environment per developer. • Often can use individual development environment. • Integration test environments. • System test environments. • May need multiple environments to handle different types of testing at each scope. • Isolation between different test environments prevents interference between tests.
- 42. Mixed Environment • Risky! • May include connectivity to public Internet • This is a great way to look stupid • Don't do it! • Risks • Injection of test data into production environment can cause all kinds of havoc • Interference between production service and testing • Corruption of customer accounts (where money is involved, expect lawsuits) • Exposure of customer data (including to insiders) • Unsafe operation of customer equipment • Disruption of service (how to DOS yourself) • Incomplete security • Remember the movie "War Games"?
- 43. Health Testing • Don’t confuse testing in mixed environment with health testing of production environment. • Built-in service tests used in normal operation of production environment. • Active probes, pings, queries, traces, measurements, stats collection, data sampling, etc. • All the virtual dials and gauges showing how your production environment is running. • This is a whole discussion in itself: • How to know about problems proactively before your customers do.
- 44. Test Automation
- 45. Problems With Manual Testing • People find it onerous, so may skip test steps. • Ad hoc, too much variability in how it gets done. • Poor repeatability. • Results between test runs may not be consistent enough to measure effects. • Labor intensive.
- 46. Tests Must Be Repeatable • Part of the "Show Me" method. • Regression testing: verify that things that used to work still work, didn't get broken by other work. • This can happen over the life of a system as different developers work on it. • Repeated testing exposes issues that were masked by other issues that have now been fixed.
- 47. Automation Makes It Easy • Automate everything that can be automated! • "Push-button" automatic: just need one step to run the test. • Easy to repeat. • This means people will actually use the tests. • Can be incorporated into toolchain and build pipelines. • For devops: provide push-button spin-up of virtual test environments.
- 48. Costs • Automation code itself requires development and ongoing maintenance. • Staffing • Dedicated test environments, real or virtual.
- 49. Risks (these are also risks of manual testing) • Automate only the easy stuff. • Automated suites don't sufficiently exercise enough of the code. • Automated suites grow stale. • Poor automated suites lead to false sense of security. • "The code is good, it passed the smoke test." • "Yeah, but the smoke test doesn't even trigger that functionality." • Resource contention on dedicated test environments can block development progress.
- 50. Performance, Scaling, And Endurance Testing
- 51. Differing Requirements • Different for each part of the IOT triad. • Different for each part of the communications infrastructure. • Conditions should include normal behavior (happy day testing), and with communication difficulties (bad day testing). • Anybody can look good when everything’s going right. It’s when things are going wrong that you see the true test of mettle. • Different test types may be intertwined. • Load/scaling/stress tests are variations of same things. • As system matures, today’s stress test becomes tomorrow’s normal load test. • Meanwhile, what are the performance and resource consumption in each scenario?
- 52. IOT Devices • Embedded systems with a range of soft-to-hard real-time requirements. • May have extremely long-duration endurance requirements. • Months to years. • High fault-tolerance requirements. • Need to keep working no matter what. • The communication path to the apps and backend can be very messy. • Event rate scaling/stress/performance. • Data acquisition. • Control output and feedback. • Communication events. • Resource contention under high load.
- 53. Backend Servers • Response time and scaling requirements to support many IOT devices and app instances. • Long-duration endurance requirements. • Fault-tolerance requirements. • Storage requirements. • May use the Erlang/Google model for endurance/fault-tolerance: • Cattle, not pets. • Just fail quickly (or kill) and restart. So test that!
- 54. Frontend Apps • Human interaction response requirements. • Robustness across usage and communication problems. • Data streaming requirements from IOT devices. • Local storage requirements.
- 55. Communications Infrastructure • Connection rates • Connection load • Traffic distribution and balancing • Data rates • Aggregate bandwidth • Latency • Filtering
- 56. Varying Scales • Test performance and endurance at different scales. • Starting with 1 IOT device and 1 app instance, decimal exponential growth is a reasonable progression (10, 100, 1000, etc.). • Each step may expose pain points in all sorts of places. • This is a continuous learning process. • Beware of caching effects skewing results at low scale. • Results don't necessarily scale up. • Large scale testing becomes impractical with real IOT devices. • Scale up with simulators.
- 57. Synchronizing Events • Test these at high scale • What is the effect when thousands of IOT devices or apps take some action at the same time in response to an event? • "Thundering herds" • Often pose a transient stress load. • Can have cascading effects that take a long time to settle out. • Examples: • Server restarts • Server-side network events • Domain-specific events, such as start of a live video broadcast, or item going on sale.
- 58. Performance Parameters • Response time to establish connection/session • Response time to perform various requests • Request rate • Event latency • CPU utilization • Server memory consumption per idle connection • Server memory consumption per connection in interesting use cases • IOT device memory consumption in interesting use cases • IOT device real-time response latency under various conditions • App UI responsiveness under various conditions
- 59. Security Testing
- 60. Security Threats • Security is easy to get wrong, hard to get right • False sense of security is a huge risk • Systems come under threat from multiple directions • Outsiders • Via public Internet connectivity • By breaching access into private/secure connectivity • Physical access to IOT devices • Social engineering • The easiest path in is the low-tech method: get people to undermine their own security. • Bruce Schneier: “Only amateurs attack machines; professionals target people.” • Insiders • Misuse authorized access • Take inadequate steps to secure resources • Vulnerable to outsider attack
- 61. What Do Malicious Attackers Want To Do? • Co-opt your IOT devices into botnets. • Access your customer private data: • Financial data such as credit cards, account information. • Device data such as recorded imagery. • Use your IOT devices against nearby systems and equipment. • Stuxnet • Disrupt your customers. • Disrupt your company and business.
- 62. Attack Yourself • You can be sure the outside world will be trying it. • You need a devious mind, because that's what will be brought to bear against you. • Attack at each concentric ring of security, for each part of the IOT triad and all interconnects • Assume outer rings have been breached by high- or low-tech means • Can the next level of defense hold? • Test to verify that you can trigger detection, mitigation, alerting, and auditing mechanisms at that level. • Attack data in flight and data at rest. • Across communication media, devices, ports, and buses. • In IOT device and app platform volatile/non-volatile memory, in backend storage (databases and file/block stores). • Use a test environment that matches the security profile of the production environment. • Don’t use the production environment! • Use separate security health monitoring for production. • How much time/effort/money should you invest in this? • How valuable are the things you're trying to protect? • How much time/effort/money would attackers invest in trying to get to them? • What are the potential consequences of security failures for your customers and your company?
- 63. Test Attacks • Pen test (penetration testing) of IOT devices, servers (including storage systems), and apps • Connection/login paths • Traffic capture/analysis • Cryptanalysis • DDOS attacks • Replay attacks • Man-in-the-middle attacks • Injection and buffer overflow attacks • Spoof attacks • Test with factory default, invalid, forged, expired, and “stolen” credentials (accounts/passwords, OTP codes, certificates, keys, etc.) • Spoofed IOT devices talking to servers and apps • Spoofed servers talking to IOT devices and apps • Spoofed apps talking to servers and IOT devices • IOT device and app platform external ports • IOT device teardown • Component pin/lead/bus sniffing/signal injection • Volatile/non-volatile memory/storage dump • Load foreign code
- 64. Monitoring Tests • Capture the traffic from all components of the IOT triad. • Check what they are sending and receiving, and who they are communicating with. • Scan for open ports. • Make sure you can account for all traffic and port usage as valid system functioning. • Third-party code or insiders may be trying to send data to unauthorized destinations, or receive commands from unauthorized sources.
- 65. Update Testing
- 66. Update Model • IOT devices are typically Over The Air (OTA) updatable. • Backend servers are typically directly updatable. • Apps are typically app store/web download updatable. • Need to maintain compatibility between all triad components at all times in order to maintain control, regardless of version skew. • This can get extremely complex.
- 67. Test Cases • Functional: • Version updates for each component. • Version rollbacks if supported. • Partial/failed updates in each direction. • Recovery mechanisms for bricked devices. • Rejection of unauthorized updates. • Regression test after each version test. • Performance and scaling: • For single instance of a component. • For many instances of a component. • During the update. • Post-update behavior. • Rollout model (expanding, regionalized, etc.) • Affects on normal operation.
- 69. Test Frameworks • Language-specific • Help to reduce effort in building tests. • Help automate tests.
- 70. A Random Partial List Of Tools • Google Test/Google Mock • JUnit • CxxUnit • Mocha • Chai • NYC/Istanbul • Sinon/Mockery/Rewire • Karma • Protractor • Supertest • NodeJS Debug • Cucumber • Ava • Selenium • Capybara • JMeter • Jasmine • Scriptable app simulators
- 71. Thank you!