Continuous Testing
Download as PPTX, PDF2 likes2,642 views
The document discusses continuous testing in DevOps. It introduces SOASTA and their products for load and performance testing (CloudTest) and mobile functional test automation (TouchTest). SOASTA advocates for continuous integration with small batches, version control, automation, and continuous feedback to align development and testing teams. The presentation provides examples of how SOASTA's clients implement continuous integration by running unit tests and load tests on every code change through Jenkins and addressing issues before deployment.
Continuous Testing
- 1. September 9, 2015 “Continuous Testing in DevOps” Dan Boutin – Senior Product Evangelist dboutin@soasta.com Mobile (404) 304-9529 @DanBoutinSOASTA
- 2. © 2014 SOASTA. All rights reserved. 2 Agenda: • SOASTA Introduction • The need for “Continuous” • The “Big 3” • CI with mobile and performance testing
- 3. © 2014 SOASTA. All rights reserved. 3 100 Billion User Experiences Tested 10 Million Tests Performed Actual CloudTest view
- 4. © 2014 SOASTA. All rights reserved. 4 Agenda: • SOASTA Introduction • The need for “Continuous” • CI with mobile and performance testing
- 5. © 2014 SOASTA. All rights reserved. 5
- 6. © 2014 SOASTA. All rights reserved. 6 o Small Batch Sizes
- 7. © 2014 SOASTA. All rights reserved. 7 o Small Batch Sizes o Comprehensive Version Control
- 8. © 2014 SOASTA. All rights reserved. 8 o Small Batch Sizes o Comprehensive Version Control o Simple Branch Strategies
- 9. © 2014 SOASTA. All rights reserved. 9 o Small Batch Sizes o Comprehensive Version Control o Simple Branch Strategies o Automation
- 10. © 2014 SOASTA. All rights reserved. 10 o Small Batch Sizes o Comprehensive Version Control o Simple Branch Strategies o Automation o Continuous Feedback
- 11. © 2014 SOASTA. All rights reserved. 11 o Small Batch Sizes o Comprehensive Version Control o Simple Branch Strategies o Automation o Continuous Feedback o Emphasis on working builds
- 12. © 2014 SOASTA. All rights reserved. 12 o Small Batch Sizes o Comprehensive Version Control o Simple Branch Strategies o Automation o Continuous Feedback o Emphasis on working builds o Consistent environments
- 13. © 2014 SOASTA. All rights reserved. 13 o Small Batch Sizes o Comprehensive Version Control o Simple Branch Strategies o Automation o Continuous Feedback o Emphasis on working builds o Consistent environments o Developer – Tester Collaboration
- 14. © 2014 SOASTA. All rights reserved. 14 Agenda: • SOASTA Introduction • The need for “Continuous” • CI with mobile and performance testing
- 15. © 2014 SOASTA. All rights reserved. 15 • We can only test as fast as the conveyor belt allows. • If the conveyor belt is moving slow, testing is delayed. • The Big 3 drives testing cycle. o Why is CI critical? https://youtu.be/8NPzLBSBzPI • Speed • New features, functions, etc. need to get to the customer, ASAP! “We didn’t have enough testing for a very complicated project” US Health and Human Services Secretary, Kathleen Sebelius
- 16. © 2014 SOASTA. All rights reserved. 16 s Memory leaks Scale of Test Stage/Team Dev & Test Release & Deploy Network & Ops CDN file placement Load Balancer configuration Network bandwidth Network configuration DNS routing Inadequate server resources Default configuration settings Unbalanced web serversAuto-scaling failures Latency between systems Slow third-party plug-ins Garbage collection Database thread counts Inefficient database queriesSlow pages Conflict with other apps Test Lab Staging Production (100% +++) Search technology limits Method-level tuning Max sockets exceeded Firewall max capacity Global latency variance Security bottlenecks Tuning for full-scale…and well beyond
- 17. © 2014 SOASTA. All rights reserved. 17 Does it work as designed? Does it perform under extreme circumstances? Approve materials Real Drivers What happens in actual conditions? Assembly Line
- 18. © 2014 SOASTA. All rights reserved. 18 Functional validation Performance, reliability, scalability tests Real Users Build What happens in actual conditions? Unit-tested code
- 19. © 2014 SOASTA. All rights reserved. 19
- 20. © 2014 SOASTA. All rights reserved. 20 Jenkins runs any unit tests as usual
- 21. © 2014 SOASTA. All rights reserved. 21 Run iterative load tests via Jenkins Spin up suitable, consistent test environments in the cloud or the lab Jenkins runs any unit tests as usual
- 22. © 2014 SOASTA. All rights reserved. 22 Run iterative load tests via Jenkins Mobile? Run functional automation on real devices for functional & performance validation Spin up suitable, consistent test environments in the cloud or the lab Jenkins runs any unit tests as usual
- 23. © 2014 SOASTA. All rights reserved. 23 Run iterative load tests via Jenkins Daily (or more frequent) updates keep teams quality focused Embedded failure details & bottleneck data isolate issues Mobile? Run functional automation on real devices for functional & performance validation Spin up suitable, consistent test environments in the cloud or the lab Jenkins runs any unit tests as usual
- 24. © 2014 SOASTA. All rights reserved. 24 Run iterative load tests via Jenkins Daily (or more frequent) updates keep teams quality focused Embedded failure details & bottleneck data isolate issues Manage to a performance baseline Mobile? Run functional automation on real devices for functional & performance validation Spin up suitable, consistent test environments in the cloud or the lab Jenkins runs any unit tests as usual
- 25. © 2014 SOASTA. All rights reserved. 25 Use real user data to iterate dev & test plans Run iterative load tests via Jenkins Daily (or more frequent) updates keep teams quality focused Embedded failure details & bottleneck data isolate issues Manage to a performance baseline Mobile? Run functional automation on real devices for functional & performance validation Spin up suitable, consistent test environments in the cloud or the lab Jenkins runs any unit tests as usual
- 26. © 2014 SOASTA. All rights reserved. 26 o Load and Performance Testing with Cloudtest o Mobile Functional Test Automation with TouchTest
- 27. © 2014 SOASTA. All rights reserved. 27 o SOASTA’s Process Metrics • 3 Main Products • (2) Builds per day of each product (Dev branch + Customer branch) • 8,000 tests executed per build • For Mobile (TouchTest), Over 300 fully-automated tests each for iOS & Android
- 28. © 2014 SOASTA. All rights reserved. 28 o Understand your requirements o Identify the cloud advantages for you o Automate the obvious and most critical o Connect to a continuous process o Align teams with actionable information
- 29. © 2014 SOASTA. All rights reserved. 29 o Understand your requirements o Identify the cloud advantages for you o Automate the obvious and most critical o Connect to a continuous process o Align teams with actionable information
- 30. © 2014 SOASTA. All rights reserved. 30 o Understand your requirements o Identify the cloud advantages for you o Automate the obvious and most critical o Connect to a continuous process o Align teams with actionable information
- 31. © 2014 SOASTA. All rights reserved. 31 o Understand your requirements o Identify the cloud advantages for you o Automate the obvious and most critical o Connect to a continuous process o Align teams with actionable information
- 32. © 2014 SOASTA. All rights reserved. 32 o Understand your requirements o Identify the cloud advantages for you o Automate the obvious and most critical o Connect to a continuous process o Align teams with actionable information
- 33. © 2014 SOASTA. All rights reserved. 33CONFIDENTIAL – Not for Distribution Revenue Brand Competitive advantage
- 34. © 2014 SOASTA. All rights reserved. 34 We will close with some Mobile Trivia @DanBoutinSOASTA
- 35. © 2014 SOASTA. All rights reserved. 35 Questions? Dan Boutin – Senior Product Evangelist dboutin@soasta.com Mobile (404) 304-9529 @DanBoutinSOASTA o Visit our Blog: http://goo.gl/TA2HKP o Questions? @DanBoutinSOASTA
- 36. 78 days 16 hours
- 37. September 9, 2015 “Continuous Testing in DevOps” Dan Boutin – Senior Product Evangelist dboutin@soasta.com Mobile (404) 304-9529 @DanBoutinSOASTA