Performance Testing in the Cloud
- 1. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo Performance Testing in the Cloud Stuart Moncrieff & Chris De Lorenzo Iqnite Australia 2014
- 2. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo 2 www.myloadtest.com About Us… Stuart Moncrieff •Performance Testing consultant for over 10 years •More than 10,000 people read something I have written about performance testing every month −www.myloadtest.com Chris De Lorenzo •Performance & Automation Test Lead at Tabcorp •Lives and Breaths Performance Testing
- 3. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo 3 www.myloadtest.com Presentation Roadmap Part 1 •Tabcorp – Auto-scaling •Vic Emergency – testing with a CDN •Monitoring •Auto-scaling Case Studies & Examples •Tabcorp – load testing at massive scale •SA Health – using cloud- based load gens to test an internal app Load Generators in the Cloud •Testing outside the firewall •When you need a lot of load generators quickly and cheaply •Cloud costs Load Testing Cloud- based applications Case Studies & Examples Part 2
- 4. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo 4 www.myloadtest.com Local Load Gens Cloud-based Load Gens Cloud-based Test Envt Local Test Envt traditional load testing Previous Cloud Experience
- 5. 5 © Copyright 2014 Stuart Moncrieff, www.myloadtest.com www.myloadtest.com Part 1: Load Generators in the Cloud
- 6. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo 6 www.myloadtest.com Public Cloud (IaaS) Cost Calculation Example costings for a 2-week test cycle •50 Large Windows instances in the Sydney Region •10 days of test execution •4 hours of test execution per day •4 GB of data transferred per hour during testing On-Demand Instance Type (in Sydney Region) Cost/Hour Small (1.7GB, 1 CPU core/1 thread) $0.12 Medium (3.75GB, 1 CPU core/2 threads) $0.23 Large (7.5GB, 2 CPU cores/4 threads) $0.46 Extra Large (15GB, 4 CPU cores/8 threads) $0.92 Data transfer = $0.18/GB (charged for outbound data only) Elastic IPs = $0.005/hour (when not in use) Line item Calculation Cost Instances 50 x 10 x 4 x $0.46 $920 Data 10 x 4 x 4 x $0.18 $29 IP Addresses 50 x 10 x (24-4) x $0.005 $50 TOTAL $999 A Performance Testing workload suits cloud-based infrastructure perfectly. Why have servers sitting idle for 20 hours/day?
- 7. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo 7 www.myloadtest.com Use Case 1: Testing outside of the firewall Your users are on the Internet, but your load generators are in the Data Centre. What potential bottlenecks are you not testing? network devices load balancer firewall system under test end user load generator Data Centre the Internet Bottleneck: Internet link capacity Bottleneck: internal firewall Bottleneck: Network devices
- 8. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo 8 www.myloadtest.com Why aren’t you testing from the cloud? If you are testing a public-facing app, ask yourself these questions: •What potential bottlenecks am I not testing? •What additional risk is the project carrying due to this test approach? •Is this the right person to be doing my performance testing?
- 9. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo 9 www.myloadtest.com Use Case 2: You need a lot of load generators It may be impractical to organise enough load generators for large tests Virtual User Type Vusers/Load Gen Web (HTTP/HTML) 2000 Flex 500 TruClient 50 RDP 20 Citrix ICA 20 SAP GUI 20 Note: these are approximate numbers only.
- 10. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo 10 www.myloadtest.com Case Study: Load testing an internal app at SA Health Load testing an internal app using RDP virtual users. Impossible to get 50 load generators set up in time for test execution dates. •Set up load generators and RDS farm in AWS cloud. •Main problem was connectivity from cloud to internal network of large government department. •First time doing anything “cloud” for SA Health. Security team had no process for it. Escalated exemptions. •Set up Amazon “Virtual Private Cloud”, and created a VPN connection to internal network. VPN was restricted to test environment IP addresses. •Lots of meetings! system under test The Cloud LoadRunner Controller load gens Remote Desktop servers Data Centre Install client application here VPN
- 11. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo 11 www.myloadtest.com Case Study: Melbourne Cup Race 7 Jumps @ 15:00 The race that stops a nation and hopefully not our site! •Total transactions supported: 68,000,000 •Total bets: 11,000,000 •Total logins: 1,500,000 •Unique users per hour: 321,000 •Peak transaction rates per second −Betting Peak: 3,046 −Paying Peak: 10,916 −Customer Transactions Peak: 1,317 −Mobile Native Apps Peak 3,846 $90.6 million in turnover on just the Melbourne Cup Race Mobile Growth Trending Up Large focus for performance testing Melbourne Cup Year # of Bets From Mobile Native Apps % Growth 2011 186,024 2012 837,489 350% 2013 1,281,022 53% 2014 Tuesday 4th November
- 12. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo 12 www.myloadtest.com Drivers to Cloud Infrastructure Driver: Large revenue growth continuing on digital distribution channels •Cost •Supports testing objectives −Ease of use −Elastic IP Capability •Industry is Maturing −Get On It! -> {Natural Selection} −High Compute! −API Integration •Challenges −Privacy | Terms and Conditions | Security | Accounts Payable The driver to migrate load testing capability to the cloud?
- 13. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo 13 www.myloadtest.com Cloud Performance Testing - Tabcorp Physical / Internal Cloud and External Cloud Lesson’s learned from the TAB •Still have internal physical and internal cloud machines •Trial a number of products prior to defining final solution •Amazon worked for us due to: −Automation •Small, Medium, Large Executions −Firewall = No worries −Quick Provisioning (min) / Cleanup (sec) −Up and running in 1 week −External location •DDOS, Bandwidth, Data Cleansing services
- 14. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo 14 www.myloadtest.com Managing a large number of Load Generators What does it take? Final Product •Final Product – Getting There… −Invest in automation to take out the manual bottlenecks −Execution Scheduled •Bamboo and Ansible Tower •All sized executions −Status of execution is published −Screens are available for everyone to see −Large scale testing with minimal effort
- 15. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo 15 www.myloadtest.com So what does it look like? This is just a sneak peak
- 16. 16 © Copyright 2014 Stuart Moncrieff, www.myloadtest.com www.myloadtest.com Part 2: Load Testing Cloud-based apps
- 17. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo 17 www.myloadtest.com Infrastructure Monitoring Good instrumentation is critical for performance testing and for your Operations team Performance testing should also validate that the system has sufficient monitoring for future trouble-shooting •Amazon CloudWatch vs. Azure Diagnostics −Azure gives average of last hour −CloudWatch counters are 1 minute averages (i.e. 1-minute granularity) •CloudWatch console access is free, but you need to pay for API access •Can write own values to counters. This is worth doing. Maximise your visibility into system health and application behaviour
- 18. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo 18 www.myloadtest.com Hardware won’t save you! Myth 1: Adding server capacity solves all performance problems Myth 2: Infinite cloud capacity means that you don’t need to do any performance testing •Bottlenecks could be due to code or configuration, not just capacity •What are the operational costs of paying for infinite hardware?
- 19. 19 © Copyright 2014 Stuart Moncrieff, www.myloadtest.com www.myloadtest.com “A backup that you haven't done a successful test recovery from is #notabackup.” -- Some guy on Twitter “Don’t assume your auto-scaling works unless you have actually tested it under load.” -- Stuart Moncrieff
- 20. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo 20 www.myloadtest.com Auto-Scaling is not a Silver Bullet If your system uses auto-scaling, you must test auto-scaling under load. •What metric will trigger auto-scaling? −Is it actually the metric that indicates a bottleneck? •Optimise before auto-scaling. −Tune until your workload is either CPU-bound, or I/O bound. •How fast will auto-scaling kick in? −Azure metrics based off last hour (moving average). A sudden spike in demand will not trigger auto-scaling. −How long does it take an instance to spin up? Auto-scaling never seems to work correctly the first time
- 21. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo 21 www.myloadtest.com War Story: Auto-scaling at Tabcorp Considerations •Even with a slow ramp-up solution failed due to 6 min provisioning time −Based off CPU% > 50% −Realized there are too many variables… •Settled with setting “enough” capacity for “predictable peaks” −Works for Tabcorp
- 22. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo 22 www.myloadtest.com Traps for new players New (or important) test cases: •Can you redeploy while under load? •Does alerting/monitoring work? −Will your Ops team have enough visibility to solve problems in Production? •Does Autoscaling work as expected? •Failover/HA/DR test cases −Between availability zones, between regions −Do instances need to be “warmed up” before putting them under load? •How will you test something behind a CDN? −All traffic hits single POP of CDN, means less traffic hits backend (VicEmergency example) Watch out for: •First time “cloud architects” −You can always get help through your AM/SA •How reproducible are your test results? −“noisy neighbours”, CPU steal −Inconsistent hardware for same instance type •Routes for traffic from load gens −All traffic routed over small pipe (MFB example) •Restricting access to your test environment to load generators (using a whitelist on your firewall) −Default limit of 5 static IP addresses (Elastic IPs) •Monitoring in a PaaS environment −You won’t have OS-level access to servers It’s a trap!
- 23. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo 23 www.myloadtest.com Summary Load generators in the Cloud •Cost model suits servers which are not needed 24 hours/day. Low-cost, pay as you go. •Great for testing public-facing applications from outside the firewall. •Useful for situations where you need a large number of load generators. Load testing Cloud-based applications •Infrastructure monitoring is different. Invest the time to set this up properly. •Don’t assume auto-scaling will work perfectly without testing under load. New Cloud-specific test cases!
- 24. 24 © Copyright 2014 Stuart Moncrieff, www.myloadtest.com www.myloadtest.com Thank you Please see us afterwards if you have questions Email: •stuart@myloadtest.com Twitter: •@StuartMoncrieff LinkedIn •http://au.linkedin.com/in/stuartmoncrieff Website: •http://www.myloadtest.com Email: •cdelorenzo@gmail.com Twitter: •@cjdelorenzo LinkedIn •http://au.linkedin.com/in/cjdelorenzo