Production Performance Testing in the Cloud
- 1. TK PM Tutorial 4/30/13 1:00PM Production Performance Testing in the Cloud Presented by: Dan Bartow SOASTA, Inc. Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
- 2. Dan Bartow At SOASTA Dan Bartow is vice president of product management. Prior to joining SOASTA, Dan was senior manager of engineering at Intuit where his team was responsible for the speed and stability of TurboTax Online. During the past decade he has been responsible for the performance of websites for leading consumer brands. Dan has set multiple industry precedents including launching the world's largest stateful JBoss cluster and using more than 2,000 cloud computing cores to generate load against a live website. He is credited with authoring the industry’s first testing in production methodology. A frequent industry presenter, Dan has spoken at leading testing and cloud computing conferences.
- 3. Production Performance Testing in the Cloud Dan Bartow SOASTA Objectives 1. 2. 3. 4. 5. Explain the fundamentals of cloud computing and its application to fullscale performance validation Describe how to create a performance testing strategy to give your team critical data about how your online application performs and scales Learn how to create a robust lab-to-production ecosystem that delivers the answers about what will happen when peak traffic hits your application Share the best practices and techniques needed to design and execute a successful testing-in-production strategy Give practical approaches to mitigate the three most common problems security, test data, and live customer impact - that arise when designing production test plans 1
- 4. Agenda 1. Performance Engineering Methodology Categorization of Sites by Traffic Effort vs. Gain 2. Cloud Computing Overview What is cloud computing? The killer app for cloud Does it work for testing? 3. Lab to Launch Ecosystem 4. Production Performance Testing Security Test data in production Potential live customer impact Discipline and rigor Most common bottlenecks 5. Case Studies and Discussion TurboTax Online Myspace Green Button Essential components - lab and live Evolution of the testing cycle Designing good test plans Monitoring About the Speaker SOASTA VP Product Management CloudTest Evangelist Intuit Sr. Manger, Engineering TurboTax Online and E-com ATG Sr. Deployment Engineer Work American Airlines, Best Buy, Target, Turbotax Online, QuickenOnline, MySpace, Dennys, Dominos, Mattel, Hallmark, FAA, US Army, AT&T Wireless, Alcatel, Newsweek, Oprah, NeimanMarcus, SBC, Plantronics, Kodak, JCrew, Cingular, Newell Rubbermaid 2
- 5. Poor performance. How Deep Should We Go? 3
- 6. Black Friday | Cyber Monday - 2011 http://www.internetretailer.com/2011/11/26/outage-hits-victoriassecretcom-black-friday http://www.internetretailer.com/2011/11/28/thanksgiving-weekend-brings-multiple-site-headaches No Outages in the Top Ten US Online Retailers! 4
- 7. In 2011 Wal-Mart Kroger Target Walgreen The Home Depot Costco CVS Caremark Lowe’s Best Buy Sears Holdings At Least Six of the Top Ten Online Retailers in the US Tested Their Apps – In Production – from the Cloud http://www.stores.org/2011/Top-100-Retailers Methodology SOASTA CloudTest Methodology Strategy Implementation Execution Measurement Define People Define Analyze Integrate Processes Design Adjust Test Assess 5
- 8. Defining Performance What does it mean? Capacity? Page Speed? Reliability? Scalability? Transaction Speed? Code Execution Speed? Defining Performance 1. Capacity 2. Customer Experience 3. Reliability 4. Scalability 5. Transaction 6. Code Level 6
- 10. Categorizing Sites by Traffic Categorizing Sites by Traffic 8
- 11. Categorizing Sites by Traffic The top 10 websites get 42% of the visitors to the top 100, and 21% of the visitors to the top 1,000* The top 100 get 50% of the visitors to the top 1,000. I.e. the top 100 together get as many visitors as the following 900 websites counted together* If you want to break into the top 100 websites, you’ll need at least 22 million visitors per month. If you “only” aim for the top 1,000, you’ll need at least 4.1 million visitors per month* Gee, that can’t be so hard, can it…? *SOURCE: http://royal.pingdom.com/2010/07/05/what-it-takes-to-be-a-top-100-websitecharts/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+RoyalPingdom+%28Royal+Pingdom%29 Categorizing Sites by Traffic Top 10 – Mega Sites* 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Google Facebook Youtube Yahoo! Windows Live Baidu.com Wikipedia Blogger.com Twitter QQ.com *SOURCE: Alexa Top 500 Worldwide - http://www.alexa.com/topsites 9
- 12. Categorizing Sites by Traffic Top 10-500 – Massive Sites MSN Amazon.com Wordpress eBay LinkedIn Craigslist MySpace Flickr Apple Paypal Reddit Mashable Skype Amazon WS Yelp WSJ Match.com Walmart Salesforce AT&T *SOURCE: Alexa Top 500 Worldwide - http://www.alexa.com/topsites Categorizing Sites by Traffic Top 500 – 2000 XL Sites Top 2000-10000 Large Sites Orbitz Travelocity NBA United AmericanAirlines Marriott NHL Dominos Avis Mint WhiteHouse.gov *SOURCE: Alexa - http://www.alexa.com/ 10
- 13. The Point? Where should we focus the performance engineering efforts? Testing in Production Most common problems during peak traffic periods Bandwidth Load Balancers Database Scale 11
- 14. Production Performance Testing Anyone can do it - the NYSE did production performance testing in the eighties Some enterprises have teams dedicated to production performance testing Most Commonly Used 12
- 15. Typical Approach to Testing Test in a lab at a smaller scale and then use . Extrapolation Extrapolation Extrapolation Extrapolation Extrapolation Extrapolation to Prod Breaks Down Load Environment Invisible walls can only be identified out in the production environment and through testing at scale – as load and environment changes, so do problems 13
- 16. Thank You, Public Cloud! Some Cloud Providers and Locations Amazon EC2 – Arlington, San Francisco, Portland, Dublin, Japan, Singapore GoGrid – San Francisco IBM BlueCloud – Toronto, Denver, Raleigh Microsoft Azure – Chicago, San Antonio, Amsterdam, Dublin, China, Singapore RackSpace – London, Dallas, Chicago 14
- 17. Cloud Computing Q: What is Cloud Computing? A: Access to affordable infinite compute power. Accessibility, Availability and Affordability. SAAS PAAS IAAS A killer application (commonly shortened to killer app), in the jargon of technologists, has been used to refer to any computer program that is so necessary or desirable that it proves the core value of some larger technology…* *source: wikipedia Lab to Launch Ecosystem Essential components – lab and live 15
- 18. Lab to Launch Ecosystem Evolution of the testing cycle 100,000 10,000 1,500 Lab Stage Production Lab to Launch Ecosystem Designing good test plans Heat Map Scenario Guide Test Plan Guide 16
- 19. Lab to Launch Ecosystem Monitoring What Where How Production Performance Testing Security Test data in production Potential live customer impact Tools and techniques 17
- 20. Production Performance Testing Discipline and rigor The Play Book Production Performance Testing Most common bottlenecks 18
- 21. Practice #1 Test accounts in production Practice #2 Place orders in production 19
- 22. Practice #3 Test on the live site Practice #4 Test the top 6-8 critical transaction flows 20
- 23. Practice #5 Don’t test from the same cloud you’re hosted in Case Studies Source: http://www.soasta.com/customers/financial/ss_turbotax/ 21
- 24. Case Studies Source: http://highscalability.com/blog/2010/3/4/how-myspace-tested-their-live-site-with-1-million-concurrent.html Case Studies Source: http://www.greenbutton.net/blog#post349 22
- 25. Contact Information Dan Bartow VP Product Management CloudTest Evangelist SOASTA Inc. Email: Twitter: LinkedIn: Blog: dan@soasta.com www.twitter.com/PerfDan www.linkedin.com/in/danbartow/ www.soasta.com 23