Performance Testing in Context; From Simple to Rocket Science
2 likes982 views
The document discusses the complexities of performance testing, emphasizing that while it may seem like 'rocket science,' understanding its fundamental aspects can help improve software systems. It outlines the importance of defining performance, the responsibilities involved, the significance of performance throughout the product lifecycle, and the impact of performance testing on business value. Additionally, it provides practical strategies for effectively managing and advocating for performance testing within organizations.
![Thoughts on Questions
How is the performance today?
How will this [change] effect performance?
Go to dev, test, & management meetings – and ask
those questions.
Advocate performance through questions.
Earning the “annoying” label for asking
questions is ok, but becoming “annoying”
via your response to their answers isn’t.
Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.](https://image.slidesharecdn.com/simple2rocketscience-120513214533-phpapp01/85/Performance-Testing-in-Context-From-Simple-to-Rocket-Science-29-320.jpg)
Performance Testing in Context; From Simple to Rocket Science
- 1. Context Appropriate Performance Testing: From Simple to Rocket Science Created for: By: Scott Barber Chief Technologist PerfTestPlus, Inc. Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 2. “Let’s face the truth, performance testing *IS* rocket science.” --Dawn Haynes … but even rocket science involves *SOME* easy stuff. --Addendum added by: Scott Barber Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 3. Context Appropriate Performance Testing: From Simple to Rocket Science Scott Barber Chief Technologist, PerfTestPlus, Inc. sbarber@perftestplus.com www.perftestplus.com @sbarber Co-Founder: Workshop On Performance and Reliability www.performance-workshop.org Author: Co-Author: Contributing Author: Books: www.perftestplus.com/pubs About me: about.me/scott.barber Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 4. Let’s Get Organized… Introductions: Who are you? What do you do? Performance experience? Desired outcome? Team Formation: Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 5. Let’s Start with an Exercise As a team, come up with one answer each to: What is “Performance”? What is “Performance Testing”? Who is responsible for Testing Performance? How do you know if Performance is good or bad? De-Brief & Discuss: My Answers: Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 6. What is Performance? System or application characteristics related to: Speed: - responsiveness - user experience Scalability: - capacity - load - volume Stability - consistency - reliability - stress Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 7. What is Performance Testing? What mom tells people: I help people make websites go fast. What I tell people: I help and/or teach individuals and organizations to optimize software systems by balancing: • Cost • Time to market • Capacity while remaining focused on the quality of service to system users. Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 8. Who is Responsible? Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 9. The Performance Lifecycle is: Conception to Headstone Not Cradle to Grave Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 10. Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 11. Good/bad, How Do You Know? (But mostly, you don’t until it’s “too late”) (Yet we can know if our Performance Testing is adding value via…) Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 12. Performance Testing Objectives What we actually hope to gain by testing performance Are sometimes completely unrelated to stated requirements, goals, thresholds, or constraints Should be the main drivers behind performance test design and planning Usually indicate the performance-related priorities of project stakeholders Will frequently override good/bad in “go-live” decisions Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 13. “With an order of magnitude fewer variables performance testing could be a science, but for now, performance testing is at best a scientific art.” --Scott Barber Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 14. Fact: As an activity, performance testing is widely misunderstood, particularly by executives and managers. This misunderstanding can lead to a variety of difficulties -- including outright project failure. Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 15. Fact: Managers and executives do not need to understand the technical details of performance testing to make good decisions or effectively manage performance testing projects. They do need to understand what performance testing is, what it is not and what value it adds. Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 16. Fact: Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 17. Fact: Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 18. Fact: Poor Performance is Bad for Business Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 19. Fact: One does not need to be a performance testing rock star to have a significant positive impact on performance… …and thus add significant business-value… …quickly and simply. Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 20. “There is no such thing as a ‘junior performance tester’… but there are people who are new to performance testing.“ --Scott Barber Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 21. So then, its hopeless? Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 22. Simple Item #1 Make Performance a: Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 23. Thoughts on Priority Focusing on performance adds value and mitigates risk from “bar napkin to delete key”. Get performance in the dev, test, & delivery plans. Don’t let performance fall off the plate. Be the advocate, even if it makes you “annoying”. Goals: Soft Boundaries (User Satisfaction) Requirements: Firm Boundaries (Business or Legal) Thresholds: Hard Boundaries (Laws of Physics) Constraints: Arbitrary Boundaries (Budget or Timeline) Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 24. Simple Item #2 Give Performance: Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 25. On Visibility Acceptance criteria Ask questions Priority setting Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 26. Executive Visibility • Current Capacity • Single User • Projections Responsiveness • Scalability Plan • Resource • Reliability Management • Component-Level Concurrency Architects, DevOps & Analysts Architects & Scrum Prod Scrum & Beta CapEx Alpha Load Scrum & Team & Load Dev Ops Team • Prepare for Prod • Tune/Optimize • Multi-User • Predict Responsiveness • Early Warning • Resource Trending Identification • Limit Identification Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 27. Process Visibility Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 28. Simple Item #3 Ask lots of: Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 29. Thoughts on Questions How is the performance today? How will this [change] effect performance? Go to dev, test, & management meetings – and ask those questions. Advocate performance through questions. Earning the “annoying” label for asking questions is ok, but becoming “annoying” via your response to their answers isn’t. Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 30. Simple Item #4 Research the Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 31. How are “leaders” doing? Keynote Systems http://www.keynote.com/keynote_competitive_research/ Gomez Benchmarks http://www.gomez.com/benchmarks/ WebMetrics http://www.webmetrics.com/resources/ The eService Index http://www.vertain.com/?pesi Others?? Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 32. “Ok, that stuff is ‘simple’ and I can see the value, but what about the *testing*?! Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 33. Testing Item #1 Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 34. Thoughts on Annoyance Why am I annoyed? How annoyed am I? Does this annoy me all the time, or just sometimes? What impact is this likely to have on product value? Advocate something better. Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 35. Testing Item #2 Who else is annoyed? Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 36. More Thoughts on Annoyance Who matters? How do I get their feedback? Are they annoyed with performance, or workflow, or, or, or…? Advocate something better. Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 37. “I guess that counts as ‘testing’ and yes, there is value, but what about testing *performance*?! Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 38. Performance Testing Item #1 Determine: Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 39. Speedy Speed Collection http://www.websiteoptimization.com/services/analyze/ http://www.websitepulse.com/help/tools.php http://webwait.com/ If you don’t have speed targets, don’t fret... Trends are trendy! Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 40. Speedy Speed Collection (and more) Visual Round Trip Analyzer IBM Page Detailer Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 41. Performance Testing Item #2 Make use of: Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 42. On Taking Snapshots http://www.webpagetest.org/ https://developers.google.com/pagespeed/ http://www.softwareqatest.com/qatweb1.html Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 43. Performance Testing Item #3 Test the Front-End with Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 44. What is SCORN, anyway? Media, HTML, styles & scripts – compress & minify. The end-user’s browser cache can be your best friend, or your worst nightmare, use it wisely. Get the load order of your scripts and styles wrong, and you’ll lose your users every time – even though response time hasn’t changed! 3, 4, & 5xx series response codes on individual objects are bad things. When it comes to performance, less is more (usually). Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 45. SCORN References • High Performance Web Sites: Essential Knowledge for Front-End Engineers, by Steve Souders, O’Reilly, 2007. • Yahoo! YSlow for Firebug High Performance Web Sites: Essential Knowledge for Front-End Engineers • Page Speed • Right Click -> View Source and other Tips for Performance Testing the Front End, by Scott Barber, for AST Update, 2007. www.amazon.com/dp/0596529309 Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 46. Performance Testing Item #3 Proceed with caution, the following is only easy for hard-core Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 47. Easy Stuff for Geeks Firefox Performance Tester’s Pack https://addons.mozilla.org/en-US/firefox/collection/performance Web Site Test Tools and Site Management Tools http://www.softwareqatest.com/qatweb1.html Fiddler http://www.fiddler2.com/fiddler2/version.asp Web Development Helper http://projects.nikhilk.net/WebDevHelper Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 48. “Alright, that covers ‘performance testing’ and value, but you said something about *rocket science*?! Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 49. Measurements and Metrics For each goal, determine what information will answer: • Has this goal been achieved? • To what degree? • What needs to be done to achieve this goal? Decide what data must be collected to provide that information Figure out how to collect that data Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 50. Analysis All three have an average of 4. Which has the “best” performance”? How do you know? Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 51. “Only performance testing at the conclusion of system or functional testing is like ordering a diagnostic blood test after the patient is dead.” --Scott Barber Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 52. Which Transactions to Test Common activities (get from logs) e.g. Resource hogs (get from developers/admins) Even if these activities are both rare and not risky SLA’s, Contracts and other stuff that will get you sued What the users will see and are mostly likely to complain about. What is likely to earn you bad press New technologies, old technologies, places where it’s failed before, previously under-tested areas Don’t argue with the boss (too much) Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 53. Which Transactions to Test Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 54. “Linear extrapolation of performance test results is, at best, black magic. Don’t do it (unless your name is Connie Smith, PhD. or Daniel Menasce, PhD.)” --Scott Barber Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 55. Reporting Inspired by “ET”: Edward Tufte, Ph.D., Professor Emeritus of political science, computer science and statistics, and graphic design at Yale. According to ET: Power Corrupts... Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 56. Reporting PowerPoint Corrupts Absolutely. Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 57. So, what is Performance Testing? In effect: Performance testing helps stakeholders make decisions regarding product value and project risk; Specifically value and risk related to speed, scalability, and stability attributes of a system and it’s components throughout the product life-cycle. Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 58. Review & Questions Did we learn anything? Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.
- 59. Contact Info Scott Barber Chief Technologist PerfTestPlus, Inc E-mail: Web Site: sbarber@perftestplus.com www.PerfTestPlus.com Blog: Twitter: scott-barber.blogspot.com @sbarber Simple to Rocket Science © 2012 PerfTestPlus, Inc. All rights reserved.