ConFoo: Moving web performance testing to the left

o What is the impact of Web Performance?
o Your Web Site and the need for Performance Testing
o Typical Performance Testing
o Issues found in Performance Testing
o Making Performance Testing Continuous

What do users care about?
Survey Says:

Source: http://www.soasta.com/blog/web-performance-mothers-day/
Fast: they buy more.
Slow: they buy less.

Performance affects every business KPI
• Revenue
• Conversions
• User satisfaction
• User retention
• Time on site
• Page views
• Bounce rate
• Organic search traffic
• Brand perception

Impact of a 1-second delay
A site that earns $100K per day
could lose $2.5M in sales
in one year

…website slowdowns occur
10X more often than outages.
Source: TRAC Research

Free download:
soasta.io/timeismoneybook

12
Anatomy of Your Web App
Responsive web site Web Servers
• Desktop Users
• Mobile Users
• Same or similar content
o Adapting to different
screen sizes
Mobile optimized
• “m-dot” site for mobile
• Custom built for mobile devices
All have html, images, JavaScript, css, plus third-party content

13
Native App
• Android – Java
• iOS – Objective C/Swift
• Mobile app framework
o Same source code
Hybrid App
• Mix of native code + html
All have backend calls for login/authentication, account
settings, database calls, transactions, plus third-party SDK
calls
Mobile Apps and Back End Services
WebSite/Mobile
Back end

14
The Need for Load Testing
• Everyone runs load testing on their website, right?
o Load and Performance testing is often not done or too late
o Black Friday, Cyber Monday, Holiday failures
• What’s the need for mobile web load testing?
o Mobile is growing faster than desktop
o Responsive sites and mobile sites deliver different content to
mobile devices over mobile network
• What about mobile app load testing?
o Mobile app back ends serve different content to mobile apps
o Remember Pokemon Go? It wasn’t the app that failed.

15
When is your Peak Season?
• Peak is year-round
o Super Bowl Sunday – 110 million viewers
o Bi-weekly or monthly payroll processing
o Valentine’s Day, President’s Day, Mother’s Day, Father’s Day
o Quarterly processing and filing
o U.S. Tax filing
o Spring, Summer, Fall & Winter season sales
o July 4th, Christmas in July
o Back to School
o Olympics
o Halloween
o Election Day
o Singles Day – world’s biggest shopping day, $9B in 12 hours
o Black Friday, Cyber Monday and Holidays
Back to top to repeat again next year!

s
Memory leaks
Scale of Test
Stage/Team
Dev
& Test
Release
& Deploy
Network
& Ops
CDN file placement
Load Balancer configuration
Network bandwidth
Network configuration
DNS routing
Inadequate server resources
Default configuration settings
Unbalanced web serversAuto-scaling failures
Latency between systems
Slow third-party plug-ins
Garbage collection
Database thread counts
Inefficient database queriesSlow pages
Conflict with other apps
Test Lab Staging Production (100% +++)
Search technology limits
Method-level tuning
Max sockets exceeded
Firewall max capacity
Global latency variance
Security bottlenecks
Tuning for full-scale…and well beyond

Testing In General
Linear approaches include delays between testing
Ut
Unit Testing
Ft
Functional
Testing
Development Testing Operations
Pt
Performance
Testing
Mo
Monitoring
Legacy Approach: Silos and waiting
TIME

Eliminating Delays Compresses Time
Enabling more testing to be done earlier and more often
Ut
Unit Testing
Ft
Functional
Testing
Testing
Pt
Performance
Testing
Mo
Monitoring
Legacy Approach: Silos and waiting
Agile Approach: Automation and Continuous Testing
OperationsDevelopment
Dev Ops
Testing
TIME

Load Testing with Specific Objectives
Tests at different stages deliver higher coverage
Component
scalability
Performance
baseline
Integration
testing
End to End
w/ Service
Virtualization
Unit scalability
Database testing
3rd party
services
Live E2E
tests
Stress
testing Soak test
(reliability)
Failover
Capacity
Latency
Performance
Regression
(smoke tests)
Network
Testing (staging) Operations (prod)Development (sandbox)

o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation
o Continuous Feedback
o Emphasis on working builds
o Consistent environments
o Developer – Tester Collaboration

21
Shifting Left
• Agile means shorter development cycles
o Used to be months, now weeks
o Quality sign-off within sprint
• Developers and QA working together
o Developers write unit tests; some teams swap roles
• Performance testing is moving to left
o Meet performance SLA in all new code
• Automation with CI tools including Jenkins
o Code that meets quality goals gets promoted automatically
Testing is moving to the left, including unit, functional, and performance

22
Shifting Right
• Performance testing in development
o Development integrating performance tests into SDLC
o Strong adoption of open source test tools, such as JMeter
• Performance engineers using commercial tools
o Execute performance tests at scale, from multiple geos
o Testing not just software but infrastructure
• Testing tools addressing both sets of users
o Adding JMeter support, Continuous Integration automation
Moving performance tests from development to production

• Scalability issues
• Capacity issues
• Concurrency issues
• Reliability issues

• What does a problem look like?
o Longer response times is a clue
o ”High” CPU / Memory / Storage / Network utilization
• Solutions?
o Ops Problem? Can often be “solved” with hardware
o Tall stacks -> Wirth’s Law: “Software is getting slower more rapidly than
hardware gets faster”
• Testing in development
o Test at loads greater than 1 user
o Monitor resource usage across builds and compare to baseline / acceptable

o Response times vary related to load on system
o Increasing CPU / Memory / Network utilization
o Increasing I/O latency
• Solutions?
o Cannot be “solved” with hardware
o Re-architect solution for high-capacity
o Test at loads greater than 1 user, compare as load increases
o Monitor resource usage across builds and compare to baseline / acceptable

o Infrequent functional issues that seem only to occur under load
o Process crashes and restarts
o Not easily reproducible
• Solutions?
o Re-architect solution for high-concurrency
o Test at loads greater than 1 user
o Monitor functional results while testing at increasing loads
o Monitor processes for crashes and restarts

o Process crashes and restarts
o Memory or object leaks
o More frequent garbage collection
o Decaying response times
• Solutions?
o Find resource leaks based on failure points
o Test at loads greater than 1 user for increasing amounts of time
o Monitor processes for crashes and restarts
o Compare behavior and resource usage as time increases

• Additional measurements and aggregation
o Response Time Average / 90th (92/95/98th) Percentile
o Minimum / Maximum Response Times
o Completion / Error Rate %
o Transactions per second
o Size of Response
o Network Throughput
o Measurement against previous run or baseline
o Use more than one of the above!

• The Path to a Performance Culture
Run load tests via
JenkinsDaily (or more frequent) updates keep
teams “performance focused”
Identify
bottlenecks &
inform teams,
just like unit
and functional
tests in
Jenkins
Jenkins launches
suitable cloud-
based test
environment
Manage with a new metric!

30
Test Continuously at Different Load
Test from Development to Integration to Staging to Production

31
JMeter
• Hosted by Apache Software Foundation
o http://jmeter.apache.org
• Designed to load test functional behavior and measure
performance
o Web – HTTP, HTTPS; SOAP / REST; FTP; Database, more
• GUI design
• Plug-ins
• Tests as xml
Popular Open Source performance testing tool

Have you met Jenkins?
#1 CI/CD Automation Engine
• Open Source
• Highly Extensible
About 8 years old
Easy to install/use
1000+ plugins
Widely adopted
• 100K+ installations

Over 1000 Jenkins Plugins for Tool Integration
Integrated Development Environments
(IDEs), editors/compilers/debuggers, code
review and collaboration tools.
Source code control
systems and related asset
management tools.
Automated API-based
testing frameworks.
Release
management and
release automation
software
Automated API-based
testing frameworks, static
code analysis tools.
Load Testing Tools and
Security Scanning
technology
Automated and
manual UAT tools
Release management,
release automation, and
change control software.
Environment and Configuration
Management SoftwareDevelop
Commit
Build &
Integrate
Unit Tests
Release
(to Test)
Automated
Regression
Tests
Release
(to Prod)
User
Acceptance
Tests
Load &
Security
Tests

Jenkins is the Hub of the CD Ecosystem
On-premise or
in the cloud
Plug-ins for all
your tools
RunDeployStageTestBuildCommit

SOASTA Jenkins Plug-in
o Open Source: online at jenkins-ci.org
o Automatically available on CloudBees DEV@cloud
o Complete control over test automation from within your Jenkins builds:
o Totally integrated into Jenkins environment
o Similar plug-ins available for HP LoadRunner, BlazeMeter, JMeter, Load.io, etc.

36
Automate Your Testing and Promotion
Use Continuous Integration tools to automate testing

37
Results of Shifting Performance Testing Left
Using Continuous Integration tools to automate testing
• Run incremental performance tests automatically with
each iteration
• Establish a trend line for performance early
• Developers find code and architecture issues early
• Performance teams focus on end-to-end infrastructure
and user related testing
• Minimize production escapes and implement
production testing

38
Questions, Resources & Next Steps
• Holiday Readiness
o soasta.com/holiday-readiness
• The Performance Beacon
o soasta.com/blog
o soasta.com/blog/2015-popular-posts-web-performance
• Webinars
o soasta.com/webinars
o Load Testing for Peak; RUM-Based Testing
• CloudTest Lite
o soasta.com/cloudtest: Free community version for 100 VUs
Resources

How fast should your site be?

Speed costs money…

…and how much load should it handle?

Your customers will be your guide…

RUM
How fast is my site?
What locations?
From where?
Which pages?
Which flows?
Test
Full scale load
From any location
Going through
real user flows
With real-time
analytics
Real-Time Analytics with Real User Measurement

• Tying Business Metrics to (ROI) with website
performance metrics and real user data

CONFIDENTIAL – Not for Distribution © 2015 SOASTA. All rights reserved. January 13, 2015
Building performance tests that model user
behavior on your site
First you need to determine the workload for the test.
How are companies doing that today?
• Guessing.
• Logs.
• 3rd Party tools.
What if I have a new system?

o What information I need?
● When is your peak second/minute/hour/day?
● What is your peak volume? (active sessions, unique users, page
visits, orders, ...)
● What is the median session length?
● What is the distribution of user paths?
● What is the distribution of devices?
● What is the distribution of locations?
o Good Test Data
● Search Terms
● Product Distribution
● Browse Distribution
● Cart Size
● Order Size
How to get a realistic load test
Factors to consider

Use real user data to model real user tests

Focus on the ROI of Performance

Test how you want, when you want,
where you want

• Real Users (RUM) provide deep data about your visitors
o No other way to know everything about your users on your site,
from desktop and mobile
• Virtual Users tell you how your site or app will do at scale
o You can’t get a million friends to visit your site or app at the same
time!
JMeter, Load.io, LoadRunner, SOASTA CloudTest let you build and
execute load tests at scales from hundreds to millions of virtual users

• Session paths
o How are your users going through your site
• Mobile App Performance
o How are your mobile apps performing?
o What are users doing in your mobile apps?
• Revenue
o How much money are you making from each user and where?

What was the busiest Day, Hour, Minute, Second?

What were most popular page flows with think times? Which APIs?

• Testing to match realistic situations
o Users are not all from one location on a fast network
o Packets get delayed, lost, re-ordered
• Apps perform differently as network conditions worsen
o Reduced throughput
o Timeouts occur, connections are lost
o Transactions take longer, longer sessions
Adding real network modeling to load and performance testing

Walmart found…
Converted shoppers were served pages
that were 2X faster than pages served to
non-converted shoppers.
Non-buyers were served category pages that
were 2-3 seconds slower than category pages
served to buyers.
For every 100 milliseconds of improvement,
incremental revenue increased by up to 1%.

http://www.soasta.com/blog/mobile-web-performance-monitoring-conversion-rate/

Conversion rate barely decreases
when load times for “checkout” pages degrade

Conversion rate shrinks by about 50%
when load time for “browse” pages
increases from 1 to 6 seconds

What is the Conversion Impact Score?
The Conversion Impact Score (CIS) is a relative score that ranks
page groups by their propensity to negatively impact conversions
due to high load times. For each page group, the Conversion
Impact Score is calculated using the proportion of overall requests
that are associated with that group, along with the Spearman
Ranked Correlation between its load times and number of
conversions. The Conversion Impact Score will always be a
number between -1 and 1, though scores much greater than zero
should be very rare. The more negative the score, the more
detrimental to conversions that high load times for that page
group are, relative to the other page groups.

How much impact does the performance of this
page have on conversions?
1. Product
2. Category
3. Shopping cart
4. Home
5. Search results

http://www.soasta.com/blog/website-monitoring-conversion-impact-score/

69
Test in Production
Some issues will only be found at scale load on production systems
Don’t forget your third party providers: CDNs, external scripts

70
Measure Mobile App Performance Under
Load
All mobile apps perform well when there is no load

71
Now That the Test is Over
• Analyze results for future areas to address
o Weak or stress areas that aren’t likely this time but …
• Identify third parties that were affected by load
o Coordinate with suppliers to address issues
o Align schedules or at least be aware of their changes
Even if the test passed, you’re not done

72
Issues Will Happen
• “I am 100% confident that everything we tested will work
just fine.” – Nordstrom Performance Leader
o When loads are different than what you modeled, may have
problems
o E.g. At Nordstrom, during a recent anniversary sale, mobile
sign-ins were 9x greater than anticipated
o Used RUM to get to the root cause of the issue
Realize that Issues Will Happen Where You Can’t Predict Them

74
CloudTest – Load Testing for mobile web, apps
• Load testing from development to production
o Continuous testing at any scale, on-prem or in the cloud
• Real-time analytics with customizable dashboards
o Safely test in production with full visibility
• Run tests live or automatically
o Full hands-on control or hands-off automation
• Full end-to-end view
o Front-end performance to back-end systems
Fast, scalable, affordable load testing for performance optimization

75
Questions, Resources & Next Steps
• Holiday Readiness
o soasta.com/holiday-readiness
• The Performance Beacon
o soasta.com/blog
o soasta.com/blog/2015-popular-posts-web-performance
• Webinars
o soasta.com/webinars
o Load Testing for Peak; RUM-Based Testing
• CloudTest Lite
o soasta.com/cloudtest: Free community version for 100 VUs
Resources

ConFoo: Moving web performance testing to the left

More Related Content

What's hot (20)

Similar to ConFoo: Moving web performance testing to the left (20)

More from Tom Chavez (7)

Recently uploaded (20)

ConFoo: Moving web performance testing to the left