QA standup - workload analysis

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Eloquentperformance testing.

Serghei Radov
Current position:
Principal Performance Engineer
Contacts :
sergey.radov@gmail.com
Github: github.com/sergeyradov
Skype : serghei.radov

1. Amazon: 100-millisecond increase in page speed translated to a 1% increase in its
revenue.
1. Shopzilla improved their revenues by 5% to 12% after drop their page load times
from 6 to 9 seconds down to 1.2 seconds.
1. The Aberdeen Group in 2015 showing that performance issues impact revenues by
up to 9%. An average loss of $117 million per year.
1. Strangeloop (bought by Radware in 2013) showed that a 2-second delay in load
time during checkout could result in abandonment rates reaching up to 87%.
Why it matters?

Why it matters?

➢ Define acceptance criteria
➢ Select tools for monitoring and testing
➢ Workload Characterization
➢ Test tools for testing
➢ Run tests, analyze, scale, re-run <- cycle
➢ Report to stakeholders

Define performance tests SLA
● Throughput
● Expected Response times
● User count
● Thread limitation
● Resource Usage
● Time-out and e.t.c
Exceptions that can be included in the SLA:
● Failure
● Network issues
● Denial of service
● Scheduled maintenance

NRQL - NewRelic query language
SELECT uniqueCount(session) FROM PageView SINCE 1 week ago
SELECT uniqueCount(session) FROM PageView SINCE 1 week ago
COMPARE WITH 1 week ago
SELECT count(*) FROM PageView SINCE 1 day ago COMPARE
WITH 1 day ago TIMESERIES AUTO
SELECT uniqueCount(uuid) FROM MobileSession FACET osVersion
SINCE 7 days ago

Gathering response times

Transactions throughput

Finding peaks
● Target PEAK load will be 1.14K RPM
● Lowest point will be 430 RPM
(extracted from New relic for presentation only instead of DataDog)

Monitoring targets
● Response times
● Resource utilisation at SUT
● Resource utilisation at Test Tool
● Exceptions
● Workload behaviour

CPU usage per 1 server (DataDog)

Grafana

Resource usage

to catch exceptions

Workload Characterization
- Catch traffic patterns
- Resource utilisation
- Distribution of response times
- Distribution of response sizes
- Characterizations of users
behaviour
- Analyse input data
- Use performance analysis toolkit

Traffic patterns
“Keep workload as real as possible.”

- Ramp up to 430
RPM slowly to 700
RPM in 4 hours
- Run test for 6 hours
- Ramp up to 1.14K
rpm
- Run test for 11
hours
- Ramp down slowly
Scenario per one server

Characterize user behaviour
Investigate user actions by help of
- New Relic Browser (session+funnel functions)
- Universal Analytics with User behaviour path
- Mixpanel.com (needs code injection)
- Server’s logs at NGINX
- (http requests, REST calls)
- Sumo-logic (apache access logs)
- Server’s App logs (HP ALM has QC sense)
- DB activity logs (applied solution)

● Jmeter
● Gatling
● Locust
● Grinder
● Tsung
Open Source load tools - 54 found

Distributed JMeter testing

● BlazeMeter - (JMeter)
● Visual Studio Team Services - (JMeter)
● Flood IO - (Jmeter , Gatling, Ruby DSL)
● Redline 13 - (Jmeter , Gatling, Ruby DSL)
● OctoPerf - (JMeter)
Load Tool as Service Providers

Create a Grid ( Docker containers)

Create a Flood (upload jmx & data)

Let’s wait till a test is completed.

JMeter HTML Report

Reports
● Goals & achievements (e.g 150% of Daily RPM is reached)
● Side effects are found (DB connections limit reached due to quick
ramp up)
● Exceptions caught during testing (e.g. ELB lost connections)
● Run-time notes and fixes made by DevOps (EC2 change during the
test iterations)
● Observations ( CPU usage was critical resource during RPM
increase)
● Recommendations ( EC2 - add more VM, add more Shards DB)

Thank you!

Questions and Answers

Books

QA standup - workload analysis

More Related Content

What's hot (20)

Similar to QA standup - workload analysis (20)

Recently uploaded (20)

QA standup - workload analysis

Editor's Notes