![2/18/2014
9
Server Software: nginx
Server Hostname: dustinwhittle.com
Server Port: 80
Document Path: /
Document Length: 6642 bytes
Concurrency Level: 10
Time taken for tests: 10.042 seconds
C l 286Complete requests: 286
Failed requests: 0
Write errors: 0
Keep-Alive requests: 0
Total transferred: 2080364 bytes
HTML transferred: 1899612 bytes
Requests per second: 28.48 [#/sec] (mean)
Time per request: 351.133 [ms] (mean)
Time per request: 35.113 [ms] (mean, across all concurrent requests)
Transfer rate: 202.30 [Kbytes/sec] received
Connection Times (ms)Connection Times (ms)
min mean[+/-sd] median max
Connect: 9 20 14.2 17 148
Processing: 117 325 47.8 323 574
Waiting: 112 317 46.3 314 561
Total: 140 346 46.0 341 589
Percentage of the requests served within a certain time (ms)
50% 341
66% 356
75% 366
80% 372
Siege](https://image.slidesharecdn.com/w14-hirschauer-140711075138-phpapp02/85/Techniques-for-Agile-Performance-Testing-11-320.jpg)
![2/18/2014
17
import requestsimport requests
class Transaction(object):
def run(self):
r = requests.get('http://dustinwhittle.com/')
d()r.raw.read()
class Transaction(object):
def run(self):
br = mechanize.Browser()
br.set_handle_robots(False)
start_timer = time.time()
resp = br.open('http://www.dustinwhittle.com/')
resp.read()
latency = time.time() - start_timer
self custom timers['homepage'] = latencyself.custom_timers[ homepage ] latency
start_timer = time.time()
resp = br.open('http://www.dustinwhittle.com/blog')
resp.read()
latency = time time() start timer](https://image.slidesharecdn.com/w14-hirschauer-140711075138-phpapp02/85/Techniques-for-Agile-Performance-Testing-19-320.jpg)
![2/18/2014
18
[global]
run_time = 10
5rampup = 5
results_ts_interval = 1
progress_bar = on
console_logging = off
xml_report = on
[user_group-1]
threads = 1
script = demo.py
multimech-run
demo](https://image.slidesharecdn.com/w14-hirschauer-140711075138-phpapp02/85/Techniques-for-Agile-Performance-Testing-20-320.jpg)
![2/18/2014
22
pip install
beeswithmachineguns
# ~/.boto
[Credentials]
aws_access_key_id=xxx
aws_secret_access_key=xxx
[Boto]
ec2_region_name = us-west-2
ec2_region_endpoint = ec2.us-west-
2.amazonaws.com](https://image.slidesharecdn.com/w14-hirschauer-140711075138-phpapp02/85/Techniques-for-Agile-Performance-Testing-24-320.jpg)
![2/18/2014
25
bees attack -n 1000 -c 50 -
u http://dustinwhittle.com/
Read 2 bees from the roster.
Connecting to the hive.
Assembling bees.
Each of 2 bees will fire 50000 rounds, 125 at a time.
Stinging URL so it will be cached for the attack.
Organizing the swarm.
Bee 0 is joining the swarm.
Bee 1 is joining the swarm.
Bee 0 is firing his machine gun. Bang bang!
Bee 1 is firing his machine gun. Bang bang!
Bee 1 is out of ammo.
Bee 0 is out of ammo.
Offensive complete.
Complete requests: 100000
Requests per second: 1067.110000 [#/sec] (mean)
Time per request: 278.348000 [ms] (mean)
50% response time: 47.500000 [ms] (mean)
90% response time: 114.000000 [ms] (mean)
Mission Assessment: Target crushed bee offensive.
The swarm is awaiting new orders.](https://image.slidesharecdn.com/w14-hirschauer-140711075138-phpapp02/85/Techniques-for-Agile-Performance-Testing-27-320.jpg)
Techniques for Agile Performance Testing
- 2. Jim Hirschauer AppDynamics Technology evangelist for AppDynamics Jim Hirschauer has an extensive background in highly available, business critical, large enterprise IT operations environments. Jim has been interested in application performance testing and monitoring since his days as a systems administrator in a retail bank. His passion for performance analysis led him down a path to design, implement, and manage the cloud computing monitoring architecture for a top-ten investment bank. During his tenure at the bank, Jim created new processes and procedures that reduced the time spent in the test and QA cycle while simultaneously increasing overall code release quality and dramatically improving end user experience.
- 3. 2/18/2014 1 PERFORMANCE TESTING CRASH COURSE Jim Hirschauer @HirschOnAPM Lockheed Martin Aeronautical Systems Duke Energy Wachovia Wells Fargo AppDynamics
- 4. 2/18/2014 2 Wh dWhy does performance matter?matter? Mi ft f d th t BiMicrosoft found that Bing searches that were 2 seconds slower resulted in a 4.3% drop in revenue per userdrop in revenue per user
- 5. 2/18/2014 3 Wh M ill h d 2 2When Mozilla shaved 2.2 seconds off their landing page, Firefox downloads increased 15.4%increased 15.4% Making Barack Obama’s website 60% faster increased donation conversions by 14%
- 6. 2/18/2014 4 Amazon and WalmartAmazon and Walmart increased revenue 1% for every 100ms of improvementimprovement
- 7. 2/18/2014 5 Performance directly impacts the bottom linee
- 8. 2/18/2014 6 T l f th t dTools of the trade for performance testingtesting
- 9. 2/18/2014 7 Understand your baseline performance Static vs Hello World vsvs App
- 10. 2/18/2014 8 Apache Bench ab -c 10 -t 10 -k http://dustinwhittle.com/
- 11. 2/18/2014 9 Server Software: nginx Server Hostname: dustinwhittle.com Server Port: 80 Document Path: / Document Length: 6642 bytes Concurrency Level: 10 Time taken for tests: 10.042 seconds C l 286Complete requests: 286 Failed requests: 0 Write errors: 0 Keep-Alive requests: 0 Total transferred: 2080364 bytes HTML transferred: 1899612 bytes Requests per second: 28.48 [#/sec] (mean) Time per request: 351.133 [ms] (mean) Time per request: 35.113 [ms] (mean, across all concurrent requests) Transfer rate: 202.30 [Kbytes/sec] received Connection Times (ms)Connection Times (ms) min mean[+/-sd] median max Connect: 9 20 14.2 17 148 Processing: 117 325 47.8 323 574 Waiting: 112 317 46.3 314 561 Total: 140 346 46.0 341 589 Percentage of the requests served within a certain time (ms) 50% 341 66% 356 75% 366 80% 372 Siege
- 12. 2/18/2014 10 siege -c 10 -b -t 10S http://dustinwhittle.com/ ** SIEGE 2.72 ** Preparing 10 concurrent users for battle. The server is now under siege... Lifting the server siege... done. Transactions: 263 hits Availability: 100.00 % Elapsed time: 9.36 secs Data transferred: 0.35 MB Response time: 0.35 secs Transaction rate: 28.10 trans/sec Throughput: 0.04 MB/sec Concurrency: 9.82y Successful transactions: 263 Failed transactions: 0 Longest transaction: 0.54 Shortest transaction: 0.19
- 13. 2/18/2014 11 Crawl the entire app to discover all urls sproxy -o ./urls.txt
- 14. 2/18/2014 12 SPROXY v1.02 listening on port 9001 ...appending HTTP requests to: ./urls.txt ...default connection timeout: 120 seconds wget -r -o verbose txt -l 0 -t 1 --spider -w 1 -ewget -r -o verbose.txt -l 0 -t 1 --spider -w 1 -e robots=on -e "http_proxy = http://127.0.0.1:9001" "http://dustinwhittle.com/" sort -u -o urls.txt urls.txt
- 15. 2/18/2014 13 Benchmark traffic across all unique urls with siegeu s t s ege siege -v -c 50 -i -t 3M -f urls.txt - d 10
- 17. 2/18/2014 15 M lti M h i iMulti-Mechanize is an open source framework for performance and load testingtesting
- 19. 2/18/2014 17 import requestsimport requests class Transaction(object): def run(self): r = requests.get('http://dustinwhittle.com/') d()r.raw.read() class Transaction(object): def run(self): br = mechanize.Browser() br.set_handle_robots(False) start_timer = time.time() resp = br.open('http://www.dustinwhittle.com/') resp.read() latency = time.time() - start_timer self custom timers['homepage'] = latencyself.custom_timers[ homepage ] latency start_timer = time.time() resp = br.open('http://www.dustinwhittle.com/blog') resp.read() latency = time time() start timer
- 20. 2/18/2014 18 [global] run_time = 10 5rampup = 5 results_ts_interval = 1 progress_bar = on console_logging = off xml_report = on [user_group-1] threads = 1 script = demo.py multimech-run demo
- 21. 2/18/2014 19
- 22. 2/18/2014 20 What about when you need more than one machine?o e ac e Who lives in the cloud?
- 23. 2/18/2014 21 Bees with Machine Guns A utility for arming (creating)A utility for arming (creating) many bees (micro EC2 instances) to attack (load test) t t ( b li ti )targets (web applications)
- 24. 2/18/2014 22 pip install beeswithmachineguns # ~/.boto [Credentials] aws_access_key_id=xxx aws_secret_access_key=xxx [Boto] ec2_region_name = us-west-2 ec2_region_endpoint = ec2.us-west- 2.amazonaws.com
- 25. 2/18/2014 23 bees up -s 2 -g default -z us-west- 2b -i ami-bc05898c -k appdynamics- dustinwhittle-aws-us-west-2 -l ec2- user Connecting to the hiveConnecting to the hive. Attempting to call up 2 bees. Waiting for bees to load their machine guns... . . . . Bee i-3828400c is ready for the attack. Bee i-3928400d is ready for the attack. The swarm has assembled 2 bees.
- 26. 2/18/2014 24 bees report Read 2 bees from the roster. Bee i-3828400c: running @ 54.212.22.176 Bee i-3928400d: running @ 50.112.6.191
- 27. 2/18/2014 25 bees attack -n 1000 -c 50 - u http://dustinwhittle.com/ Read 2 bees from the roster. Connecting to the hive. Assembling bees. Each of 2 bees will fire 50000 rounds, 125 at a time. Stinging URL so it will be cached for the attack. Organizing the swarm. Bee 0 is joining the swarm. Bee 1 is joining the swarm. Bee 0 is firing his machine gun. Bang bang! Bee 1 is firing his machine gun. Bang bang! Bee 1 is out of ammo. Bee 0 is out of ammo. Offensive complete. Complete requests: 100000 Requests per second: 1067.110000 [#/sec] (mean) Time per request: 278.348000 [ms] (mean) 50% response time: 47.500000 [ms] (mean) 90% response time: 114.000000 [ms] (mean) Mission Assessment: Target crushed bee offensive. The swarm is awaiting new orders.
- 28. 2/18/2014 26 bees down What about the client side?
- 30. 2/18/2014 28
- 34. 2/18/2014 32 Automate client-side performance testing with Gruntt G u t Use Bower (forUse Bower (for dependencies), Grunt (for automation), and Yeoman (for bootstrapping)
- 35. 2/18/2014 33
- 36. 2/18/2014 34 H lHow many people understand exactly how fast their site runs in production?production?
- 37. 2/18/2014 35
- 38. 2/18/2014 36
- 39. 2/18/2014 37
- 40. 2/18/2014 38
- 42. 2/18/2014 40
- 43. 2/18/2014 41 L d t tiLoad testing services from the cloudcloud
- 44. 2/18/2014 42
- 45. 2/18/2014 43 I t t t t dIntegrate automated performance testing into continuous integration for server-side and client-sideserver side and client side
- 46. 2/18/2014 44 U d t d thUnderstand the performance implications of every deployment and package upgradepackage upgrade Monitor end user experience from end to end in productionto e d p oduct o