Velocity 2014 nyc WebPagetest private instances

WebPagetest
Private Instances
Patrick Meenan
pmeenan@webpagetest.org
@patmeenan

The Basics
● Presentation is available on slideshare
○ http://www.slideshare.net/patrickmeenan
○ Mostly contains links to everything I’ll be showing
● Video will be uploaded to YouTube
○ https://www.youtube.com/user/patmeenan
● I’ll send links and announcements of both to Twitter and Google+
○ https://twitter.com/patmeenan - @patmeenan
○ https://plus.google.com/u/0/+PatrickMeenan/posts

WPT Web Server
Sites being tested
System Architecture
Desktop Agent
Node.js Mobile Agent
Mobitest Agent

WPT Web Server
Sites being tested
Agent
Data Flow
Submit test (UI or API)
http://<server>/runtest.php
User

Server:
- Creates test ID
YYMMDD_<hash>_<num>
- Creates directory for test
results/YY/MM/DD/<hash>/<num>
- Writes test job to work directory for location
work/jobs/<location>/<ID>.<priority>
- Adds test to queue file (lossy, can be re-built from the work directory)
tmp/<location hash>.queue
- Redirects to result page

WPT Web Server
Sites being tested
Agent
Poll for test status
http://<server>/result/<ID>/
User

WPT Web Server
Sites being tested
Agent
User
Poll for work for location
http://<server>/work/getwork.php

WPT Web Server
Sites being tested
Agent
User
- Launch Browser
- Navigate to page
- Measure/Collect Data

WPT Web Server
Sites being tested
Agent
User
Upload large files individually
(images, tcpdump, timeline, etc)
http://<server>/work/resultimage.php

WPT Web Server
Sites being tested
Agent
User
Complete run
(or full test if all runs are done)
http://<server>/work/workdone.php

WPT Web Server
Sites being tested
Agent
View completed test
http://<server>/result/<ID>/
User

Server Config
● Any server that can run php
○ Linux tends to work best
○ Can run on the same PC as test agents if needed
● Copy www/ directory from distribution to site docroot
● Configure rewrites for friendly URLs (optional)
○ .htaccess included for apache
○ nginx.conf can be included into nginx server config
● Fix anything that http://<server>/install/ complains about
○ PHP ini settings: allow_url_fopen, memory and post file size limits
○ PHP modules: GD, zip, zlib, curl, apc, sqlite
○ Command-line utilities: ffmpeg, imagemagick, jpegtran, exiftool
○ Filesystem permissions (and directories)

Nginx config
server {
listen 80;
server_name webpagetest.example.com;
root /var/www;
location ~ .php$ {
fastcgi_pass 127.0.0.1:9000;
fastcgi_param SCRIPT_FILENAME /var/www$fastcgi_script_name;
fastcgi_param HTTP_MOD_REWRITE On;
include fastcgi_params;
}
include /var/www/nginx.conf;
}

Server WPT Config
● Copy/modify settings/*.sample
○ settings/settings.ini
■ Contact information
■ Maximum number of runs
■ Screen shot quality
■ Show Slow integration
■ Test archiving (covered later)
○ settings/locations.ini
■ Definition for all of the test locations

Locations.ini
Grouping
Browsers

[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
[NYC_Mobile]
browser=Motorola G - Chrome,Motorola G - Chrome Beta
[Group_2]
...

[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
[NYC]
[NYC_Mobile]
[Group_2]
...
List of the group IDs in order to be displayed

[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
[NYC]
[NYC_Mobile]
[Group_2]
...
“Location ID”s that are part of the group in the
order the browsers are to be displayed.
String displayed for the “Test Location”

[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
[NYC]
[NYC_Mobile]
[Group_2]
...
These are the location ID’s the agents connect to
and that are used when submitting tests through
the API

[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
[NYC]
[NYC_Mobile]
[Group_2]
...
All of the browsers from all of the locations in a
group are collected and displayed in the Browser
select box.

[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
[NYC]
[NYC_Mobile]
[Group_2]
...
All of the agents for a given location ID must
support all of the browsers listed

[NYC]
browser=Chrome,Firefox
[NYC_IE10]
browser=IE 10
[NYC_IE11]
browser=IE 11
[NYC_Mobile]
A single agent can pull from multiple locations
(configure it as a comma-separated list).
location=NYC,NYC_IE10

[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
lat=40.712918
lng=-74.005469
group=North America
[NYC]
[NYC_Mobile]
[Group_2]
...
Add map=1 to settings.ini

[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
lat=40.712918
lng=-74.005469
group=North America
[NYC]
[NYC_Mobile]
[Group_2]
...

[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
[NYC]
connectivity="Google Fiber (1Gbps)"
key=123456
agents=6
notify=pmeenan@webpagetest.org,admin@webpagetest.org
hidden=1
When traffic shaping is not supported.
Shows up in the UI as the connectivity information.

[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
[NYC]
key=123456
agents=6
hidden=1
Shared key for securing agent communications
(optional)

[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
[NYC]
key=123456
agents=6
hidden=1
Expected number of agents (for alerting)

[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
[NYC]
key=123456
agents=6
hidden=1
Email addresses for alerting when agents stop
connecting.

[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
[NYC]
key=123456
agents=6
hidden=1
Hide the location from the UI (still works through the API
and from http://<server>/?hidden=1 )

Desktop Agent Config
● Prepare Windows
○ Auto logon
○ Disable screen savers and screen power management (never sleep)
○ Disable UAC
○ Copy agent/* to local disk (c:/webpagetest/ in the examples)
○ Install dummynet (optional for traffic shaping)
○ Create startup shortcut to wptdriver.exe (and urlblast.exe if used)
● Install browsers
○ Firefox can self-install (necessary for updates)
● Configure wptdriver.ini
○ Server URL
○ Location ID
○ Browsers

wptdriver.ini
[WebPagetest]
url=http://127.0.0.1/
location=Test
browser=chrome
Time Limit=120
;key=TestKey123
;Automatically install and update support software (Flash, Silverlight, etc)
software=http://www.webpagetest.org/installers/software.dat
[chrome]
exe="%PROGRAM_FILES%GoogleChromeApplicationchrome.exe"
options='--load-extension="%WPTDIR%extension" --user-data-dir="%PROFILE%" --no-proxy-server'
installer=http://www.webpagetest.org/installers/browsers/chrome.dat
[Firefox]
exe="%PROGRAM_FILES%Mozilla Firefoxfirefox.exe"
options='-profile "%PROFILE%" -no-remote'
installer=http://www.webpagetest.org/installers/browsers/firefox.dat
template=firefox
[Safari]
exe="%PROGRAM_FILES%SafariSafari.exe"
[IE]
exe="%PROGRAM_FILES%Internet Exploreriexplore.exe"

EC2 Test Agents
● AMI’s available in all regions
○ IE8-11, Chrome, Firefox, Safari 5 (Windows)
● Pass configuration through user data when launching
○ wpt_server=www.webpagetest.org wpt_loc=EC2_East
● No need to ever touch/login to instance
● Kill/Start as needed
● c1.medium or better recommended

Sites being tested
WPT Web Server
Mobile Agent (Node)

Mobile Agents - Android
● Android 4.4+ recommended
● Root (Nexus and Moto phones are easiest)
● Install Chrome and Chrome Beta from Play store
● Configure Play to auto-update
● Disable screen lock (security settings)
● Configure developer mode
○ Enable USB debugging
○ Disable screen sleep

Mobile Agents - iOS (alpha)
● Jailbreak
● install/configure iOS webkit debug proxy
● build openurl ipa
○ Needs to be signed with your dev key

Mobile Agents - Tethered host config
● Linux or Windows (Mac may work)
○ Multiple phones per tethered agent
● Node.js
● imagemagick
● Launch wptdriver.bat (or wptdriver.sh)
● Config is passed by command-line
○ Device ID
○ Server URL
○ Location ID
● Windows - consider using adbwatch
○ Kills/restarts adb is it gets hung
○ Launches/restarts agents if they crash

Sites being tested
WPT Web Server
Mobile Agent (Node) - Cell Network

Sites being tested
WPT Web Server
Mobile Agent (Node) - RNDIS
IPFW
(Optional)

WPT Web Server
Sites being tested
Mobile Agent (Node) - WiFi
IPFW Bridge
WiFi Access (Optional)
Point

Priority Queues
● Tests are pulled from queues based on test priority
● 10 Priority levels (0-9)
○ 0 = highest, 9 = lowest
● Defaults
○ 0 - Tests submitted through the UI
○ 4 - Bulk tests submitted through the UI
○ 5 - Tests submitted through the API
● Queue lengths reported on location check UI
○ http://<server>/getLocations.php

https://www.flickr.
com/photos/lunchtimemama/110765169/

API Keys
settings/keys.ini (disabled if file is missing)
[server]
secret=7a5c742c021c414ca0dd1bec42e8f64b
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
contact=me@example.com
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
contact=batch@example.com
limit=5000
priority=8
[acd5ffd840094cdda054a0b7e57d4546]
description=API Key
contact=notme@example.com
limit=100

API Keys
[server]
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
limit=5000
priority=8
limit=100
Arbitrary string
used for hashing keys sent to browser
(I usually use GUIDs)

API Keys
[server]
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
limit=5000
priority=8
limit=100
Server API key
To be used by the Web UI

API Keys
[server]
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
limit=5000
priority=8
limit=100
Each API key defined as
an ini file section

API Keys
[server]
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
limit=5000
priority=8
limit=100
Contact info shows up in
usage stats

API Keys
[server]
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
limit=5000
priority=8
limit=100
Allowed “Page Loads” per day
(tests x runs)
x 2 if also recording repeat view

API Keys
[server]
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
limit=5000
priority=8
limit=100
Forced test priority
Defaults to 5 for API
(can be specified in request)

API Keys
[server]
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
limit=5000
priority=8
limit=100
Minimal key configuration

Firewalls...
Intranet WPT Web Server
Desktop Agent
Private Sites
Public Desktop
Agent
Public Sites

Server Lock-down
● Use keys for test locations
● Open access to /work/* for test agents
● Auth access (or IP rules) for rest of the UI

WPT Web Server
Firewalls...
Private Sites
Intranet
Desktop Agent
Public Desktop
Agent
Public Sites
WPT Relay
Server

Relay Server
● Configure locations for public agents
● Configure API keys for internal instance to use for relay
● add headless=1 to settings.ini
○ Disbles UI for submitting tests, requires API key for all tests
● Results are pulled back to internal instance
○ never visible on external

Relay Server - locations.ini
● Configure locations.ini like normal on the relay server
● Local server configures 1:1 mapping of locations
○ forwarded to the relay server
[Public_EC2_East]
browser=IE 11,Chrome,Firefox
label="EC2 East (Public)"
relayServer="http://<relay server>/"
relayKey=<your API key>
relayLocation=EC2_East
Config like normal location
API Key and Location ID
on the Relay Server

Results Storage
● Each test is self-contained in a directory under results/
● Grouped by year, month and day
○ results/YY/MM/DD/…
● Prune old tests? Delete directories.

Results Archiving
● WPT can seamlessly restore missing tests from “archive”
● Supports archiving to local path or S3-like storage
● Configure archive settings in settings.ini
● Test copied to archive after it completes (as 1 zip file per test)
● To delete old archived tests need to run/schedule script
○ php cli/archive.php
● Local Path:
○ zip files copied to specified directory
○ YY/MM/DD/<hash>/<id>.zip
● S3-like
○ zip files uploaded to bucket with <id>.zip filename
○ Can restore from URL or S3 API

Results Archiving - Local Path
● Settings.ini
○ archive_dir=/archive/
○ archive_days=2
● Local Path:
○ zip files copied to specified directory
○ YY/MM/DD/<hash>/<id>.zip
● 2nd Level archiving
○ settings.ini: archive2_dir=/mnt/archive/
○ archive zip files merged into daily zip files
○ cli/archive2.php archive script

Results Archiving - S3
● Settings.ini
○ archive_s3_server=s3.amazonaws.com
○ archive_s3_key=<access key>
○ archive_s3_secret=<secret>
○ archive_s3_bucket=<bucket>
● zip files uploaded to bucket with <id>.zip filename
● Can use URL access to restore
○ Defaults to using API
○ archive_s3_url=http://s3.amazonaws.com/

Bulk Testing in the UI (Private Instances)

Bulk Testing in the UI (Private Instances)
● Can be disabled through settings.ini
○ noBulk=1
● Submit list of URLs to test
● UI options apply to each test
● Scripts can be run against each URL
○ %URL% in script gets replaced with test URL
○ %HOST% in script gets replaced with host from test URL
○ %HOSTR% in script gets replaced with host from URL after redirects
● Aggregate results

API - Submitting a test
● Anything you can do with the UI you can do with the API
○ If a field isn’t documented, just inspect the HTML on the UI to get the
field name :-)
● Make sure to encode parameters
○ urlencode if GET
○ form encode if POST

API Response
http://www.webpagetest.org/runtest.php?url=www.aol.com&f=json
{
"statusCode":200,
"statusText":"Ok",
"data":{
"testId":"140610_FY_N2F",
"ownerKey":"04121a1e5b17a59ba9ddd666ae020cf52cee4abe",
"jsonUrl":"http://www.webpagetest.org/jsonResult.php?test=140610_FY_N2F",
"xmlUrl":"http://www.webpagetest.org/xmlResult/140610_FY_N2F/",
"userUrl":"http://www.webpagetest.org/result/140610_FY_N2F/",
"summaryCSV":"http://www.webpagetest.org/result/140610_FY_N2F/page_data.csv",
"detailCSV":"http://www.webpagetest.org/result/140610_FY_N2F/requests.csv"
}
}

Node API Wrapper
Created by Marcel Duran (@marcelduran)
https://github.com/marcelduran/webpagetest-api
$ npm install webpagetest -g
webpagetest test -k <key> http://twitter.com/marcelduran
webpagetest status 140610_F0_T5K
webpagetest results 140610_F0_T5K
webpagetest test http://twitter.com/marcelduran --poll 5 --timeout 60

CI Integration
https://github.com/marcelduran/webpagetest-api/wiki/Test-Specs
Match result to provided specs
{
"median": {
"firstView": {
"requests": 20,
"render": 400,
"loadTime": 3000,
"score_gzip": {
"min": 90
}
}
}
}

CI Integration cont.
Uses Mocha with reporters suitable for integration with:
● Jenkins
● Travis-CI
● Drone.io
● Just about anything else you can imagine

grunt-perfbudget
Created by Tim Kadlec (@tkadlec)
https://github.com/tkadlec/grunt-perfbudget

Results logging
● As each test comes in, full result data is written to log files
○ Page Data (load time, Speed index, requests, custom metrics, etc)
○ Request Data (details for every individual request)
● Each test run or request is on it’s own log file line
● JSON-formatted
● Config information is included
○ Test URL
○ Run #
○ First/Repeat View
○ Test label
○ Location
○ Browser
○ Connectivity
○ Test ID
○ URL to test result

Results logging - integration
● Splunk
● Logster
○ Logster -> StatsD = Arbitrary metrics trending automatically
● Track individual requests across all pages
○ ads JS performance
○ CDN response times
○ Effectiveness of image compression
○ so much more

Benchmarks (Private Instances)
● Still pretty experimental
● Recurring tests run automatically
○ Configuration is a bit ugly but flexible
● Tests are scheduled, run and aggregated automatically
○ Top-level trended aggregate view
○ All-metrics trended aggregate view
○ Per-page trended view
○ Scatter Plot for a given run
○ Filmstrip comparisons

Monitoring the System
● Hourly Agent Monitoring
● getLocations.php
● getTesters.php
● checkTesters.php
● Custom cron jobs/scripts

Velocity 2014 nyc WebPagetest private instances

More Related Content

What's hot (20)

Similar to Velocity 2014 nyc WebPagetest private instances (20)

More from Patrick Meenan (14)

Recently uploaded (20)

Velocity 2014 nyc WebPagetest private instances