Sauce Labs Beta Program Overview

Beta Program Overview Al Sargent [email_address] +1 415 730 1430 June 5, 2009

DimDim tip 06/05/09 Sauce Labs Confidential Choose Fit Screen

The Problem Many, many functional tests… 06/05/09 Sauce Labs Confidential Hundreds of test flows Happy path Corner cases Negative tests Before check-in, after check-in

… and test platforms Dozens of test platforms Hundreds of test flows Happy path Corner cases Negative tests Before check-in, after check-in 06/05/09 Sauce Labs Confidential 6, 7 (and soon 8) 9.5 (and soon 9.6) 1.0 3.0, 3.1 (and soon 4) 2, 3 (and soon 3.1) Add-ons Browser configurations Languages/locales

… and test data 06/05/09 Sauce Labs Confidential Lots of test data Correct data, Incorrect data, Missing data, Data overflow Dozens of test platforms 6, 7 (and soon 8) 9.5 (and soon 9.6) 1.0 3.0, 3.1 (and soon 4) 2, 3 (and soon 3.1) Add-ons Browser configs Languages/locales

… run on minimal test hardware. 06/05/09 Sauce Labs Confidential Test jobs In-house test lab Minutes 10 20 30 40 50 60 70 80 90 Test results Can’t run all scripts + browsers Test lab rathole Too long to find and fix bugs

How many tests? Which browsers/OS combos? How much data? 06/05/09 Sauce Labs Confidential

Concurrent Test Execution 06/05/09 Sauce Labs Confidential Minutes 0 5 Test jobs 2000 nodes Test results Run all scripts + browsers Find and fix bugs fast No configuration needed

How many test machines? How long to run today? How long, ideally? How often run today? How often, ideally? 06/05/09 Sauce Labs Confidential

User Experience 06/05/09 Sauce Labs Confidential Agile developer / tester Source control 1 Check in code & test scripts 5 Run scripts HTTP(s) 7 View results Web, RSS REST 8 Delete scripts & results 6 Download results REST 4 Upload scripts Sauce Labs FTP App Under Test 3 Promote new build to cloud-accessible location (Your infrastructure) Start new build Polling or trigger Build System 2 cron rake

How does this map to your workflow? Which CI or build system? Build script language? Test script language? 06/05/09 Sauce Labs Confidential

What’s in our cloud? 06/05/09 Sauce Labs Confidential Sauce Labs Scripts Selenium test scripts Written in Python, Ruby (now) Selenese, PHP, Java, C#, Perl (future) Configs Which browser, addons, & OS that a job should run on Jobs Similar to a print job waiting, running, done… Results Output of Script Stdout, stderr, and more… Users Can access certain Scripts, Configs, Jobs, and Results

What’s in our cloud? 06/05/09 Sauce Labs Confidential Sauce Labs Waiters (small number) Listen to API requests Dispatch Job execution to Chefs Handle other requests directly Chefs (many, up to 2000) Selenium RC + browser Run Job and return Results

What’s the app under test? Internet-accessible? Max # of Selenium RCs? 06/05/09 Sauce Labs Confidential

06/05/09 Sauce Labs Confidential

require 'rubygems' require 'couchrest' # http://github.com/jchris/couchrest # Link to user's database db = CouchRest.database("http://jdoe.saucelabs.com") # Upload a script # ----------------------------------------------------------------------- # A SauceLabs script is simply a regular ruby script wrapped in JSON response = db.save( { "Type" => "script", "Name" => "Google Search", "Language" => "ruby", "Content" => "open ('google.com')\ntype ('q', 'selenium')\nclick'btnG')\nwaitForPageToLoad()\nverifyTextPresent('openqa.org')" }) script_id = response['id’] 06/05/09 Sauce Labs Confidential 4 Upload script

# Run a job # ------------------------------------------------------------------ # You run a job by uploading a new "job" document, linking to the # script id. # A job is instantly queued for execution upon upload. response = db.save( { "Type" => "job", "ScriptId" => script_id, }) job_id = response['id’] 06/05/09 Sauce Labs Confidential 5 Run job

# Check job status # ---------------------------------------------------------------------------- status = db.get(job_id)['Status'] # 'status' will be one of: # "new" # "queued" # "in progress" # "complete" # "error” # Download result # ---------------------------------------------------------------------------- # Once the job is complete, a "result" document is created. The "job" document # is updated with a reference to the result. # First, retrieve the ResultId from the job result_id = db.get(job_id)['ResultId'] # Second, retrieve the complete result document: result = db.get(result_id) # You can now introspect the results # View standard output standard_output = result['Stdout'] # View standard error standard_error = result['Stderr’] 06/05/09 Sauce Labs Confidential 6 Download result

# Delete result # ------------------------------------------------------------------ result = db.get(result_id) db.delete(result) # Delete script # ------------------------------------------------------------------ script = db.get(script_id) db.delete(script) 06/05/09 Sauce Labs Confidential 8 Delete result and script

What we’ll do for you Provide free access to product Provide coaching on: How to integrate our service into your build scripts How to parallelize your test scripts Demonstrate prototypes for your feedback Notify you of downtime and updates (planned or not) Take precautions to ensure that your information remains secure 06/05/09 Sauce Labs Confidential

What we need from you Information about your existing testing environment Perform actual integration of Sauce Labs into your test environment Use Sauce Labs' product in your testing process Report any bugs or feature shortcomings you find Assign a primary point of contact Participate in weekly check-in calls Provide a product review (good or bad) that we may publish Speak with potential Sauce Labs investors about your experiences 06/05/09 Sauce Labs Confidential

Next steps 06/05/09 Sauce Labs Confidential

Where do we go from here? xUnit, TestNG Fit(nesse), RSpec Hudson Continuum CruiseControl TeamCity Bamboo Git(hub) Subversion Cron Eclipse Internet Explorer Firefox Safari Chrome Opera Flash/Flex iPhone Android Selenium output Screencasts Screenshots DOMshots JavaScript state Yslow HTML errors Accessibility HTTP traffic Dtrace … all tracked over time Python Ruby Java Perl Selenese C# PHP Selenium on Rails 06/05/09 Sauce Labs Confidential Platforms Languages Build Systems Telemetry Frameworks

# CouchDB library from http://code.google.com/p/couchdb-python/ from couchdb import Server # Link to user's database server = Server('http://sauceapp.com/') db = server['jsmith’] # Upload a script # ---------------------------------------------------------------------- # A Sauce Labs script is simply a regular python script wrapped in JSON script_id = db.create({ "Type" : "script", "Name" : "Google Search", "Language" : "python", "Content" : "open ('google.com')\ntype ('q', 'selenium')\nclick'btnG')\nwaitForPageToLoad()\nverifyTextPresent('openqa.org')" }) 06/05/09 Sauce Labs Confidential 4 Upload script

# Run a job # ---------------------------------------------------------- # You run a job by uploading a new "job" document, linking to the script id. # A job is instantly queued for execution upon upload. job_id = db.create( { "Type" :"job", "ScriptId" :script_id, }) 06/05/09 Sauce Labs Confidential 5 Run job

# Check job status # -------------------------------------------------------------------------- status = db[job_id]["Status"] # 'status' will be one of: # "new" # "queued" # "in progress" # "complete" # "error” # Download result # -------------------------------------------------------------------------- # Once the job is complete, a "result" document is created. The "job" document # is updated with a reference to the result. # First, retrieve the ResultId from the job result_id = db[job_id]['ResultId'] # Second, retrieve the complete result document result = db[result_id] # You can now introspect the results. # View standard output standard_output = result['Stdout'] # View standard error standard_error = result['Stderr'] 06/05/09 Sauce Labs Confidential 6 Download result

# Delete result # ----------------------------------------------------------------- del db[result_id] # Delete script # ----------------------------------------------------------------- del db[script_id] 06/05/09 Sauce Labs Confidential 8 Delete result and script

# Using CouchDB library by Michael Hendricks (not in CPAN, yet!) # http://git.ndrix.com/?p=Net-CouchDB use Net::CouchDB; # Link to user's database my $server = Net::CouchDB->new('http://localhost:5984'); my $db = $server->db('perltest'); # Upload a script # -------------------------------------------------------------------------- # A Sauce Labs script is simply a regular perl script wrapped in JSON my $script = $db->insert( { Type => "script", Name => "Google Search", Langage => "perl", Content => "open ('google.com')\ntype ('q', 'selenium')\nclick'btnG')\nwaitForPageToLoad()\nverifyTextPresent('openqa.org')" }); my $script_id = $script->id; 06/05/09 Sauce Labs Confidential 4 Upload script

# Run a job # ---------------------------------------------------------- # You run a job by uploading a new "job" document, linking to # the script id. # A job is instantly queued for execution upon upload. my $job = $db->insert( { Type => "job", ScriptId => $script_id }); my $job_id = $job->id; 06/05/09 Sauce Labs Confidential 5 Run job

# Check job status # -------------------------------------------------------------------------- my $status = $db->document($job_id)->{'Status'}; # 'status' will be one of: # "new" # "queued" # "in progress" # "complete" # "error” # Download result # -------------------------------------------------------------------------- # Once the job is complete, a "result" document is created. The "job" document # is updated with a reference to the result. # First, retrieve the ResultId from the job my $result_id = $db->document($job_id)->{'ResultId'}; # Second, retrieve the complete result document: my $result = $db->document($result_id); # You can now introspect the results. # View standard output my $standard_output = result->{'Stdout'}; # View standard error my $standard_error = result->{'Stderr'}; 06/05/09 Sauce Labs Confidential 6 Download result

# Delete result # ------------------------------------------------------------------ my $result = $db->document($result_id); $result->delete; # Delete script # ------------------------------------------------------------------ my $script = $db->document($script_id); $script->delete; 06/05/09 Sauce Labs Confidential 8 Delete result and script

Curl command: curl -X POST - u jdoe:password -d ' { "Type":"script", "Name":"Google Search", "Language":"python", "Content":"open ('google.com')\ntype ('q', 'selenium')\nclick'btnG')\nwaitForPageToLoad()\nverifyTextPresent('openqa.org')", "CreatedBy":"jdoe", "CreatedDate":"Mon, 18 Aug 2008 14:26:10 +0000", "LastModifiedBy":"jdoe", "LastModifiedDate":"Mon, 18 Aug 2008 14:26:10 +0000" }' http://saucedb.com/jdoe/ 06/05/09 Sauce Labs Confidential 4 Upload script

Curl returns: {"ok":true,"id":"4749f6c4159ae5a097e586f3e5040cbe","rev":"668367075"} To view in browser: (You'll probably need to provide name/password) http://jdoe.sauceapp.com/4749f6c4159ae5a097e586f3e5040cbe 06/05/09 Sauce Labs Confidential 4 Upload script

Curl command: curl -X POST -u jdoe:password -d ' { "Type":"job", "Status":"new", "ScriptId":"3ece08113906d84325f9341e794caefa", "ResultId":null, "CreatedBy":"jdoe", "CreatedDate":"Mon, 18 Aug 2008 14:54:55 +0000" }' http://saucedb.com/jdoe/ 06/05/09 Sauce Labs Confidential 5 Run job

Curl returns: {"ok":true,"id":"3c4c1f2f21ea4e346aebc244f21dde76","rev":"1359430379"} To view in browser: (You'll probably need to provide name/password) http://jdoe.sauceapp.com/3c4c1f2f21ea4e346aebc244f21dde76 06/05/09 Sauce Labs Confidential 5 Run job

Partnering with Sauce Labs Sauce focuses on hosted testing Looking for independent consultants, and consulting firms, who use our service during their engagements Guru pricing, training, etc. available 06/05/09 Sauce Labs Confidential

Agile is now mainstream Source: Dr. Dobb’s Journal. 5/7/08. http://bit.ly/4EjcDH 06/05/09 Sauce Labs Confidential

Continuous Integration is now mainstream 06/05/09 Sauce Labs Confidential Source:VersionOne. 8/22/08. http://www.versionone.com/pdf/3rdAnnualStateOfAgile_FullDataReport.pdf

Functional tests tend to run at end of day 06/05/09 Sauce Labs Confidential Source: Viewtier Systems. http://bit.ly/2qQ5LX

Our current thinking about pricing (Subject to change) 06/05/09 Sauce Labs Confidential $ per test minute Tests queued no later than… Five minutes Express : Relatively high per test minute 8 hours Economy : Low amount per test minute

Express entails lower chef utilization (We’ll need beta usage to quantify) 06/05/09 Sauce Labs Confidential Express Chef # minute Economy Chef # minute script 1 script 6 script 10 script 2 script 7 script 11 script 12 script 3 script 8 script 13 script 4 script 9 script 14 script 5 10 20 30 40 50 60 1 2 3 4 5 10 20 30 40 50 60 script 1 script 2 script 3 script 4 script 5 script 6 1

Thinking about pricing the right way? How much Express would you use? How much Economy? 06/05/09 Sauce Labs Confidential

Sauce Labs Beta Program Overview

More Related Content

What's hot (20)

Similar to Sauce Labs Beta Program Overview (20)

Recently uploaded (20)

Sauce Labs Beta Program Overview