SlideShare a Scribd company logo
Online Testing – the 
O li T i          h
 light in the tunnel
 light in the tunnel

 October, 2010
AGENDA

•   What is Online Testing?
    What is Online Testing?
•   How is it working?
•   Variables of Success
       i bl    fS
•   Case Studies
•   Conversion Optimization Consulting
What is Online Testing?

Online Testing (OL) Definition:
Online Testing (OL) Definition:
  In a nutshell, online testing 
  involves testing one or more 
                  g
  variable from a webpage to 
  identify which ones can create a 
  positive difference in the 
  conversion. 
  The winning variable then 
    h                bl h
  becomes the control, or bench 
  mark, which future ones are 
  mark which future ones are
  measured against.
Online Testing Steps

Scientific testing involves the following steps:
1) Determining which element(s) of a page you want to test. 
2) Creating the test / the test page
3) Setting up the tool use for the testing
3) Setting up the tool use for the testing
4) Keeping good, accurate test records.
5) Running the test campaign.
6) Tracking and comparing results to determine which test variable is the 
6)     ki      d         i        l    d      i   hi h        i bl i h
    winner. (100 conversions)
7) Repeating the cycle, constantly attempting to beat your existing control
Categories of Online Testing

• A/B test, Split test
   /      , p                     • MVT ‐Multivariable test
 Page 1: Control (the original)      Page 1: Control (the original)

                                                         SUBSCRIBE 
                                                         SUBSCRIBE
                                                           NOW
        SEND ME THE FREE 
           GUIDE NOW



 Page 2: Test page                   Page 2: Test page

                                                          SEND ME 
        SEND ME THE FREE                                  THE INFO
           GUIDE NOW
           GUIDE NOW
Time needed for testing?

                        LONGER Test Period    SHORTER Test Period
# of Alternatives /     More (MVT)            Less (A/B)
Variables
Difference in Results   Lesser                Greater
Traffic Volume          Less                  More
Confidence Level        Higher > 90%          Lower < 90%




                                        100 conversions
Case Study 1 – Relevancy 




ORIGINAL                 TEST
Partial results
Relevant? 



NOT YET !
NOT YET !
Risks? What Risks?

• All Test Types are not
  All Test Types are not 
  Equal

• Comparing & Contrasting 
  A/B and MVT
  A/B d MVT

• The Risk of the Endless 
  Test
KEY principles for online tests

Test Methodology
     – Is A Process !
     – Single Variable Test Formula
         • The most important tool for performing a useful experiment
           The most important tool for performing a useful experiment. 
     – The (relevant) Academic Vocabulary
         • There are Concepts, Terminology, Differences between Variable
         • Understand and Use them Properly
              d       d d        h            l
     – Test Protocol
         • Use proper tool organizes each stage of the test process and 
           automatically calculates all of the key equations.
KEY principles for online tests

Test Validity
            y
• Understanding of validity principles, so that you can be 
   confident in the results of your test.
    – The Threats to Test Validity
       • "history effect", 
       • "instrumentation error"
          "instrumentation error"
       • "sample distortion”
    – The Sample Size Validity
      The Sample Size Validity
       • The key mathematical equations for determining test 
          size.
    – The Validity Formula
KEY principles for online tests
Test Planning
• Which variables to test? 
     – Knowing what to test is as important as knowing how to test. 
     – The Optimization Formula
         • Prioritize your testing goals
           Prioritize your testing goals. 
         • More revenue = right order.
     – The Test ROI Calculation Tool
         • Calculates the compounding ROI on a given series of tests. 
         • Saves time and money as it helps you determine the impact of each 
           test on your marketing budget.
     – Radical Redesigns
         • How to capture multi‐variable test results with single variable testing 
           techniques
                  q
What’s the big deal about Online 
                                      Testing?

• Online Testing:
  Online Testing:
  – Motivation
  – Value proposition
    Value proposition
  – Incentive
  –A i t
    Anxiety
  – Friction 



        C = 4m + 3v + 2(i f) -2a ©
                      2(i-f) 2a
When?
• How would you Know WHEN need TO APPLY
  this strategy?
   hi          ?

• The right answer: 

    • ALL THE TIME there is an
      ALL THE TIME there is an 
            Opportunity
             pp       y
• Amazon – case study: 
    200 live online tests running every day
    200 li     li t t         i         d
Case Study 2‐ The Formula!




816% Growth in
New Subscribers

ORIGINAL                TEST
Case Study 3 – Prove It!
Case Study 3‐ Treatment
 100% Money BacK Guarantee – Eliminating anxietY




  Call to Actions highlighting the benefits
Let’s compare …




ORIGINAL      TREATMENT
Case Study 4: Is It Worth it?

Professional Photographer
                  g p
• Status:
   –   New site
   –   Artistic / flash
                /
   –   Minimum content
   –   No commercial aspect
• Goals:
   – To get New Clients
• Strategies:
   – They decided to do PPC
   – Adding some wedding pictures on the actual site
     Adding some wedding pictures on the actual site
Case Study 4 – cont.

• Our proposal:
  Our proposal:
  – Landing Page
           g g
• Reaction:
  – Fighting
• Agreement:
  – Online Testing 
Case Study 4 – cont.

• Results:
  – Our page performed better: 71.6% improvement 
    in conversion (contacts)
    in conversion (contacts)
• The client got 4 wedding contracts, with an 
  online marketing budget of 3.5% spent
  online marketing budget of 3 5% spent
Oak Photography LP




Differences
Oak Photography LP




       Differences
ATTENTION


• Controversy about Best Practices
  Controversy about Best Practices

• Example: Video on AutoStart or 
  Press Play
         l
Case Study 5 ‐ controversy

           Video Play Not Play
           Vid – Pl or N t Pl ?




           63% improvement
           in conversion with
           Video on autoplay
CONCLUSIONS
• Do not go Blind with 
  your marketing 
  your marketing

• Focus your money and 
  resources where it 
  counts

• Get confirmation you 
  are on the right track
               g
Conversion Optimization
            Eworks WSI Cyprus:
•   Consulting Services
    Consulting Services
•   Landing Page / Website / Email 
    Marketing
    M k ti
•   Report (14‐20 variables analyzed)
•   100% Money Back Guarantee 
    (certain conditions apply)
•   Our Clients Implement 95‐100 % 
    of our recommendations
    of our recommendations
EworksWSI Cyprus
SERVICES:
- Webdesign               - Interactive Content
                          ( g
                          (digital p
                                   publishing)
                                            g)
- Landing Pages Design
                          -Digital Marketing
- Website Conversion
Optimization
Optimi ation              - Flash Content

- Landing Pages           - Print Design
Conversion Optimization
             p
                          - Logo Design
- Email Marketing
Conversion Optimization                           CONTACT INFO: 
                                                  info@eworkswsi.com.cy
                                                  i f @    k i
                                                  +35799‐555747
                                                  www.EworksWSI.com.cy
                                                  www EworksWSI com cy
PROMO !

ONLY for our clients:
ONLY for our clients:
‐ Until Dec 31, 2010
‐ O li
  Online testing discounted f
             i di         d from €500 to €300
                                 € 00 €300




‐ A d hi i
  And this is not all ……
                   ll
PROMO !

Package 1:          Package 2:          Package 3:
1 sample design     2 sample design     2 sample design
Cost Package:       Cost Package:       1 Online Testing
€450 
€450                €600 
                    €600                Cost Package: 
                                        Cost Package:
(Information        (Information        €685
Architecture Doc 
Architecture Doc    Architecture Doc 
                    Architecture Doc    (Information 
                                        (Information
completed)          completed)          Architecture Doc 
                                        completed)
Contact  Eworks WSI Cyprus

Contact us for more info
Contact us for more info

   Eworks WSI Cyprus
       k S C

 info@eworkswsi.com.cy
 www.eworkswsi.com.cy
 www eworkswsi com cy
    Tel: +357‐99555747

More Related Content

PDF
Catalogue spanish courses_in_spain_fabrica_de_idiomas
PDF
Investigaton of the links between the Natural Environment, Biodiversity and M...
PPT
Gubad Ibadoglu - PWYP Montreal Conference 2009
PPTX
Grab Infotech Portfolio
PPT
Kerstdienst 2014 25 dec. 2014
PPT
Online Marketing für kleinere Unternehmen
PDF
Optimizing Shopping Carts for the Holidays
Catalogue spanish courses_in_spain_fabrica_de_idiomas
Investigaton of the links between the Natural Environment, Biodiversity and M...
Gubad Ibadoglu - PWYP Montreal Conference 2009
Grab Infotech Portfolio
Kerstdienst 2014 25 dec. 2014
Online Marketing für kleinere Unternehmen
Optimizing Shopping Carts for the Holidays

Similar to Eworks WSI Cyprus online testing oct 2010 (20)

PDF
How to design powerful experiments - Ying Zhang
PDF
Cro webinar what you're doing wrong in your cro program (sharable version)
 
PDF
Can I Test More Than One Variable at a Time? Statisticians answer some of th...
PPTX
Conversion Rate Optimisation: The Science Behind Turning Visitors Into Customers
PPTX
#Measurefest : 20 Simple Ways to Fuck Up your AB tests
PDF
Materials And Information Flow Map
PPTX
John Fodeh - Spend Wisely, Test Well
PDF
See the Research in Action
PDF
Pricing strategies 2017 nz
PDF
Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experienc...
PDF
Meta-Analyses in Experimentation: The Whats and Hows
 
PPTX
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
PDF
Brooks Bell Interactive Tama Presentation
PDF
What is the true ROI on your technology investments?
PDF
4 ie 2015 rapid research
PDF
Lean Startup 301
PPT
Metrics and Analytics, Guest Lecture, UCLA
PDF
4 Steps Toward Scientific A/B Testing
PDF
Eye see - increase your marketing impact presentation2
PDF
EyeSee improve marketing impact
How to design powerful experiments - Ying Zhang
Cro webinar what you're doing wrong in your cro program (sharable version)
 
Can I Test More Than One Variable at a Time? Statisticians answer some of th...
Conversion Rate Optimisation: The Science Behind Turning Visitors Into Customers
#Measurefest : 20 Simple Ways to Fuck Up your AB tests
Materials And Information Flow Map
John Fodeh - Spend Wisely, Test Well
See the Research in Action
Pricing strategies 2017 nz
Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experienc...
Meta-Analyses in Experimentation: The Whats and Hows
 
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Brooks Bell Interactive Tama Presentation
What is the true ROI on your technology investments?
4 ie 2015 rapid research
Lean Startup 301
Metrics and Analytics, Guest Lecture, UCLA
4 Steps Toward Scientific A/B Testing
Eye see - increase your marketing impact presentation2
EyeSee improve marketing impact
Ad

More from EworksWSI Cyprus (14)

PDF
EworksWSI Cyprus 2016 Content Marketing Whitepaper
PDF
WSI Video Marketing Packages 2016
PDF
Eworks WSI Cyprus Digital Marketing Services Overview 2016
PDF
EworksWSI Productsheet Mobilemarketing
PDF
Eworks WSI Cyprus Productsheet Email Marketing
PDF
Nov2014 whitepaper digitallandscape2015
PDF
2013 EworksWSI Social media tactical plan
PDF
Eworks WSI Cyprus bulk SMS platform User Guide
PDF
Eworks WSI Cyprus 2012 Mobilemarketing simplified
PDF
Social media in travel & tourism best practices
PDF
Eworks WSI Cyprus: Online reputation Mangement
PDF
Converting visitors into customers
PDF
3 fifty7 Music Agency Presentation by EworksWSI
PDF
Eworks wsi cyprus wsi google adwords
EworksWSI Cyprus 2016 Content Marketing Whitepaper
WSI Video Marketing Packages 2016
Eworks WSI Cyprus Digital Marketing Services Overview 2016
EworksWSI Productsheet Mobilemarketing
Eworks WSI Cyprus Productsheet Email Marketing
Nov2014 whitepaper digitallandscape2015
2013 EworksWSI Social media tactical plan
Eworks WSI Cyprus bulk SMS platform User Guide
Eworks WSI Cyprus 2012 Mobilemarketing simplified
Social media in travel & tourism best practices
Eworks WSI Cyprus: Online reputation Mangement
Converting visitors into customers
3 fifty7 Music Agency Presentation by EworksWSI
Eworks wsi cyprus wsi google adwords
Ad

Recently uploaded (20)

PDF
TyAnn Osborn: A Visionary Leader Shaping Corporate Workforce Dynamics
PDF
Ôn tập tiếng anh trong kinh doanh nâng cao
PDF
NISM Series V-A MFD Workbook v December 2024.khhhjtgvwevoypdnew one must use ...
PDF
BsN 7th Sem Course GridNNNNNNNN CCN.pdf
PDF
Deliverable file - Regulatory guideline analysis.pdf
PDF
Building a Smart Pet Ecosystem: A Full Introduction to Zhejiang Beijing Techn...
PDF
Module 2 - Modern Supervison Challenges - Student Resource.pdf
PDF
NewBase 12 August 2025 Energy News issue - 1812 by Khaled Al Awadi_compresse...
PDF
SIMNET Inc – 2023’s Most Trusted IT Services & Solution Provider
PPTX
Negotiation and Persuasion Skills: A Shrewd Person's Perspective
PDF
ANALYZING THE OPPORTUNITIES OF DIGITAL MARKETING IN BANGLADESH TO PROVIDE AN ...
PDF
Outsourced Audit & Assurance in USA Why Globus Finanza is Your Trusted Choice
PPTX
Slide gioi thieu VietinBank Quy 2 - 2025
PPTX
2025 Product Deck V1.0.pptxCATALOGTCLCIA
PPTX
TRAINNING, DEVELOPMENT AND APPRAISAL.pptx
PDF
Solaris Resources Presentation - Corporate August 2025.pdf
PDF
pdfcoffee.com-opt-b1plus-sb-answers.pdfvi
PPTX
Principles of Marketing, Industrial, Consumers,
PDF
Tata consultancy services case study shri Sharda college, basrur
PDF
Introduction to Generative Engine Optimization (GEO)
TyAnn Osborn: A Visionary Leader Shaping Corporate Workforce Dynamics
Ôn tập tiếng anh trong kinh doanh nâng cao
NISM Series V-A MFD Workbook v December 2024.khhhjtgvwevoypdnew one must use ...
BsN 7th Sem Course GridNNNNNNNN CCN.pdf
Deliverable file - Regulatory guideline analysis.pdf
Building a Smart Pet Ecosystem: A Full Introduction to Zhejiang Beijing Techn...
Module 2 - Modern Supervison Challenges - Student Resource.pdf
NewBase 12 August 2025 Energy News issue - 1812 by Khaled Al Awadi_compresse...
SIMNET Inc – 2023’s Most Trusted IT Services & Solution Provider
Negotiation and Persuasion Skills: A Shrewd Person's Perspective
ANALYZING THE OPPORTUNITIES OF DIGITAL MARKETING IN BANGLADESH TO PROVIDE AN ...
Outsourced Audit & Assurance in USA Why Globus Finanza is Your Trusted Choice
Slide gioi thieu VietinBank Quy 2 - 2025
2025 Product Deck V1.0.pptxCATALOGTCLCIA
TRAINNING, DEVELOPMENT AND APPRAISAL.pptx
Solaris Resources Presentation - Corporate August 2025.pdf
pdfcoffee.com-opt-b1plus-sb-answers.pdfvi
Principles of Marketing, Industrial, Consumers,
Tata consultancy services case study shri Sharda college, basrur
Introduction to Generative Engine Optimization (GEO)

Eworks WSI Cyprus online testing oct 2010

  • 1. Online Testing – the  O li T i h light in the tunnel light in the tunnel October, 2010
  • 2. AGENDA • What is Online Testing? What is Online Testing? • How is it working? • Variables of Success i bl fS • Case Studies • Conversion Optimization Consulting
  • 3. What is Online Testing? Online Testing (OL) Definition: Online Testing (OL) Definition: In a nutshell, online testing  involves testing one or more  g variable from a webpage to  identify which ones can create a  positive difference in the  conversion.  The winning variable then  h bl h becomes the control, or bench  mark, which future ones are  mark which future ones are measured against.
  • 4. Online Testing Steps Scientific testing involves the following steps: 1) Determining which element(s) of a page you want to test.  2) Creating the test / the test page 3) Setting up the tool use for the testing 3) Setting up the tool use for the testing 4) Keeping good, accurate test records. 5) Running the test campaign. 6) Tracking and comparing results to determine which test variable is the  6) ki d i l d i hi h i bl i h winner. (100 conversions) 7) Repeating the cycle, constantly attempting to beat your existing control
  • 5. Categories of Online Testing • A/B test, Split test / , p • MVT ‐Multivariable test Page 1: Control (the original) Page 1: Control (the original) SUBSCRIBE  SUBSCRIBE NOW SEND ME THE FREE  GUIDE NOW Page 2: Test page Page 2: Test page SEND ME  SEND ME THE FREE  THE INFO GUIDE NOW GUIDE NOW
  • 6. Time needed for testing? LONGER Test Period SHORTER Test Period # of Alternatives /  More (MVT) Less (A/B) Variables Difference in Results Lesser Greater Traffic Volume Less More Confidence Level Higher > 90% Lower < 90% 100 conversions
  • 10. Risks? What Risks? • All Test Types are not All Test Types are not  Equal • Comparing & Contrasting  A/B and MVT A/B d MVT • The Risk of the Endless  Test
  • 11. KEY principles for online tests Test Methodology – Is A Process ! – Single Variable Test Formula • The most important tool for performing a useful experiment The most important tool for performing a useful experiment.  – The (relevant) Academic Vocabulary • There are Concepts, Terminology, Differences between Variable • Understand and Use them Properly d d d h l – Test Protocol • Use proper tool organizes each stage of the test process and  automatically calculates all of the key equations.
  • 12. KEY principles for online tests Test Validity y • Understanding of validity principles, so that you can be  confident in the results of your test. – The Threats to Test Validity • "history effect",  • "instrumentation error" "instrumentation error" • "sample distortion” – The Sample Size Validity The Sample Size Validity • The key mathematical equations for determining test  size. – The Validity Formula
  • 13. KEY principles for online tests Test Planning • Which variables to test?  – Knowing what to test is as important as knowing how to test.  – The Optimization Formula • Prioritize your testing goals Prioritize your testing goals.  • More revenue = right order. – The Test ROI Calculation Tool • Calculates the compounding ROI on a given series of tests.  • Saves time and money as it helps you determine the impact of each  test on your marketing budget. – Radical Redesigns • How to capture multi‐variable test results with single variable testing  techniques q
  • 14. What’s the big deal about Online  Testing? • Online Testing: Online Testing: – Motivation – Value proposition Value proposition – Incentive –A i t Anxiety – Friction  C = 4m + 3v + 2(i f) -2a © 2(i-f) 2a
  • 15. When? • How would you Know WHEN need TO APPLY this strategy? hi ? • The right answer:  • ALL THE TIME there is an ALL THE TIME there is an  Opportunity pp y • Amazon – case study:  200 live online tests running every day 200 li li t t i d
  • 16. Case Study 2‐ The Formula! 816% Growth in New Subscribers ORIGINAL TEST
  • 18. Case Study 3‐ Treatment 100% Money BacK Guarantee – Eliminating anxietY Call to Actions highlighting the benefits
  • 20. Case Study 4: Is It Worth it? Professional Photographer g p • Status: – New site – Artistic / flash / – Minimum content – No commercial aspect • Goals: – To get New Clients • Strategies: – They decided to do PPC – Adding some wedding pictures on the actual site Adding some wedding pictures on the actual site
  • 21. Case Study 4 – cont. • Our proposal: Our proposal: – Landing Page g g • Reaction: – Fighting • Agreement: – Online Testing 
  • 22. Case Study 4 – cont. • Results: – Our page performed better: 71.6% improvement  in conversion (contacts) in conversion (contacts) • The client got 4 wedding contracts, with an  online marketing budget of 3.5% spent online marketing budget of 3 5% spent
  • 24. Oak Photography LP Differences
  • 25. ATTENTION • Controversy about Best Practices Controversy about Best Practices • Example: Video on AutoStart or  Press Play l
  • 26. Case Study 5 ‐ controversy Video Play Not Play Vid – Pl or N t Pl ? 63% improvement in conversion with Video on autoplay
  • 27. CONCLUSIONS • Do not go Blind with  your marketing  your marketing • Focus your money and  resources where it  counts • Get confirmation you  are on the right track g
  • 28. Conversion Optimization Eworks WSI Cyprus: • Consulting Services Consulting Services • Landing Page / Website / Email  Marketing M k ti • Report (14‐20 variables analyzed) • 100% Money Back Guarantee  (certain conditions apply) • Our Clients Implement 95‐100 %  of our recommendations of our recommendations
  • 29. EworksWSI Cyprus SERVICES: - Webdesign - Interactive Content ( g (digital p publishing) g) - Landing Pages Design -Digital Marketing - Website Conversion Optimization Optimi ation - Flash Content - Landing Pages - Print Design Conversion Optimization p - Logo Design - Email Marketing Conversion Optimization CONTACT INFO:  info@eworkswsi.com.cy i f @ k i +35799‐555747 www.EworksWSI.com.cy www EworksWSI com cy
  • 30. PROMO ! ONLY for our clients: ONLY for our clients: ‐ Until Dec 31, 2010 ‐ O li Online testing discounted f i di d from €500 to €300 € 00 €300 ‐ A d hi i And this is not all …… ll
  • 31. PROMO ! Package 1: Package 2: Package 3: 1 sample design 2 sample design 2 sample design Cost Package: Cost Package: 1 Online Testing €450  €450 €600  €600 Cost Package:  Cost Package: (Information  (Information  €685 Architecture Doc  Architecture Doc Architecture Doc  Architecture Doc (Information  (Information completed) completed) Architecture Doc  completed)
  • 32. Contact  Eworks WSI Cyprus Contact us for more info Contact us for more info Eworks WSI Cyprus k S C info@eworkswsi.com.cy www.eworkswsi.com.cy www eworkswsi com cy Tel: +357‐99555747