SlideShare a Scribd company logo
experience
Advanced Experimentation Workshop
James Spittal, CEO, Web Marketing ROI
Holly Joshi, Sr. Customer Success Manager, Optimizely
Mansoor Omar, Technical Support Engineer, Optimizely
1. Tackling data problems
• Misconceptions/theories
• Data between GA and Optimizely
• How to ensure what you’re seeing is accurate
2. Overcoming common challenges faced operationally
• Writing regular expressions
• Cross-domain Optimizely
3. Making your campaign development faster and more efficient
• Optimizely Extensions
• Reducing flickering
Agenda
experience
CRO/MarTech Leadership = GSD
experience
experience
Modernize 2018: Advanced Experimentation Workshop - Web Marketing ROI & Optimizely
experience
All About The Data
What Optimizely measures and integrating with your data systems
1. Resolving the misconceptions
2. Implementation is the solution
3. Get the most common/most critical
right
Optimizely Data
experience
73%
… the percentage of Google Analytics implementations that are wrong (leading to
incorrect traffic numbers, incorrect revenue, etc).
Source: https://econsultancy.com/blog/10925-80-of-online-retailers-are-using-google-
analytics-incorrectly
Modernize 2018: Advanced Experimentation Workshop - Web Marketing ROI & Optimizely
Apples & Oranges
experience
If you’re not comparing the same thing –
you’re not going to see the same numbers.
Apples & Oranges
experience
Google Analytics Session Time-Outs (and
Optimizely End User Id)
By default, a Google Analytics session
expires after 30 minutes of user inactivity.
This is entirely different to how Optimizely
tracks a “unique visitor” (tied to Optimizely
End User Id cookie, 10 year expiry).
experience
Impressions (new Optimizely measurement
model) – How it actually works
Optimizely logs an event whenever an experiment activates (this can be seen as a ‘decision’
under the hood as an XMLHTTPRequest)
The decision carries a few pieces of information, including:
Visitor ID
The ID of the experiment / campaign which is activating
The ID of the variation the visitor is allocated to
UUID: A unique decision ID (used for deduplication)
Again, decision events are generated whenever an experiment activates, even if the visitor
has been targeted by the same experiment previously. So in a non-technical sense, you can
imagine that a decision is generated each time an experience is delivered to a visitor.
Impressions (new Optimizely measurement
model)
select
count(distinct(UUID))
from
decisions;
This deduplication on UUID is important because Optimizely clients sometimes send multiple
decisions for the same experiment activation (this is due to the batching behavior of the web
snippet).
Typically, impressions are allocated on an annual basis. So we just count up the number of
unique decision events, subtract them from the annual quota, and voila, you've got your
"impressions remaining" count for the year.
experience
Optimizely Results Screen Example
Pages misfiring/misactivating
experience
Audience Conditions – Custom JavaScript
experience
QA everything (DIY)
1. Make a spreadsheet of all of your
events, pages and audiences
2. Create a QA campaign for each, to verify
it is triggering when intended and not
triggering when it shouldn’t.
experience
Optimizely Accelerate
experience
Optimizely Accelerate
Total cost around $15k AUD, includes full events/pages/URLs/audiences QA and campaign build,
delivered by approved Optimizely Solutions Partners.
experience
experience
URLs
Custom, Cross-Domain, Multi-Page
1. Regular expressions (versus Simple
Match, Exact Match and Substring Match)
2. Multi-page tests
3. Making Optimizely work cross-domain
(and/or cross-project)
Pages & URLs
experience
1. Matching too freely
2. ‘example.com’ Matches (Substring) for:
yourdomain.com.au/example.com
Common ‘Pages’ mistakes
experience
Regular Expressions Cheat Sheet
experience
Regular Expressions Practice
1. This makes Optimizely cross-domain
work
2. In short - Injects an iFrame that allows
local storage/cookie to traverse
3. Put waitForOriginSync into your Project
JS
WaitForOriginSync
experience
WaitForOriginSync
experience
experience
Campaign Development
Extensions, Efficiencies and Performance
1. Extensions = more campaigns, faster
• Extensions 101
• Why extensions can be tricky
• Extension of extensions
2. Experiment flickering minimisation
Campaign Development
experience
Extensions
Extensions are extendable!
experience
1. Templates, modularity, performance,
strategy, operational savings…
2. Object- orientated
101
experience
1. How many input fields should we have?
2. Why is my extension not showing up/not
accessible?
3. Where did my extension go?
4. Data drops out of an extension
Challenges
experience
1. Translation
[:en]
Hello World
[:zh]
(Traditional Chinese)
2. Markdown
3. Extending extensions
Cool Use Cases
experience
Page Flash/Flickering
https://media.giphy.com/media/wdh1SvEn0E06I/giphy.g
if
Modernize 2018: Advanced Experimentation Workshop - Web Marketing ROI & Optimizely
1. WaitForElement & ES6
2. WaitUntil
3. See the Optimzely Developer Docs
4. Async vs. Sync
Document object model
challenges
experience
jQuery – what you need to know about
Optimizely X
Var $ = window.optimizely.get(‘jQuery’) [Optimizely jQuery] when
instrumenting jQuery in your Optimizely campaigns goes wrong versus
var $ = window.jQuery;
var $ = window.jQuery || window.optimizely.get(‘jQuery’);
experience
1. Use ‘utils’ (WaitForElement, etc)
2. Use ES6/Native JavaScript
3. Asynchronous versus synchronous
Most important things to know
experience
JavaScript Hell
Asynchronous versus synchronous
Don’t always believe what Google
PageSpeed Insights tells you 
experience
Document Object Model
experience
experience
Takeaways
Recap & Resources
1. If you need help with data analysis
problems or auditing your GA data, speak
to an Optimizely Solutions Partner or
GACP
2. Manny and Holly are available for on-
demand Optimizely support for Optimizely
customers
Next steps
experience
experience
Questions?
james@webmarketingroi.com.au
James Spittal
THANK YOU

More Related Content

PDF
Modernize 2018: Customer Data Orchestration Across Every Experience in Real-T...
PDF
Modernize 2018: The Need for Speed - Contentful
PDF
Modernize 2018: Activating Customer Experiences for Modern Organisations - Ac...
PDF
Modernize 2018: Running High-Performance Experience Teams - Isobar
PDF
The Wall Street Journal - Optimizing Membership
PPTX
An Experimentation Framework: How to Position for Triple Digit Growth
PDF
How to Reduce Customer Acquisition Costs and Optimize Advertising Spend
PDF
Optimizing Your B2B Demand Generation Machine
Modernize 2018: Customer Data Orchestration Across Every Experience in Real-T...
Modernize 2018: The Need for Speed - Contentful
Modernize 2018: Activating Customer Experiences for Modern Organisations - Ac...
Modernize 2018: Running High-Performance Experience Teams - Isobar
The Wall Street Journal - Optimizing Membership
An Experimentation Framework: How to Position for Triple Digit Growth
How to Reduce Customer Acquisition Costs and Optimize Advertising Spend
Optimizing Your B2B Demand Generation Machine

What's hot (20)

PPTX
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
PDF
Run High Impact Experimentation with High-quality Customer Discovery
PDF
How Clorox Experiments Across Brands to Turn Visitors into Consumers
PPTX
Making Your Hypothesis Work Harder to Inform Future Product Strategy
PPTX
The Personalization Playbook: Secrets from successful real-world experiments
PPTX
Triple Your Experiment Velocity by Integrating Optimizely with Your Data Ware...
PPTX
Optimizely Product Vision: The Future of Experimentation
PDF
Definition of A/B testing and Case Studies by Optimizely
PPTX
UX Analytics and Experimentation for eCommerce Growth
 
PPTX
Becoming a True Experimentation Organization to Drive Innovation
PPTX
Optimizely & Photobox - DON'T PANIC: The No-Confusion Experimentation Startup...
PPTX
Experimentation Excellence Webinar Series: Time to scale up
PPTX
Testing Your Testing Program
PPTX
[Webinar] Getting Buy-in for Your Experimentation and Personalisation Program
PPTX
Digital Personalization 101: The building blocks for a powerful strategy
PPTX
Outperform Webinar Series: How to Capture Your Customers at the Top of the Fu...
PPTX
The Science of Getting Testing Right
PPTX
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
PDF
World Class Optimization: Benchmarking 1,000+ Companies
PDF
Product Experimentation | Forming Strong Experiment Hypotheses
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Run High Impact Experimentation with High-quality Customer Discovery
How Clorox Experiments Across Brands to Turn Visitors into Consumers
Making Your Hypothesis Work Harder to Inform Future Product Strategy
The Personalization Playbook: Secrets from successful real-world experiments
Triple Your Experiment Velocity by Integrating Optimizely with Your Data Ware...
Optimizely Product Vision: The Future of Experimentation
Definition of A/B testing and Case Studies by Optimizely
UX Analytics and Experimentation for eCommerce Growth
 
Becoming a True Experimentation Organization to Drive Innovation
Optimizely & Photobox - DON'T PANIC: The No-Confusion Experimentation Startup...
Experimentation Excellence Webinar Series: Time to scale up
Testing Your Testing Program
[Webinar] Getting Buy-in for Your Experimentation and Personalisation Program
Digital Personalization 101: The building blocks for a powerful strategy
Outperform Webinar Series: How to Capture Your Customers at the Top of the Fu...
The Science of Getting Testing Right
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
World Class Optimization: Benchmarking 1,000+ Companies
Product Experimentation | Forming Strong Experiment Hypotheses
Ad

Similar to Modernize 2018: Advanced Experimentation Workshop - Web Marketing ROI & Optimizely (20)

PDF
Maximizing Optimizely Website Performance
PDF
Opticon 2017 Decisions at Scale
PPTX
How Optimizely Uses Optimizely - Opticon 2017
PDF
The Optimizely Experience Keynote by Matt Althauser - Optimizely Experience L...
PPTX
eBay Partner Network & Optimizely: Optimization Best Practices
PPTX
Confessions of an uber optimiser conversion summit - craig sullivan - v 1.9
PPTX
Optimizely Experience Keynote - Dan Siroker
PPTX
How We Do It: Proven Website Personalization Strategies
PDF
Opticon 2015- Powerful Integrations with Optimizely
PDF
Optimization Summer Games - Test Ideation
PDF
Optimization Summer Games - Test Ideation
PDF
eBay Partner Network & Optimizely: Optimization Best Practices
PDF
Better Decisions with Experimentation
PDF
A/B Testing for WordPress & Drupal
PPTX
Opticon 2017 How Optimizely Uses Optimizely
PDF
Optimizely for Travel
PPTX
Developer Night Opticon 2017
PPTX
Opticon18: Developer Night
PDF
Atlassian's Mystique CLI, Minimizing the Experiment Development Cycle
PPTX
Myths and Illusions of Cross Device Testing - Elite Camp June 2015
Maximizing Optimizely Website Performance
Opticon 2017 Decisions at Scale
How Optimizely Uses Optimizely - Opticon 2017
The Optimizely Experience Keynote by Matt Althauser - Optimizely Experience L...
eBay Partner Network & Optimizely: Optimization Best Practices
Confessions of an uber optimiser conversion summit - craig sullivan - v 1.9
Optimizely Experience Keynote - Dan Siroker
How We Do It: Proven Website Personalization Strategies
Opticon 2015- Powerful Integrations with Optimizely
Optimization Summer Games - Test Ideation
Optimization Summer Games - Test Ideation
eBay Partner Network & Optimizely: Optimization Best Practices
Better Decisions with Experimentation
A/B Testing for WordPress & Drupal
Opticon 2017 How Optimizely Uses Optimizely
Optimizely for Travel
Developer Night Opticon 2017
Opticon18: Developer Night
Atlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Myths and Illusions of Cross Device Testing - Elite Camp June 2015
Ad

More from Optimizely (20)

PDF
Clover Rings Up Digital Growth to Drive Experimentation
PPTX
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
PPTX
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
PDF
The Future of Optimizely for Technical Teams
PPTX
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
PDF
Building an Experiment Pipeline for GitHub’s New Free Team Offering
PPTX
AMC Networks Experiments Faster on the Server Side
PDF
Evolving Experimentation from CRO to Product Development
PDF
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
PPTX
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
PPTX
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
PPTX
Experimentation through Clients' Eyes
PPTX
Shipping to Learn and Accelerate Growth with GitHub
PPTX
Test Everything: TrustRadius Delivers Customer Value with Experimentation
PDF
Optimizely Agent: Scaling Resilient Feature Delivery
PDF
The Future of Software Development
PPTX
Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...
PDF
Using Empathy to Build Custom Solutions at Scale
PPTX
How to find data insights that will drive a 10X impact
PPTX
Targeted Rollouts: How to Release Features to Multiple Audiences
Clover Rings Up Digital Growth to Drive Experimentation
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
The Future of Optimizely for Technical Teams
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Building an Experiment Pipeline for GitHub’s New Free Team Offering
AMC Networks Experiments Faster on the Server Side
Evolving Experimentation from CRO to Product Development
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Experimentation through Clients' Eyes
Shipping to Learn and Accelerate Growth with GitHub
Test Everything: TrustRadius Delivers Customer Value with Experimentation
Optimizely Agent: Scaling Resilient Feature Delivery
The Future of Software Development
Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...
Using Empathy to Build Custom Solutions at Scale
How to find data insights that will drive a 10X impact
Targeted Rollouts: How to Release Features to Multiple Audiences

Recently uploaded (20)

PDF
Approach and Philosophy of On baking technology
PDF
NewMind AI Weekly Chronicles - August'25-Week II
PDF
Network Security Unit 5.pdf for BCA BBA.
PPTX
Spectroscopy.pptx food analysis technology
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
PDF
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
PDF
Encapsulation_ Review paper, used for researhc scholars
PDF
Empathic Computing: Creating Shared Understanding
PDF
Accuracy of neural networks in brain wave diagnosis of schizophrenia
PPTX
Machine Learning_overview_presentation.pptx
PDF
Univ-Connecticut-ChatGPT-Presentaion.pdf
PDF
MIND Revenue Release Quarter 2 2025 Press Release
PDF
Heart disease approach using modified random forest and particle swarm optimi...
PDF
Getting Started with Data Integration: FME Form 101
PDF
Mushroom cultivation and it's methods.pdf
PPTX
OMC Textile Division Presentation 2021.pptx
PDF
A comparative study of natural language inference in Swahili using monolingua...
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
PDF
gpt5_lecture_notes_comprehensive_20250812015547.pdf
Approach and Philosophy of On baking technology
NewMind AI Weekly Chronicles - August'25-Week II
Network Security Unit 5.pdf for BCA BBA.
Spectroscopy.pptx food analysis technology
Mobile App Security Testing_ A Comprehensive Guide.pdf
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Advanced methodologies resolving dimensionality complications for autism neur...
Encapsulation_ Review paper, used for researhc scholars
Empathic Computing: Creating Shared Understanding
Accuracy of neural networks in brain wave diagnosis of schizophrenia
Machine Learning_overview_presentation.pptx
Univ-Connecticut-ChatGPT-Presentaion.pdf
MIND Revenue Release Quarter 2 2025 Press Release
Heart disease approach using modified random forest and particle swarm optimi...
Getting Started with Data Integration: FME Form 101
Mushroom cultivation and it's methods.pdf
OMC Textile Division Presentation 2021.pptx
A comparative study of natural language inference in Swahili using monolingua...
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
gpt5_lecture_notes_comprehensive_20250812015547.pdf

Modernize 2018: Advanced Experimentation Workshop - Web Marketing ROI & Optimizely