Measure Right the First Time - Infusionsoft Partnercon

Editor's Notes

• #5: Jonathan Analytics platform of choice is irrelevant, as long as it has the major features of something like Google analytics. We ’ ll be acting as if you use Google Analytics exclusively. It ’ s our preferred application. Optimizely is our favorite A/B testing tool - Adobe ’ s offering is great but complicated and expensive There are many other tools Call tracking is crucial if there ’ s a phone number on the website. We roll our own using Twimlbin & Twilio Just get the data into Analytics. You must get in the habit of tagging links so they track in analytics. Google has a utility that does this which we ’ ll review later If you ’ re doing online advertising, configure those tools to properly push into Analytics. Cost and all Campaign data can be automatically configured to push into analytics, but you ’ ll need to properly tag your AdCenter and Facebook advertising URLs.
• #6: Jonathan It ’ s crucial that whether you ’ re working on your own or a client ’ s site, you understand the revenue model in play. CPA is minimizing the cost of customer acquisition on a per customer basis. ROI is maximizing the return on every advertising dollar spent. Margin is maximizing advertising dollars spent as taken from revenue to the point of diminishing marginal returns - it ’ s minimizing ad spend as a percent of revenue while increasing revenue.
• #7: Jonathan
• #8: Jonathan Advertising section, AdWords, Campaigns, and click on the “ Clicks ” subsection This is only useful for AdWords. Apply the formula mentioned previously to all other traffic sources
• #9: Jonathan
• #10: Jonathan
• #12: Jonathan
• #13: Jonathan Goals are everything Clients or previous consultants have often setup goals that are totally irrelevant. You need to track ecommerce as accurately as possible. Consult external system for accuracy If advertising, you MUST completely understand their costs. Simply asking what the target margin is is unacceptable, and often times inaccurate. When setting up goals, funnels are often put in place to show drop off in the steps. Be careful - if someone can complete your goal or purchase without completing the funnel steps, your goal will not fire correctly, and your metrics will be off.
• #14: Jonathan Everything belongs to a campaign - SO TRACK IT Source and medium are self explanatory Name/Content can be many things - keyword, twitter campaign, hashtag, email campaign, etc. Don ’ t just fill out the url builder - keep it CONSISTENT
• #15: Jonathan
• #16: Jonathan To track video plays, bind an event to the "Play" action of the player. You can do this with almost every embedable player, our favorite is YouTube.
• #17: Jonathan Track social events with _trackSocial() method Treats social interactions as a special event in Google Analytics Attributed to engagement and conversion
• #18: Jonathan Knowing how your form converts is crucial - we go down to individual fields Track blur on each field and hit an event if it ’ s empty or full Determine which form fields cause problems
• #19: Jonathan
• #20: Jonathan
• #23: Scott This is the simplest way to experiment - one factor at a time Make sure your Y-value (what you ’ re measuring) is aligned with what ’ s critical to the client In this case, CTR of a button
• #24: Scott This is where you test multiple factors at one time in combination with one another This requires more data because you need to test every combination of factors More Factors = more unique tests in order to gain statistically significant data More Variations also = more unique tests “” If you ’ re short on time, decide what ’ s more important - the number of factors you ’ re testing, or number of variations of each factor
• #25: Scott
• #26: Scott: We recommend one factor for simple cases where you ’ re trying to influence one output Often times this is with a single drastic change We recommend full factorial for things like designing a new landing page, where you want to test copy, images, buttons, layout, and more all at the same time
• #27: Scott
• #28: Scott
• #30: Scott This is the data that ’ s available out of either a one factor or full factorial test Trying to get “ N ” value, which is the minimum number of test participants necessary to achieve statistical significance Sample mean is the arithmetic average number of your output of your new variant U is the arithmetic average of all of your traffic (it ’ s the Y value) E is the difference between those - is your test variant different from your typical traffic? Z is a measure of how confident you want to be in your results (90%, 95%, 98%, etc) Alpha is the distance from 100% Sigma is standard deviation - a measure of how much variation there is in your output
• #31: Scott This is a normal curve Important to determine how confident you want to be such that your change is actually significant As Za/2 creates a narrower range, you are less confident in your results
• #32: Scott To make this valuable, need to solve for n in terms of the other variables What ’ s important here isn ’ t the actual algebra, but how each factor drives the number of unique visitors necessary to achieve valid test results As Za/2 increases (this means you want to have greater confidence) it requires more test participants in order to meet that confidence requirement As sigma increases (there is more variation in your output) it requires more tests to prove that any change is more than just a statistical anomaly As E increases (there is a larger gap between the output of your test and the output of the rest of your traffic) it requires less participants since the difference between the test and the baseline is so great
• #33: Scott This is what this looks like in A/B testing software such as optimizely - it does most of the work for you The conversion rate of each variation is your x-bar, and the change to beat baseline is your confidence (which is correlated to Za/2) Usually 95% is acceptable to declare a winning variation.
• #35: Scott
• #36: Scott Remember to think of your digital marketing such that Y is a function of X ’ s, which are variables Some of these you can control, some you can ’ t These are some typical X ’ s that impact the typical outputs: CPA, Margin, ROI
• #37: Scott A typical digital marketing model is a funnel where a portion of impressions become clicks and a portion of those clicks become conversions Between impressions and clicks we measure CTR, and between clicks and conversions we measure CR Each of these Y-values are controlled by different levers - things you can change to influence the output
• #38: Scott Each traffic source has its own performance level in terms of the output Y The overall output of your marketing system is the weighted average of all your traffic sources You can increase or decrease marketing spend on each traffic source to influence the weighting and thus the overall output
• #39: Scott
• #40: Scott
• #41: Scott
• #42: Scott
• #43: Scott
• #44: Scott
• #45: Scott
• #46: Scott
• #47: Scott
• #48: Scott
• #49: Scott
• #51: Scott Not all levers are created equal Some have a much larger impact on output than others Some have zero impact on output Some only influence other levers It is important to know which levers have large and small impacts so you can budget time and effort accordingly
• #52: Scott It is important to always think about the entire system - moving one lever doesn ’ t only change one metric, often times it influences many or all metrics The further down in the funnel you go (i.e. the closer to conversion you are), the more drastic an effect the lever has on output In order to ensure that all changes to the funnel are trackable and accounted for, you need to design great experiments
• #53: Scott We ’ re going to run through three scenarios, each moving only one lever at a time We will then move all three at once to maximize for our outcome
• #54: Scott
• #55: Scott
• #56: Scott
• #57: Scott
• #59: Jonathan