Startup analytics
1 like440 views
This document provides guidance on setting up analytics for a startup to understand user behavior and business metrics. It recommends beginning with high-level business metrics like growth and churn rates before analyzing lower-level usage data. Google Analytics is suggested to start tracking events and user behavior. Funnel analysis and A/B testing can help identify where users drop off and test improvements. Cohort analysis of retention rates over time helps evaluate user engagement and churn. The goal is to use analytics to answer questions that influence key business metrics.
Startup analytics
- 1. STARTUP ANALYTICS Getting Started Down the Path to Understanding Your Business and Your Users Dale Beermann Chief Technology and Analytics Officer dale@studyblue.com
- 2. THE GOAL OF ANALYTICS: IMPROVING YOUR BUSINESS BY ANSWERING AND ACTING ON QUESTIONS With every question answered, ask yourself if it is the desired result. If not, determine what needs to be done to improve it.
- 3. BUSINESS METRICS VERSUS USAGE METRICS You should always be reporting on your business metrics. Analytics is the way to understand what is driving them. Effectively, business metrics are the aggregate result of your usage metrics.
- 4. BUSINESS METRICS The ultimate goal of business metrics is to evaluate the health of your business. Examples: How fast is your business growing? What is your churn rate? What is your cost per acquisition for each channel? What is your Average Revenue per Active User?
- 5. USAGE METRICS The ultimate goal of usage metrics is to evaluate the health of your product. Examples: What percentage of your users are realizing your value propositions? Is your new feature reaching the expected audience? What percentage of users make it through the onboarding process? What percentage of users are using social channels?
- 6. INFLUENCING BUSINESS METRICS Know the answers to your high-level business metrics before digging into your usage. Use your usage metrics to determine how you can influence your business metrics.
- 8. THE RIGHT TIME TO START Have you found your product/market fit? There may be some high level business metrics that help you get there, but don’t start your analysis on a product that is going through a massive amount of change.
- 9. WHERE TO START Have you filled out a Business Model Canvas? What are your business’ most important metrics? How well are each of your customer segments doing when it comes to realizing your value propositions? Take your value propositions and work backwards through the paths that your users take to get there.
- 10. DEVELOP GOOD HABITS Make Analytics a core part of your development workflow. Ensure you are creating both good behavioral habits as well as good programming habits.
- 11. GOOD BEHAVIORAL HABITS Review your metrics on a regular basis Continually log changes that are going into your product There will inevitably be a point in the future where you ask yourself what happened six months ago to influence a particular metric.
- 12. GOOD PROGRAMMING HABITS Create guidelines and tools that require you to implement metrics as you build out your software E.g. Use abstract click handlers that can be easily refactored: display.addClassesHandler(new SBClickHandler(SBAnalytic.HOME_FIND_CLICK) { @Override public void doOnClick(ClickEvent event) { ... } });
- 13. AVOID VANITY METRICS Page views don't matter (impressions may). Time On Site can be interesting, but doesn't necessarily convey usage. It’s very difficult to influence metrics like Page Views or Time on Site. Attempting to do so will be a waste of your time.
- 14. FOCUS ON ACTIONABLE METRICS These are going to be different for every business. Again, you want to find the metrics that mean the most to your company and determine how you can influence them.
- 15. MAXIMIZING A METRIC CAN HAVE SIDE EFFECTS Providing multiple options splits your usage between them. Similarly, forcing users down one particular path means they can’t take another. This can arise in subtle ways. In some cases, such as with a payment page, you may be able to find the optimal solution without many side affects. Ask yourself: What user segments are affected by this change? Will any side effects be worth it?
- 16. IMPLEMENTATION
- 17. CAVEAT: I DO NOT SUBSCRIBE TO THE IDEA THAT YOU SHOULD LIMIT WHAT YOU TRACK. If you are smart about how you’re doing your analysis, you will not fall into the trap of “analysis paralysis.”
- 18. START WITH GOOGLE ANALYTICS It’s free and you can throw everything at it without worrying about usage tiers. We don’t use the high level (vanity) metrics for much. Rather, by sending our events through Google Analytics, we have the ability to answer a lot of questions.
- 19. GETTING THE MOST OUT OF GOOGLE ANALYTICS Track all of your events (views, clicks, actions). This isn’t limited to your click stream. Track final events for workflows (e.g. completed_onboarding). This allows you to create Advanced Segments for those events. Set up profiles for each platform (web, iOS, Android, etc.). You’re going to have very different usage patterns for each platform, and they should be analyzed separately.
- 20. GETTING THE MOST OUT OF GOOGLE ANALYTICS Make use of custom variables. At the very least, you should be setting your (non personally-identifiable) user ID as one of the variables. This will let you find some per-user data that is otherwise difficult with Google Analytics. If you have organizational data, or if your users are segmented in pre-defined ways, this can help look at those segments more closely.
- 21. I’M TRACKING MY EVENTS. NOW WHAT? Funnel Analysis The goal of a funnel analysis is to determine where your users are falling off. Take one of your core metrics and walk through the steps it takes to get there.
- 22. FUNNEL ANALYSIS EXAMPLE StudyBlue and Indexable Content We want to maximize the amount of content created that is “paired” with a class. How does that happen?
- 23. HOW DO YOU IMPROVE YOUR FUNNELS? Think about how can you change an experience to improve the end result. Sometimes this is as simple as changing a button’s color or using a modal popup (while thinking about the side effects). A/B Testing A/B Testing can be a reliable way to evaluate multiple paths. Caveat: Do your homework and understand statistical significance. Learn what a chi-squared test is.
- 24. TOOLS FOR FUNNEL ANALYSIS Google Analytics does make it possible to do some of this. Their goal conversions are annoying if you don’t use page views the way they expect. Create advanced segments for users with particular events. Other good for-pay tools are KissMetrics and Mixpanel. Roll your own. In all honesty, doing this stuff yourself isn’t that hard.
- 25. TANGENT: YOUR OWN IMPLEMENTATION You’ll want to use partitioned tables under the hood (if your data store supports it). In postgresql, we use triggers to write data to the correct table. Queries then only hit the necessary tables for the time span you’ve defined. We got away with a table per week for about 5 years. Our table schema: user_id, session_id, platform, activity_id, activity_timestamp, activity_detail
- 27. COHORT ANALYSIS A cohort is a set of users grouped in a particular fashion. Typical cohorts are time-based (week of registration). Cohorts can also be based on acquisition campaigns (e.g. Adwords vs. Direct vs. SEO). The purpose of a cohort analysis is to understand user retention and if your changes are making an impact between cohorts.
- 28. WHY COHORT ANALYSIS Most educated investors are going to ask for cohort analyses. Cohort analyses, and their corresponding retention rates help determine: Engagement levels. Are you a one-and-done sort of site? Churn rates. If users aren’t coming back to your site, or if churn is higher than acquisition, your site will not grow. Quantifying the value of your existing userbase.
- 29. A COHORT ANALYSIS EXAMPLE Sadly, I can’t provide some of our own data here. But this is what your cohort analyses will look like:
- 30. COHORT ANALYSIS QUERIES In Posgtresql: crosstab. In MySQL: Pivot Tables (still pretty manual). In everything else: pull your data into one of the above. Or write a lot of code.
- 31. THE HOLY GRAIL A full fledged Customer Relationship Management system driven from your analytics solutions: Adaptive in-app user education Drip email campaigns Churn prediction Re-engagement
- 32. QUESTIONS?