Actionable Analytics: Using Better Data Better
This document discusses using analytics and data to improve game performance and retention. It emphasizes that vanity metrics are not actionable and that deeper analysis of user behavior and testing hypotheses through experiments is needed. Daily retention cohorts only show symptoms, not causes of churn. Good analysis requires exploring where users are pooling and dropping out, and what differs between progressing and non-progressing users. The document provides an example of hypothesizing that early achievement earning reduces churn, and testing that hypothesis through targeted experiments without changing game code. Measuring experiment results can indicate if changes reduced churn. It recommends tools for direct SQL access to rich event data and easy experimentation on mobile.
Actionable Analytics: Using Better Data Better
- 1. WHAT YOU DON’T KNOW CAN KILL YOU ACTIONABLE ANALYTICS – USING BETTER DATA BETTER Reach Engage Earn JAMES GWERTZMAN CEO, PLAYFAB DAMON DANIELI CEO, APPURI
- 2. About Us • Former Founder/CTO, Z2 (acquired by King.com in 2015) • Core Designer of Xbox Live (Gamertag ‘d’) • Filed over 20 patents • Masters in Computer Science, University of Washington • BSEE, University of Washington James Gwertzman CEO, PlayFab • Former Founder, Sprout Games (acquired by PopCap in 2005) • Launched and led PopCap’s 1st F2P studio in China • At Code.org, led Hour of Code campaign • Named on several patents for server technology developed while at Microsoft • A.B. CS, Harvard Damon Danieli CEO, Appuri
- 3. Vanity metrics are not actionable Game Events Metrics ?
- 4. Good Acquisition Stagnant Cohort Performance Poor Retention Daily retention cohorts only show the symptoms Now what?
- 5. Using metrics more effectively is about learning Source: Lean Startup Framework Idea Try Data
- 6. 21,340 46,953 982 1,273 484 1,162 4,881 5,495 3,846 3,728 3,651 14,200 -44,257 -91,368 -12,092 -11,355 -3,308 -7,042 -19,343 -9,996 -2,423 -891 -543 -852 Acquired Level 1 Level 2 Level 3 Level 4 Level 5 Level 6+ Level 10+ Level 20+ Level 30+ Level 40+ Level 50+ Population Churned “Ideas” takes work – explore the data Where are users pooling and dropping out? What is different between users who progress and those who don’t? Idea Try Data
- 7. 21,340 46,953 982 1,273 484 1,162 4,881 5,495 3,846 3,728 3,651 14,200 -44,257 -91,368 -12,092 -11,355 -3,308 -7,042 -19,343 -9,996 -2,423 -891 -543 -852 Acquired Level 1 Level 2 Level 3 Level 4 Level 5 Level 6+ Level 10+ Level 20+ Level 30+ Level 40+ Level 50+ Population Churned Come up with a hypothesis Idea Try Data “Users who progress are earning achievements early”
- 11. Test your idea with experiments Idea Try Data Targeted push notification campaign
- 12. Test your idea with experiments Idea Try Data Changing game behavior in real time
- 13. Test your idea with experiments Idea Try Data A/B test to try multiple tests at the same time
- 14. 21,340 46,953 1,212 1,354 578 1,239 5,388 6,323 4,231 3,923 4,123 15,300 -44,257 -76,368 -12,092 -10,447 -3,043 -6,479 -17,796 -9,196 -2,229 -820 -500 -784 Acquired Level 1 Level 2 Level 3 Level 4 Level 5 Level 6+ Level 10+ Level 20+ Level 30+ Level 40+ Level 50+ Population Churned Measure the results of your change Idea Try Data Did the change reduce the churn?
- 15. Requirements • Deeper access to your data than many analytics tools allow – Consider Amazon Redshift – provides direct SQL access • Rich set of events – you can’t measure what you don’t capture • Tools to easily run experiments w/o writing new code – Especially on mobile… don’t resubmit through Apple!
- 16. Questions? Tweet @playfabnetwork Learn more: https://playfab.com http://appuri.com Follow us: @playfabnetwork, @appuricorp https://playfab.com/blog/