Data Gone Wrong - GDCNext 2013
- 1. 1
- 2. It’s the cornerstone of many of the biggest businesses in the US, including Google & Amazon, and the backbone of most scientific undertakings. 2
- 3. There’s plenty of cheerleading for data so I want to spend today on cautionary tales & advice, in the hope of helping keep data more on the Iron Man side of the Robert Downey Jr spectrum. 3
- 4. I love using numbers & testing to understand the world, I’m not a data hater by any means. If you want to know about medieval Transylvania or the Ottoman invasion of Hungary, I’m you’re woman. That didn’t get me far on the job market. Partly to do something different, partly to do games. But despite that intention it hasn’t been as different as I expected – user acquisition for games and catalogs is fundamentally the same. But Kongregate, because we’re an open platform with over 70k games and now a mobile game publisher, has provided some incredibly rich opportunities for data mining. 4
- 5. Part of the reason I’m telling you this is to make my first point: And for an organization to do data right you can’t toss analysis back and forth over a wall to quants. It takes intimate knowledge of a game (and the development) to do good analysis and multiple perspectives and theories are good. 5
- 6. Sometimes it’s immediately obvious – we had a mobile game we launched recently, an endless runner, that wasn’t filtering purchases from jailbroken phones and was showing an ARPPU of $500, not very plausible and easily caught. But most issues are much more subtle – tracking pixels not firing correctly for a particular game on a particular browser, tutorial steps being completed twice by some players but not by others, clients reporting strange timestamps, etc. For this reason you should never rely on any analytic system where you can’t go in and inspect individual records. If you can’t check the detail you’ll never be able to find and fix problems. We use Google Analytics for reference and corroboration but nothing very crucial, and are using it less and less because of this. 6
- 7. This looks like 4 separate pictures photoshopped together to create an appealing color grid, right? 7
- 8. Wrong. So much of data is like these pictures – a set-up that appears straightforwardly to be one thing from one angle, turns out to be completely different from another. 8
- 9. Except of course you know I’m setting you up 9
- 10. I mentioned lifetime conversion and showed daily ARPPU. Lifetime conversion may be similar between the two games, but daily conversion is 40% higher for game 1. This is why $/DAU is not a very interesting stat on its own. If someone quotes just D1 retention and $/DAU that’s not enough information to judge how a game monetizes. 10
- 11. It’s a living, changing system. Flat views are not enough. 11
- 12. So here are a series of likely traps analysts can fall into. I know I have. They’re not in a particular order of importance because they’re all important. We tend to think of playerbases as monolithic but really they are aggregations of all sorts of subgroups. It’s sort of like watching a meal go through a snake. Though with time cohorts it’s easy to lose track of events and changes in the game, so you can’t rely on those, either. 12
- 13. 13
- 14. 14
- 15. You may have noticed that win rates got a bit wacky towards the later missions of the graph of the last chart – this is a sample size issue. Even games that overall have very substantial playerbases like Tyrant may end up with small sample sizes when you’re looking at uncommon behavior in subgroups. Early test market data is often tantalizing & fascinating, but it’s often the most unreliable because you’re combining small sample sizes and a non-representative subgroup – the people who discover you first are the most hard-core. 15
- 16. not normal (bell-curve) distributions, which affects everything. Theoretically it’s not even possible to calculate the average value of a power distribution since the infrequent top values could be infinite. The sample size depends on the frequency of the event – tutorial completion & D1 retention should be fine with just a few hundred users, % buyer with 500+, but I don’t like to look at ARPPU with much less than 5,000. These are just my rules of thumb based on experience and probably have no mathematical basis. 16
- 17. 17
- 18. 18
- 19. If you ask small questions you’ll usually get small answers. And the dirty secret of testing is that most test are inconclusive anyway. It’s hard to move important metrics. So prioritize tests that significanly affect the game, like energy limits and regeneration over button colors. 19
- 20. Your existing players are used to things working a certain way – a change in UI that makes things clearer for a new player may annoy/confuse an elder player. Where possible I like to test disruptive changes on new players only, and then roll out the test to other players if the test proves successful. A pricing change that increases non-buyer conversion might reduce repeat buyer revenue. For example if you’re A/B testing your store, don’t assign people to the test unless they interact with the store. It’s often easier to split people as they arrive in your game, or some other thing, but a) there’s a chance you would end up with non-equal distribution of interaction with the tested feature and b)any signal from the test group would get lost in the noise of a larger sample. 20
- 21. Early results tend to be both volatile and fascinating – differences are exaggerated or totally change direction. People tend to remember the early, interesting results rather than the actual results. People also often want to end the test early if they see a big swing, which is a bad idea. We tested to see what gain we were getting from bonusing buying larger currency packages, which had to be judged on total revenue to make sure we were capturing both transaction size and repeat purchase factors. To make sure the 15% lift was real we broke buyers into cohorts by how much they’d spent ($0-$10, $100-$200, $200-$500, etc) and checked the distribution in each test. On the bonus side of the test we saw fewer buyer <$20 and 30%+ gains on all the cohorts above $100+, so we were confident that the gain was not being driven but a few big spenders. Again this should be worked backward from the frequency and distribution of the metric you’re judging the test on. There’s internet calculators to help you figure out what you need to get to statistical significance given an expected lift. My advice (if you have the playerbase and patience) is to then double or triple that. Why do I want my sample sizes so much bigger than the minimum? 21
- 22. It comes down to some of the issues with judging results by statistical significance itself. It doesn’t mean what you probably think it means. Statistical significance tests assume that there is some true difference in lift, and that if you test there will be a bell curve distribution of results, with the true lift as the average. Your 5% result could be right on the mean, or it could be an outlier on either end. If it’s statistically significant then the chance is low (usually 5% or less) that there’s no lift at all. But the true lift could be 1% or 10%. It’s possible you’d get two outlier results in the same direction, but becomes less and less likely, and more likely that your test results represent the true mean. And the size of the effect you are testing does matter as it helps you understand the relative importance of different factors, and what to prioritize testing next. 22
- 23. For example we’ve had a lot of tests that increased registration and reduced retention, so much so that we now judge tests based on % retained registrations because that’s what we really care about, but that’s not always possible. 23
- 24. A good example of this is adding a Facebook login button on our website. If a player comes back on a different browser they need to be able to login. 24
- 25. 25
- 26. This is about how you think about your business. 26