Can one lens see bi and ux developconf

Editor's Notes

• #2: Hiya, welcome. My name’s Heather. I work on analytics, as an advisor and consultant. What I’m going to be talking about today is THIS.
• #3: That is to say...
• #4: What’s the relationship between business intelligence type measures, and ways of understanding user experience, in games analytics?I’m talking about this because I’m trying to work out the answer, and there’s nothing like having to explain something to make you try to get it straight in your head. So if you have ideas I’d like to hear them. There will be some time for questions, drop me a line, shout out on twitter. I can’t reply in real time as I’ll be up here. I’ll be putting the slides up on slideshare, and I’d also encourage you to pick up one of my new biz cards. There’s a choice of colours but I’m not running an experiment. Honest.Just so I know a bit about you - how many people here have MBA’s? Just need to know in case I diss anyone. And how many people are reasonably experienced with game analytics? You can help me answer questions.
• #5: To set the scene a bit – here’s the business environment we’re in. This is from one of the tool vendors in the games analytics space, who has a good view of what’s going on.What this says, is that genres that are on the up and up have shifted in the past 6 months. TV show trackers are up, casino is way way up. The momentum’s been stolen from RPG, Trivia, and Casual.So - what? So - the market is fickle – and you need to make the most of the window you’ve got. That means being ready willing and able to change your design based on what happens when people use it.
• #6: I’m going to cut to the chase – here’s where we’re at. Launch and learn. There’s two different melody lines going on.On the one hand this is a key belief of a cult whose HQ is somewhere between San Francisco and San Jose. And back on Planet Earth, on the other hand, there is also this ‘oh shit’ vibe. We’re changing it because it didn’t work.So there’s a bit of dissonance. But its also a huge, huge opportunity to get out in front of it and surf it. Change is part of the design.
• #7: So here’s the start of a syllogism. Design is change, and analytics powers change. Therefore....Analytics is part of the design. It’s not the only part. By a huge stretch. But it’s a smart part.
• #8: Here’s a little bit of level-setting about analytics. Hang onto your hats, I’ll be quick.
• #9: You see this?This is such a biggie. Repeat after me.
• #10: If you don’t believe me believe him.It’s a myth that doing analytics will turn you into Zynga. For better or worse.And putting stats in isn’t the same as doing analytics.But he’s not wrong.
• #11: So – change is part of your design. And that’s ok. In fact, it’s good. And Analytics powers change.Where this leads to, is that analytics is part of the design.Where people get their knickers in a twist about this is they think that it’s weird and evil for analytics to be THE design process.And that’s a good point.But for sure it’s a design process.
• #12: Analytics isn’t complicated. It’s really simple. Here’s what you do. You can use big JCBs or whatnot to do it, but this is the basic process.This is simple but useless.
• #13: Here’s where it gets interesting.
• #14: There is some change that’s going to be of the ‘uh oh’ variety. This doesn’t work. That doesn’t work. Something’s on fire. This is really useful.
• #15: There’s also planned analytics-powered design change. Here, you start out with some questions, AND an idea of what impact the answers will have on how you will change your design. There are different flavours. This can be done in conjunction with multivariate testing in a very straightforward way – where you simply pick the winner. Red or blue, blue’s better, ok we’ll have the blue one, next please. But you can also use investigation – observational and experimental - to ask questions that have a more subtle relationship to design evolution. And you can use testing and observational analysis to titrate quantifiable elements like difficulty, or price, or level of competition – changing it progressively, as the result of testing.
• #16: Ok, enough with the context, and on to the question at hand. Let’s talk about BI. Business intelligence. Who here is a beancounter? If you’re a beancounter, there’s only one thing that’s interesting. Beans.
• #17: Also known as Lifetime Customer Value. There’s a guy called Andrew Chen who’s written on this extensively. Check him out. This is where the chickens come home to roost. What this means is value OF the customer. Not value TO the customer. There is some relationship - “it’s complicated!”.
• #18: So LTCV is the good stuff. How do you get it?Here’s the secret of the secret sauce. There isn’t a recipe that suits everyone. Here are some KPIs (key performance indicators) that are – quite rightly! – held up as holy grails to chase.But the takeaway point is that the relationship between these intervening variables and LTCV is not the same for every product. Cost and monetisation are pretty direct – but they are also driven by other factors that have more complicated inter-relationships.
• #19: Let’s drill down. Virality – user-to-user contagiousness is clearly good for lowering acquisition costs. But they can also be good for engagement, and whatever it is that flows from that. For some games, players invited via friends are much more likely to stick. So the inviter’s value to the game can be greater than the sum of their own purchases. And here’s another pattern to look for – some people are super-inviters. They have network value. There’s also an interesting point here about
• #20: The point here is that ‘normal’ patterns of play are different for different contexts. How they relate to paying is something that you have to work to findout. Whatever you do don’t use DAU/MAU. I’ve got a long rant on that you can find on google. Games Analytics (a predictive analytics service provider) and others have found that regardless of what the ‘normal’ pattern is, momentum – the rate of change of can be a good predictor of churn. And churn means no more retention, and no more monetisation, bye bye acquisition cost. I would also be interested in looking at the 2nd derivative. It’s important to get straight about what you think the causal relationship is. A slowdown may be indicative of churn, but somehow making a person play more frequently it might not stop them from churning.
• #21: Here’s a typical pattern of how many people pay how much in a typical freemium game. What isn’t clear is why people end up in one category or another. Differences in game experience, or player type (however you define this) are worth looking at, though you might not get an answer.
• #22: Here’s an interesting breakdown from Sonamine – it shows that the time it takes people to monetise for the first time has a long tail. Even 20 days in they might still convert. You don’t necessarily know what someone’s lifetime value is for quite some time.
• #23: Ok so much for BI. The point I have been trying to make is that the relationships between these various high level BI measures aren’t set in stone – they are there to be discovered. This has all been pretty straightforward I hope but also pretty abstract so I thought I’d give you all a break and show you a picture of something.Switching gears from BI to user experience, more particularly the visible bits of experience that we see – user behaviour. And while you can get a lot of info from videoing sessions and running focus groups, etc, all of which I’ve done, I’m going to be focussing on what you get from interaction records. Which are composed of clicks and other stuff like that.
• #24: My point is this – that you have to boil your clicks to distill them into something useful. There is very seldom the nuclear button click – the one that changes everything. You have to look carefully to discover the effective context in which an action occurs.
• #25: The main point here is that part of the context is what happens before an action or event, and part happens afterwards. What this ‘stuff’ is isn’t necessarily best looked at as a specific action, but as an example of a more abstract type. As does the event itself. It needs to be boiled. There are as many ways to do this as there are ideas about game play, and psychology. Which is for sure the subject of another talk. But that shouldn’t hold you back. It’s better to do something than nothing. One thing that’s quite simple to look at, that doesn’t require much boiling, is looking at attribution analysis and the nature of someone’s engagement. This can help align the messaging and delivery.
• #26: There’s another dimension of context - which is what is happening concurrently. Again it’s open season what you grab onto here. And again you may well want to use derived measures that make some kind of abstraction of the situation. Being simple and obvious is a good idea.
• #27: Let’s say you murdered Professor Plum in the library with a samurai sword at midnight. Didn’t you?This can also be seen more abstractly. Hostile action.
• #28: What you grab in the way of concurrent context can include factors from the past. To leap ahead to the tools topic, this where some logging and reporting services can run into difficulty. They’re a great way to get started. If you find yourself shipping over the kitchen sink to describe every event, it might be time to consider bringing it inhouse.
• #29: So, to get back to my original question – can one lens see BI and UX. Yes. I think the answer is that it’s analytics which reveals those relationships – it’s what helps you pull focus, relating one level of description to another.
• #30: Now we’ve built up a picture of what we’re doing – let’s look at how it relates to tools and processes.
• #31: Here’s the simplest picture I’ve come up with about tools. There’s really 3 bits of the market that typically don’t connect up very well. You’ve got specialist cloudbased services for app tracking, which might have a platform focus, and might have some game-specific vocab built in. You’ve got a variety of BI front ends. You’ve got some hard core stats tools – from traditional stats packages like SPSS and SAS, to specialised machine learning algorithm implementations. These assume you’ve got your data all lined up – somewhere - and standing in an orderly array ready to be consumed. Which it might not be, so you will need to do some work to spoon feed it properly.
• #32: Here’s a pop quiz. How many think 12 is bigger than 16?
• #33: This is the right answer.
• #34: This is why you need stats. In the first case, a group with a mean of 12 is a different group from the group with the mean of 16. In the 2nd case, the right answer is ‘dunno’.This is the perspective that’s lacking from some BI approaches, and some app trackers. You get an answer but you don’t know how hard to lean on it.
• #35: On the app tracking side, you’ve got xxx, and MixPanel, enabling more exploratory analysis.Kontagent moving into data mining, and professional services.For BI front ends, Tableau is the current darling, on the strength of its approach to visual exploratory analysis.On the hard core side, there’s a specialist strand of infrastructure to do with graph db. And you’ve got traditional stats tools moving more into machine learning style predictive analytics e.g. IBM Clementine, and more cloud-based services, from SAS but not yet from IBM. There is also an uptick in activity for service-based approaches specific to predictive analytics in games, Playnomics, Games Analytics, Sonamine. Typically what you get here is access to expertise in model building as well as a bit of secret sauce.
• #36: When it comes to predictive analytics, there’s a strong focus on identifying cues to segments which predict future behaviour.A lot of this is transcribed tune from another key, in telco CRM.
• #37: Sometimes when you do a segmentation it gives you some pretty strong clues about what action to take. And here, with Games Analytics, they have a focus on a specific type of intervention, in-game messaging. The groupings here are fairly interpretable.
• #38: But sometimes when you use heavy artillery, you get useful results that are meaningless. As in: the system can predict really well that someone’s about to churn. But you don’t know why.
• #39: Then it’s difficult to understand the right change.
• #40: Do tools matter? I mean you could do it all with a lego turing machine, couldn’t you?
• #41: Well, no. Try doing multiplication in roman numerals. Or something really modular in Excel.Don’t blame your tools. Use them, get more if you need to. For predictives you want to leverage stuff that’s out there.
• #42: So now I want to tie all this in with processes. Specifically, the role of A/B testing in design change.
• #43: Which is better?The top one’s Google, the bottom one’s Bing. Google tested 42 different shades of blue to get the one that performed the best. The designer ran screaming out of the room and went to work for Twitter. If you want someone to blame for the mania for A/B testing you could try them. Or Amazon. Or Facebook. Last month they were testing an option where you could pay to promote your stories to your friends, like Badoo.Daring. Decisive. Conviction. SXSWDoug Bowman on Design at GoogleWhen a company is filled with engineers, it turns to engineering tosolve problems. Reduce each decision to a simple logic problem.Remove all subjectivity and just look at the data. Data in your favor?Ok, launch it. Data shows negative effects? Back to the drawing board.And that data eventually becomes a crutch for every decision,paralyzing the company and preventing it from making any daringdesign decisions.every 100 ms increase in load time of Amazon.com decreased sales by 1% (Kohavi and Longbotham 2007) invented the amazon.com recommendation engineGoogle VP Marissa Mayer asked a group of Google searchers how many search results they wanted to see. Users asked for more, more than the ten results Google normally shows. More is more, they said.So, Marissa ran an experiment where Google increased the number of search results to thirty. Traffic and revenue from Google searchers in the experimental group dropped by 20%.Ouch. Why? Why, when users had asked for this, did they seem to hate it?After a bit of looking, Marissa explained that they found an uncontrolled variable. The page with 10 results took .4 seconds to generate. The page with 30 results took .9 seconds.Half a second delay caused a 20% drop in traffic.
• #44: A/B testing is a demanding discipline. And it sometimes does make people run screaming from the room.
• #45: What you see a lot are people playing with quant variations. Prices. Offers. More, or less, rather than different. It’s easier to get your head round.
• #46: So you optimise, but that optimisation doesn’t lead you to any further insights. Why not 2? Why not 4? Totally a black box.Useful, but not magical. Can the answer suggest more questions? No.
• #47: And again what you get here is that the answers aren’t always what you’d expect. So they’ve hired an economist. Possibly a better choice than a banker – we’ll have to see.
• #48: I’ve been noticing that the casino sites often feature breasts. The interesting question here is not ‘is breast A better than breast B’, and optimising along some continuum via some hill climbing algorithm.. You need to link the independent variable to a hypothesis – about, say, whether being exploited into a state of mild excitement makes players more likely to take risks. That leads to more fruitful ideas about possible design interventions, in addition to plastic surgery. So the test isn’t just about A cup vs a B cup.
• #49: The test I showed in the last slide isn’t a real one. It’s two different views of the same thing. A time series.These are a suggested parameters for of body physics add-on in 2nd Life. And the controls that control it are in the hands of the user, for something they have created. There’s a fundamental difference in attitude, between designs which are extractive, and designs which are inclusive, and engage people in adding the value of the experience. I owe this distinction to Niall Ferguson in the recent Reith lecture. Although there are probably casino’s on 2nd life.
• #50: What I’m thinking about, and arguing for, is to look beyond data-driven design, and think about it as discovery driven. Or as design-driven analytics.Who likes the first, and who likes the 2nd best?
• #51: You can’t drive just by looking at the rear view mirror.
• #52: You need to look ahead, and plan your investigations and analysis to feed into your design change. Not just wind tunnel, but a guidance system.
• #53: Bon voyage!