When and how to use statistics in a UX world
Download as PPTX, PDF0 likes461 views
The document discusses the effective use of statistics in user experience (UX) design, emphasizing the importance of measuring and comparing metrics. It highlights challenges in statistical interpretation and provides examples, such as using Fermi equations to estimate potential sales of products based on user engagement data. The conclusion stresses the need for an action plan to translate statistical insights into effective UX strategies.
When and how to use statistics in a UX world
- 1. WHEN AND HOW TO USE STATISTICS IN A UX WORLD Niki Lin | niki.lin@internetarchitects.be | +32 471 01 34 39 A primer about good use of statistics
- 2. Des McHale THE AVERAGE HUMAN HAS ONE BREAST AND ONE TESTICLE
- 3. WHY ARE WE BAD WITH STATISTICS We are good at assessing probability, but odds are we’re a bit off 1.
- 4. WE ARE AFRAID OF SNAKES NOT OF CARS
- 5. 5 1. Why are we bad with statistics Current Situation A website generates statistics that cause ideas that lead to action
- 6. MEASURE AND COMPARE Compare to understand your measurement better 2.
- 7. 77 2. Measure and Compare Is this already being measured? › If you are measuring it, chances are, someone else already has • For UX on websites: SUS › Review published literature to see how other researchers measure • How are items phrased and compared • What scales are being used • How close do our methods match published findings
- 8. 88
- 9. 99
- 10. THE NEEDED PRECISIONDo we shoot for the The Moon or Mars? 3.
- 11. 11 3. Precision Continuous statistics • Google Analytics measures everything continuously • This results in a different approach to define a “test group” • When do we have enough data to decide on an A/B test?
- 12. 12 3. Precision Continuous statistics › Strive for stabilization
- 13. 13 3. Precision Estimate example • Idea: let’s set up a service to sell t-shirts printed with memes from 9gag. • How can we estimate the sales of this?
- 14. 14
- 15. 15 3. Precision Estimate: Fermi equation › Famously used by Drake to estimate the number of extraterrestrial life › Estimating things through the relation of entities
- 16. 16 3. Precision Fermi equation: 9gag T-shirt › How many users are on 9gag each month › What percentage of users requires a shirt that month › What percentage of shirts sold are shirts with memes › How long are users on the site each month › How many memes are exposed per time frame › What percentage of meme are liked enough to want a shirt
- 17. 17 3. Precision Fermi equation: 9gag T-shirt Shirts Sold = Market Size x Need x Niche-interest x Time x Exposure x Selection
- 18. 18 3. Precision Fermi equation: 9gag T-shirt 10 000 000 visitors/month (in the US) times 5/12 shirts needed/visitor/month Times 5/30 000 meme-shirts bought/ shirts needed (global) Times 297 view-minutes / user / month Times 11 memes-seen / view-minutes Times 1/300 memes-liked-to-buy / memes-seen Equals 7 562 average number of meme shirts bought per month by the users
- 19. ACTIONS Define actions before you even see the results 4.
- 20. 20 Extreme Results? › Reiterate: what part of estimating went wrong › Learn from the experience
- 21. 2121
- 22. 22 4. What to do with all those results? Back to the current Situation A website generates statistics that cause ideas that lead to action
- 23. 23 4. What to do with all those results? Deciding afterwards… Statistics that cause ideas that leads to action
- 24. 24 4. What to do with all those results? … results in an emotional decision Statistics that cause ideas that leads to action are often emotional
- 25. 25 4. What to do with all those results? So what’s wrong? A website generates statistics that cause ideas that lead to action
- 26. 26 4. What to do with all those results? Add a plan Plan an action plan about statistics that defines action plans B A
- 27. 27
- 28. CONCLUSIONS 5.
- 29. Niki Lin | niki.lin@internetarchitects.be | +32 471 01 34 39
Editor's Notes
- #3: Run the half-a bra analogy Happens every day
- #4: http://www.geneticliteracyproject.org/2015/04/06/humans-have-innate-ability-to-assess-probability-but-odds-are-were-a-bit-off/
- #5: Psychology today notes 10 reasons why we are bad with odds https://www.psychologytoday.com/articles/200712/10-ways-we-get-the-odds-wrong Risk and emotion are inseparable
- #6: Current situation
- #9: literature for keywords that worked well in discriminating between high and low luxury items see if they have been specifically tested in cars reference database that can help us interpret the results when we do collect data
- #10: User Experience context factors Tom Van de Zande
- #12: So when is a measurement good enough to use it for any type of action? How long? One day? Two days? A week? How often? Based on number of qualitative or percentual vistors Does it depend on the action you want to take?
- #13: Results for a treejack they tend to stabilize around 40-60 users
- #14: All too often we see budgets blown on unnecessarily large sample sizes. You'll often reach the same conclusions with margins of errors three to five times as wide
- #15: So how are we estimating how many shirts we are going to sell?
- #16: (current estimation ranges from 2 to 280 million)
- #19: http://techcrunch.com/2015/01/12/9gag-steps-into-gaming/ http://www.alexa.com/siteinfo/9gag.com
- #21: 7562 t-shirts estimated What if only sell 700 What if we sell 70 000
- #22: So we have measurements but how are we now going to take action
- #23: Recap on the current situation
- #24: We can actually drop the website, because in any statistical process it actually starts with the stats (not what generated the stats)
- #25: The problem is that this is emotional, you could quickly invest into certain areas without really knowing what went wrong. Do you need to invest to increase certain numbers or do you simply need to drop the prodcut, without a plan or estimate up front this is a choice!
- #26: Is it the order that is wrong? Is it the website that created (continues) statistics that does not fit into this type of standardized working? What’s wrong with this proces?
- #27: This makes sense: your estimate gives you a rough idea and depending on different estimates and different course of actions certain action plans can be executed Because we have have used a Fermi equation we can also look back at the equation and decide Action plans should focus on the “hard to guess” entities of the fermi equation