10 Reasons Why Data-driven App Design Needs Social Science | Julian Runge
Download as PPTX, PDF2 likes294 views
The document outlines ten reasons why social science is essential in data-driven app design, particularly in enhancing user engagement and profitability through insights that address the limitations of machine learning. It emphasizes the importance of understanding user behavior, cognitive biases, and the long-term effects of design choices, demonstrating how social science can inform better modeling, incentive structures, and treatment designs. The integrated approach aims to create more effective and personalized user experiences while addressing potential challenges and compliance with regulatory frameworks.
10 Reasons Why Data-driven App Design Needs Social Science | Julian Runge
- 1. 10 Reasons Why Data-driven App Design Needs Social Science Julian Runge & Yavuz Acikalin Data Science / User Research
- 3. Why Does Data-Driven App Design Need Social Science? Let’s get on the same page on terminology.. By data-driven app design, we mean the application of quantitative methods – in particular machine learning and experimentation – to the design of mobile apps. Our examples will center around this definition.
- 4. Why Does Data-Driven App Design Need Social Science? Five broader angles 1. Noisy signals and lack of scale can fail machine learning 2. Unanticipated longer term and side effects of ML 3. Modeling and choice of method 4. Identifying the right incentives and treatments 5. Understanding experiment results
- 5. 1) Noisy signals and lack of scale can fail machine learning • Good design wants to maximize users’ engagement with an app and ultimately be profitable • Engagement and revenue signals in the app economy are outlier infested and noisy • Additionally, e.g. during soft launch, apps commonly lack the scale for meaningful inference and analysis
- 6. Ex. 1a) Price personalization • Example: Personalize a game’s starter pack • Goal: Increase revenue • Seems sensible to optimize for overall revenue per user • If the app has strong monetization, most algorithms get lost though as the treatment effect disappears in strong downstream revenue • Social science can offer guidance, e.g. complementary goods (Allen 1934): • Give large starter packs to users who paid a lot for their mobile device
- 7. Ex. 1b) Unsupervised learning • Noise can also make unsupervised learning problems difficult • Example: Payer segmentation, Solution: Guide ML with conceptual insights from social science, e.g. RFM, jobs-to-be-done
- 8. 2) Unanticipated longer term and side effects of ML • Even well-fitted and accurate models can generate policies with unanticipated outcomes or negative long-term consequences • Social science can help us anticipate and address such outcomes
- 9. Ex. 2a) “Filter bubbles” • Personalization can lead to "filter bubbles" reinforcing repetitive user experiences • E.g., targeting users with offers that improve performance in their preferred game modes may prevent them from exploring better monetizing game features • Social science: Variety in experiences increases engagement
- 10. Ex. 2b) Price belief formation • Staying with starter pack example, say we want to use a bandit approach • Revenue too noisy to be the bandit’s reward, consider conversion • Conversion excessively favors low price and small start packs • Lower LTV :o • Social science: Consumers form price beliefs (e.g. Kalwani and Yim 1992) • Consider composite reward blending conversion and revenue to take price belief formation into account
- 11. 3) Modeling and choice of method • A good modeler does not just throw algorithms at data, but understands the data generating process • Why do we see these patterns in the data? • What are underlying decision situations? What choices does a consumer have to make? • Asking such substantive questions helps refine models and identify the right choice of method
- 12. Ex. 3a) LTV prediction in freemium settings • Consumers have non-compensatory decision rules • In freemium app context: “I will not spend on this app no matter how much I use it.” → Do not use linear methods for LTV prediction • Intuitively: Some free users have the same usage patterns as payers RMSE of per-user LTV predictions with Linear Regression, Random Forest and Deep Neural Net: RF and Deep-NN can accommodate non-compensatory decision rules
- 13. Ex. 3b) Feature engineering • Analysis requires aggregation of raw data to extract the most valuable information available • Examples: Price sensitivity – Discount targeting, temporal discounting – cost of impatience -> insights about a user's characteristics that don't emerge from common go-to aggregations
- 14. 4) Identifying the right incentives and treatments • Social science helps identify better ways to interact with users • E.g. what incentive will work well for stimulating a certain kind of behavior
- 15. Ex. 4a) Variable rewards schedules • Variable reward schedules - insight from social science -> apply across contexts • E.g., to subscription offers
- 16. Ex. 4b) Intrinsic motivation and extrinsic rewards • Churn prediction: Set up a well-working system predicting churn of high-value players • Sent out a free pack of in-game goods worth 10 USD to churning players • No effect on churn • Social science: Rewards need to align with players’ motivation; e.g. intrinsically motivated players are unlikely to be affected by extrinsic rewards (Deci and Ryan 1985)
- 17. 5) Understanding experiment results • Social science can help understand and explain counterintuitive or undesired AB test results • Embedding such learnings in a broader context helps avoid them in the future
- 18. Ex. 5a) Goal valence • Introduction of usage-based reward system: Players can collect tokens by achieving certain level scores and can use these tokens to access further levels • Expected results: Users play more and access higher levels • Actual result: Users play more, but access higher levels less • Additional play of lower levels substituted play of higher levels • Social science: Goal valence (Kernan and Lord 1990) • The new goal of achieving scores affected the original goal’s (advance to higher levels) priority
- 19. Ex. 5b) Enrich big with small-scale experiments • Combining ‘big data’ with highly controlled lab experiments
- 20. Bonus: Social science can help comply with regulatory frameworks • A user experience routed in social science rather than only in data and algorithms is less opaque and can be more readily explained • E.g.: Why am I getting a large starter pack and my friend is getting a small one? • Personalize user experience to reduce frictions and cognitive load • Device quality and game consumption are complementary – on average, users on higher quality devices wish for larger starter packs
- 21. N3TWORK Platform: End-to-end publishing • Data and behavioral science are part of N3TWORK’s Platform services • These services are available to games published on the Platform and minimize acquisition cost and maximize engagement and LTV • E.g.: • Automated creative generation and campaign optimization • AB testing / experimentation • Personalization service
- 22. Thank you! •(Data) scientists and analysts: Let’s chat. •Game devs: We publish – reach out! julian@n3twork.com yavuz@n3twork.com
Editor's Notes
- #20: Enrich field experiments with lab experiments Brain experiment as an example of how small samples in the lab with correct data source can accurately predict market level behavior Can explain over 70% of variation in market level video engagement from 40 people's brain data