User Sentiment to Determine App Quality
Download as PPTX, PDF3 likes643 views
The document discusses using user sentiment analysis from app reviews and feedback to determine app quality. It describes how Applause analyzes app reviews to classify sentiment, extract key attributes mentioned, identify trends, and cluster similar reviews to provide insights. This helps app developers focus on the right issues to improve their apps based on what users are saying.
User Sentiment to Determine App Quality
- 1. 1 User Sentiment to Determine App Quality Heidi Young VP Engineering, Applause
- 2. User Sentiment to Determine App Quality Heidi Young VP of Engineering 2
- 3. Getting to Know Each Other 3
- 4. Who am I? • Roles/Titles: Engineer, Data Scientist, Software Junkie • Worked at/for: – Applause (App Quality) – Bing (Microsoft, Search) – Rotella Capital Management (Financial Industry) – SAIC (Defense Contracting) – Other smaller startups • Passionate about: – Search – Information Retrieval – Optimizing and Quantifying Large Data Sets • Member of the Apigee Technical Advisory Board 4©2015 Apigee. All Rights Reserved.
- 5. Applause: Who we are and what we do 5©2015 Apigee. All Rights Reserved.
- 7. Building an App is Easy 7 Step 1: Write code Step 2: Ship to App Store ©2015 Apigee. All Rights Reserved.
- 8. Building a Good App is Hard • Device Fragmentation • Multiple App Stores • Native vs Hybrid vs Web • Server/Client Tradeoffs • International, globalization, localization, etc. • Security 8©2015 Apigee. All Rights Reserved. ? ? ?
- 9. Building a High Quality App is Harder • No crashes • High star ratings • Lots of downloads • Happy customers 9©2015 Apigee. All Rights Reserved.
- 10. Different Needs at Different Times 10©2015 Apigee. All Rights Reserved. Ideation Design Development TestingBeta Production Maintenance
- 11. Ideation Design Development Testing Beta Production Maintenance We think about things like this 11©2015 Apigee. All Rights Reserved. • Use humans to touch your apps in the wild, Who better than an actual user? • Use software to collect data and measure quality of your your apps, Too much data, too little time • Use data to analyze where to focus, Actionable Metrics are the only way to get things done Crowd Testers SD K User Sentiment
- 13. User Reviews and Feedback are Important 1. They are very public 2. They end up making headlines 3. They inform user download decisions 13©2015 Apigee. All Rights Reserved.
- 14. Store Star Rating vs Review Star Rating • In a lot of cases, what you can extract from review text is very different than the store star rating • Don’t let rating inflation fool you! 14©2015 Apigee. All Rights Reserved. Store Store Star Rating App Reviews (only) Rating Delta Google Play (US) 3.97 3.83 -0.14 Apple iTunes 3.76 3.81 +0.05 App Store Star Rating App Reviews (only) Rating Delta HBO Go (Android) 3.89 2.64 -1.25 Netflix (iOS) 3.43 2.67 -0.76 Bank of America 4.16 2.94 -1.22
- 15. A Systematic Approach to Measuring App Quality 15©2015 Apigee. All Rights Reserved.
- 16. Extracting something out of the noise 16©2015 Apigee. All Rights Reserved. • Step 1: Classify Every Review • Step 2: Use ratings to assign a score/importance to put a measurement stake in the ground Max Score Mean for Category Your app score
- 17. Using Signals to Help with Classification 17©2015 Apigee. All Rights Reserved.
- 18. Applause Analytics Scoring and Sentiment Analysis 18©2015 Apigee. All Rights Reserved. • Histogram of Attribute Scores Across App Stores • Mean Attribute Scores Across App Stores
- 19. Applause Analytics Scoring and Sentiment Analysis 19©2015 Apigee. All Rights Reserved. • Histogram of Attribute Scores Across App Stores • Mean Attribute Scores Across App Stores People talk about these things a LOT
- 20. Applause Analytics Scoring and Sentiment Analysis 20©2015 Apigee. All Rights Reserved. • Histogram of Attribute Scores Across App Stores • Mean Attribute Scores Across App Stores People talk about these things a LOT People talk more positively about these things People talk more negatively about these things
- 21. Opens Up Competitive Benchmarking 21©2015 Apigee. All Rights Reserved.
- 22. Applause Analytics Clustering and Ranking 22©2015 Apigee. All Rights Reserved. The How: • Modified Agglomerative Clustering Algo (Patent Pending) • Represent reviews per app as term review vectors and calculate cosine similarities • Cluster based on size of review corpus, vocabulary for app, etc. • Rank based on magnitude, date, avg sentiment, and rarity The What: • Grouped, related app reviews • Most actionable reviews Example: Can’t turn off notifications
- 23. So… what does this help us do? 23©2015 Apigee. All Rights Reserved.
- 24. Smarter, More Context to Alert You to the Right Issues 24©2015 Apigee. All Rights Reserved. Step 1: Use ratings and review count trends for detection Step 2: Use clusters for smart recommendation of focus areas Security issues with UK accounts Bidding isn’t sticking Items are no longer copyable Search is broken on HTC devices Auto-detected, pushed to your inbox, or app notification Trend over N timeunits, then pop alert if >90th% over N-1
- 25. Food for thought: Pre- vs Post- Release Issues 25©2015 Apigee. All Rights Reserved. Top Pre Release Bug Issues – Creating/Modifying Account information – Navigation flow breaks – Order forms – Video playback (typically on mobile due to rotated screens) – Search experiences – Broken pictures/photos Top Post Release User Issues - Latest update is regression in functionality - Bad graphics - Music or video playback is poor - Slow to load, freezes on load - Login broken - Upsell is annoying
- 26. Food for thought: Pre- vs Post- Release Issues 26©2015 Apigee. All Rights Reserved. Top Pre Release Bug Issues – Creating/Modifying Account information – Navigation flow breaks – Order forms – Video playback (typically on mobile due to rotated screens) – Search experiences – Broken pictures/photos Top Post Release User Issues - Latest update is regression in functionality - Bad graphics - Music or video playback is poor - Slow to load, freezes on load - Login broken - Upsell is annoying What’s common before and after release? So that perhaps we can tell you…
- 27. Credit Where Credit is Due • Everyone at Applause! • Especially the people doing all of the actual work behind Applause Analytics: – Jason Stredwick – Matthew Johnson – Scott Ferguson – Yashas Mavinakere – Lin Zhong – Rob Reid – Josh Wilcox – Christina Green – Peter Bell 27©2015 Apigee. All Rights Reserved.