Why Twitter Is All the Rage: A Data Miner's Perspective
1 like2,922 views
1. The document discusses Matthew Russell's perspective on why Twitter is popular for data mining. He is a data scientist and author who created an open-source software project and book to make data science experiments on social media accessible to everyone. 2. Russell provides an overview of using Twitter as a data science platform and tools for mining Twitter data. He discusses analyzing real-time data about world events using the #Syria topic as an example. 3. Examples are given of analyzing Twitter data about Costa Rican presidential candidates and their potential influence. Methods for measuring influence on social media are also considered.
Why Twitter Is All the Rage: A Data Miner's Perspective
- 1. 1 Why Twitter Is All The Rage: A Data Miner's Perspective Matthew A. Russell O'Reilly Webcast 15 Oct 2013
- 2. 2 Hello, My Name Is ... Matthew Educated as a Computer Scientist CTO @ Digital Reasoning Systems Data mining; machine learning Author @ O'Reilly Media 5 published books on technology Principal @ Zaffra Selective boutique consulting
- 3. 3 Transforming Curiosity Into Insight An open source software (OSS) project http://bit.ly/MiningTheSocialWeb2E A book http://bit.ly/135dHfs Accessible to (virtually) everyone Virtual machine with turn-key coding templates for data science experiments Think of the book as "premium" support for the OSS project
- 4. 4 Overview Background Twitter as a data science platform Politics, influence, world events Data science tools for mining Twitter Q&A
- 5. 5 Background
- 6. 6 Data Science Data => Actionable information Highly interdisciplinary Nascent Necessary http://wikipedia.org/wiki/Data_science
- 7. 7 Digital Signal Explosion A model for the world: signal and sinks Growth in data exhaust is accelerating Digital fingerprints "Software is eating the world" Data mining opportunities galore...
- 8. 8 Digital Data Stats 100 terabytes of data uploaded daily to Facebook. Brands and organizations on Facebook receive 34,722 Likes every minute of the day. According to Twitter’s own research in early 2012, it sees roughly 175 million tweets every day 30 Billion pieces of content shared on Facebook every month. Data production will be 44 times greater in 2020 than it was in 2009 According to estimates, the volume of business data worldwide, across all companies, doubles every 1.2 years. See http://wikibon.org/blog/big-data-statistics
- 9. 9 Social Media Is All the Rage World population: ~7B people Facebook: 1.15B users Twitter: 500M users Google+ 343M users LinkedIn: 238M users ~200M+ blogs (conservative estimate)
- 10. 10 Why Does Social Media Matter? It's the frontier for predictive analytics Understanding world events Swaying political elections Modeling human behavior Analyzing sentiment Making intelligent recommendations
- 11. 11 Twitter Is All the Rage It satisfies fundamental human desires We want to be heard We want to satisfy our curiosity We want it easy We want it now Accessible, rich, and (mostly) "open" data RESTful APIs and JSON responses Great proving ground for predictive analytics
- 12. 12 Twitter's Network Dynamics 500M curious users 100M curious users actively engaging Real-time communication Short, sweet, ... and fast Asymmetric Following Model An interest graph
- 13. 13 Twitter as a data science platform
- 14. 14 What's in a Tweet? 140 Characters ... ... Plus ~5KB of metadata! Authorship Time & location Tweet "entities" Replying, retweeting, favoriting, etc.
- 15. 15 Twitter and Facebook Compared Twitter Facebook Accounts Types: "Anything" Accounts Types: People & Pages "Following" Relationships Mutual Connections Favorites "Likes" Retweets "Shares" Replies "Comments" (Almost) No Privacy Controls Extensive Privacy Controls
- 18. 18 A (Social) Interest Graph U2 Roberto Mercedes Juan Luis Luís Guerra Ana Jorge Nina
- 19. 19 A (Political) Interest Graph Johnny Araya Roberto Mercedes Rodolfo Hernández Ana Jorge Nina
- 20. 20 Costa Rican Presidential Candidates @ElDoctor2014 @Johnny_Araya
- 21. 21 ~3 Months on Twitter Aug 2013 Sept 2013 % Change Johnny Araya 14,573 15,506 6.40% Otto Guevara Guth 114 159 39.47% José María Villalta FlorezEstrada 8,160 8,990 10.17% 745 858 15.17% 1,192 1,487 24.75% Dr. Rodolfo Hernández Luis Guillermo Solís Rivera
- 22. 22 Who are Candidates Following?
- 23. 23 What are Candidates Tweeting?
- 25. 25 Potential Twitter Influence Araya Hernández Followers ~14k ~750 Theoretical Reach ~40M ~550k Reach (10) 490 673 Reach (100) 289 702 Reach (1000) 2782 X Reach (10,000) 2832 X "Suspect" Followers 3,246 94 See also http://wp.me/p3QiJd-2a
- 26. 26 Considerations for Measuring Influence Spam bot accounts that effectively are zombies and can’t be harnessed for any utility at all Inactive or abandoned accounts that can’t influence or be influenced since they are not in use Accounts that follow so many other accounts that the likelihood of getting noticed (and thus influencing) is practically zero The network effects of retweets by accounts that are active and can be influenced to spread a message See also http://wp.me/p3QiJd-2a
- 27. 27 Social Media Popularity: Araya vs Hernández Twitter Popularity Facebook Popularity Araya% Araya% Hernandez% Hernandez%
- 28. 28 Realtime Analysis: #Syria Monitor Twitter's firehose for realtime data using filters such as #Syria Keep in mind the sheer volume of data can be considerable Analysis at MiningTheSocialWeb.com
- 36. 36 #Syria: Why? That's for you (as the data scientist) to decide Quantitative automation can amplify human intelligence Qualitative analysis is still requires human intelligence
- 37. 37 Data science tools for mining Twitter
- 38. 38 MTSW Virtual Machine Experience Goal: Make it easy to transform curiosity into insight Vagrant-based virtual machine Virtualbox or AWS IPython Notebook User Experience Point-and-click GUI 100+ turn-key examples and templates Social web mining for the masses
- 39. 39 Social Media Analysis Framework A memorable four step process to guide data science experiments: Aspire Acquire Analyze Summarize
- 40. 40
- 41. 41
- 42. 42
- 43. 43
- 44. 44 Free Resources Mining the Social Web 2E Chapter 1 (Chimera) http://bit.ly/13XgNWR Source Code (GitHub) http://bit.ly/MiningTheSocialWeb2E http://bit.ly/1fVf5ej (numbered examples) Screencasts (Vimeo) http://bit.ly/mtsw2e-screencasts http://MiningTheSocialWeb.com
- 45. 45 Q&A