DataCon Talk
Download as PPTX, PDF0 likes703 views
The document discusses a presentation on the telecommunications market, emphasizing the saturation of subscriber growth and the importance of attracting switchers to drive industry growth. It details the use of both human and machine scoring for sentiment analysis of tweets related to carrier switching, providing insights into market trends and consumer behavior. Future work includes improving scoring algorithms and integrating additional data sources to enhance analysis capabilities.
![12 M-SCORING EXAMPLES
Phone
iPhone 6 Switch to T-Mobile
Real Time Market Data & Analysis for the Telecom Industry
to T-Mobile Mention
Switch from [carrier1] to [carrier2]
Switch from [carrier1] to [carrier2]
Switch from [carrier1]
Switch to [carrier1]](https://image.slidesharecdn.com/datacontalk-140915123635-phpapp01/85/DataCon-Talk-12-320.jpg)
![13 M-SCORING EXAMPLES
T-Mobile
Mention
Phone
Real Time Market Data & Analysis for the Telecom Industry
Switch from ATT
From ATT
Switch from [carrier1] to [carrier2]
Switch from [carrier1] because I love [carrier2]
Assume carrier2 is the ‘switch to’ carrier](https://image.slidesharecdn.com/datacontalk-140915123635-phpapp01/85/DataCon-Talk-13-320.jpg)
DataCon Talk
- 1. DATA-CON BOSTON 2014 TANYA CASHORALI @TANYACASH21 This presentation and the information contained herein is confidential. By your acceptance and review of this presentation, each recipient agrees that it will not, and shall assure that its employees, agents, representatives and advisors will not copy, reproduce or distribute this presentation, in whole or in part, without the prior written consent of Comniscient Technologies LLC, and that it will keep confidential all information contained herein which is not already in the public domain. Further, the recipient will use the presentation only to obtain background information on the Company and its activities.
- 2. 2 THE TELECOMMUNICATIONS MARKET IS FULLY SATURATED 67% of new subscriber growth comes from switchers.* Last year, switchers put $29B up for grabs in the wireless industry. WINNING SWITCHERS IS KEY FOR GROWTH *Percentage of net adds that are not activations, tablets or wholesale Real Time Market Data & Analysis for the Telecom Industry
- 3. 3 CARRIER VISION – THE WHAT, WHERE, WHEN Real Time Market Data & Analysis for the Telecom Industry
- 4. 4 VOICE OF THE CUSTOMER – THE WHY Real Time Market Data & Analysis for the Telecom Industry
- 5. 5 OBSERVED DATA ON TWITTER Real Time Market Data & Analysis for the Telecom Industry
- 6. 6 STARTED GATHERING DATA USING STREAMR Real Time Market Data & Analysis for the Telecom Industry
- 7. 7 HUMAN SCORED DATA We “S-scored” about 150-500 tweets per day until we had ~4,000 human scored tweets. Used this data set to learn how to systematically crowd source the same process and also automate the process using machine scoring “M-scoring” in R Training Set of 4,000 tweets Real Time Market Data & Analysis for the Telecom Industry Crowd sourced ~80,000 tweets Derived rules M-scoring rules in R
- 8. 8 CLASSIFY CARRIERS Real Time Market Data & Analysis for the Telecom Industry
- 9. 9 NLP AND SENTIMENT ANALYSIS IS HARD Real Time Market Data & Analysis for the Telecom Industry
- 10. 10 BASIC PATTERN MATCHING IS COMPLICATED ENOUGH • Phone Price • Customer Service • Coverage Quality • Upgrade Plan • Coverage Availability • Family Plan • Plan Price • Service Promo • Outage • Data Plan • Phone Availability • Device Promo • Switching to / from Real Time Market Data & Analysis for the Telecom Industry iPhone + expensive Samsung Galaxy + money Phone + cost Data, unlimited Switch from Verizon to ATT
- 11. Real Time Market Data & Analysis for the Telecom Industry 11
- 12. 12 M-SCORING EXAMPLES Phone iPhone 6 Switch to T-Mobile Real Time Market Data & Analysis for the Telecom Industry to T-Mobile Mention Switch from [carrier1] to [carrier2] Switch from [carrier1] to [carrier2] Switch from [carrier1] Switch to [carrier1]
- 13. 13 M-SCORING EXAMPLES T-Mobile Mention Phone Real Time Market Data & Analysis for the Telecom Industry Switch from ATT From ATT Switch from [carrier1] to [carrier2] Switch from [carrier1] because I love [carrier2] Assume carrier2 is the ‘switch to’ carrier
- 14. 14 CATEGORY CLASSIFICATION Real Time Market Data & Analysis for the Telecom Industry
- 15. 15 SWITCHING TO/FROM Real Time Market Data & Analysis for the Telecom Industry
- 16. 16 ARE WE IMPROVING? Sensitivity = TP / (TP + FN) Specificity = TN / (TN + FP) Real Time Market Data & Analysis for the Telecom Industry
- 17. 17 MATURING INTO A REAL PRODUCT Real Time Market Data & Analysis for the Telecom Industry
- 18. 18 GNIP PROCESS Table Daily Average Total Wireless 655 111,483 Wireline 152 25,264 Other 11,803 2,006,649 Real Time Market Data & Analysis for the Telecom Industry
- 19. STREAMR VS GNIP VS DATASIFT Real Time Market Data & Analysis for the Telecom Industry 19 StreamR GNIP DataSift Full firehose data - X X Historical Twitter - X X data Real-time X X X Data Destinations - - X Data Buffer - 5 days ($$$) 2 hours (included) Demographics - - X Sentiment - - X Gender Detection - - X Topic Detection - - X Entity Extraction - - X Link Analysis - - X
- 20. Deployment Architecture Comlinkdata JSON Real Time Market Data & Analysis for the Telecom Industry 20 Data Source Amazon RDS GNIP Amazon EC2 Technology Stack R, PostgreSQL PostgreSQL Kantar Events PostgreSQL /home/ubuntu/Documents/Scripts/TwitterScrapes/GNIP/GNIP Extract.R /home/ubuntu/Documents/Scripts/TwitterScrapes/voc-dataops/ GNIP/Json_Interpreter.R /home/ubuntu/Documents/Scripts/TwitterScrapes/voc-dataops/M-Scoring/ simple_MScore_GNIP.R /home/ubuntu/Documents/Scripts/TwitterScrapes/voc-dataops/ Aggregates/af_tweets.R HTTPS Application and Web Servers MG Tomcat Internet Chrome, Safari, Firefox, IE Web Service (REST) GUI Java 6, Spring (Security, MVC, JDBC) Angular JS/Ajax, HTML5, CSS3, D3, Bootstrap Shiny server@ubuntu gnip_wireless_raw gnip_landline_raw gnip_other_raw tweet_mwl tweet_mll a_tweets f_tweets /home/ubuntu/GNIP/data/
- 21. 21 INITIAL PROTOTYPE IN TABLEAU Real Time Market Data & Analysis for the Telecom Industry
- 22. 22 WHAT WE LEARNED • Always store raw unprocessed data somewhere • Beware of UTF encodings and special characters • Ensure time zones are synched across databases / applications • Don’t be afraid to cast a larger net of tweets given the ~1M tweet/month limit provided by most vendors • Consider how to deal with blast tweets, retweets. Source can be used to help identify blasts (TweetCaster, Scoop.it, etc.) Real Time Market Data & Analysis for the Telecom Industry
- 23. 23 ENTIRE PRODUCT LIFECYCLE Real Time Market Data & Analysis for the Telecom Industry
- 24. 24 CONSIDERATIONS Many variations of changing carriers: • Bye Sprint! • Getting rid of Verizon • Peace out T-Mobile • Going to AT&T • Twitter data is not necessarily representative of the entire population • Other languages • Geo-tagged is only ~5% • Expanding to Canada Real Time Market Data & Analysis for the Telecom Industry
- 25. 25 FUTURE WORK • Migrate to DataSift from GNIP • Improve M-scoring using more complex NLP algorithms • Integrate additional data sources including downdetector.com, ad spend, and other relevant sources • Integrate additional Twitter data (mentions to carriers as well as tweets from the carriers, various language around switching) • Develop landline version and Canada version • Trending words / Keyword search • ??? Real Time Market Data & Analysis for the Telecom Industry
- 26. 26 ACKNOWLEDGEMENTS Real Time Market Data & Analysis for the Telecom Industry Jacob Tobias Software Developer Sarah Bolt Marketing Manager Dylan Doyle Data Scientist Mallorie Ekstrom Graphic Designer Alan Tam Web Developer Josh Walker COO Ken Yeoh Data Engineer For more information, hiring, or questions, email contact@comlinkdata.com
- 27. 27 REFERENCES • streamR - http://cran.r-project.org/web/packages/streamR/index.html • sqldf - http://cran.r-project.org/web/packages/sqldf/index.html • GNIP - http://gnip.com/ • DataSift - http://datasift.com/ Real Time Market Data & Analysis for the Telecom Industry
Editor's Notes
- #18: Topsy for historical data, acquired by Apple
- #19: Rules in JSON format? Put on GNIP servers Uses a https stream to enable transfer of Json formatted data to customer Must be processed from Json extracting useful data from the stream No buffer available customer must always be connected to the stream Adding 5 day buffer costs money Historical jobs are prohibitively expensive. Lots of issues, migrate to datasift 2.1M tweets total
- #20: Priced up to 1 million tweets per month (GNIP = $1560/month)