Scaling big data mining infrastructure thetwitte experience - Jimmy Lin and Dmitriy Ryaboy
Download as PPT, PDF1 like530 views
The document discusses the challenges and insights gained from big data mining at Twitter, emphasizing the complexities of handling unstructured data and the inadequacies of existing tools. It highlights the importance of effective data cleaning and preparation, with 80% of the work involving these processes rather than mining itself. Recommendations include adopting specific logging practices and embracing various algorithms to improve the efficiency of data analytics efforts.
Scaling big data mining infrastructure thetwitte experience - Jimmy Lin and Dmitriy Ryaboy
- 1. Scaling Big Data Mining Infrastructure: The Twitter Experience Jimmy Lin and Dmitriy Ryaboy Presented by Ohud Saud The paper can be found on KKD and appear in SIGKDD explorations Dec, 2012 http://www.kdd.org/sites/default/files/issues/14-2-2012-12/V14-02-02-Lin.pdf
- 2. Overview • Jimmy Lin and Dmitriy Ryaboy worked in Twitter • presents insights about Big Data mining infrastructures. • the experience of doing analytics at Twitter. • due to the current state of the data mining tools, performing analytics is not straightforward process. • Time consumed in preparatory work to the application of data mining methods..
- 3. Motivation • Big data is not easy to clean and manage. • Twitter has one of the highest unstructured data, yet a big target of research. • Twitter has experienced tremendous growth over the past few years in terms of size, complexity, number of users, and variety of use cases. • Some of well-known algorithms simply does not work with Twitter environment.
- 4. The problem • First: schemas play an important role in helping data scientists understand petabyte-scale data stores, but they’re insufficient to provide an overall “big picture” of the data available to generate insights. • Second: a major challenge in building data analytics platforms stems from the heterogeneity of the various components that must be integrated together into production workflows.
- 5. • It was estimated that 2007 was the first year in which it was not possible to store all the data that we are producing.
- 6. Approach Block Diagram Twitter's Java regular expression to parse Scribe infrastructure log messages at Twitter 2010
- 7. Approach Block Diagram JSON log message Apache Thrift message
- 8. Lessons Learned • Data science in the is not so glamorous, it is about get the plumbing right). • Storing logs in a traditional RDBMS such as MySQL simply does not . • JSON not a good alternative (does not scale). • Hadoop infrastructure is bad at iterative algorithms. • Naming things is hard (when diff. codes combined!).
- 9. Recommendations • Log directly into DB. • Use Scribe for logging before aggregating data and then writing it to HDFS and to Hadoop. • Thrift provides a great balance between flexibility and structure. • Use stochastic gradient descent and have some sort of online learning. • ensemble learning is a good option (Pig). • XRX tool is powerful.
- 10. Insights Gained • 80% of work with data is data cleaning not mining. • successful big data mining in practice is more than what most academics would consider data mining is. • life “in the trenches” is occupied by much preparatory work that precedes the application of data mining algorithms. • Open sources improving continuously. Thus, can be the solution you are looking for.