Link extraction and classification
0 likes1,771 views
Bruno Pedro discusses link extraction and classification strategies using Tarpipe, a tool for processing large amounts of data from social media streams like Twitter. The document outlines techniques for immediate storage, content extraction, and classification of information received through tweets, emphasizing the need for efficient data handling. It also mentions various resources and APIs relevant to the topic.
Link extraction and classification
- 1. http://www.flickr.com/photos/pio1976/3330670980/ Link extraction and classification Bruno Pedro December 2010
- 2. Bruno Pedro A n e x p e r i e n c e d We b d e v e l o p e r a n d entrepreneur. Has extensive background in large scale projects and technical writing. http://tarpipe.com/user/bpedro
- 3. What is tarpipe? User
- 5. twitter source: mashable.com • Average ~1.1M tweets/hour • ~300 new tweets every second
- 6. Challenge • ~300 reads/second • 160 X 300 = 48 KB/second = 4 GB/day (approximate calculation) • How to process all this information?
- 7. Strategy • Read and store immediately: • High performance write storage • No locks allowed • Prepare for lots of reading errors
- 8. Strategy • Process later: • Regular expressions • Term extraction • Machine learning
- 10. Link extraction extract user extract URL new tweet yes launch process has URL? save show no dump
- 11. Content extraction • Short URLs • HTTP redirects • HTTP errors — retry? • Content analysis
- 12. Content extraction URL retry? yes yes no fetch contents redirect? error? yes no save
- 13. Content analysis • Assume malformed (X)HTML • Regular expressions or • Convert to XHTML • DOM traversing
- 14. Content classification • HTML title, description, keywords tag cloud { • H1, H2, H3, ... • Paragraphs graph placement { • Who shared the link? • Internal and external links
- 15. Content classification (X)HTML extract head extract text extract text elements yes yes yes H1,H2,... paragraphs head found? save found? found? no no
- 16. The big picture new tweet classify content save extract URL extract content
- 17. Food for thought • PubsubHubbub http://code.google.com/p/pubsubhubbub/ • Activity Streams http://activitystrea.ms/ • twitter streaming API http://dev.twitter.com/pages/streaming_api
- 18. tarpipe streamlines your tarpipe is one of the most Today I had a chance to updates to various social curious experiments in spend time experimenting web sites, creating simple social media that I've with tarpipe and I have to or complex workflows to seen lately. The service say that I am intrigued by update several buckets in has the potential to be the concept and impressed one fell swoop. the answer to the lament by the implementation. I first talked about in The Adam Pash looming crisis: Personal Jeff Barr lifehacker syndication overload. Amazon.com Rafe Needleman CNET news thank you automated publishing