Feature Extraction for Effective Microblog Search and Adaptive Clustering Algorithms for TTG

Editor's Notes

• #2: Good morning, everybody. This is Runwei. I’m a senior three student of Peking University. Our report name is feature extraction for effective microblog search and adaptive clustering algorithms for TTG.
• #3: During this year’s track, we still focus more on the ad hoc search task. That is, given a query Q at T, find the most relevant tweets about Q before time T. I’ll first describe our system overview, then present our feature extraction method, and at last show out experimental results.
• #4: We know that there are many challenges in microblog search. As the tweet is very short, we will meet sever vocabulary-mismatch problem. Thus it is necessary to apply query expansion technologies. There are also a lot of shortened urls, we should use them to expand the original tweets. What’s more, there are all kinds of babbles in the microblogosphere, we should define tweet quality and filter the non-informative tweet.
• #5: learning to rank framework can make full use of various factors that affect the microblog search. Thus our system is based on the learning to rank framework.
• #6: With the help of the provided common API, we first get the topic related candidate with original query. To improve the recall of the retrieved results, query expansion is used to get more candidate results. Then, we generate features for the query-document pair. With the labels of TREC 13 topics, we trained a model with Ranking SVM. Then for the test set, we can re-rank the tweets according to the ranking function. Finally, we return the top 1000 tweets.
• #7: The most important part of our system is feature extraction, which directly affect the final retrieval performance. We investigated a lot of related work of learning to rank in microblog search and find that many features have been proven useful. However, the features used in previous work are actually very few.
• #8: When it comes to the traditional web search, we found that hundreds or even thousands of features are used in the full ranker. For example, Microsoft published a dataset for evaluating the effectiveness of different learning to rank algorithms. They provided 136 features for each query and url pair. Following this, we also want to extract more effective features for microblog search.
• #9: We want to generate the features from three perspectives: query, document and retrieval models. For retrieval models, we use the traditional BM25, Language Model and TFIDF models. Specially, we use three different smoothing methods in language modeling framework. Thus, we have totally 5 retrieval methods. 𝑘 1 = 0.3 and b = 0.05
• #10: From the query side, we expand the query using three different query expansion techniques. That is PRF based, web based and freebase based query expansion. As in this year’s track, external evidences are encouraged for improving the retrieval performance. We seek for Google and Freebase’s help. As the freebase related features don’t perform very well in our training data, we don’t use them in our submitted runs.
• #11: Take the query “Ron Weasley birthday” for an example, we search the top related 5 web documents using Google search API with a time limitation. Then we recognize all the verbs, nouns and adjectives with python nltk tool, top 5 terms are then used as a new query named WebQuery. We also search the top tweet as issue tweet with common API, then we extract topical terms to generate the IssueQuery. Besides, we also combine original query and issue query with an interpolation parameter 0.6 (origin) to form a MergeQuery. http://jwebnet.net/advancedgooglesearch.html#advSearchURI
• #12: From the document side, we crawl all the urls embedded in the tweets and extract the title information of each page. We name this corpus as TopicInfo. We also merge the OriginInfo and TopicInfo to generate the DocExInfo corpus. The texts are preprocessed in several ways: … Note that aside from the corpus retrieved by api, we also crawl a copy of the official corpus and name it local corpus.
• #13: We further compare the differences of the two corpora. …
• #14: Besides the relevance score related features, we also define some traditional quality features to filter babbles.
• #15: Here are the experimental results. 4:query 3:document 2:prf 5: retrieval models From the table we can see that l2r related runs are better than the unsupervised run (i.e. PKUICST4). Learning to rank runs show significant superiority over the unsupervised run (PKUICST4). Api corpus is better than the local corpus candidate. Combining api and local corpus can get a little improvement in terms of MAP compared with run with only api corpus features.
• #16: Tweet Timeline Generation (TTG) is a new task for this year's Microblog track with a putative user model as follows: "I have an information need expressed by a query Q at time t and I would like a summary that captures relevant information." 
• #17: Determine how many results to return : that is, systems will be penalized for results that are too verbose. Detect (and eliminate) redundant tweets: this is equivalent to saying that systems must detect novelty. Systems will be penalized for returning tweets that contain redundant information.
• #18: Given a query, we first get 1000 results using the ad hoc search system, Then a candidate selection component is used to determine how many results to returns for clustering. With the selected tweets, we apply two clustering algorithms to get the final summarized tweets. Topics for 11-12 are used for parameter tuning.
• #19: Three methods are used to determine …
• #20: Now I’ll describe the first clustering algorithm, that is star clustering. We first build a graph according to the tweet similarity. Each tweet is a vertex, when two tweets’ cosine similarity score is larger than a threshold \sigma, there will be a edge between the two tweets. Then for each iteration, we found the vertex with the highest degree and regard it as the star tweet, and other tweets that have an edge with it will be added to that cluster. This cluster is then deleted from the graph, then we conduct this procedure iteratively until all stars are found.
• #21: We also attempt using the hierarchical clustering algorithm. At first, each tweet is a cluster, then we compute the distance between each cluster. The nearest cluster are then combined to a new cluster. Then we do this iteratively until the minimum cluster distance is less than a tuning parameter beta.
• #22: Here are the experimental results for our submitted runs. PKUICST1 uses … From the table, we can observe that TTGPKUICST1 shows significant superiority in terms of unweighted recall and weighted recall over other runs, while it performs poorly in terms of precision. TTGPKUICST2 performs better in precision compared with other runs. TTGPKUICST3 and TTGPKUICST4 which utilize manually selected top N parameter have medium and stable performance over all metrics.