Using hash tag graph based topic model to connect semantically-related words without co-occurrence in micro blogs
1 like103 views
This paper introduces a novel hashtag graph-based topic model (HGTM) for analyzing microblogging environments, specifically to address the challenges posed by the informal and sparse nature of tweets. By utilizing user-contributed hashtags, HGTM connects semantically related words even when they do not co-occur, thus enhancing topic discovery and alleviating issues related to sparseness and noise in tweet data. Experimental evaluations demonstrate that HGTM outperforms traditional topic models and achieves effective clustering and classification on real-world tweet datasets.
Using hash tag graph based topic model to connect semantically-related words without co-occurrence in micro blogs
- 1. #13/ 19, 1st Floor, Municipal Colony, Kangayanellore Road, Gandhi Nagar, Vellore – 6. Off: 0416-2247353 / 6066663 Mo: +91 9500218218 Website: www.shakastech.com, Email - id: shakastech@gmail.com, info@shakastech.com Using Hash tag Graph-Based Topic Model to Connect Semantically-Related Words without Co-Occurrence in Micro blogs ABSTRACT: In this paper, we introduce a new topic model to understand the chaotic micro blogging environment by using hashtag graphs. Inferring topics on Twitter becomes a vital but challenging task in many important applications. The shortness and informality of tweets leads to extreme sparse vector representations with a large vocabulary. This makes the conventional topic models (e.g., Latent Dirichlet Allocation and Latent Semantic Analysis) fail to learn high quality topic structures. Tweets are always showing up with rich user-generated hashtags. The hashtags make tweets semi-structured inside and semantically related to each other. Since hashtags are utilized as keywords in tweets to mark messages or to form conversations, they provide an additional path to connect semantically related words. In this paper, treating tweets as semi-structured texts, we propose a novel topic model, denoted as Hashtag Graph-based Topic Model (HGTM) to discover topics of tweets. By utilizing hashtag relation information in hashtag graphs, HGTM is able to discover word semantic relations even if words are not co-occurred within a specific tweet. With this method, HGTM successfully alleviates the sparsity problem. Our investigation illustrates that the user-contributed hashtags could serve as weakly-supervised information for topic modeling, and the relation between hashtags could reveal latent semantic relation between words. We evaluate the effectiveness of HGTM on tweet (hashtag) clustering and hashtag classification problems. Experiments on two real-world tweet data sets show that HGTM has strong capability to handle sparseness and noise problem in tweets. Furthermore, HGTM can discover more distinct and coherent topics than the state-of-the-art baselines. EXISTING SYSTEM: Although traditional methods have achieved success in uncovering topics for normal documents (e.g., news articles, technical papers), the characteristics of tweets bring new challenges and opportunities to them.
- 2. #13/ 19, 1st Floor, Municipal Colony, Kangayanellore Road, Gandhi Nagar, Vellore – 6. Off: 0416-2247353 / 6066663 Mo: +91 9500218218 Website: www.shakastech.com, Email - id: shakastech@gmail.com, info@shakastech.com Several methods have been proposed to tackle the serious noise and lack of context problems in tweets. One intuitive method is to aggregate tweets as a long document. Hong, et al. aggregated tweets by the same user, the same word or the same hashtag. Mehrotra, et al. investigated different pooling schemes with hashtags for the later LDA process. Weng, et al. introduced “a pseudo document” by collecting tweets under the same author. Yan, et al. clustered tweets by a non-negative matrix factorization. DISADVANTAGES OF EXISTING SYSTEM: Compared with normal texts, tweets usually contain only a few words. The usage of informal language enlarges the size of the dictionary. They consider tweets as flat texts and ignore tag-related information contained in twitter data. ATM (Author-Topic Model) just leverages tag information by a uniform distribution of tags, but ignores the potential tag relation that is vitally helpful to build the latent semantic relationship between words. PROPOSED SYSTEM: We construct different kinds of hashtag graphs based on statistical information of hashtag occurrence in a crowd sourcing manner that can be acquired without human efforts such as labeling. Based on these hashtag graphs, we propose a novel framework of Hashtag Graph based Topic Model (HGTM). The basic idea of HGTM is to project tweets into a coherent semantic space by using latent variables via user-contributed hashtags. HGTM provides a robust way for noisy and sparse tweets, which is different from traditional topic models since they normally consider only content information and ignore explicit and potential semantic connection via noisy hashtags. HGTM is a probability generative model that incorporates such weakly-supervised information based on a weighted hash tag graph. The model links tweets via both explicit and potential tweet-hash tag relationship, so that hash tag relationship can connect
- 3. #13/ 19, 1st Floor, Municipal Colony, Kangayanellore Road, Gandhi Nagar, Vellore – 6. Off: 0416-2247353 / 6066663 Mo: +91 9500218218 Website: www.shakastech.com, Email - id: shakastech@gmail.com, info@shakastech.com semantically-related words with or without co-occurrences, which alleviates severe sparse and noise problem in short texts. In this paper, we extend the work and further explore the influence of different hash tag graph construction methods and discuss more details about HGTM, including time complexity analysis and the key process of hash tag assignment analysis. ADVANTAGES OF PROPOSED SYSTEM: We evaluate HGTM on two real-world Twitter data sets to understand different kinds of hash tag graphs and the working of HGTM on extensive tweet mining tasks such as clustering, classification, and topic quality evaluation. Compared to the state-of-the-art methods, HGTM shows the ability of handling the sparseness and noise problem in mining tweets by exploiting both explicit and potential relations between hash tags and tweets. SYSTEM REQUIREMENTS: HARDWARE REQUIREMENTS: System : Pentium Dual Core. Hard Disk : 120 GB. Monitor : 15’’ LED Input Devices : Keyboard, Mouse Ram : 1GB. SOFTWARE REQUIREMENTS: Operating system : Windows 7. Coding Language : JAVA/J2EE Tool : Netbeans 7.2.1 Database : MYSQL