Tweet segmentation and its application to named entity recognition
Download as DOCX, PDF1 like625 views
The document presents a novel framework called HybridSeg for tweet segmentation to enhance named entity recognition in tweets, addressing issues caused by their noisy and informal nature. The proposed system utilizes both global and local contexts to improve segmentation quality and effectiveness in extracting semantic information. Experiments show that local linguistic features provide more reliable support for segmentation compared to term-dependency models.
Tweet segmentation and its application to named entity recognition
- 1. 1. #13/ 19, 1st Floor, Municipal Colony, Kangayanellore Road, Gandhi Nagar, vellore – 6. Off: 0416-2247353 / 6066663 Mo: +91 9500218218 /8870603602, Project Titles: http://shakastech.weebly.com/2015-2016-titles Website: www.shakastech.com, Email - id: shakastech@gmail.com, info@shakastech.com TWEET SEGMENTATION AND ITS APPLICATION TO NAMED ENTITY RECOGNITION ABSTRACT: Twitter has attracted millions of users to share and disseminate most up-to-date information, resulting in large volumes of data produced everyday. However, many applications in Information Retrieval (IR) and Natural Language Processing (NLP) suffer severely from the noisy and short nature of tweets. In this paper, we propose a novel framework for tweet segmentation in a batch mode, called HybridSeg. By splitting tweets into meaningful segments, the semantic or context information is well preserved and easily extracted by the downstream applications. HybridSeg finds the optimal segmentation of a tweet by maximizing the sum of the stickiness scores of its candidate segments. The stickiness score considers the probability of a segment being a phrase in English (i.e., global context) and the probability of a segment being a phrase within the batch of tweets (i.e., local context). For the latter, we propose and evaluate two models to derive local context by considering the linguistic features and term-dependency in a batch of tweets, respectively. HybridSeg is also designed to iteratively learn from confident segments as pseudo feedback. Experiments on two tweet data sets show that tweet segmentation quality is significantly improved by learning both global and local contexts compared with using global context alone. Through analysis and comparison, we show that local linguistic features are more reliable for learning local context compared with term-dependency. As an application, we show that high accuracy is achieved in named entity recognition by applying segment-based part-of-speech (POS) tagging.
- 2. 1. #13/ 19, 1st Floor, Municipal Colony, Kangayanellore Road, Gandhi Nagar, vellore – 6. Off: 0416-2247353 / 6066663 Mo: +91 9500218218 /8870603602, Project Titles: http://shakastech.weebly.com/2015-2016-titles Website: www.shakastech.com, Email - id: shakastech@gmail.com, info@shakastech.com EXISTING SYSTEM: Many existing NLP techniques heavily rely on linguistic features, such as POS tags of the surrounding words, word capitalization, trigger words (e.g., Mr., Dr.), and gazetteers. These linguistic features, together with effective supervised learning algorithms (e.g., hidden markov model (HMM) and conditional random field (CRF)), achieve very good performance on formal text corpus. However, these techniques experience severe performance deterioration on tweets because of the noisy and short nature of the latter. In Existing System, to improve POS tagging on tweets, Ritter et al. train a POS tagger by using CRF model with conventional and tweet-specific features. Brown clustering is applied in their work to deal with the ill-formed words. DISADVANTAGES OF EXISTING SYSTEM: Given the limited length of a tweet (i.e., 140 characters) and no restrictions on its writing styles, tweets often contain grammatical errors, misspellings, and informal abbreviations. The error-prone and short nature of tweets often make the word-level language models for tweets less reliable. PROPOSED SYSTEM: In this paper, we focus on the task of tweet segmentation. The goal of this task is to split a tweet into a sequence of consecutive n-grams, each of which is called a segment. A segment can be a named entity (e.g., a movie title “finding nemo”), a semantically meaningful information unit (e.g., “officially released”), or any other types of phrases which appear “more than by chance”
- 3. 1. #13/ 19, 1st Floor, Municipal Colony, Kangayanellore Road, Gandhi Nagar, vellore – 6. Off: 0416-2247353 / 6066663 Mo: +91 9500218218 /8870603602, Project Titles: http://shakastech.weebly.com/2015-2016-titles Website: www.shakastech.com, Email - id: shakastech@gmail.com, info@shakastech.com To achieve high quality tweet segmentation, we propose a generic tweet segmentation framework, named HybridSeg. HybridSeg learns from both global and local contexts, and has the ability of learning from pseudo feedback. Global context. Tweets are posted for information sharing and communication. The named entities and semantic phrases are well preserved in tweets. Local context. Tweets are highly time-sensitive so that many emerging phrases like “She Dancin” cannot be found in external knowledge bases. However, considering a large number of tweets published within a short time period (e.g., a day) containing the phrase, it is not difficult to recognize “She Dancin” as a valid and meaningful segment. We therefore investigate two local contexts, namely local linguistic features and local collocation. ADVANTAGES OF PROPOSED SYSTEM: Our work is also related to entity linking (EL). EL is to identify the mention of a named entity and link it to an entry in a knowledge base like Wikipedia. Through our framework, we demonstrate that local linguistic features are more reliable than term-dependency in guiding the segmentation process. This finding opens opportunities for tools developed for formal text to be applied to tweets which are believed to be much more noisy than formal text. Helps in preserving Semantic meaning of tweets. ALGORITHM EXPLANATION: As an application of tweet segmentation, we propose and evaluate two segment-based NER algorithms. Both algorithms are unsupervised in nature and take tweet segments as input. One algorithm exploits co-occurrence of named entities in targeted Twitter streams by applying random walk (RW) with the assumption that named entities are more likely to co-occur together.
- 4. 1. #13/ 19, 1st Floor, Municipal Colony, Kangayanellore Road, Gandhi Nagar, vellore – 6. Off: 0416-2247353 / 6066663 Mo: +91 9500218218 /8870603602, Project Titles: http://shakastech.weebly.com/2015-2016-titles Website: www.shakastech.com, Email - id: shakastech@gmail.com, info@shakastech.com The other algorithm utilizes Part-of-Speech (POS) tags of the constituent words in segments. SYSTEM ARCHITECTURE: SYSTEM REQUIREMENTS: HARDWARE REQUIREMENTS: System : Pentium IV 2.4 GHz. Hard Disk : 40 GB. Floppy Drive : 1.44 Mb. Monitor : 15 VGA Colour. Mouse : Logitech. Ram : 512 Mb. SOFTWARE REQUIREMENTS: Operating system : Windows XP/7. Coding Language : JAVA/J2EE IDE : Netbeans 7.4 Database : MYSQL