UNDERSTAND SHORTTEXTS BY HARVESTING & ANALYZING SEMANTIKNOWLEDGE
Download as DOCX, PDF1 like347 views
The document proposes a system for understanding short texts by exploiting semantic knowledge from external knowledge bases and web corpora. It discusses challenges in analyzing short texts using traditional NLP methods due to their ambiguous and non-standard language. The system addresses this by harvesting semantic knowledge and applying knowledge-intensive approaches to tasks like segmentation, POS tagging and concept labeling to better understand semantics. An evaluation shows these knowledge-based methods are effective and efficient for short text understanding.
UNDERSTAND SHORTTEXTS BY HARVESTING & ANALYZING SEMANTIKNOWLEDGE
- 1. UNDERSTAND SHORTTEXTS BY HARVESTING & ANALYZING SEMANTIKNOWLEDGE ABSTRACT: Understanding short texts is crucial to many applications, but challenges abound. First, short texts do not always observe the syntax of a written language. As a result, traditional natural language processing tools, ranging from part-of-speech tagging to dependency parsing, cannot be easily applied. Second, short texts usually do not contain sufficient statistical signals to support many state-of-the-art approaches for text mining such as topic modeling. Third, short texts are more ambiguous and noisy, and are generated in an enormous volume, which further increases the difficulty to handle them. We argue that semantic knowledge is required in order to better understand short texts. In this work, we build a prototype system for short text understanding which exploits semantic knowledge provided by a well-known knowledgebase and automatically harvested from a web corpus. Our knowledge-intensive approaches disrupt traditional methods for tasks such as text segmentation, part-of-speech tagging, and concept labeling, in the sense that we focus on semantics in all these tasks. We conduct a comprehensive performance evaluation on real-life data. The results show that semantic knowledge is indispensable for short text understanding, and our knowledge- intensive approaches are both effective and efficient in discovering semantics of short texts. ARCHITECTURE DIAGRAM:
- 2. EXISTING SYSTEM: Many problems in natural language processing, data mining, information retrieval, and bioinformatics can be formalized as string transformation, which is a task as follows. Given an input string, the system generates the k most likely output strings corresponding to the input string. This paper proposes a novel and probabilistic approach to string transformation, which is both accurate and efficient. The approach includes the use of a log linear model, a method for training the model, and an algorithm for generating the top k candidates, whether there is or is not a predefined dictionary. The log linear model is defined as a conditional probability distribution of an output string and a rule set for the transformation conditioned on an input string. The learning method employs maximum likelihood estimation for parameter estimation. The string generation algorithm based on pruning is guaranteed to generate the optimal top k candidates. The proposed method is applied to correction of spelling errors in queries as well as reformulation of queries in web search. Experimental results on
- 3. large scale data show that the proposed approach is very accurate And efficient improving upon existing methods in terms of accuracy and efficiency in different settings. PROPOSED SYSTEM: Understanding short texts is crucial to many applications, but challenges abound. First, short texts do not always observe the syntax of a written language. As a result, traditional natural language processing methods cannot be easily applied. Second, short texts usually do not contain suffi cient statistical signals to support many state-of-the-art approaches for text processing such as topic modeling. Third, short texts are usually more ambiguous. We argue that knowledge is needed in order to better understand short texts. In this work, we use lexicalsemantic knowledge provided by a well-known semantic network for short text understanding. Our knowledge-intensive approach disrupts traditional methods for tasks such as text segmentation, part-of-speech tagging, and concept labeling, in the sense that we focus on semantics in all these tasks. We conduct a comprehensive performance evaluation on real-life data. The results show that knowledge is indispensable for short text understanding, and our knowledge- intensive approaches are effective in harvesting semantics of short texts. ADVANTAGES: • user can search realated words • view chart based on mostword searching
- 4. SYSTEM CONFIGURATION: HARDWARE REQUIREMENTS: Hardware - Pentium Speed - 1.1 GHz RAM - 1GB Hard Disk - 20 GB Key Board - Standard Windows Keyboard Mouse - Two or Three Button Mouse Monitor - SVGA SOFTWARE REQUIREMENTS: Operating System : Windows Technology : Java and J2EE Web Technologies : Html, JavaScript, CSS IDE : My Eclipse Web Server : Tomcat Database : My SQL Java Version : J2SDK1.8