Automated Abstracts and Big Data
2 likes1,196 views
The document discusses automated abstracts, focusing on extracting significant sentences from large data corpora using algorithms like Luhn's and techniques such as tf-idf and NLP. It emphasizes domain-specific summaries that cater to user-defined queries and the importance of leveraging domain knowledge for accurate abstraction. Axiomine's platform utilizes big data technologies for real-time abstract generation and analytics on massive scale data.
Automated Abstracts and Big Data
- 1. Automated Abstracts By - Sameer Wadkar Big Data Architect / Data Scientist December 8th, 2012 © 2012 Axiomine LLC
- 2. Agenda What are Automated Abstracts? Process of Automated Abstracts Extracting significant words Scoring Sentences using Luhn’s algorithm Domain specific abstracts Automated Abstracts on a Massive Data Corpus The Axiomine Platform
- 3. What are Automated Abstracts? • Abstracts comprise of the key sentences in the document • Key challenges • Generate Automated Abstracts on massive Terabyte Scale or Streaming Data • Exploit valuable domain knowledge. • Allow abstracts to be based on user-defined query • If user declares her interest in “Risk”, the abstracts will be focussed around the term “Risk” and its related words. In practice Automated Abstracts is Automated Extracts
- 4. Process of Automated Abstracts Define Corpus Extract Score Generate & Summary significant Sentences per Abstracts Size Criteria words document (Extracts) • Define Document Corpus • Find imp. words in the • Calculate a importance • Pick the top sentences • A Corpus is collection of Corpus score for a sentence based based on score and “Text” documents in digital • Word frequency is the on frequency and co- chosen criteria format simplest measure. location of significant • Define criteria for key • Words like “and”, words. sentence selection. “the” occur • Luhn’s Algorithms Examples include frequently but not • Score of sentences • Top 20 sentences informative depends on relative • Top 5% of the • Likewise very low importance of significant sentences frequency words words. like “preposterous” not informative • Statistical and Natural Language Processing (NLP) offer stronger methods • TF-IDF (Term Freq. - Inverse Doc Freq.) is statistical technique to evaluate word importance • NLP techniques like Parts of Speech Tagging and Named Entity Extraction can be used. Pick sentences based on location & occurrence of important words
- 5. Extracting significant words • Number of times the word occurs is an inadequate measure. • Stop words like “and”, “the” occur frequently but are not important • Very rarely occurring words like, “preposterous” are also not very significant • Pick words often but not too often and also not too rarely • Two popular methods • Statistical measures like TF-IDF can be used • Linguistic methods like Natural Language Processing can be used • Hybrid of Statistical and Linguistic methods Discovery of key words algorithmically is a non-trivial problem
- 6. Extracting significant words- Statistical Technique • TF-IF stands for Term Frequency-Inverse Document Frequency • TF-IDF = Term Frequency * Log(Inverse Document Frequency) • TF = Number of times a word occurs in a corpus • DF = Proportion of documents containing the word 𝟏 • IDF= log( ) 𝑫𝑭 • Pick words with TF-IDF above a predefined threshold • Ex. Consider a News corpus with 10000 news articles Word in TF DF 𝟏 IDF = 𝐥𝐨𝐠( 𝟏 ) TF-IDF 𝑫𝑭 corpus 𝑫𝑭 and 10 million 10000 (all docs) 10K/10K=1 0 0 football 1000 100 10K/100=1000 3 3000 “and” occurs more but “football” is significant TF-IDF combines two conflicting measures into a “significance” score
- 7. Extracting significant words- NLP Techniques • Rules • Sentences containing a proper noun are important. • Sentences containing a place, person, medical / technology term, a custom domain dictionary, are important • Two main techniques • Parts of Speech Tagging • Named Entity Extraction • Parts of Speech Tagging • Identifies grammatical form of the words in the sentence. Is the word a proper noun, noun, adjective, adverb etc. • Named Entity Extraction • Discover from a text of document named entities like “person”, “place”, “medical term”. Try out Calais Viewer • Examples of COTS and Open Source Software – Open Calais, GATE, UIMA, Autonomy Exploit your domain knowledge - No glory in full automation
- 8. Sentence Scoring (Luhn’s Algorithm) • Find a cluster of important words in a sentence. For a cluster to be formed important words have to be within a pre-specified number of words of each other. Ex. 3 • Score each cluster and use cluster scores to score the sentence All bolded words are All significant words within 1 “discovered” to be word of each other significant words in a medical corpus A 15-year-old liver transplant patient is the first person in the world to take on the immune system and blood type of her donor. “patient” and “immune” within 12 words of each other. Hence different All significant words within a maximum of 3 words of each other clusters in 1 sentence. Important sentences have important words close together
- 9. Scoring Sentences • Sample Scoring Criteria • Cluster Score = (No of Significant Words)2/(No of words in the cluster) • Sentence Score = Max of all cluster scores for the given sentence • Pick to N or N% of sentences for the abstract Phrase No of Significant No of words in Cluster Score Words in cluster cluster Liver transplant 3 3 (3)2/3=3 patient immune system 6 9 (6)2/9=4 and blood type of her donor Sentence Score = Max(3,4) 4 All words have same weight. Limitation(?) or Opportunity(!)
- 10. Domain Specific Abstracts • Give each significant word a different weight during cluster • We can get Domain/Query specific abstracts! • Ex. In the previous example, if we wanted abstracts related to “Liver Transplants”, we would weigh the words “Liver” and “Transplant” higher (Ex. 5 vs.1 for the rest) Phrase Weight of Weight of all words Cluster Score Significant Words in cluster Liver transplant 5+5+1=11 5+5+1 (11)2/11=11 patient immune system 6*1=6 9*1=9 (6)2/9=4 and blood type of her donor Sentence Score = Max(11,4) 11 Sentences with words “liver” or “transplant” will get weighed higher now. Abstracting process is not a black box - The user & domain can drive it
- 11. Examples of Domain Specific Abstracts • Imagine a large Project Review Document • Find the Project Risk Summary (Give more weight to words related to “Risk”) • Find the Project Execution Summary (Give more weight to words related to Project Management) • Imagine a Medical Corpus • Find sentences to “Transplant” and “Grafting” procedures • Find sentences related to “Heart Surgery” (Provides more weight to words like “Cardiac”, “Heart”, “Cardiovascular”, etc. Domain dictionaries and expert knowledge improve abstracts
- 12. Automated Abstracts on Big Data Scale (Process) Large TF-IDF Document MapReduce Corpus process Weighing Significant Rules Words Named Entity Extraction MapReduce process Automated Abstracts Document MapReduce Abstracts process Domain Knowledge Abstracts generation techniques work well with MapReduce technique
- 13. What Axiomine can do? • At Axiomine we have developed methods to • Generate abstracts on a massive scale. • Generate abstracts on new documents in real-time • Allow incorporation of domain knowledge in real-time • We utilize various Big Data Technologies • Natural Language Processing on Hadoop • Real time NLP using General Purpose Graphics Programming (GPGPU) using NVIDIA graphics chips At Axiomine we handle large scale Text Analytics
- 14. Intuitive Insights Information Access Platform Integration platform for diverse data sources comprising of Structured and Unstructured Data Intuitively navigate Big Data Corpus at the Speed of Thought Methodology and Implementation to perform Topic Modeling on Massive Text Corpora A high fidelity algorithm to estimate Document Similarity based on results of Topic Modeling Develop Automated Domain Specific Abstracts in Real Time Business Intelligence Layer that can query Terabyte scale corpuses in Real-Time Axiomine’s I3AP supports access to unlimited data at the speed of thought
- 15. Q.E.D 15