Concepts and Challenges of Text Retrieval for Search Engine
1 like1,312 views
This document discusses concepts and challenges in text retrieval for search engines. It provides an overview of text retrieval and search engine concepts. Some key challenges discussed are semantics and specificity in queries. The document also uses an example of an expert search engine to illustrate a case study. It describes various components involved in text retrieval including document representation, indexing, inverted indexing, retrieval functions and evaluation metrics.
Concepts and Challenges of Text Retrieval for Search Engine
- 1. CONCEPTS AND CHALLENGES OFTEXT RETRIEVAL FOR SEARCH ENGINES PRE CONFERENCETUTORIAL by Gan Keng Hoon 16th August 2016 1
- 2. THISTUTORIAL Overview:Text Retrieval & Search Engine Concept : Basics ofText Retrieval Challenges: Semantics & Specific Case: Expert Search Engine 2
- 3. Search 3
- 4. What Do People Search for? FuYuanhui How to get free Pokeball ? How to write thesis in three month ? keynote speaker ICAICTA 2016 4
- 5. What Do People Expect ? How to get free Pokeball 5
- 7. Quiz:Which one is not a Search Engine? 7
- 8. Type of Search Engine Web Search Engine Google,Yahoo, Bing Domain Specific Search Engine Medline/Pubmed Microsoft Academic Desktop Search Engine Copernic 8
- 9. ConnectingTwo Ends Search Collection Web Domain Specific Personal Enterprise Etc. Information Needs I want to know more about the keynotes speech of ICAICTA 2016. I need more Pokeballs Free Of Charge..… What’s so funny about FuYuan Hui?? Scholarship ending soon, three months left to submit my thesis…. Web Sites Journal Articles News Images Videos Audio Scanned Documents Tweets Posts Reviews Etc… 9
- 10. A Conceptual Model forText Retrieval Information Needs Query Search Collection Document Representation Retrieved Documents Indexing Formulation Retrieval Function Relevance Feedback Natural Language Content Analysis 10
- 11. Natural Language Content Analysis 11
- 12. SearchCollection (Retrieval Unit) Web pages, email, books, news stories, scholarly papers, text messages,Word™, Powerpoint™, PDF, forum postings, patents, etc. Retrieval unit can be Part of document, e.g. a paragraph, a slide, a page etc. In the form different structure, html, xml, text etc. In different sizes/length. 12
- 13. Document Representation FullText Representation Keep everything. Complete. Require huge resources.Too much may not be good. Reduced (partial) Content Representation Remove not important contents e.g. stopwords. Standardization to reduce overlapped contents e.g. stemming. Retain only important contents, e.g. noun phrases, header etc. 13
- 14. Document Representation Think of representation as some ways of storing the document. Bag of Words Model Store the words as the bag (multiset) of its words, disregarding grammar and even word order. Document 1: "The cat sat on the hat" Document 2: "The dog ate the cat and the hat" From these two documents, a word list is constructed: { the, cat, sat, on, hat, dog, ate, and } The list has 8 distinct words. Document 1: { 2, 1, 1, 1, 1, 0, 0, 0 } Document 2 : { 3, 1, 0, 0, 1, 1, 1, 1} 14
- 15. Information Needs & Query Information Needs != Query Recall the information needs Query: icaicta 2016 keynote Information Need: I want to know more about the keynotes speech of ICAICTA 2016 Query: free pokeball Information Need: I need more Pokeballs. I don’t want to pay. No cheat codes. 15
- 16. Retrieved Documents From the original collection, a subset of documents are obtained. What is the factor that determines what document to return? SimpleTerm Matching Approach 1. Compare the terms in a document and query. 2. Compute “similarity” between each document in the collection and the query based on the terms they have in common. 3. Sorting the document in order of decreasing similarity with the query. 4. The outputs are a ranked list and displayed to the user - the top ones are more relevant as judged by the system. 16
- 17. Indexing Convert documents into representation or data structure to improve the efficiency of retrieval. To generate a set of useful terms called indexes. Why? Many variety of words used in texts, but not all are important. Among the important words, some are more contextually relevant. Some basic processes involved •Tokenization •StopWords Removal •Stemming •Phrases •Inverted File 17
- 18. Indexing (Tokenization) Convert a sequence of characters into a sequence of tokens with some basic meaning. “The cat chases the mouse.” “Bigcorp's 2007 bi-annual report showed profits rose 10%.” the cat chases the mouse bigcorp 2007 bi annual report showed profits rose 10% 18
- 19. Indexing (Tokenization) Token can be single or multiple terms. “Samsung Galaxy S7 Edge, redefines what a phone can do.” samsung galaxy s7 edge redefines what a phone can do samsung galaxy s7 edge redefines what a …. or 19
- 20. Indexing (Tokenization) Common Issues 1. Capitalized words can have different meaning from lower case words Bush fires the officer. Query: Bush fire The bush fire lasted for 3 days. Query: bush fire 2. Apostrophes can be a part of a word, a part of a possessive, or just a mistake rosie o'donnell, can't, don't, 80's, 1890's, men's straw hats, master's degree, england's ten largest cities, shriner's 20
- 21. Indexing (Tokenization) 3. Numbers can be important, including decimals nokia 3250, top 10 courses, united 93, quicktime 6.5 pro, 92.3 the beat, 288358 4. Periods can occur in numbers, abbreviations, URLs, ends of sentences, and other situations I.B.M., Ph.D., cs.umass.edu, F.E.A.R. Note: tokenizing steps for queries must be identical to steps for documents 21
- 22. Indexing (Stopping) Top 50 Words from AP89 News Collection Recall, Indexes should be useful term links to a document. Are the terms on the right figure useful? 22
- 23. Indexing (Stopping) Stopword list can be created from high-frequency words or based on a standard list Lists are customized for applications, domains, and even parts of documents e.g., “click” is a good stopword for anchor text Best policy is to index all words in documents, make decisions about which words to use at query time? 23
- 24. Indexing (Stemming) Many morphological variations of words inflectional (plurals, tenses) derivational (making verbs nouns etc.) In most cases, these have the same or very similar meanings Stemmers attempt to reduce morphological variations of words to a common stem usually involves removing suffixes Can be done at indexing time or as part of query processing (like stopwords) 24
- 25. Indexing (Stemming) Porter Stemmer Algorithmic stemmer used in IR experiments since the 70s Consists of a series of rules designed to the longest possible suffix at each step Produces stems not words Example Step 1 (right figure) 25
- 26. Indexing (Phrases) Recall, token, meaningful tokens are better indexes, e.g. phrases. Text processing issue – how are phrases recognized? Three possible approaches: Identify syntactic phrases using a part-of-speech (POS) tagger Use word n-grams Store word positions in indexes and use proximity operators in queries 26
- 27. Indexing (Phrases) Example Noun Phrases * Other method like N-Gram 27
- 28. Indexing (Inverted Index) Recall, indexes are designed to support search. Each index term is associated with an inverted list Contains lists of documents, or lists of word occurrences in documents, and other information. Each entry is called a posting. The part of the posting that refers to a specific document or location is called a pointer Each document in the collection is given a unique number Lists are usually document-ordered (sorted by document number) 28
- 29. Indexing (Inverted Index) Sample collection. 4 sentences fromWikipedia entry for Tropical Fish 29
- 30. Indexing (Inverted Index) Simple inverted index. 30
- 31. Indexing (Inverted Index) Inverted index with counts. Support better ranking algorithms. 31
- 32. Indexing (Inverted Index) Inverted index with positions. Support proximity matching. 32
- 33. Retrieval Function Ranking Documents are retrieved in sorted order according to a score computing using the document representation, the query, and a ranking algorithm 33
- 34. Retrieval Function (Vector Space Model) Ranked based method. Documents and query represented by a vector of term weights. Collection represented by a matrix of term weights. 34
- 35. Retrieval Function (Vector Space Model) borneo daily new north straits times D1 0 0 1 0 1 1 D2 0 1 1 0 1 0 D3 1 0 0 1 0 1 D1: new straits times D2: new straits daily D3 : north borneo times Vector of useful terms 35
- 36. Retrieval Function (Vector Space Model) borneo daily new north straits times D1 0 0 0.176 0 0.176 0.176 D2 0 0.477 0.176 0 0.176 0 D3 0.477 0 0 0.477 0 0.176 idf (borneo) = log(3/1) =0.477 idf (daily) = log(3/1) = 0.477 idf (new) = log(3/2) =0.176 idf (north) = log(3/1) = 0.477 idf (straits) = log(3/2) = 0.176 idf (times) = log(3/2) = 0.176 then multiply by tf tf.idf weight Term frequency weight measures importance in document: Inverse document frequency measures importance in collection: Note: Doc Length,Term Location,Term Semantic Meaning 36
- 37. Retrieval Function (Vector Space Model) Documents ranked by distance between points representing query and documents Similarity measure more common than a distance or dissimilarity measure e.g. Cosine correlation 37
- 38. Retrieval Function (Vector Space Model) Consider two documents D1, D2 and a query Q Q = “straits times” Compare against collection, D1 = “new straits times” (borneo, daily, new, north, straits, times) Q = (0, 0, 0, 0, 0.176, 0.176) D1 = (0, 0, 0.176, 0, 0.176, 0.176) D2 = (0, 0.477, 0.176, 0, 0.176, 0) 𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶 𝐷𝐷𝐷, 𝑄𝑄 = 0∗0 + 0∗0 + 0.176∗0 + 0∗0 + 0.176∗0.176 +(0.176∗0.176) 0.1762 +0.1762 +0.1762 (0.1762 +0.1762 ) =0.816 Find Cosine (D2,Q). Which document is more relevant? 38
- 39. Evaluation A must to evaluate the retrieval function, preprocessing steps etc. StandardCollection Task specific Human experts are used to judge relevant results. Performance Metric Precision Recall 39
- 40. Evaluation (Collection) Test collections consisting of documents, queries, and relevance judgments, e.g., 40
- 41. Evaluation (Collection) Example query and narrative for golden standard. 41
- 42. Evaluation (Effectiveness Measures) A is set of relevant documents, B is set of retrieved documents 42
- 44. Evaluation (Ranking Effectiveness) Recall@4 = 3/4 Precision@4 = 3/4 Recall@2 = 2/4 Precision@2 = 2/2 44
- 45. Challenges SocialTexts, e.g.Tweets, Posts Hard question. Hard Disk ? Named Entity Various levels and aspects of annotations 45
- 46. Challenges Small Data Specific search Improve semantics extensively Big Data Multi modal retrieval Connecting many medias 46
- 47. Case: Adding Semantics Bibliography
- 48. Improve Search Results Display Facet-based semantic UsefulTerms Demo: ir.cs.usm.my