Boolean Retrieval
- 2. Information retrieval is defined,,, To find material (usually documents) of an unstructured nature (usually text) that satisfies an information need from within large collections (usually stored on computers). The field of information retrieval also covers supporting users in browsing or filtering document collections or further processing a set of retrieved documents. Ex) Given a set of documents, clustering is the task of coming up with a good grouping of the documents based on their contents. It is similar to arranging books on a bookshelf according to their topic. Given a set of topics, standing information needs, or other categories
- 3. Information retrieval systems can also be distinguished by the scale at which they operate, and it is useful to distinguish three prominent scales. 1. In web search the system has to provide search over billions of documents stored on millions of computers. 2. personal information retrieval 3. enterprise, institutional, and domain-specific search for collections such as a corporation's internal documents, a database of patents, or research articles on biochemistry.
- 4. RETRIEVAL example grepping simple retrieval sort of linear scan through documents With modern computers, for simple querying of modest collections , you really need nothing more. But for many purposes, you do need more: 1. To process large document collections quickly 2. To allow more flexible matching operations 3. To allow ranked retrieval
- 5. The way to avoid linearly scanning the texts for each query is to index the documents in advance
- 6. term-document incidence matrix
- 7. To answer the query Brutus AND Caesar AND NOT Calpurnia, we take the vectors for Brutus, Caesar and Calpurnia, complement the last, and then do a bitwise AND: 110100 AND 110111 AND 101111 = 100100 The answers for this query are thus Anthony and Cleopatra and Hamlet
- 8. two key statistics Precision : What fraction of the returned results are relevant to the information need? Recall : What fraction of the relevant documents in the collection were returned by the system?
- 9. We now cannot build a term-document matrix in a naive way. A 500k * 1 M matrix has half-a-trillion 0's and 1's - too many to fit in a computer's memory. A much better representation is to record only the things that do occur, that is, the 1 positions. it is the inverted index
- 10. dictionary we use dictionary for the data structure and vocabulary for the set of terms this is commonly kept in memory posting for each term, we have a list that records which documents the term occurs in. Each item in the list - which records that a term appeared in a document (and, later, often, the positions in the document) this Is stored on disk
- 11. To gain the speed benefits of indexing at retrieval time, we have to build the index in advance 1. Collect the documents to be indexed: 2. Tokenize the text, turning each document into a list of tokens: 3. Do linguistic preprocessing, producing a list of normalized tokens, which are the indexing terms: 4.Index the documents that each term occurs in by creating an inverted index, consisting of a dictionary and postings.
- 12. ・ The sequence of terms in each document is sorted alphabetically ・ Instances of the same term are then grouped by word and then by documentID ・ The terms and documentID are then separated out ・ The dictionary stores the terms, and has pointer to the postings list for each term
- 13. Processing Boolean queries simple conjunctive query Locate Brutus in the Dictionary Retrieve its postings Locate Calpurnia in the Dictionary Retrieve its postings Intersect the two postings lists The intersection is the crucial one: we need to efficiently intersect postings lists so as to be able to quickly find documents that contain both terms.
- 14. Algorithm for the intersection of two postings lists p1 and p2.
- 16. four additional things we would like to be able to do 1 We would like to better determine the set of terms in the dictionary and to provide retrieval that is tolerant to spelling mistakes and inconsistent choice of words. 2 “operating system “ ,” Gates near Microsoft” To answer such queries , the index has to be augmented to capture the proximities of terms in documents. 3 to be able to accumulate evidence, we need term frequency information in postings lists. 4 Boolean queries just retrieve a set of matching documents, but commonly we wish to have an effective method to order (or ``rank'') the returned results. This requires having a mechanism for determining a document score which encapsulates how good a match a document is for a query.