20100528 distributedinformationretrieval crestani_lecture01-02
0 likes498 views
This document provides an overview of distributed information retrieval (DIR) and is presented by Prof. Fabio Crestani. It covers the background of information retrieval (IR) including indexing, querying, IR models and evaluation. The document then outlines topics in DIR including introduction, architectures, broker-based systems, evaluation and applications of DIR.
![Background
DIR: Introduction
Resource Description
DIR Architectures
Resource Selection
Broker-Based DIR
Results Merging
DIR Evaluation
Applications of DIR
Capture-Recapture
The idea
X - event that a randomly sampled document is already in a
sample
Y - number of X in n trials
Two samples S1 and S2
|S| |S|
E[X ] = , E[Y ] = n · E[X ] = n ·
|C | |C |
|S1 ||S2 | ˆ |S1 ||S2 |
|S1 ∩ S2 | ≈ =⇒ |C | =
|C | |S1 ∩ S2 |
Prof. Fabio Crestani Distributed Information Retrieval 69](https://image.slidesharecdn.com/20100528distributedinformationretrievalcrestanilecture01-02-100818052600-phpapp01/85/20100528-distributedinformationretrieval-crestani_lecture01-02-69-320.jpg)
![Rv
Rv
0.4 Background 0.4
DIR: Introduction
Resource Description
0.2 DIR Architectures 0.2
Resource Selection
Broker-Based DIR
Results Merging
0.0 DIR Evaluation 0.0
0 5
Applications of DIR
10 15 20 0 5 10 15 20
Cutoff (nonrelevant) Cutoff (GOV2)
Resource Selection Comparison
Fig. 3. R values for the cori, redde and crcs algorithms on the trec123-FR-DOE-
81col (left) and 100-col-gov2 (right) testbeds.
Table 2. Performance of different methods for the Trec4 (trec4-kmeans) testbed.
Trec topics 201–250 (long) were used as queries
Cutoff=1 Cutoff=5
P@5 P@10 P@15 P@20 P@5 P@10 P@15 P@20
CORI 0.3000 0.2380 0.2133† 0.1910† 0.3480 0.2980 0.2587 0.2380
ReDDE 0.2160 0.1620 0.1373 0.1210 0.3480 0.2860 0.2467 0.2190
CRCS(l) 0.2960 0.2260 0.2013 0.1810† 0.3520 0.2920 0.2533 0.2310
CRCS(e) 0.3080 0.2400 0.2173† 0.1910† 0.3880 0.3160 0.2680 0.2510
Table 3. Performance of collection selection methods for the uniform (trec123-100col-
bysource) testbed. Trec topics 51–100 (short) were used as queries
evance judgments. We only report the results for cutoff=1 and cutoff=5. The
former shows the system outputs when only the best collection is selected while
Cutoff=1 Cutoff=5
for the latter, P@5 P@10collections are chosen to P@10 P@15for the query.
the best five P@15 P@20 P@5 get searched P@20
We do not report the results for larger cutoff values because 0.2867 0.2730 shown
CORI 0.2520 0.2140 0.1960 0.1710 0.3080 0.3060 cutoff=5 has
to beReDDE 0.1920threshold0.1413 0.1280
a reasonable 0.1660 for dir experiments on the real web collections
0.2960 0.2820 0.2653 0.2510
[Avrahami et al., 2006]. The 0.1520‡values † 0.3440 calculated precision on the
CRCS(l) 0.2120 0.1760 P @x 0.1330show the 0.3240 0.3067 0.2860
‡ ‡
top xCRCS(e) 0.3800 0.3060 0.2613 0.2260
results. 0.3960 0.3700† 0.3480† 0.3310†
We select redde as the baseline as it does not require training queries and
Table 4. Performance of collection selection methods for the representative (trec123-
its effectiveness is found to be higher than cori and older alternatives [Si and
2ldb-60col) testbed. Trec topics 51–100 (short) were used as queries
Callan, 2003a]. The following tables compare the performance of discussed meth-
Prof. Fabio Crestani Distributed Information Retrieval 85](https://image.slidesharecdn.com/20100528distributedinformationretrievalcrestanilecture01-02-100818052600-phpapp01/85/20100528-distributedinformationretrieval-crestani_lecture01-02-85-320.jpg)
![Background
DIR: Introduction
Resource Description
DIR Architectures
Resource Selection
Broker-Based DIR
Results Merging
DIR Evaluation
Applications of DIR
SSL with Identical Retrieval Models
The idea
SSL uses documents found in CSI to train a single regression model to
estimate the normalised score (Di,j ) from resource document scores (Di,j )
and the score of the same document computed from the CSI (Ei,j ).
Normalised scores
Having:
D1,1 C1 D1,1 E1,1
D1,2 C1 D1,2 E1,2
× [a b] =
... ... ...
Dn,m Cn Dn,m En,m
Train:
Di,j = a ∗ Ei,j + b ∗ Ei,j ∗ Ci
Prof. Fabio Crestani Distributed Information Retrieval 95](https://image.slidesharecdn.com/20100528distributedinformationretrievalcrestanilecture01-02-100818052600-phpapp01/85/20100528-distributedinformationretrieval-crestani_lecture01-02-95-320.jpg)
![Background
DIR: Introduction
Resource Description
DIR Architectures
Resource Selection
Broker-Based DIR
Results Merging
DIR Evaluation
Applications of DIR
SSL with Different Retrieval Models
The idea
SSL uses documents found in CSI to train a different regression
models for each resource.
Normalised scores
Having:
D1,1 1 E1,1
D1,2 1 E1,2
. . . . . . × [ai bi ] =
...
Dn,m 1 En,m
Train:
Di,j = ai ∗ Ei,j + bi
Prof. Fabio Crestani Distributed Information Retrieval 96](https://image.slidesharecdn.com/20100528distributedinformationretrievalcrestanilecture01-02-100818052600-phpapp01/85/20100528-distributedinformationretrieval-crestani_lecture01-02-96-320.jpg)
![Background
DIR: Introduction
Objectives
DIR Architectures
Test Collections
Broker-Based DIR
Evaluation Metrics
DIR Evaluation
Applications of DIR
Datasets available
78 Federated search testbeds
Table 6.1 Testbed statistics.
# docs (×1000) Size (MB)
Testbed Size (GB) Min Avg Max Min Avg Max
trec123-100col-bysource 3.2 0.7 10.8 39.7 28 32 42
trec4-kmeans 2.0 0.3 5.7 82.7 4 20 249
trec-gov2-100col 110.0 32.6 155.0 717.3 105 1 126 3 891
SYM236 & UDC236. SYM236 [French et al., 1998; 1999; Powell,
2001; Powell and French, 2003] includes 236 collections of varying sizes,
and is generated from documents on TREC disks 1–4 [Harman, 1994;
1995]. UDC236 [French et al., 1999; Powell and French, 2003], also con-
tains 236 collections, and is generated from the same set of documents
(i.e., TREC disks 1–4). The difference is only in the methodology used
for assigning documents to collections. In UDC236, each collection con-
tains almost the same number of documents; in SYM236, documents
are distributed between collections according to their publication date,
Prof. Fabio Crestani 2
Distributed Information Retrieval 109](https://image.slidesharecdn.com/20100528distributedinformationretrievalcrestanilecture01-02-100818052600-phpapp01/85/20100528-distributedinformationretrieval-crestani_lecture01-02-109-320.jpg)
20100528 distributedinformationretrieval crestani_lecture01-02
- 1. Background DIR: Introduction DIR Architectures Broker-Based DIR DIR Evaluation Applications of DIR Distributed Information Retrieval Prof. Fabio Crestani Faculty of Informatics University of Lugano, Switzerland fabio.crestani@usi.ch Prof. Fabio Crestani Distributed Information Retrieval 1
- 2. Background DIR: Introduction DIR Architectures Broker-Based DIR DIR Evaluation Applications of DIR Acknowledgements Large part of this material were presented at the Tutorial on Distributed Information Retrieval at ECIR 2010, in Milton Keynes, UK. The tutorial was presented jointly with Ilya Markov, a PhD student at the University of Lugano, whose help was invaluable in the preparation of this material. Prof. Fabio Crestani Distributed Information Retrieval 2
- 3. Background DIR: Introduction DIR Architectures Broker-Based DIR DIR Evaluation Applications of DIR Objectives of the Course This short course aims at providing the listener with an overview of the area of research of Distributed Information Retrieval (DIR). Highlighting the key elements of this technology and showing how the methods developed in the context of DIR can be applied also in other areas of research. A set of references accompanies this material to help deepen the knowledge on specific aspects of the topics covered. Prof. Fabio Crestani Distributed Information Retrieval 3
- 4. Background DIR: Introduction DIR Architectures Broker-Based DIR DIR Evaluation Applications of DIR Topics covered in this Course 1 Background 2 DIR: Introduction 3 DIR Architectures 4 Broker-Based DIR 5 DIR Evaluation 6 Applications of DIR Prof. Fabio Crestani Distributed Information Retrieval 4
- 5. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR Outline 1 Background 2 DIR: Introduction 3 DIR Architectures 4 Broker-Based DIR 5 DIR Evaluation 6 Applications of DIR Prof. Fabio Crestani Distributed Information Retrieval 5
- 6. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR Background needed DIR is a subarea of research of Information Retrieval (IR). It shares with IR the objectives and much of the underlying technology for indexing and retrieving information. So, it is not possible to understand DIR without some background knowledge of IR. The objectives of this part of the course is to quickly get you up to speed with the main concepts of IR. Prof. Fabio Crestani Distributed Information Retrieval 6
- 7. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR Topics Covered 1 Background Information Retrieval Indexing Querying IR Models Evaluation Prof. Fabio Crestani Distributed Information Retrieval 7
- 8. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR What is Information Retrieval? Search on the Web is a daily activity for many people throughout the world Applications involving search are everywhere The field of computer science that is most involved with R&D of search technology is Information Retrieval (IR) A definition: ”Information retrieval is a field concerned with the structure, analysis, organisation, storage, searching, and retrieval of information.” (Salton, 1968) Primary focus of IR since the 50s has been on text and textual documents Prof. Fabio Crestani Distributed Information Retrieval 8
- 9. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR What is a Document? A document in IR can be anything with some semantic content and (maybe) with some structure Documents are very different from database records The core issue of IR is: comparing the query text to the documents’ text and determining what is a good match The exact match of words is not sufficient: there are many different ways to write the same thing in a natural language Prof. Fabio Crestani Distributed Information Retrieval 9
- 10. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR IR Tasks IR deals with many different types of content, applications and tasks. In particular, the most well known tasks are: Ad-hoc search: find relevant documents for an arbitrary text query Filtering: identify new documents relevant to user profiles Classification: identify relevant labels for documents Question answering: give a specific answer to a question Prof. Fabio Crestani Distributed Information Retrieval 10
- 11. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR The Big Issue in IR: Relevance A simplistic definition of relevance: a document is relevant to a query if it contains the information that a person was looking for when he/she submitted the query to the IR system Many factors influence a persons decision about what is relevant: task, context, novelty, style, the usefulness, etc. An important distinction: topical relevance (same topic) vs. user relevance (everything else) An IR system can only estimate the relevance of a document to a query, using a document representation, and a retrieval model (that defines a view of relevance) It is important to verify if the system’s estimate is correct, hence the importance of evaluation in IR Prof. Fabio Crestani Distributed Information Retrieval 11
- 12. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR The IR Process Prof. Fabio Crestani Distributed Information Retrieval 12
- 13. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR The Indexing Process Text acquisition: identifies and stores documents for indexing Text transformation: transforms documents into index terms or features Index creation: takes index terms and creates data structures (indexes) to support fast searching Prof. Fabio Crestani Distributed Information Retrieval 13
- 14. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR Text Acquisition Could be carried out in different ways: Collection: documents could be provided directly (e.g., digital libraries) Crawler: identifies and acquires documents for IR system (e.g., web, enterprise, desktop search) Feeds: real-time streams of documents (e.g., web feeds for news, blogs, video, radio, tv) Documents end up in a Document Data Store that stores text, metadata, and other related content for documents for fast access to document contents in other phases of the IR process Prof. Fabio Crestani Distributed Information Retrieval 14
- 15. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR Text Transformation It usually involves: 1 Parsing: processing the sequence of text tokens in the document to recognise structural elements and words in the text 2 Stopping: remove words with no semantic meaning and common words 3 Stemming: group words derived from a common stem 4 Link analysis: makes use of links and anchor text in web pages 5 Information Extraction: identify classes of index terms that are important for some applications 6 Classification: identifies class-related metadata for documents Prof. Fabio Crestani Distributed Information Retrieval 15
- 16. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR Index Creation It usually involves: 1 Calculating document statistics: gathers counts and positions of words and other features 2 Weighting: computes weights for index terms (used by ranking algorithm) 3 Inversion: converts document-term information to term-document for indexing (the core of indexing process: building an inverted file) 4 Index distribution: distributes indexes across multiple computers and/or multiple sites (essential for fast query processing with large numbers of documents) Prof. Fabio Crestani Distributed Information Retrieval 16
- 17. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR The Querying Process User interaction: supports creation and refinement of query, and the display of results Ranking: uses query and indexes to generate ranked list of documents Evaluation: monitors and measures effectiveness and efficiency of the search process (primarily offline) Prof. Fabio Crestani Distributed Information Retrieval 17
- 18. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR User Interaction It usually involves: 1 Query input: provides interface and parser for query language 2 Query transformation: improves initial query, both before (same text transformation techniques used for documents) and after the initial search (query expansion and relevance feedback) 3 Results output: constructs the display of ranked list of documents for a query (including document surrogates) Prof. Fabio Crestani Distributed Information Retrieval 18
- 19. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR Ranking It usually involves: 1 Scoring: calculates scores of documents using a ranking algorithm (core component of the IR engine, with many variations depending on the retrieval model) 2 Performance optimisation: designing ranking algorithms for efficient processing 3 Query processing distribution: processing queries in a distributed environment (involving a query broker and caching) More about this when we talk about retrieval models Prof. Fabio Crestani Distributed Information Retrieval 19
- 20. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR Evaluation It usually involves: 1 Logging: logging user queries and interaction to mine for improving search effectiveness and efficiency 2 Ranking analysis: measuring and tuning ranking effectiveness 3 Performance analysis: measuring and tuning system efficiency More about this when we talk about IR evaluation Prof. Fabio Crestani Distributed Information Retrieval 20
- 21. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR Information Retrieval Models 1 A retrieval model is a mathematical framework for defining the search process, including the underlying assumptions and the interpretation of the concept of relevance 2 Progress in retrieval models has corresponded with improvements in effectiveness 3 Retrieval models make various assumptions about document representations (e.g., bag of words, document independence) and relevance (e.g., topical relevance, binary relevance) Prof. Fabio Crestani Distributed Information Retrieval 21
- 22. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR A Taxonomy of Information Retrieval Models Source: Wikipedia Prof. Fabio Crestani Distributed Information Retrieval 22
- 23. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR Information Retrieval Evaluation Evaluation is a key factor to building effective and efficient IR system Measurement usually carried out in controlled laboratory experiments, but online testing can also be done Core to IR evaluation are: collections (corpora), queries, relevance judgements, evaluation metrics IR evaluation has developed over 40 years thanks to evaluation frameworks like Cranfield and TREC Prof. Fabio Crestani Distributed Information Retrieval 23
- 24. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR Corpora Prof. Fabio Crestani Distributed Information Retrieval 24
- 25. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR Queries and Relevance Judgements Prof. Fabio Crestani Distributed Information Retrieval 25
- 26. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR Evaluation Metrics: Precision and Recall Prof. Fabio Crestani Distributed Information Retrieval 26
- 27. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR Other Evaluation Metrics There are many other evaluation metrics False Positive (Fall out or type I error) and False Negative (or type II error) F Measure: harmonic mean of recall and precision Mean Average Precision (MAP) Discounted Cumulative Gain (DCG, for graded relevance) and Normalised Discounted Cumulative Gain (NDCG) Prof. Fabio Crestani Distributed Information Retrieval 27
- 28. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR Essential Information Retrieval References Keith Van Rijsbergen Information Retrieval. Second Edition. Butterworths, London, 1979. Bruce Croft, Donald Metzler, Trevor Strohman Search Engines: Information Retrieval in Practice. Addison Wesley, New York, 2009. Maristella Agosti, Fabio Crestani, Gabriella Pasi. Lectures on Information Retrieval, Third European Summer-School, ESSIR 2000. Varenna, Italy, September 11-15, 2000, Revised Lectures, Springer-Verlag, Berlin, 2001. Prof. Fabio Crestani Distributed Information Retrieval 28
- 29. Background Information Retrieval DIR: Introduction Indexing DIR Architectures Querying Broker-Based DIR IR Models DIR Evaluation Evaluation Applications of DIR Questions? Prof. Fabio Crestani Distributed Information Retrieval 29
- 30. Background DIR: Introduction DIR Architectures What is DIR? Broker-Based DIR Motivations for DIR DIR Evaluation Applications of DIR Outline 1 Background 2 DIR: Introduction 3 DIR Architectures 4 Broker-Based DIR 5 DIR Evaluation 6 Applications of DIR Prof. Fabio Crestani Distributed Information Retrieval 30
- 31. Background DIR: Introduction DIR Architectures What is DIR? Broker-Based DIR Motivations for DIR DIR Evaluation Applications of DIR Topics Covered 2 DIR: Introduction What is DIR? Motivations for DIR Deep Web Federated Search Metasearch Aggregated Search Prof. Fabio Crestani Distributed Information Retrieval 31
- 32. Background DIR: Introduction DIR Architectures What is DIR? Broker-Based DIR Motivations for DIR DIR Evaluation Applications of DIR What is DIR? A DIR system is an IR system that is designed to search for information that is distributed across different resources Each resource is composed on a search engine and one or more collection of documents. Each resource is assumed to handle the search process on its own collection in a independent way Other names for DIR are: federated search and federated information retrieval Example of DIR systems are: FedStast, PubMed, US Census Bureau, Westlaw, MedLine, Cheshire, etc. Prof. Fabio Crestani Distributed Information Retrieval 32
- 33. Background DIR: Introduction DIR Architectures What is DIR? Broker-Based DIR Motivations for DIR DIR Evaluation Applications of DIR Motivations Why do we need DIR? There are limits to what a search engines can find on the web 1 Not everything that is on the web is or can be harvested 2 The ”one size fits all” approach of web search engine has many limitations 3 Often there is more than one type of answer to the same query Thus: Deep Web, Federated Search, MetaSearch, Aggregated Search Prof. Fabio Crestani Distributed Information Retrieval 33
- 34. Background DIR: Introduction DIR Architectures What is DIR? Broker-Based DIR Motivations for DIR DIR Evaluation Applications of DIR Deep Web There is a lot of information on the web that cannot be accessed by search engines (deep or hidden web) There are many different reasons why this information is not accessible to crawlers This is often very valuable information! All current search engines are able to identify deep web resources Web search engines can only be used to identify the resource (if possible), then the user has to deal directly with it Prof. Fabio Crestani Distributed Information Retrieval 34
- 35. Background DIR: Introduction DIR Architectures What is DIR? Broker-Based DIR Motivations for DIR DIR Evaluation Applications of DIR Deep Web: Example Prof. Fabio Crestani Distributed Information Retrieval 35
- 36. Background DIR: Introduction DIR Architectures What is DIR? Broker-Based DIR Motivations for DIR DIR Evaluation Applications of DIR Federated Search Federated Search is another name for DIR Federated search systems do not crawl a resource, but pass a user query to the search facilities of the resource itself Why would this be better? Preserves the property rights of the resource owner Search facilities are optimised to the specific resource Index is always up-to-date The resource is curated and of high quality Examples of federate search systems: PubMed, FedStats, WestLaw, and Cheshire Prof. Fabio Crestani Distributed Information Retrieval 36
- 37. Background DIR: Introduction DIR Architectures What is DIR? Broker-Based DIR Motivations for DIR DIR Evaluation Applications of DIR Federated Search: Example Prof. Fabio Crestani Distributed Information Retrieval 37
- 38. Background DIR: Introduction DIR Architectures What is DIR? Broker-Based DIR Motivations for DIR DIR Evaluation Applications of DIR Metasearch Even the largest search engine cannot crawl effectively the entire web Different search engines crawl different disjoint portions of the web Different search engines use different ranking functions Metasearch engines do not crawl the web, but pass a user query to a number of search engines and then present the fused results set Examples of federate search systems: Dogpile, MataCrawler, AllInOneNews, and SavvySearch Prof. Fabio Crestani Distributed Information Retrieval 38
- 39. Background DIR: Introduction DIR Architectures What is DIR? Broker-Based DIR Motivations for DIR DIR Evaluation Applications of DIR Metasearch: Example Prof. Fabio Crestani Distributed Information Retrieval 39
- 40. Background DIR: Introduction DIR Architectures What is DIR? Broker-Based DIR Motivations for DIR DIR Evaluation Applications of DIR Aggregated Search Often there is more that one type of information relevant to a query (e.g. web page, images, map, reviews, etc) These type of information are indexed and ranked by separate sub-systems Presenting this information in an aggregated way is more useful to the user Prof. Fabio Crestani Distributed Information Retrieval 40
- 41. Background DIR: Introduction DIR Architectures What is DIR? Broker-Based DIR Motivations for DIR DIR Evaluation Applications of DIR Aggregated Search: Example Prof. Fabio Crestani Distributed Information Retrieval 41
- 42. Background DIR: Introduction DIR Architectures What is DIR? Broker-Based DIR Motivations for DIR DIR Evaluation Applications of DIR Questions? Prof. Fabio Crestani Distributed Information Retrieval 42
- 43. Background Peer-to-Peer Network DIR: Introduction Broker-Based Architecture DIR Architectures Crawling Broker-Based DIR Metadata Harvesting DIR Evaluation Hybrid Applications of DIR Outline 1 Background 2 DIR: Introduction 3 DIR Architectures 4 Broker-Based DIR 5 DIR Evaluation 6 Applications of DIR Prof. Fabio Crestani Distributed Information Retrieval 43
- 44. Background Peer-to-Peer Network DIR: Introduction Broker-Based Architecture DIR Architectures Crawling Broker-Based DIR Metadata Harvesting DIR Evaluation Hybrid Applications of DIR Topics Covered 3 DIR Architectures Peer-to-Peer Network Broker-Based Architecture Crawling Metadata Harvesting Hybrid Prof. Fabio Crestani Distributed Information Retrieval 44
- 45. Background Peer-to-Peer Network DIR: Introduction Broker-Based Architecture DIR Architectures Crawling Broker-Based DIR Metadata Harvesting DIR Evaluation Hybrid Applications of DIR A Taxonomy of DIR Systems A taxonomy of DIR architectures can be build considering where the indexes are kept This suggest 4 different types of architectures: broker-based, peer-to-peer, crawling, and meta-data harvesting Global collection Distributed indexes Centralised index P2P Broker Crawling Meta-data harvesting Hybrid Prof. Fabio Crestani Distributed Information Retrieval 45
- 46. Background Peer-to-Peer Network DIR: Introduction Broker-Based Architecture DIR Architectures Crawling Broker-Based DIR Metadata Harvesting DIR Evaluation Hybrid Applications of DIR Peer-to-Peer Networks Indexes are located with the resources Some part of the indexes are distributed to other resources Queries are distributed across the resources and results are merged by the peer that originated the query Prof. Fabio Crestani Distributed Information Retrieval 46
- 47. Background Peer-to-Peer Network DIR: Introduction Broker-Based Architecture DIR Architectures Crawling Broker-Based DIR Metadata Harvesting DIR Evaluation Hybrid Applications of DIR Broker-Based Architecture Indexes are located with the resources Queries are forwarded to resources and results are merged by a broker Prof. Fabio Crestani Distributed Information Retrieval 47
- 48. Background Peer-to-Peer Network DIR: Introduction Broker-Based Architecture DIR Architectures Crawling Broker-Based DIR Metadata Harvesting DIR Evaluation Hybrid Applications of DIR Crawling Resources are crawled and documents are harvested Indexes are centralised Queries are evaluated out in a centralised way and documents are fetched from resources or from a storage Prof. Fabio Crestani Distributed Information Retrieval 48
- 49. Background Peer-to-Peer Network DIR: Introduction Broker-Based Architecture DIR Architectures Crawling Broker-Based DIR Metadata Harvesting DIR Evaluation Hybrid Applications of DIR Metadata Harvesting Indexes are located with the resources, but metadata are harvested according to some protocol (off-line phase), like for example the OAI-PMH Queries are evaluated at the broker level (on-line phase) to identify relevant documents by the metadata, that are then requested from the resources. Prof. Fabio Crestani Distributed Information Retrieval 49
- 50. Background Peer-to-Peer Network DIR: Introduction Broker-Based Architecture DIR Architectures Crawling Broker-Based DIR Metadata Harvesting DIR Evaluation Hybrid Applications of DIR The Open Archive Initiative The Open Archives Initiative (OAI) develops and promotes interoperability standards that aim to facilitate the efficient dissemination of content The OAI developed a Protocol for Metadata Harvesting (OAI-PMH) and a set of tools that implements that Only Dublin Core type metadata (or some extension of that set) is exchanged, via HTTP in a XML like format OAI has its origin in library world and is very popular in federated digital libraries Prof. Fabio Crestani Distributed Information Retrieval 50
- 51. Background Peer-to-Peer Network DIR: Introduction Broker-Based Architecture DIR Architectures Crawling Broker-Based DIR Metadata Harvesting DIR Evaluation Hybrid Applications of DIR Indexing Harvesting It is possible to crawl the indexes, instead of the metadata according to some protocol (off-line phase), like for example the OAI-PMH Queries are evaluated out at the broker level (on-line phase) to identify relevant documents by the documents’ full content, that are then requested from the resources Prof. Fabio Crestani Distributed Information Retrieval 51
- 52. Background Peer-to-Peer Network DIR: Introduction Broker-Based Architecture DIR Architectures Crawling Broker-Based DIR Metadata Harvesting DIR Evaluation Hybrid Applications of DIR Questions? Prof. Fabio Crestani Distributed Information Retrieval 52
- 53. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Outline 1 Background 2 DIR: Introduction 3 DIR Architectures 4 Broker-Based DIR 5 DIR Evaluation 6 Applications of DIR Prof. Fabio Crestani Distributed Information Retrieval 53
- 54. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Architecture of a Broker-based DIR System Indexes are located with the resources Queries are forwarded to resources and results are merged by the broker Prof. Fabio Crestani Distributed Information Retrieval 54
- 55. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Phases of the DIR Process The DIR process is divided in the following phases: 1 Resource discovery 2 Resource description 3 Resource selection 4 Results fusion 5 Results presentation Prof. Fabio Crestani Distributed Information Retrieval 55
- 56. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Topics Covered 4 Broker-Based DIR Resource Description Resource Selection Results Merging Prof. Fabio Crestani Distributed Information Retrieval 56
- 57. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Objectives of the Resource Description Phase The resource description phase is concerned with building a description of each and every resource the broker has to handle. This phase is required for all other subsequent phases. Prof. Fabio Crestani Distributed Information Retrieval 57
- 58. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR DIR Cooperation There are two kinds of environments that determine the way resource description is carried out: Cooperative environments: the resource provides full access to the documents and the indexes and responds to queries Uncooperative environments: the resource does provides any access to the document and the indexes; it only respond to queries Prof. Fabio Crestani Distributed Information Retrieval 58
- 59. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Resource Description in Cooperative Environments Resource Description in cooperative environments can be very simple as the broker has full access to the collection(s) held at the resource The broker could crawl or harvest the full collection(s) and deal with the query locally, but this might not be a good idea The broker could receive from the resource information (a description) useful for the selection Prof. Fabio Crestani Distributed Information Retrieval 59
- 60. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Stanford Protocol Proposal for Internet and Retrieval Search (STARTS) STARTS is similar to OAI. It stores for each resource some resource metadata and content summary: Query Language Statistics (term frequency, document frequency, number of documents) Score range Stopwords list Others (sample results, supported fields, etc) Prof. Fabio Crestani Distributed Information Retrieval 60
- 61. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Stanford Protocol Proposal for Internet and Retrieval Search (STARTS) STARTS provides a query language with: Filter expressions Ranking expressions Retrieved documents are provided by each resource with: Unnormalised score Source indication Using the source metadata and content summary the broker can produce a normalised score for each document Prof. Fabio Crestani Distributed Information Retrieval 61
- 62. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Resource Description in Un-Cooperative Environments Resource Description in uncooperative environments is far more difficult as the broker does not have to full collections, or access to resource metadata and content summary The broker needs to acquire this information without any help from the resource Important information to acquire for the resource description includes: collection size, term statistics, document scores The required information can only be estimated and will contain estimation errors Prof. Fabio Crestani Distributed Information Retrieval 62
- 63. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Query-based Sampling The idea Query −→ Documents (2-10) ←− Resource Analyse Documents Build description Questions How do we select the queries? When do we stop (stopping criterium)? Prof. Fabio Crestani Distributed Information Retrieval 63
- 64. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Selecting the Sampling Queries Queries can be selected by: Other Resource Description (ORD): selecting terms from a reference dictionary Learned Resource Description (LRD): selecting terms from the retrieved documents based on term statistics ORD produces more representative samples, but is sensitive to out of vocabulary terms (OOV) that do not return any document Prof. Fabio Crestani Distributed Information Retrieval 64
- 65. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Selecting the Sampling Queries The best experimental strategy proved to be based on random selection of terms that have an Average Term Frequency, where CollectionTermFrequency AverageTermFrequency = DocumentFrequency Another important strategy would be to use personal query-logs to achieve a personalised resource description Prof. Fabio Crestani Distributed Information Retrieval 65
- 66. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Stopping Criteria Not a well studied problem, mostly approached in a heuristic way Experimental studies suggest to stop after downloading 300-500 unique terms But this depends of the collection size Different regions of the resource term space could be unequally sampled Prof. Fabio Crestani Distributed Information Retrieval 66
- 67. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Adaptive Sampling Ideally we would need an adaptive stopping criterium, related to: The proportion of documents sampled in relation of the size of the collection The proportion of term sampled in relation to the size of the term space The vocabulary growth There have been some attempts to propose methods for adaptive quey-based sampling Prof. Fabio Crestani Distributed Information Retrieval 67
- 68. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Estimating Collection Size The size of the collection is an important element of the resource description Knowing the size of the collection is useful for a better stopping criterium of query-based sampling It is also a useful parameter of the resource selection phase Two techniques have been proposed: capture-recapture and sample-resample Prof. Fabio Crestani Distributed Information Retrieval 68
- 69. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Capture-Recapture The idea X - event that a randomly sampled document is already in a sample Y - number of X in n trials Two samples S1 and S2 |S| |S| E[X ] = , E[Y ] = n · E[X ] = n · |C | |C | |S1 ||S2 | ˆ |S1 ||S2 | |S1 ∩ S2 | ≈ =⇒ |C | = |C | |S1 ∩ S2 | Prof. Fabio Crestani Distributed Information Retrieval 69
- 70. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Capture-Recapture Take two samples Count the number of common documents ˆ Estimate collection size |C | Not very clear how the random samples should be generated Prof. Fabio Crestani Distributed Information Retrieval 70
- 71. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Sample-Resample The idea Randomly pick a term t from a sample A - event that some sampled document contains t B - event that some documents from the resource contains t dft,S dft,C P(A) = , P(B) = |S| |C | ˆ |S| P(A) ≈ P(B) =⇒ |C | = dft,C · dft,S Prof. Fabio Crestani Distributed Information Retrieval 71
- 72. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Sample-Resample Send the query t to the resource to estimate dft,C ˆ Repeat several times and estimate collection size |C | as average value of estimates The method relies on the resource giving the correct document frequency of the query terms Prof. Fabio Crestani Distributed Information Retrieval 72
- 73. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Essential Resource Description References Mark Baillie, Leif Azzopardi, and Fabio Crestani. An adaptive stopping criteria for query-based sampling of distributed collections. In String Processing and Information Retrieval (SPIRE), pages 316–328, 2006. Jamie Callan and Margaret Connell. Query-based sampling of text databases. ACM Trans. Inf. Syst., 19(2):97–130, 2001. James Caverlee, Ling Liu, and Joonsoo Bae. Distributed query sampling: a quality-conscious approach. In Proceedings of the ACM SIGIR, pages 340–347, ACM, 2006. Luis Gravano, Chen-Chuan K. Chang, H´ctor Garc´ e ıa-Molina, and Andreas Paepcke. Starts: Stanford proposal for internet meta-searching. SIGMOD Rec., 26(2):207–218, 1997. Luo Si and Jamie Callan. Relevant document distribution estimation method for resource selection. In Proceedings of the ACM SIGIR, pages 298–305. ACM, 2003. Milad Shokouhi, Justin Zobel, Seyed M. M. Tahaghoghi, and Falk Scholer. Using query logs to establish vocabularies in distributed information retrieval. Inf. Process. Manage., 43(1):169–180, 2007. Prof. Fabio Crestani Distributed Information Retrieval 73
- 74. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Questions? Prof. Fabio Crestani Distributed Information Retrieval 74
- 75. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Objectives of the Resource Selection Phase The resource selection phase is concerned with the broker, given a query, selecting only those resources that are likely to retrieve relevant documents. Resource selection uses the representations built by the resource description phase and other parameters. Prof. Fabio Crestani Distributed Information Retrieval 75
- 76. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Approaches to Resource Selection There are three approaches: Lexicon-Based: used for cooperative environments, the broker calculates the similarity of the query with the resource description using the detailed lexicon statistics of the resource Document Surrogate-Based: used for uncooperative environments, in addition to the above the broker use also sampled documents from each resource Classification-Based: used for uncooperative environments, the broker classify resources into a topic hierarchy and make selection decisions based on the matching between the query and the categories of the hierarchy Prof. Fabio Crestani Distributed Information Retrieval 76
- 77. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Lexicon-Based Approaches In Lexicon based approaches collections at resources are treated as a large single documents or vocabulary distributions. The “super document” that is most relevant to the query identifies the collection to select. The two best known methods are CORI and GlOSS Prof. Fabio Crestani Distributed Information Retrieval 77
- 78. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Collection Retrieval Inference Network (CORI) Collection =⇒ Super-Document Bayesian inference network on super-documents Adapted Okapi dft,i T = dft,i + 50 + 150 · cwi /avg cw log( Nccft ) +0.5 I = log(Nc + 1.0) p(t|Ci ) = b + (1 − b) · T · I Collections are ranked according to p(Q|Ci ) Prof. Fabio Crestani Distributed Information Retrieval 78
- 79. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Glossary-of-Servers Server (GlOSS) We assume cooperation and the availability of documents and terms statistics Rank(q, l, C ) = {d ∈ C |sim(q, d)>l} Goodness(q, l, C ) = sim(q, d) d∈Rank(q,l,C ) Prof. Fabio Crestani Distributed Information Retrieval 79
- 80. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Document Surrogate-Based Approaches In document surrogate-based approaches collections at resources are selected based on their similarity with some sample documents retrieved from each resource Away from the super document approach and retaining document boundaries The two best known methods are ReDDE and CRCS Prof. Fabio Crestani Distributed Information Retrieval 80
- 81. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Relevant Document Distribution Estimation (ReDDE) Idea Estimate number of relevant documents in the collection and rank them If one sampled document is relevant to a query ⇐⇒ |SC|| similar |C documents in a collection c are relevant to a query. |C | R(C , Q) ≈ P(R|d) |SC | d∈SC where P(R|d) is the prob of relevance of an arbitrary document in the description Prof. Fabio Crestani Distributed Information Retrieval 81
- 82. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Relevant Document Distribution Estimation (ReDDE) Ranked sampled documents =⇒ Ranked documents in a centralised retrieval system Idea A document dj appears before a document di in a sample ⇐⇒ |Cj | |SC | documents appear before di in a centralised retrieval system. j |Cj | Rankcentralized (di ) = |SCj | dj :Ranksample (dj )<Ranksample (di ) Prof. Fabio Crestani Distributed Information Retrieval 82
- 83. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Relevant Document Distribution Estimation (ReDDE) |C | R(C , Q) ≈ P(R|d) |SC | d∈SC |Cj | Rankcentralized (di ) = |SCj | dj :Ranksample (dj )<Ranksample (di ) α if Rankcentralized (d) < β · i |Ci | P(R|d) = 0 otherwise. where α is a constant positive probability of relevance and β is a percentage threshold separating relevant from non-relevant documents Prof. Fabio Crestani Distributed Information Retrieval 83
- 84. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Centralised-Rank Collection Selection (CRCS) Different variation of the ReDDE algorithms, based on analysis of the top ranked sampled documents to estimate relevance documents distribution in the collection |C | R(C , Q) ≈ P(R|d) |SC | d∈SC Linear γ − Ranksample (d) if Ranksample (d) < γ R(d) = 0 otherwise. Exponential R(d) = α exp(−β · Ranksample (d)) R(d) P(R|d) = |Cmax | Prof. Fabio Crestani Distributed Information Retrieval 84
- 85. Rv Rv 0.4 Background 0.4 DIR: Introduction Resource Description 0.2 DIR Architectures 0.2 Resource Selection Broker-Based DIR Results Merging 0.0 DIR Evaluation 0.0 0 5 Applications of DIR 10 15 20 0 5 10 15 20 Cutoff (nonrelevant) Cutoff (GOV2) Resource Selection Comparison Fig. 3. R values for the cori, redde and crcs algorithms on the trec123-FR-DOE- 81col (left) and 100-col-gov2 (right) testbeds. Table 2. Performance of different methods for the Trec4 (trec4-kmeans) testbed. Trec topics 201–250 (long) were used as queries Cutoff=1 Cutoff=5 P@5 P@10 P@15 P@20 P@5 P@10 P@15 P@20 CORI 0.3000 0.2380 0.2133† 0.1910† 0.3480 0.2980 0.2587 0.2380 ReDDE 0.2160 0.1620 0.1373 0.1210 0.3480 0.2860 0.2467 0.2190 CRCS(l) 0.2960 0.2260 0.2013 0.1810† 0.3520 0.2920 0.2533 0.2310 CRCS(e) 0.3080 0.2400 0.2173† 0.1910† 0.3880 0.3160 0.2680 0.2510 Table 3. Performance of collection selection methods for the uniform (trec123-100col- bysource) testbed. Trec topics 51–100 (short) were used as queries evance judgments. We only report the results for cutoff=1 and cutoff=5. The former shows the system outputs when only the best collection is selected while Cutoff=1 Cutoff=5 for the latter, P@5 P@10collections are chosen to P@10 P@15for the query. the best five P@15 P@20 P@5 get searched P@20 We do not report the results for larger cutoff values because 0.2867 0.2730 shown CORI 0.2520 0.2140 0.1960 0.1710 0.3080 0.3060 cutoff=5 has to beReDDE 0.1920threshold0.1413 0.1280 a reasonable 0.1660 for dir experiments on the real web collections 0.2960 0.2820 0.2653 0.2510 [Avrahami et al., 2006]. The 0.1520‡values † 0.3440 calculated precision on the CRCS(l) 0.2120 0.1760 P @x 0.1330show the 0.3240 0.3067 0.2860 ‡ ‡ top xCRCS(e) 0.3800 0.3060 0.2613 0.2260 results. 0.3960 0.3700† 0.3480† 0.3310† We select redde as the baseline as it does not require training queries and Table 4. Performance of collection selection methods for the representative (trec123- its effectiveness is found to be higher than cori and older alternatives [Si and 2ldb-60col) testbed. Trec topics 51–100 (short) were used as queries Callan, 2003a]. The following tables compare the performance of discussed meth- Prof. Fabio Crestani Distributed Information Retrieval 85
- 86. significantly worse than redde at the 99% confidence interval. Background On the relevant testbed (Table 5), all Resource Description for cori are signifi- DIR: Introduction precision values DIR Architectures cantly inferior to that Broker-Based for both Resourcevalues. Redde in general pro- of redde DIR cutoff Selection Results Merging duces higher precision valuesEvaluation methods. However, none of the gaps are DIR than crcs detected statistically Applications of DIR t-test at the 99% confidence interval. significant by the The results for crcs(l), redde and cori are comparable on the non-relevant Resource Selection Comparison testbed (Table 6). Crcs(e) significantly outperforms the other methods in most cases. On the gov2 testbed (Table 7), cori produces the best results when Table 6. Performance of collection selection methods for the non-relevant (trec123- FR-DOE-81col) testbed. of collection 51–100 (short) were used as queries(trec123-AP- Table 5. Performance Trec topics selection methods for the relevant WSJ-60col) testbed. Trec topics 51–100 (short) were used as queries Cutoff=1 Cutoff=5 P@5 Cutoff=1 P@15 P@20 P@5 P@10 P@15 P@20 P@10 Cutoff=5 CORI 0.2520 0.2100 P@15 P@20 P@5 0.2980 0.2707 P@20 P@5 P@10 0.1867 0.1690 0.3200 P@10 P@15 0.2670 ReDDE 0.2240 0.1900 0.1160 0.1090 0.2440 0.2980 0.2707 0.2210 CORI 0.1440 0.1280 0.1813 0.1750 0.3480 0.2340 0.2333 0.2610 CRCS(l) 0.2040 0.1860 0.3360 0.3270 0.3920 0.3220 0.2973 0.3490 ReDDE 0.3960 0.3660 0.1813 0.1800 0.3360 0.3900 0.3640 0.2860 CRCS(e) 0.3200† 0.2820† 0.3293 † 0.3120 0.3800 0.3600 0.3387† 0.3210† CRCS(l) 0.3840 0.3580 0.2533 0.2240 0.3880 0.3640 0.3467 0.3250 CRCS(e) 0.3080 0.2860 0.2813 0.2680 0.3480 0.3420 0.3280 0.3170 Table 7. Performance of collection selection methods for the gov2 (100-col-gov2) testbed. Trec topics 701–750 (short) were used as queries Cutoff=1 Cutoff=5 P@5 P@10 P@15 P@20 P@5 P@10 P@15 P@20 CORI 0.1592† 0.1347† 0.1143† 0.0969† 0.2735 0.2347 0.2041 0.1827 ReDDE 0.0490 0.0327 0.0286 0.0235 0.2163 0.1837 0.1687 0.1551 CRCS(l) 0.0980 0.0755 0.0667 0.0531 0.1959 0.1510 0.1442 0.1286 CRCS(e) 0.0857 0.0714 0.0748 0.0643 0.2776 0.2469 0.2272 0.2122 cutoff=1 while in the other scenarios there is no significant difference between the methods. Prof. Fabio Crestani Distributed Information Retrieval 86
- 87. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Essential Resource Selection References James P. Callan, Zhihong Lu, and W. Bruce Croft. Searching distributed collections with inference networks. In Proceedings of the ACM SIGIR, pages 21–28. ACM, 1995. Luis Gravano, H´ctor Garc´ e ıa-Molina, and Anthony Tomasic. GlOSS: text-source discovery over the internet. ACM Trans. Database Syst., 24(2):229–264, 1999. Luo Si and Jamie Callan. Relevant document distribution estimation method for resource selection. In Proceedings of the ACM SIGIR, pages 298–305. ACM, 2003. Milad Shokouhi. Central-rank-based collection selection in uncooperative distributed information retrieval. In ECIR, pages 160–172, 2007. Shengli Wu, Fabio Crestani. Distributed Information Retrieval: A Multi-Objective Resource Selection Approach. In International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 11, pages 83–100, 2003. Prof. Fabio Crestani Distributed Information Retrieval 87
- 88. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Questions? Prof. Fabio Crestani Distributed Information Retrieval 88
- 89. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Objectives of the Results Merging Phase The results merging phase is concerned with the broker merging the list of top-ranked documents returned from the different resources and returning a fused list to the user Not to be confused with data fusion, where the results come from a single resource and are then ranked by multiple retrieval models Prof. Fabio Crestani Distributed Information Retrieval 89
- 90. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Results Merging Issues The results merging process involves a number of issues: 1 Duplicate detection and removal 2 Normalising and merging relevance scores Different solutions have been proposed for these issue, depending in the DIR environment Prof. Fabio Crestani Distributed Information Retrieval 90
- 91. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Results Merging in Cooperative Environments The results merging in cooperative environments is much simpler and has different solutions: 1 Fetch documents from each resource, reindex and rank according to the broker IR model. 2 Get information about the way the document score is calculated and normalise score. At the highest level of collaboration it is possible to ask the resources to adopt the same retrieval model! Prof. Fabio Crestani Distributed Information Retrieval 91
- 92. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Collection Retrieval Inference Network (CORI) The idea Linear combination of the score of the database and the score of the document. Normalised scores (Ri −Rmax ) Normalised collection section score: Ci = (Rmax −Rmin ) (Dj −Dmax ) Normalised document score: Dj = (Dmax −Dmin ) Dj +0.4∗Dj ∗Ci Heuristic linear combination: Dj = 1.4 Prof. Fabio Crestani Distributed Information Retrieval 92
- 93. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Results Merging in Uncooperative Environments In uncooperative environments resources might provide scores: But the broker does not have any information on how these score are computed. Score normalisation requires some way of comparing scores. Alternatively the resources might provide only rank positions: But the broker does not have any information on the relevance of each document in the rank lists. Merging the ranks requires some way of comparing rank positions. Prof. Fabio Crestani Distributed Information Retrieval 93
- 94. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Semi-Supervised Learning (SSL) The idea Train a regression model for each collection that maps resource document scores to normalised scores. Requires that some returned documents are found in the Collection Selection Index (CSI). Two cases: 1 Resources use identical retrieval models 2 Resources use different retrieval models Prof. Fabio Crestani Distributed Information Retrieval 94
- 95. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR SSL with Identical Retrieval Models The idea SSL uses documents found in CSI to train a single regression model to estimate the normalised score (Di,j ) from resource document scores (Di,j ) and the score of the same document computed from the CSI (Ei,j ). Normalised scores Having: D1,1 C1 D1,1 E1,1 D1,2 C1 D1,2 E1,2 × [a b] = ... ... ... Dn,m Cn Dn,m En,m Train: Di,j = a ∗ Ei,j + b ∗ Ei,j ∗ Ci Prof. Fabio Crestani Distributed Information Retrieval 95
- 96. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR SSL with Different Retrieval Models The idea SSL uses documents found in CSI to train a different regression models for each resource. Normalised scores Having: D1,1 1 E1,1 D1,2 1 E1,2 . . . . . . × [ai bi ] = ... Dn,m 1 En,m Train: Di,j = ai ∗ Ei,j + bi Prof. Fabio Crestani Distributed Information Retrieval 96
- 97. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Sample-Agglomerate Fitting Estimate (SAFE) The idea For a given query the results from the CSI is a subranking of the original collection, so curve fitting to the subranking can be used to estimate the original scores. It does not require the presence of overlap documents in CSI. Prof. Fabio Crestani Distributed Information Retrieval 97
- 98. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Sample-Agglomerate Fitting Estimate (SAFE) Normalised scores 1 The broker ranks the documents available in the CSI for the query. 2 For each resource the sample documents (with non zero score) are used to estimate the merging score, where each sample document is assumed to be representative of a fraction |Sc|/|c| of the resource. 3 Use regression to fit a curve on the adjusted scores to predict the score of the document returned by the resource. Prof. Fabio Crestani Distributed Information Retrieval 98
- 99. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR More Results Merging There are other approaches to results merging: STARTS uses the returned term frequency, document frequency, and document weight information to calculate the merging score based on similarities between documents. CVV calculates the merging score according to the collection score and the position of a document in the returned collection rank list. Another approach download small parts of the top returned documents and used a reference index of term statistics for reranking and merging the downloaded documents. Prof. Fabio Crestani Distributed Information Retrieval 99
- 100. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Data Fusion in Metasearch In data fusion methods documents in a single collection are ranked with different search engines The goal is to generate a single accurate ranking list from the ranking lists of different retrieval models. There are no collection samples and no CSI. The idea Use the voting principle: a document returned by many search systems should be ranked higher than the other documents. If available, also take the rank of documents into account. Prof. Fabio Crestani Distributed Information Retrieval 100
- 101. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Metasearch Data Fusion Methods Many methods have been proposed: Data Fusion Round Robin. CombMNZ, CombSum, CombMax, CombMin. Logistic regression (convert ranks to estimated probabilities of relevance). A comparison between score-based and rank-based methods suggests that rank-based methods are generally less effective. Prof. Fabio Crestani Distributed Information Retrieval 101
- 102. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Results Merging in Metasearch We cannot use data fusion methods when collections are overlapping, but are not the same. We cannot use data fusion methods when the retrieval model are different. Web metasearch is the most typical example. The idea Normalise the document scores returned by multiple search engines using a regression function that compares the scores of overlapped documents between the returned ranked lists. In the absence of overlap between the results, most metasearch merging techniques become ineffective. Prof. Fabio Crestani Distributed Information Retrieval 102
- 103. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Essential Results Merging References James P. Callan, Zhihong Lu, and W. Bruce Croft. Searching distributed collections with inference networks. In Proceedings of the ACM SIGIR, pages 21–28. ACM, 1995. N. Craswell, D. Hawking, and P. Thistlewaite. Merging results from isolated search engines. In Proceedings of the Australasian Database Conference, pages 189-200, 1999. E. Fox and J. Shaw. Combination of multiple searches. In Proceedings of TREC, pages 105-108,1994. L. Gravano, C. Chang, H. Garcia-Molina, and A. Paepcke. STARTS: Stanford proposal for internet meta-searching. In Proceedings of the ACM SIGMOD, pages 207-218, 1997. Prof. Fabio Crestani Distributed Information Retrieval 103
- 104. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR L. Si and J. Callan. Using sampled data and regression to merge search engine results. In Proceedings of the ACM SIGIR, pages 29-26. ACM, 2002. M. Shokouhi and J. Zobel. Robust result merging using sample-based score estimates. In ACM Transactions on Information Systems, 27(3): 129, 2009. B. Yuwono and D. Lee. Server ranking for distributed text retrieval systems on the internet. In Proceedings of the Conference on Database Systems for Advanced Applications, pages 41-50, 1997. S. Wu and F. Crestani. Shadow document methods of results merging. In Proceedings of the ACM SAC, pages 1067-1072, 2004 Prof. Fabio Crestani Distributed Information Retrieval 104
- 105. Background DIR: Introduction Resource Description DIR Architectures Resource Selection Broker-Based DIR Results Merging DIR Evaluation Applications of DIR Questions? Prof. Fabio Crestani Distributed Information Retrieval 105
- 106. Background DIR: Introduction Objectives DIR Architectures Test Collections Broker-Based DIR Evaluation Metrics DIR Evaluation Applications of DIR Outline 1 Background 2 DIR: Introduction 3 DIR Architectures 4 Broker-Based DIR 5 DIR Evaluation 6 Applications of DIR Prof. Fabio Crestani Distributed Information Retrieval 106
- 107. Background DIR: Introduction Objectives DIR Architectures Test Collections Broker-Based DIR Evaluation Metrics DIR Evaluation Applications of DIR Topics Covered 5 DIR Evaluation Objectives Test Collections Evaluation Metrics Prof. Fabio Crestani Distributed Information Retrieval 107
- 108. Background DIR: Introduction Objectives DIR Architectures Test Collections Broker-Based DIR Evaluation Metrics DIR Evaluation Applications of DIR Objectives of DIR Evaluation Evaluation is very important, as in all subareas of IR The relative effectiveness of federated search methods tends to vary between different testbeds (i.e., set of test collections) Important to have different testbeds Two main categories: Testbeds with disjoint collections Testbeds with overlapping collections There are several testbeds, her I report only some examples Prof. Fabio Crestani Distributed Information Retrieval 108
- 109. Background DIR: Introduction Objectives DIR Architectures Test Collections Broker-Based DIR Evaluation Metrics DIR Evaluation Applications of DIR Datasets available 78 Federated search testbeds Table 6.1 Testbed statistics. # docs (×1000) Size (MB) Testbed Size (GB) Min Avg Max Min Avg Max trec123-100col-bysource 3.2 0.7 10.8 39.7 28 32 42 trec4-kmeans 2.0 0.3 5.7 82.7 4 20 249 trec-gov2-100col 110.0 32.6 155.0 717.3 105 1 126 3 891 SYM236 & UDC236. SYM236 [French et al., 1998; 1999; Powell, 2001; Powell and French, 2003] includes 236 collections of varying sizes, and is generated from documents on TREC disks 1–4 [Harman, 1994; 1995]. UDC236 [French et al., 1999; Powell and French, 2003], also con- tains 236 collections, and is generated from the same set of documents (i.e., TREC disks 1–4). The difference is only in the methodology used for assigning documents to collections. In UDC236, each collection con- tains almost the same number of documents; in SYM236, documents are distributed between collections according to their publication date, Prof. Fabio Crestani 2 Distributed Information Retrieval 109
- 110. Background DIR: Introduction Objectives DIR Architectures Test Collections Broker-Based DIR Evaluation Metrics DIR Evaluation Applications of DIR Datasets available 6.1. Testbeds with disjoint collections 79 Table 6.2 The domain names for the largest fifty crawled servers in the TREC GOV2 dataset. The ‘www’ prefix of the domain names is omitted for brevity. Collection # docs Collection # docs ghr.nlm.nih.gov 717 321 leg.wa.gov 189 850 nih.library.nih.gov 709 105 library.doi.gov 185 040 wcca.wicourts.gov 694 505 dese.mo.gov 173 737 cdaw.gsfc.nasa.gov 656 229 science.ksc.nasa.gov 170 971 catalog.kpl.gov 637 313 nysed.gov 170 254 edc.usgs.gov 551 123 spike.nci.nih.gov 145 546 catalog.tempe.gov 549 623 flowmon.boulder.noaa.gov 136 583 fs.usda.gov 492 416 house.gov 134 608 gis.ca.gov 459 329 cdc.gov 132 466 csm.ornl.gov 441 201 fda.gov 111 950 fgdc.gov 403 648 forums.census.gov 105 638 archives.gov 367 371 atlassw1.phy.bnl.gov 98 227 oss.fnal.gov 363 942 ida.wr.usgs.gov 90 625 census.gov 342 746 ornl.gov 88 418 ssa.gov 340 608 ncicb.nci.nih.gov 83 902 cfpub2.epa.gov 337 017 ftp2.census.gov 82 547 cfpub.epa.gov 315 116 walrus.wr.usgs.gov 81 758 contractsdirectory.gov 311 625 nps.gov 79 870 lawlibrary.courts.wa.gov 306 410 in.gov 77 346 uspto.gov 286 606 nist.time.gov 77 188 nis.www.lanl.gov 280 106 elections.miamidade.gov 73 863 d0.fnal.gov 262 476 hud.gov 70 787 epa.gov 257 993 ncbi.nlm.nih.gov 68 127 xxx.bnl.gov 238 259 nal.usda.gov 66 756 plankton.gsfc.nasa.gov 205 584 michigan.gov 66 255 trec4-kmeans. A Fabio Crestani Prof. k-means clustering algorithm [Jain and Dubes, Distributed Information Retrieval 110
- 111. Background DIR: Introduction Objectives DIR Architectures Test Collections Broker-Based DIR Evaluation Metrics DIR Evaluation Applications of DIR Evaluation measures DIR evaluation uses the same evaluation measures of IR The benchmark is the centralised IR system, that is DIR is compared with IR over the crawled set of all resources Currently DIR performs almost as well as IR, and in some cases even better Prof. Fabio Crestani Distributed Information Retrieval 111
- 112. Background DIR: Introduction Objectives DIR Architectures Test Collections Broker-Based DIR Evaluation Metrics DIR Evaluation Applications of DIR Essential DIR Evaluation References James Callan Distributed Information Retrieval. In Croft, B., Editor, Advances in Information Retrieval, chapter 5, pages 127-150. Kluwer Academic Publishers, 2000. James Callan, Fabio Crestani, and Mark Sanderson Distributed Multimedia Information Retrieval. Lecture Notes in Computer Science Vol. 2924, Springer-Verlag, 2004. Prof. Fabio Crestani Distributed Information Retrieval 112
- 113. Background DIR: Introduction Objectives DIR Architectures Test Collections Broker-Based DIR Evaluation Metrics DIR Evaluation Applications of DIR Questions? Prof. Fabio Crestani Distributed Information Retrieval 113
- 114. Background DIR: Introduction Vertical Search DIR Architectures Blog Distillation Broker-Based DIR Expert Search DIR Evaluation Desktop Search Applications of DIR Outline 1 Background 2 DIR: Introduction 3 DIR Architectures 4 Broker-Based DIR 5 DIR Evaluation 6 Applications of DIR Prof. Fabio Crestani Distributed Information Retrieval 114
- 115. Background DIR: Introduction Vertical Search DIR Architectures Blog Distillation Broker-Based DIR Expert Search DIR Evaluation Desktop Search Applications of DIR Topics Covered 6 Applications of DIR Vertical Search Blog Distillation Expert Search Desktop Search Prof. Fabio Crestani Distributed Information Retrieval 115
- 116. Background DIR: Introduction Vertical Search DIR Architectures Blog Distillation Broker-Based DIR Expert Search DIR Evaluation Desktop Search Applications of DIR Vertical Search Vertical Specialized subcollection focused on a specific domain (e.g., news, travel, and local search) or a specific media type (e.g., images and video). Vertical Selection The task of selecting the relevant verticals, if any, in response to a user’s query. Idea Classification (does the query require a vertical search?) Resource Selection = Vertical Selection Prof. Fabio Crestani Distributed Information Retrieval 116
- 117. Background DIR: Introduction Vertical Search DIR Architectures Blog Distillation Broker-Based DIR Expert Search DIR Evaluation Desktop Search Applications of DIR Blog Distillation Blog Distillation The task of identifying blogs with a recurring central interest. Idea Blog Feed ⇐⇒ Posts Federated Collection ⇐⇒ Documents Prof. Fabio Crestani Distributed Information Retrieval 117
- 118. Background DIR: Introduction Vertical Search DIR Architectures Blog Distillation Broker-Based DIR Expert Search DIR Evaluation Desktop Search Applications of DIR Expert Search Expert Search The task of identifying experts with a given expertise Idea Experts ⇐⇒ documents authored by expert Resource Selection on different collections of documents Prof. Fabio Crestani Distributed Information Retrieval 118
- 119. Background DIR: Introduction Vertical Search DIR Architectures Blog Distillation Broker-Based DIR Expert Search DIR Evaluation Desktop Search Applications of DIR Desktop Search Desktop Search The task of identifying different file and document types on a desktop relevant to a user query Idea Resource Selection on the type Results Fusion on different documents Prof. Fabio Crestani Distributed Information Retrieval 119
- 120. Background DIR: Introduction Vertical Search DIR Architectures Blog Distillation Broker-Based DIR Expert Search DIR Evaluation Desktop Search Applications of DIR Questions? Prof. Fabio Crestani Distributed Information Retrieval 120