![Evaluation - Cube Test
Task Cube
An empty
task cube for
a search task
with 6 subtopics
[Luo et al. CIKM 2013]
24](https://image.slidesharecdn.com/2014-11-20-trec-dd-181114151012/85/Text-REtrieval-Conference-TREC-Dynamic-Domain-Track-2015-24-320.jpg)
![Evaluation - Cube Test
An empty task cube for a search
task with multiple subtopics
A stream of “document water” fills
into the task cube
A new coming relevant document
will increase waters in all its
relevant subtopics
The total height of the water in one
cuboid represents the accumulated
relevance gain for a subtopic
There is a cap for Gains
Total volume in the task Cube is the
total Gain
Cube Test (CT) calculates the rates of how fast a search system
can fill up the task cube as much as possible
[Luo et al. CIKM 2013]
25](https://image.slidesharecdn.com/2014-11-20-trec-dd-181114151012/85/Text-REtrieval-Conference-TREC-Dynamic-Domain-Track-2015-25-320.jpg)
Text REtrieval Conference (TREC) Dynamic Domain Track 2015
- 1. TREC 2015 DYNAMIC DOMAIN TRACK Grace Hui Yang, Georgetown University John Frank, MIT/Diffeo Ian Soboroff, NIST 1
- 2. MOTIVATION Underexplored subsets of Web content Limited scope and richness of indexed content, which may not include relevant components of the deep web temporary pages, pages behind forms, etc. Basic search interfaces, where there is little collaboration or history beyond independent keyword search Complex, task-based, dynamic search Temporal dependency Rich interactions Complex, evolving information needs Professional users A wide range of search strategies 2
- 3. DOMAIN-SPECIFIC SEARCH STRATEGIES Browsing Boolean search & proximity search Entity Search Forward and backward search Date/location search Number/range search Personal collection search Expert search Forum Search Image search, multi-media search 3
- 4. WHY “DYNAMIC DOMAIN”? Domain-specific Search Deep Web Under explored data Professional users Complex information needs 4
- 5. DYNAMIC INFORMATION RETRIEVAL Dynamic Relevance Dynamic Users Dynamic Queries Dynamic Documents Dynamic Information Needs Users change behavior over time, user history Temporal change of Documents, Deep Web, emerging topics Time, geolocation and other contextual change, change in user perceived relevance Rich user-system interaction through queries Knowledge evolves over time Domain- specific SE 5
- 6. OUR GOAL The TREC Dynamic Domain Track envisions a new paradigm, where one can quickly and thoroughly search and organize a subset of the Internet relevant to one's interests. We aim to encourage new research and new systems that provide Fast, flexible, and efficient access to domain-specific content Valuable insight into a domain that previously remained unexplored and addresses shortcomings of centralized Web search We develop evaluation methodologies for systems that discover, organize, and present domain relevant content Technologies for cross-domain adaptation 6
- 7. OUTLINE Introduction Domains Task Evaluation Timeline Discussion 7
- 8. DOMAINS Domain Corpus Counterfeit Pharmaceutic als (Pharma) 30k forum posts from 5-10 forums (total ~300k posts) Which users are working together to sell illicit goods? Ebola One million tweets 300k docs from in-country web sites (mostly official sites) Who is doing what and where? Local Politics 300k docs from local political groups in Pacific Northwest and British Columbia. Who is campaigning for what and why? 8
- 9. DOMAIN I COUNTERFEIT PHARMACEUTICALS 9 Sell ineffective or deadly medications Sell Addictive drugs Indirectly fund botnets and hackers
- 11. UNDER GROUND FORUM ADS Learn about major affiliation programs Handles of employees and connections Activities 11
- 12. DOMAIN II – EBOLA (CRISIS IR) Ongoing crisis 3.3 million Tweets over five days for GPS tagged conversations about Ebola around the globe. 300k docs from in-country web sites (mostly official sites) A set of questions: Where (counties / country) are personalities organizing support of Ebola Viral Disease (EVD) success or perceived failure? What is causing the population to report or not report cases of flu-like symptoms within current or future Ebola Treatment Unit (ETU) sites? How will the local population conduct EVD awareness based off religious, ethnic and tribal education? Where will individuals attempt to garner support and build trust within Liberia? 12
- 13. DOMAIN III – LOCAL POLITICS Public personas Elected officials School boards First Nation activism KBA StreamCorpus: 19 months of timestamped news, blogs, forums >500M tagged by quality NER (BBN Serif) Investigating re-using the KBA query entities Part of ground truthing is already complete Subtopic truthing still required 86 online personas (people) from the Seattle – Vancouver area 13
- 14. OUTLINE Introduction Domains Task Evaluation Timeline Discussion 14
- 15. TASK An interactive, multiple runs of search Starting point: System is given a search query Iterate System returns a ranked list of 5 documents API returns relevance judgments go to next iteration of retrieval until done (system decides when to stop) The goal of the system is to find relevant information for each topic as soon as possible One-shot ad-hoc search is included If system decides to stop after iteration one 15
- 16. TOPICS Assessors know topic descriptions Topics contain multiple subtopics Chief Sean Atlio S1: Who did he meet with S2: Issues he is pushing S3: What crises are affecting his tribe The systems are given the topic/query to start the search Not the subtopics 16
- 17. MULTIPLE RUNS OF RELEVANCE JUDGMENTS Graded relevance judgments 0, 1, 2, 3 Multiple runs of relevance judgments Suppose a topic with 3 subtopics Run 1: Systems returns d1, d2, d3, d4, d5 Relevance judgments: d1: s1 4, s2 2, s3 0 d2: s1 1, s2 0, s3 0 d3: s1 0, s2 0, s3 0 d4: s1 0, s2 0, s3 2 d5: s1 0, s2 0, s3 3 Run 2: Systems returns another set of d1, d2, d3, d4, d5 Another set of relevance judgments … Run N 17
- 18. OUTLINE Introduction Domains Task Example Topics Evaluation Timeline Discussion 18
- 19. PHARMA Nick Danger, aka HellRaiser Who is he selling to What is he selling What are other aliases in other forums Tools and Techniques Motivations? 19
- 20. EBOLA Where are untrained health professionals going to provide care? Find health care locations Figure out how to tell an untrained health professional from trained Identify individuals Track them 20
- 21. LOCAL POLITICS Chief Sean Atlio Who did he meet with Issues he is pushing What crises are affecting his tribe Background knowledge (childhood, etc) Protests or events being planned Continue from KBA 21
- 22. OUTLINE Introduction Domains Task Evaluation Timeline Discussion 22
- 23. EVALUATION METRICS Find relevant information as much as possible and as fast as possible The system decides when to stop Metrics handle relevance, novelty, time/effort, and task completion Multi-dimensional evaluation Candidate Evaluation Metrics: Cube Test (Luo et al., CIKM 2013) u-ERR – cascades as user gathers results Session nDCG (Kanoulas et al., SIGIR 2011) 23
- 24. Evaluation - Cube Test Task Cube An empty task cube for a search task with 6 subtopics [Luo et al. CIKM 2013] 24
- 25. Evaluation - Cube Test An empty task cube for a search task with multiple subtopics A stream of “document water” fills into the task cube A new coming relevant document will increase waters in all its relevant subtopics The total height of the water in one cuboid represents the accumulated relevance gain for a subtopic There is a cap for Gains Total volume in the task Cube is the total Gain Cube Test (CT) calculates the rates of how fast a search system can fill up the task cube as much as possible [Luo et al. CIKM 2013] 25
- 26. UNEXPECTED EXPECTED RECIPROCAL RANK (U-ERR) Variant of ERR for multiple search iterations with feedback: 1. Submit query to search engine 2. Receive ranked list of results 3. Start reading through the list: 4. User examines position n 5. If user finds new knowledge: 6. Update profile 7. Go to 1 with updated topic as query 8. else 9. n += 1 10. Go to 4 u-ERR = 1 / (expected list position of surprise) Figure of merit: depth in the list where user discovers new knowledge 26
- 27. TIME LINE TREC Call for Participation: January 2015 Data Available: March Detailed Guidelines: April/May Topics, Tasks available: June Systems do their thing: June-July Evaluation: August Results to participants: September Conference: November 2015 27
- 28. WHY YOU SHOULD PARTICIPATE 28 Unique, underexplored research direction Good for academics New research Great funding opportunities Easy and Exciting!
- 29. Familiar, Easy Hard = Exciting • Unit of retrieval = Document • Corpus tiny: 1-2 M docs • Specific domains with rich, interesting content features • Content is cleansed, deduplicated, utf8, NER tagged, sentence parses • Iterative, explicit relevance judgment (feedback) from user (API) • Three different domains • Systems submit ranked lists in small batches of five at a time • Relevance judgment consists of: • On topic: True or False • Passage(s): • Char offsets • Subtopics_id • Graded relevance judgment 29
- 30. DISCUSSION 30 Cross-domain Tasks & Procedures
- 31. REFERENCES Jiyun Luo, Christopher Wing, Hui Yang, and Marti Hearst. The Water Filling Model and The Cube Test: Multi-Dimensional Evaluation for Professional Search. CIKM 2013. Evangelos Kanoulas, Ben Carterette, Paul D. Clough, Mark Sanderson. Evaluating Multi-Query Sessions. SIGIR 2011. 31
- 32. THANK YOU TREC Dynamic Domain Website: http://www.trec-dd.org Google group: https://groups.google.com/forum/#!forum/trec-dd/ 32
- 33. DOMAIN I COUNTERFEIT PHARMACEUTICALS 33 Simple product space (though various dosages) Viagra Cialis Vicodin Percocet Complex online advertising space Thousands of online pharmacy storefronts Spam advertising
- 34. Domain-specific SearchWeb Search everyday users one-shot query large user query logs relevance at document level a single, straightforward information need keyword search professional searchers a sequence of queries or actions (e.g. click a node to browse) rich interaction data within the session stricter requirements for relevance - evidence multiple. complex and task- based information needs a wide range of search strategies 34
- 35. AN EXPLORATORY PROCESS User Search Engine Information need Find what city and state Dulles airport is in, what shuttles ride-sharing vans and taxi cabs connect the airport to other cities, what hotels are close to the airport, what are some cheap off-airport parking, and what are the metro stops close to the Dulles airport. 35
- 37. DATA GATHERED Aug 1 – Oct 31, 2010 7 URL/spam + 5 botnet feeds 968M URLs 17M domains Crawled domains for 98% of URLs with 1000s of Firefox instances Significant IP diversity (overcome blacklisting) ~200 purchases from all major programs 37
- 38. SEARCH ENGINES AND PHARMA But the real problem is even worse…. Ephemeral websites – multiple URLs all link to one site Compromised websites Hacked sites redirect to pharmacy stores Need to ID underlying sites and hacking patterns Crawler evasion Cloaking to only show site to customers Simple crawlers won’t get to sales sites
Editor's Notes
- #11: In the pharma arena, that chain is awfully complex. Some of it we can observe with tools like those being developed here. Some we cannot and so need to draw on other sources to inform our interpretation of what we see here.
- #16: A debate of size of the returned documents: 1, 5, 10.
- #17: The importance of each subtopic can be weighted ??
- #38: What was required to do this at the scale you did? What would be required to do this on an ongoing basis as part of Memex? Why purchasing cannot be scaled – security measures they put in place: they call, identity backstops, etc..
- #40: As in many settings tracking the train from user to underlying entity mapping what the user does in cyber space to some thing that happens in the real world.