In situ evaluation of entity retrieval and opinion summarization
Download as PPTX, PDF0 likes905 views
This document discusses the findilike system, which ranks hotels and summarizes reviews based on user preferences. It describes two components that can be evaluated: opinion-based entity ranking and review summarization. For evaluation, it proposes interleaving results from different ranking algorithms and randomly mixing phrases from summarization algorithms to observe user clicks, with more clicks indicating a better algorithm. It also describes a mini test bed for local evaluation and submitting new algorithms for online performance testing.
In situ evaluation of entity retrieval and opinion summarization
- 1. In Situ Evaluation of Entity Ranking and Opinion Summarization using www.findilike.com Kavita Ganesan & ChengXiang Zhai University of Illinois @ Urbana Champaign
- 2. What is findilike? • Preference – driven search engine – Currently works in hotels domain – Finds & ranks hotels based on user preferences: Structured: price, distance Unstructured: “friendly service”, “clean”, “good views” (Based on existing user reviews) UNIQUE • Beyond search: Support for analysis of hotels – Opinion summaries – Tag cloud visualization of reviews
- 3. …What is findilike? • Developed as part of PhD. Work – new system (Opinion-Driven Decision Support System, UIUC, 2013) • Tracked ~1000 unique users from Jan - Aug ‘13 – Working on speed & reaching out to more users
- 4. Evaluating Review Summarization Mini Test-bed • Base code to extend • Set of sample sentences • Gold standard summary for those sentences • ROUGE toolkit to evaluate the results • Data set based on - Ganesan et. al 2010
- 5. Evaluating Entity Ranking Mini Test-bed • Base code to extend • Terrier Index of hotel reviews • Gold standard ranking of hotels • Code to generate nDCG scores. • Raw unindexed data set for reference
- 6. Building a new ranking model Extend Weighting Model
- 7. DEMO
- 8. 2 Components that can be evaluated through natural user interaction 1 Ranking entities based on unstructured user preferences Opinion-Based Entity Ranking (Ganesan & Zhai 2012) Summarization of reviews Generating short phrases summarizing key opinions (Ganesan et. al 2010, 2012) 2
- 9. Evaluation of entity ranking • Retrieval – Interleave results Balanced interleaving (T. Joachims, 2002) Base DirichletLM A click indicates preference… Base
- 10. Snapshot of pairwise comparison results for entity ranking # Queries B is better Algorithms DirichletLM, Base, PL2 # Queries A is Better A B CA > CB (A Better) CB > CA (B Better) CA = CB > 0 (Tie) CA = CB = 0 Total DLM Base 30 35 2 5 72 PL2 Base 10 28 3 7 48 … … … … … … …
- 11. Snapshot of pairwise comparison results for entity ranking A B CA > CB (A Better) CB > CA (B Better) Base model better, but DLM not too far behind Base model better CA = CB & > 0 PL2 not (Tie) CA = CB = 0 Total too good DLM Base 30 35 2 5 72 PL2 Base 10 28 3 7 48 … … … … … … …
- 12. Evaluation of review summarization Randomly mix top N phrases from two algorithms ALGO1 ALGO2 Monitor click-through More clicks on phrases from Algo1 vs. Algo2 Algo1 better on per entity basis
- 13. How to submit a new algorithm? Submit code Performance report A B CA > CB (A Better) … … … … CB > CA (B Better) DLM Base 30 35 PL2 Base 10 28 Online Performance Test on mini test bed Sample Code Test Data & Gold Standard Evaluator (nDCG, ROUGE) Mini Testbed Local performance Write Java based code Extend existing code Implementation
- 14. More information about evaluation… eval.findilike.com
- 15. Thanks! Questions? Links • Evaluation: http://eval.findilike.com • System: http://hotels.findilike.com/ • Related Papers: kavita-ganesan.com
- 16. References • Ganesan, K. A., C. X. Zhai, and E. Viegas, Micropinion Generation: An Unsupervised Approach to Generating Ultra-Concise Summaries of Opinions, Proceedings of the 21st International Conference on World Wide Web 2012 (WWW '12), 2012. • Ganesan, K. A., and C. X. Zhai, Opinion-Based Entity Ranking, Information Retrieval, vol. 15, issue 2, 2012 • Ganesan, K. A., C. X. Zhai, and J. Han, Opinosis: A Graph Based Approach to Abstractive Summarization of Highly Redundant Opinions, Proceedings of the 23rd International Conference on Computational Linguistics (COLING '10), 2010. • T. Joachims. Optimizing search engines using clickthrough data. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, KDD ’02, NY, 2002.