Final Presentation V3
- 1. WEB RECOMMENDER FINAL REPORT Wei Chen Yue (Jenny) Cui
- 2. OUTLINE Motivation Problem Statement Goals for this project (solution) Requirements Design Framework design Algorithm design Evaluation design Results SE Techniques used in this project Lessons Learned
- 3. MOTIVATION Background People use the web to browse information There are too much information on the web To facilitate this web-browsing process Fast Accurate Existing solution: search engines User types in a query Search engine returns relevant pages
- 4. OUTLINE Motivation Problem Statement Goals for this project (solution) Requirements Design Framework design Algorithm design Evaluation design Results SE Techniques used in this project Lessons Learned
- 5. PROBLEM STATEMENT What is a Web Recommender? A web-browsing tool Recommends relevant web pages to the user while he/she is reading a page on the web. Why is it important? Provides a convenient way to browse the web Automatically recommends relevant information Less effort to make and type in queries Reserves the benefit from the state-of-the-art search engine
- 6. PROBLEM STATEMENT (CONT.) Why is it hard? Making queries from a web page: keyword summarization Search engines are not perfect: post-processing People have different reading goals: how to define relevance?
- 7. OUTLINE Motivation Problem Statement Goals for this project (solution) Requirements Design Framework design Algorithm design Evaluation design Results SE Techniques used in this project Lessons Learned
- 8. GOALS FOR 2009 SPRING SEMESTER Provide a software framework for Web Recommendation Provide basic recommendation algorithms Basic services Baselines for future research on Web Recommendation A tutorial: how to develop your own algorithm based on our framework? Propose an evaluation prototype
- 9. OUTLINE Motivation Problem Statement Goals for this project (solution) Requirements Design Framework design Algorithm design Evaluation design Results SE Techniques used in this project Lessons Learned
- 10. REQUIREMENTS Functional Requirements Given a web page as input, the system should be able to find a list of relevant web pages Provide three recommendation algorithms Baseline HTML-Structure-based Semantic-based A simple GUI for evaluation
- 11. REQUIREMENTS (CONT.) Non-functional Requirements Results can be retrieved in 5 seconds
- 12. OUTLINE Motivation Problem Statement Goals for this project (solution) Requirements Design Framework design Algorithm design Evaluation design Results SE Techniques used in this project Lessons Learned
- 14. Util <<interface>> SearchEngine search(Query q) : List<Page> StructureFeatureRecommender recommend(Page p): List<Page> SemanticFeatureRecommender recommend(Page p): List<Page> YahooSearch search(Query q) : List<Page> BasicRecommender recommend(Page p): List<Page> <<interface>> WebRecommender recommend(Page p): List<Page> GoogleSearch search(Query q) : List<Page> <<interface>> Stemmer stem(String s) : String <<interface>> HTMLStripper strip(Page p) : String <<interface>> HTMLParser parse(Page p) : ParseTree <<interface>> StopwordRemover remove(String s) : String <<interface>> QueryTermFilter filterQueryTerms(List<String> keyTerms) : List<String> FrequencyFilter filterQueryTerms(List<String> keyTerms) : List<String> Classes in Util package are singletons QueryFormulator form(List<String> finalTerms) : Query <<interface>> OrQueryFormulator form(List<String> finalTerms) : Query PorterStemmer stem(String s) : String NaiveHTMLStripper strip(Page p) : String SmartParser parse(Page p) : ParseTree GenericStopwordRemover remove(String s) : String
- 16. create(YahooSearch ys) recommend(p) search(q) List<Page> pages create() create() and(List<String> finalTerms) Query q parse(p) ParsingResult pr stem(pr) Stemming Result sr List<Page> pages create() filterQueryTerms(List<String> keyTerms) requestRecommends(Page p) List<String> finalTerms :StructureFeatureRecommender :YahooSearch Extract features to form key terms. :EvaluationGUI :QueryFormulator :HTMLParser :Stemmer :FrequencyFilter
- 17. OUTLINE Motivation Problem Statement Goals for this project (solution) Requirements Design Framework design Algorithm design Evaluation design Results SE Techniques used in this project Lessons Learned
- 18. ALGORITHM: BASELINE Algorithm Strip off HTML tags (e.g. </html>) Remove non-word tokens (e.g. “/**/”) Remove stop words (e.g. “the”) Example Input page: http://en.wikipedia.org/wiki/Entropy Output query: Entropy, free, encyclopedia, Jump, search, article
- 19. ALGORITHM: HTML STRUCTURE Algorithm Parse HTML page Extract text content from node <title> and <a> Remove stop words (e.g. “the”) Select the 10 most frequent words Example Input page: http://en.wikipedia.org/wiki/Entropy Output query: ISBN, edit, entropy, thermodynamics, Entropy, energy, system, law, heat, thermodynamic
- 20. ALGORITHM: SEMANTIC FEATURES Algorithm Baseline + named entities with highest frequency (top 5) Example Input page: http://en.wikipedia.org/wiki/Entropy Output query: ISBN, University, Press, Boltzmann, John
- 21. OUTLINE Motivation Problem Statement Goals for this project (solution) Requirements Design Framework design Algorithm design Evaluation design Results SE Techniques used in this project Lessons Learned
- 22. EVALUATION Evaluation form: Evaluation criteria: Modified Average Precision Example: AveP = (0+1/2+2/3) /3 = 0.389 Input page Recommended page Relevance http://en.wikipedia.org/wiki/Natural_language_processing http://research.microsoft.com/jump/50176 0 http://nlp.stanford.edu/ 1 http://www.aaai.org/aitopics/html/natlang.html 1
- 23. TEST DATA SELECTION Input pages from 5 topics: “ Harry Porter” “ Waterboarding” “ Wei Chen@CMU homepage” “ Entropy (thermodynamics)” “ How to make Sushi” Dimensions Popular vs. Unpopular (“Harry Porter”, “Wei Chen”) Ambiguous vs. Unambiguous (“Entropy”, “Sushi”) New vs. Old (“Waterboarding”, “Entropy”) Procedural vs. Conceptual (“How to”, “Entropy”) Technological vs. Mass media (“Entropy”, “Harry Porter”)
- 24. OUTLINE Motivation Problem Statement Goals for this project (solution) Requirements Design Framework design Algorithm design Evaluation design Results SE Techniques used in this project Lessons Learned
- 25. TEST DATA We evaluate on 5 topics and 3 algorithms. We have total of 15 categories. Each category has 5 recommended WebPages. We have total of 5 evaluators. Each of them scored 75 web pages.
- 29. KAPPA We can achieve very good inter-coder agreement, if we revise our score criteria. We all seem to agree with Anthony (maybe we should ask him to revise our score criteria).
- 30. CONCLUSION Topics play an important role in the evaluation results. The more popular and resourceful the topic is the better the evaluation results are. The time sensitive topic has the highest invalid page rate. At this point we cannot make any conclusion about our algorithms. Only Structure algorithm seems better. We don’t know what makes the difference in the evaluation results of the three algorithms. We need to design a new experiment to analyze query terms which the three algorithms generated in order to answer this question. We should include the condition in which it uses the human generated query terms as our control condition in our new experiment.
- 31. OUTLINE Motivation Problem Statement Goals for this project (solution) Requirements Design Framework design Algorithm design Evaluation design Results SE Techniques used in this project Lessons Learned
- 32. SE TECHNIQUES Iterative Process at each stage Design Iteration 1: Initial design of framework Composite-pattern based evaluation design Iteration 2: Added query formulator and query filter Simplified evaluation design Implementation: Iteration 1: Initial implementation of framework Implemented evaluation component based on composite pattern Iteration 2: Implemented query formulator and query filter Implemented simplified version of evaluation GUI
- 33. SE TECHNIQUES (CONT.) Evaluation: Iteration 1: Pilot study Weighted average relevance score Iteration 2: 5 raters, 5 input pages Modified average precision Used Wiki to coordinate Test-driven method for implementation
- 34. OUTLINE Motivation Problem Statement Goals for this project (solution) Requirements Design Framework design Algorithm design Evaluation design Results SE Techniques used in this project Lessons Learned
- 35. WHAT CHANGED OVER THE SEMESTER Our evaluation GUI went through several rounds of changes. Relation database composite pattern Excel We planned to use our evaluation GUI in our final evaluation, but we were unable to do it because the speed of loading a webpage is too slow in the GUI. We switched to use Excel files for the evaluation.
- 36. WHAT WOULD WE CHANGE IN OUR APPROACH IN THE FUTURE We want to improve our risk analysis: one tricky thing about risk analysis is that it is unexpected. We didn’t expect that speed will be a problem of our GUI. Evaluation took more time than we had thought. We want to allow more time for evaluation, because we need time for pilot study before we conduct the experiment. Then we can have detailed and systematic analysis of the algorithms and improve our algorithms based on the analysis. Time management: We should start evaluation early so that we can improve our algorithms based on evaluation results.
- 37. ACKNOWLEDGEMENTS Thanks Dr. Nyberg, Dr. Tomasic, Shilpa and Hideki for valuable comments and suggestions on our project through out the semester Thanks our raters for the evaluation task Thanks our classmates for helpful discussions