Crowdsourced evaluation of semantic patterns for recommendations
0 likes548 views
The document describes a study that used crowdsourcing to evaluate how well semantic patterns discovered in Linked Data can improve diversity in recommender systems. Workers on Crowdflower were shown movie recommendations generated by DBpedia patterns and an Amazon recommendation, and asked to choose the best option. Their choices and reasons were collected. The study found that patterns' global frequency and length correlated most with users' choices, suggesting these statistics indicate how suitable patterns are for enabling diversity in recommendations.
Crowdsourced evaluation of semantic patterns for recommendations
- 1. Crowdsourced Evaluation of Semantic Patterns for Recommendations Valentina Maccatrozzo, Lora Aroyo and Willem R. van Hage VU University Amsterdam
- 2. The problem Recommender systems help people cope with the amount of information available on the Internet BUT standard methods tend to suggest only similar items, preventing diversity. Flickr:PeppermintFoa
- 3. What to do? Linked Data enables us to discover connections between items that otherwise do not surface Flickr: reedster Use semantic patterns to improve diversity in recommendation results
- 4. How to evaluate? Ask the crowd!! Flickr: mac steve
- 5. Want to know more?? Come & look for my poster!!! Crowdsourced Evaluation of Semantic Patterns for Recommendations Valentina Maccatrozzo, Lora Aroyo and Willem Robert van Hage Department of Computer Science, The Network Institute VU University Amsterdam PATTERN STATISTICS Global Frequency: how many times a pattern appears in the source. Local Frequency: how many times a pattern appears in the source considering the instantiation of the source type (i.e. Schindler's List). Type Frequency: how many times a pattern appears in the source considering the instantiation of the type(s) involved (i.e. person). Pattern Length: number of properties involved in the pattern. RESULTS Gentlemen Prefer Blondes Pulp Fiction The Final Cut Easy A The Town The Silence of the Lambs Lincoln Scarface Schindler’s List Black Hawk Down Ocean’s Twelve Heat 0.2 0.3 0.4 0.5 0.6 0.3 0.4 0.5 0.6 Topics distance Genresdistance Starting Movies a a a a a a a a a a a a Amadeus Collateral Crazy, Stupid, Love. Lincoln Ocean’s Eleven Saving Private Ryan Schindler’s List Seven Seven Psychopaths Some Like It Hot The Departed The Untouchables RECOMMENDATION DIVERSITY is calculated as the semantic distance between genres and topics of the recommended items and the items in the user profile. Setup CrowdFlower workers choose a movie recommendation from 5 options: • 4 recommendations generated by DBpedia patterns. • One is from Amazon recommendations. For each choice we collect also the motivation. Some numbers ✓ 36 tests ✓ 12 movies ✓ 3 genres ✓ 12 patterns (people-based) ✓ 722 contributions ✓ 28 spammers ✓ 157 hours You are buying the movie Lincoln for a friend and you want to get the "buy one, get two" promotion. Which of the allowing movies would you match with Lincoln in order to surprise your friend with something not trivially related? Lincoln is a 2012 biographical drama film directed by Steven Spielberg, starring Daniel Day-Lewis as Abraham Lincoln and Sally Field as Mary Todd Lincoln. The film is based on Doris Kearns Goodwin's biography of Lincoln, Team of Rivals: The Political Genius of Abraham Lincoln. Offered Movies The Three Burials of Melquiades Estrada is a 2005 American drama film directed by Tommy Lee Jones and written by Guillermo Arriaga. It stars Tommy Lee Jones, Barry Pepper, Julio Cedillo, and Dwight Yoakam. Inception is a 2010 science fiction action heist film which was written, co- produced, and directed by Christopher Nolan. The film features an international ensemble cast including Leonardo DiCaprio, Ken Watanabe, Joseph Gordon-Levitt, Marion Cotillard, Ellen Page, Tom Hardy, Cillian Murphy, Dileep Rao, Tom Berenger, and Michael Caine. Eight Below is a 2006 American adventure film directed by Frank Marshall and written by David DiGilio. It stars Paul Walker, Jason Biggs, Bruce Greenwood and Moon Bloodgood. It was released theatrically on February 17, 2006 and distributed by Walt Disney Pictures in the United States Zero Dark Thirty is a 2012 American action thriller film about the special operations forces mission to capture or kill Osama bin Laden. The film is directed by Kathryn Bigelow with screenplay by Mark Boal, both associated with The Hurt Locker, another military action thriller. It will star Jessica Chastain, Joel Edgerton and Mark Strong. Schindler's List is a 1993 film about Oskar Schindler, a German businessman who saved the lives of more than a thousand mostly Polish- Jewish refugees during the Holocaust by employing them in his factories. The film was directed by Steven Spielberg, and based on the novel Schindler's Ark by Australian novelist Thomas Keneally. Movie 1 Movie 2 Movie 3 Movie 4 Movie 5 1. Select one movie: (required) Movie 1 Movie 2 Movie 3 Movie 4 Movie 5 Select one of the movies you want to match in the "buy one. get two" promotion (as explained above) 2. Explain why did you select that movie (in question 1) (required) Write a full sentence with minimum 10 words 3. Type here the 3rd word of the synopsis of the movie you selected (in question 1) (required) EXPERIMENT Source type (movie) Schindler's List Steven Spielberg director Lincolnproducer Property1 Property2 Type (person) Target type (movie) SEMANTIC PATTERNS connect a source type with a target type through property-type pairs. 0.32 -0.35 -0.40 Global Frequency Local Frequency Type Frequency Length All features Global & Local Frequency Global & Type Frequency Global Frequency & Length Local & Type Frequency Local Frequency & Length Type Frequency & Length Global & Local & Type Frequency Global & Local Frequency & Length Local & Type Frequency & Length -0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 Global frequency and pattern length are the statistics which correlate higher with the users' choices. ! Recommender systems help people cope with the amount of information available on the Internet. Standard methods tend to suggest only similar items, preventing diversity. Extend existing approaches with semantic patterns to improve diversity in recommendation results. Linked Data enables us to discover connections between items that otherwise do not surface. ✓ The goal of this study is to identify statistics of the patterns that indicate their suitability for the (diversity enabled) recommendation process.