Using semantics to improve interactive information access
Download as PPTX, PDF2 likes601 views
The document discusses improving interactive information access using semantics, emphasizing the importance of understanding intended messages and context behind media assets. It outlines mechanisms for identifying and annotating media, presents examples of systems for user interaction with metadata, and highlights the role of semantics in facilitating flexible access to media collections. The document also explores applications like the Cewe Color photobook and video documentaries, showcasing how metadata can enhance user experience and content manipulation.
Using semantics to improve interactive information access
- 1. Using semantics to improve interactive information access Lynda Hardman http://www.cwi.nl/~lynda CWI, Interactive Information Access UvA, Institute for Informatics http://www.flickr.com/photos/iboy/4528401870/
- 2. Google: Rembrandt van Rijn 2
- 4. Interactive Information Access • Users need support for – the processing of information-bearing content – in one or more media types – for their specific task • We need to be aware that there is more than the information “expressed'' by the media asset itself, e.g. – the intended purpose of the creator – the context in which the media asset was created 4
- 5. We don’t care about the media! We need to enable – the processing of information-bearing content – of one or more media types – that can be interpreted by end users End-users are primarily interested in – the meaning conveyed by a combination of media assets – interacting further with the media • as part of complex search task • passing it on to someone else in media “chain” 5 http://www.flickr.com/photos/benheine/4687572408/
- 6. How can we get this to work? We need mechanisms – for identifying (part of) an individual media asset – for associating metadata with an identified fragment – for agreeing on the meaning of metadata – that enable larger meaningful structures to be composed, identified and annotated http://www.flickr.com/photos/jannem/3312115991 6
- 7. Outline of talk • Explain information processes in which media and metadata play a role – “canonical processes of media production” • Example systems showing different types of user interaction enabled by media and metadata – MultimediaN E-Culture – VoxPopuli – EventMedia 7
- 8. Understanding Multimedia Applications Workflow • Identify and define a number of canonical processes of media production • Community effort –2005: Dagstuhl seminar – 2005: ACM MM Workshop on Multimedia for Human Communication –2008: Multimedia Systems Journal Special Issue (core model and companion system papers) editors: Frank Nack, ZeljkoObrenovic and Lynda 8 Hardman
- 9. CanonicalReduced to the simplest and most significant form possible, without loss of generality
- 10. Overview of Canonical Processes 10
- 11. Example: CeWe Color PhotoBook • Application for authoring digital photo books • Automatic selection, sorting and ordering of photos – Context analysis methods: e.g., timestamp, annotation – Content analysis methods: e.g., color histograms, edge detection • Customized layout and background http://www.cewe-photobook.com 11
- 12. CeWe Color PhotoBook Processes My winter ski holidays with my friends Premeditate Construct Message 12
- 13. CeWe Color PhotoBook Processes • Media assets are captured, generated or transformed Create
- 14. CeWe Color PhotoBook Processes Annotate 14
- 15. CeWe Color PhotoBook Processes 15
- 16. CeWe Color PhotoBook Processes Query Organize 16
- 17. Organize using domain annotations chiaroscuro Caravaggists 1623 1628 1631 1628 1638 Rembrandt 17
- 18. CeWe Color PhotoBook Processes Publish Distribute 18
- 19. CeWe Color PhotoBook Processes 19
- 20. Canonical Processes SAMT 2009 Tutorial: A Semantic Multimedia Web, 1 December 20
- 21. How can we use Semantics to support Interactive Information Access? • Long term goal to find and present information to end-users – In a way that is useful to them • We understand how to design information interfaces by hand. – How can metadata help us in giving more flexible access to media collections? • We can link media assets to existing linked data, and use this to improve presentation, e.g. by – Selecting a sub-set – Grouping, ordering and linking media assets – Influencing the presentation 21
- 22. How can semantics help? • What can be expressed explicitly? – the message to be conveyed – objects that are depicted in a media asset – domain information (e.g., art, painter) – human communication roles (discourse) • What can they be used for? – disambiguating queries – grouping similar items for conveying topic breadth – visualizing items for presentation, e.g. timeline, map – finding similar items – … 22
- 23. Browsing annotated collections of cultural heritage artefacts • Users interested in cultural heritage, exploring artefactsavailable in repository • Searching across multiple, linked collections • Thesaurus structure used for identifying topics • Artworks grouped into different topic structures to present results 23
- 24. E-Culture Linked Data Cloud
- 26. Use of linked data in E-Culture • Query construction – auto-completion uses strings found in “data” and “concepts” – suggestions are grouped and ordered using links among items • Result set – uses empirical balance between “closeness” to search string and non-intuitive path • Result presentation – uses grouping of result set to show breadth of results – uses no particular ordering within each group 26
- 27. Generating video documentaries from annotated media repositories Stefano Bocconi, Frank Nack(CWI, Amsterdam)
- 28. Outline • Motivation • Example • Scenarios • Technical details – Annotations – Editing Process • Conclusions 28
- 29. Video Documentaries on the Web • Traditional video authoring: there is only one final version, what is shown is the choice of the author/editor • Proposed video authoring: – Annotate the video material semantics – Show automatically what the user asks to see, using presentation forms a film editor would use 29
- 30. Video material • Focus on video interviews about controversial issues • Interview with America video footage with interviews and background material about the opinion of American people after 9- 11www.interviewwithamerica.com 30
- 31. Example: What do you think of the war in Afghanistan? “I am never a fan of military action, in the big picture I don’t think it is ever a good thing, but I think there are circumstances in which I certainly can’t think of a more effective way to counter this sort of thing…” 31 http://www.flickr.com/photos/soldiersmediacenter/1183836576
- 32. What do you think of the war in Afghanistan? War has Two billions never solved dollar bombs anything on tents I am not a I cannot think fan of of a more military effective actions solution 32
- 33. The annotations • Rhetorical – Rhetorical Statement (mostly verbal, but visual also possible) – Argumentation model: Toulmin model • Descriptive – Question asked – Interviewee (social) – Filmic next slide 33
- 34. Filmic annotations Continuity, e.g. – lighting conditions – background sound – gaze direction of speaker left, centre, right – framing continuity close-up, medium shot, long shot We need your metadata! – camera movement none, pan left/right, shaking, tilt up/down, zoom in/out 34
- 35. Statement encoding • Statement formally annotated: – <subject><modifier><predicate> – E.g. “warbestsolution” • A thesaurus containing: – Terms (155) – Relations between terms: similar(72), opposite(108), generalization(10), spe cialization(10) – E.g. waroppositediplomacy 35
- 36. Connect statements • Using the thesaurus, generate related statements and query the repository “war best solution”, “diplomacy best solution”, “war not solution” • Create a graph of related statements – nodes are the statements (corresponding to video segments) – edges are either support or contradict 36
- 37. Semantic Graph diplomacy best solution war best solution = support war not solution = contradict 37
- 38. Toulmin model Data Claim Qualifier Warrant Condition Concession Backing 57 Claims, 16 Data, 4 Concessions, 3 Warrants, 1 38
- 39. Analysis of the Example Two billions dollar bombs on tents Claim contradict Claim I cannot think of a more effective solution weaken Concession I am not a fan of military actions Claim support War has never solved anything 39
- 40. Facts and features • Annotations: 1 hour annotated, 15 interviews, 60 interview segments, 120 statements • Partially tunable: examining the Segment graph gives feedback on the quality of the annotations and the thesaurus S8 S1 S7 S10 = support S2 S6 S9 = contradict S4 S3 S5 40
- 41. Controlling the Bias • Video documentaries are not neutral account of reality: the selection and editing of the footage expresses a point of view • Editing strategy: – Balanced – Pro opinion X – Against opinion X • We use: – Logos (the statements) – Ethos (based on user profile) – Film editing (e.g. framing, gaze) 41
- 42. Vox Populi interface 42
- 43. Conclusions • Automatic generation of video interviews augmented with supporting and/or contradicting material • The user can determine the subject and the bias of the presentation • The documentarist can add material and let the system generate new documentaries 43
- 44. Pointers & Acknowledgments • Demo available at: http://www.cwi.nl/~media/demo/VoxPopuli/ • VoxPopuliresearch was funded by the Dutch national ToKeN I2RP and CHIME projects 44
- 46. EventMedia Interlinking • Linking Agents with – Freebase, Dbpedia, MusicBrainz • Linking Venues with – Geonames, Dbpedia, Foursquare (via Uberblic) • Linking Events with – Last.fm, Upcoming, Eventful • Linking Categories with – Facebook, Eventful, Upcoming, Zevents, LinkedIn,Eventbrite, TicketMa ster • Linking Users with – Social Graph API 02/09/2010 -
- 47. EventMedia 3 • size of different events to depend on no. of participants (popularity) • image itself chosen most viewed image on flickr • use image clustering to find largest numbers of similar images -> more important -> bigger; • Looking at event (several hundreds) • use (real-time) image clustering to show most different images. • Metadata from tags to detect poster, ticket, stage, vocal 47
- 48. What are my messages? • Metadata associated with media assets can be used for different stages of interactive access • Metadata can be created and added by hand, linked automatically or automatically extracted • The message itself can be made explicit (more metadata) • Media content and metadata can be passed around and among systems • We need community agreement on how to do this (e.g. canonical processes) • Users can be given much richer and more flexible access to (semantically annotated) media content, but… • we need to understand why we are generating metadata and store it in a reusable way 48 48
- 49. Acknowledgements Jacco van Frank Nack Raphaël Troncy Ossenbruggen Stefano Alia Amin Michiel Andre Fialho Bocconi Hildebrand 49
- 50. "I have seen the dark universe yawning Where the black planets roll without aim, Where they roll in their horror unheeded, Without knowledge or luster or name.” -- H. P. Lovecraft, Nemesis http://www.flickr.com/photos/turbojoe/421680689/
Editor's Notes
- #5: For example, for creating an exhibition about Rembrandt van Rijn.If I want to use semantics for improving interactive information access, then what do I mean by Interactive information access?While user task oriented, we need to remember that a picture is not WYSIWYG…I just discussed the paintings of Rembrandt, and have now planted the idea that there may be a forgery of this painting. Next time you see a picture of this painting you will not just think of a Rembrandt painting, but will start wondering whether there was a forgery…
- #6: In some sense we don’t even care about the media.We care about the story or the information conveyed by the media. (the narrative in a broad sense of the word)E.g., in the context of a complex search task; or passing the media along to someone else in the “media chain”
- #7: Tools aren't enough - we need them to create houses - butthey don't tell us how to be architects.Similarly, if we want to communicate using media, then we need to firstunderstand what it is we want to communicate and how we are able to communicate it.
- #14: After a process of premeditation, however short or long, at some point a media asset is created. Some device or another is used to collect images or sound for a period of time, be it photo or video camera, scanner, sound recorder, heart-rate monitor, MRI etc. A media asset may already exist, and the creation process is the transformation of some exist asset to a new one (e.g. to make the sky look brighter). Images can be created with image editing programs, or generated by transforming one or more existing images. We are not interested in the method of creation per se. If the method is considered as significant, however, then this information should be recorded as part of the annotation.
- #24: Dutch project with a number of partners, technical and cultural heritage, including the Rijksmuseum in Amsterdam.Use canonical termsOriginally technology driven. The demo is to demonstrate the possibilities (2006-2008).Currently in progress is a user centric design process to prototype and evaluate an interface that supports topic search for (semi-) expert users (2009)http://eculture.cs.vu.nl/europeana/session/search
- #25: [[Some coloured circles would be nice for:]Artworks: Rijksmuseum, Louvre, Tropenmuseum, RMV, Bibliopolis, RKD, ArtchiveGeneral concepts (thesauri): Wordnet – English, Dutch & FrenchDomain-specific thesauri: AAT, ULAN (people)Links were created among these, by researchers in Guus Schreiber’s group (VUA)
- #46: Next slide:size of different events to depend on no. of participants (popularity)image itself chosen most viewed image on flickruse image clustering to find largest numbers of similar images -> more important -> bigger;Looking at event (several hundreds)use (real-time) image clustering to show most different images.Metadata from tags to detect poster, ticket, stage, vocal
- #48: Work in progressInterface motivates: how to represent image. Need no. of participants.Link to geo-location (facets).Event Categories, events from last.fm
- #49: These interactive grouping, story telling at least as important as only searchCheck the slide for details
- #51: Independent worlds will make links toattain astate of cosmic harmony among the celestial spheres