Linking media, data, and services
1 like2,018 views
The document discusses the integration of semantic technologies in multimedia analysis and the challenges associated with linking multimedia data and metadata. It outlines various components involved in the process, including knowledge bases, reasoning, and execution of semantic analysis queries. Additionally, it addresses questions regarding the effectiveness, evaluation, and potential failures of the proposed solutions in enhancing multimedia analysis.
![{
?image a foaf:Image.
}
=>
{
?regionList a rdf:List.
_:region owl:oneOf ?regionList;
rdfs:subClassOf face:FaceRegion,
[ owl:hasValue ?image;
owl:onProperty imreg:regionOf ].
}
service rule](https://image.slidesharecdn.com/linkingmediadataandservices-160530104719/85/Linking-media-data-and-services-26-320.jpg)
![_:face owl-s-sparql:bindsVariable "face";
owl-s-sparql:boundTo
[face:isFaceOf dbpedia:Koen_De_Bouw].
_:person owl-s-sparql:bindsVariable "person";
owl-s-sparql:boundTo dbpedia:Koen_De_Bouw.
<Loft.jpg#xywh=221,91,56,56>](https://image.slidesharecdn.com/linkingmediadataandservices-160530104719/85/Linking-media-data-and-services-35-320.jpg)
![{
?image imreg:region ?regionA, ?regionB.
?regionA owl:differentFrom ?regionB.
?regionA imreg:regionDepicts [face:isFaceOf ?personA].
?regionB imreg:regionDepicts [face:isFaceOf ?personB].
}
=>
{
?personA owl:differentFrom ?personB.
}.
knowledge rule](https://image.slidesharecdn.com/linkingmediadataandservices-160530104719/85/Linking-media-data-and-services-44-320.jpg)
![_:face owl-s-sparql:bindsVariable "face";
owl-s-sparql:boundTo [face:isFaceOf ?person].
_:person owl-s-sparql:bindsVariable "person";
owl-s-sparql:boundTo ?person.
({?person = dbpedia:Koen_De_Graeve.}
{?person = dbpedia:Bruno_Vanden_Broucke.})
e:disjunction [a e:T].
<Loft.jpg#xywh=45,121,51,51>](https://image.slidesharecdn.com/linkingmediadataandservices-160530104719/85/Linking-media-data-and-services-47-320.jpg)
Linking media, data, and services
- 1. Linking media, data, and services Ruben Verborgh Ghent University – iMinds
- 5. The problem The solution The problems
- 6. The problem The solution The problems
- 9. How can semantic technologies assist with multimedia analysis?
- 15. • subrequest 1 … • subrequest n Engine EXIFJPEG metadata JPEG inputrequest
- 16. The problem The solution The problems
- 17. multimedia data + metadata Supervisor requested metadata description Blackboard requested metadata description gathered metadata Collaborator 3 semantic capability description semantic requirements description Collaborator 2 semantic capability description semantic requirements description Collaborator 1 semantic capability description semantic requirements description multimedia data + metadata r requested metadata description metadata Blackboard requested metadata description gathered metadata n 1 tic ity ion multimedia data + metadata requested metadata description metadata Blackboard requested metadata description gathered metadata
- 18. multimedia data + metadata Supervisor requested metadata description Blackboard requested metadata description gathered metadata Collaborator 3 semantic capability description semantic requirements description Collaborator 2 semantic capability description semantic requirements description Collaborator 1 semantic capability description semantic requirements description multimedia data + metadata r requested metadata description metadata Blackboard requested metadata description gathered metadata c y on r 1 ntic bility ption multimedia data + metadata visor requested metadata description metadata Blackboard requested metadata description gathered metadata or3 mantic ability cription tor2 semantic capability description rator1 semantic capability description
- 23. start
- 24. <Loft.jpg> a foaf:Image. <Loft.jpg> foaf:depicts ?person. input goal
- 25. composition
- 26. { ?image a foaf:Image. } => { ?regionList a rdf:List. _:region owl:oneOf ?regionList; rdfs:subClassOf face:FaceRegion, [ owl:hasValue ?image; owl:onProperty imreg:regionOf ]. } service rule
- 27. { ?region a face:FaceRegion. } => { ?region imreg:regionDepicts ?face. ?face a face:Face; face:isFaceOf ?person. ?person a foaf:Person. }. service rule
- 28. { ?image imreg:region ?region. ?region imreg:regionDepicts ?face. ?face face:isFaceOf ?person. } => { ?image foaf:depicts ?person. }. knowledge rule
- 29. Example 7.8 Composition from image input to person request 8 >>>>>>>< >>>>>>>: Iæ := (param1 = ) √ ( ) IFD := (regionList = ) √ (image = ) IFR := (f ace = , per son = ) √ (region = ) I≤ := ( ) √ (per son = ) Iæ and I≤ are the virtual start and end services, respectively. Iæ IFD IFR I≤ 62
- 30. execution
- 31. <Loft.jpg#xywh=45,121,51,51> a face:FaceRegion; imreg:regionOf <Loft.jpg>. <Loft.jpg#xywh=221,91,56,56> a face:FaceRegion; imreg:regionOf <Loft.jpg>. <Loft.jpg#xywh=535,118,43,43> a face:FaceRegion; imreg:regionOf <Loft.jpg>. <Loft.jpg#xywh=734,83,69,69> a face:FaceRegion; imreg:regionOf <Loft.jpg>.
- 33. Example 7.8 Composition from image input to person request 8 >>>>>>>< >>>>>>>: Iæ := (param1 = ) √ ( ) IFD := (regionList = ) √ (image = ) IFR := (f ace = , per son = ) √ (region = ) I≤ := ( ) √ (per son = ) Iæ and I≤ are the virtual start and end services, respectively. Iæ IFD IFR I≤ 62
- 35. _:face owl-s-sparql:bindsVariable "face"; owl-s-sparql:boundTo [face:isFaceOf dbpedia:Koen_De_Bouw]. _:person owl-s-sparql:bindsVariable "person"; owl-s-sparql:boundTo dbpedia:Koen_De_Bouw. <Loft.jpg#xywh=221,91,56,56>
- 37. failure
- 38. alternative plan
- 39. CONSTRUCT WHERE { dbpedia:Koen_De_Bouw ?p ?o. } CONSTRUCT WHERE { ?s ?p dbpedia:Koen_De_Bouw. }
- 40. dbpedia:Koen_De_Bouw a dbpedia-owl:Actor; rdfs:label "Koen De Bouw"@nl. dbpedia:Bruno_Vanden_Broucke a dbpedia-owl:Actor; rdfs:label "Bruno Vanden Broucke"@nl. ! dbpedia:Loft_%282008_film%29 dbpprop:director dbpedia:Erik_Van_Looy; dbpprop:name "Loft"@nl; dbpprop:producer dbpedia:Woestijnvis; dbpprop:starring dbpedia:Bruno_Vanden_Broucke, dbpedia:Filip_Peeters, dbpedia:Jan_Decleir, dbpedia:Koen_De_Bouw, dbpedia:Koen_De_Graeve, dbpedia:Matthias_Schoenaerts, dbpedia:Veerle_Baetens.
- 41. { ?image foaf:depicts ?person ?person foaf:knows ?acquaintance. } => { ?image face:maybeDepicts ?acquaintance. }. knowledge rule
- 42. recomposition
- 43. acquaintances to the parameter. Note that the actual process is slightly more complex, but some details were omitted for brevity. Example 7.14 Adapted composition, recovering from the face recognition failure 8 >>>>>>>< >>>>>>>: ... I0 æ := (param1 = ) √ ( ) I0 FR := (f ace = , per son = ) √ (region = , candidates = ) Iæ IFD IFR I≤ I0 æ I0 FR 67
- 44. { ?image imreg:region ?regionA, ?regionB. ?regionA owl:differentFrom ?regionB. ?regionA imreg:regionDepicts [face:isFaceOf ?personA]. ?regionB imreg:regionDepicts [face:isFaceOf ?personB]. } => { ?personA owl:differentFrom ?personB. }. knowledge rule
- 45. execution
- 47. _:face owl-s-sparql:bindsVariable "face"; owl-s-sparql:boundTo [face:isFaceOf ?person]. _:person owl-s-sparql:bindsVariable "person"; owl-s-sparql:boundTo ?person. ({?person = dbpedia:Koen_De_Graeve.} {?person = dbpedia:Bruno_Vanden_Broucke.}) e:disjunction [a e:T]. <Loft.jpg#xywh=45,121,51,51>
- 48. solution
- 51. The problem The solution The problems
- 52. ἀπὸ μηχανῆς θεός
- 53. deus ex machina
- 55. What are the research questions? ! How can semantic technologies assist with multimedia analysis?
- 56. What are the hypotheses? ! This solution does analysis better than… what?
- 57. How to evaluate this solution? ! Is our solution a good one or a bad one?
- 58. How can this solution fail? ! Can (existing) knowledge always fill the gap?
- 59. The problem The solution The problems
- 61. Can we link images like we link data? Can we have a Linked Media Cloud?
- 62. Linking media, data, and services @RubenVerborgh ruben.verborgh.org