Enrichment of News Show Videos with Multimodal Semi-Automatic Analysis
1 like658 views
The LinkedTV project aims to enhance television broadcasts by linking them to web content through intelligent video analysis and interactive multimedia services for non-professional users. It incorporates advanced techniques such as shot segmentation, concept detection, and automatic speech recognition while addressing various challenges related to localized dialects and video content sensitivity. Future plans include incorporating new detection methods and further improving existing techniques to enhance user engagement with broadcast content.
![Segmentation
www.linkedtv.eu
Shot segmentation technique Spatio-temporal Segmentation
[Tsamoura et. al., 2008] [Mezaris et. al., 2004]
News show video performance: News show performance: Good
“remarkably well” False positives due to:
Out of 269 shots detected: Camera movement or zoom in/out (~ 55 %)
2 had wrong starting points Gradual transition between frames (~ 10 %)
Erroneous motion vectors (~ 35 %)
4 contained multiple shots
11 were too short to evaluate
Unwanted effect: false recognition of
properly moving banners which do not yield
additional information
V. Mezaris, I. Kompatsiaris, N. V. Boulgouris, and M. G. Strintzis, "Real-time compressed-domain spatiotemporal segmentation and ontologies
for video indexing and retrieval", IEEE Transactions on Circuits and Systems for Video Technology, vol. 14, no. 5, pp. 606-621, May 2004.
E. Tsamoura, V. Mezaris, I. Kompatsiaris, "Gradual transition detection using color coherence and other criteria in a video shot meta-
segmentation framework", IEEE International Conference on Image Processing, Workshop on Multimedia Information Retrieval (ICIP-MIR
2008), San Diego, CA, USA, October 2008, pp. 45-48.
8 Information Technologies Institute
Centre for Research and Technology Hellas](https://image.slidesharecdn.com/wp1presentationnem12-130131063803-phpapp02/85/Enrichment-of-News-Show-Videos-with-Multimodal-Semi-Automatic-Analysis-8-320.jpg)
![Concept Detection
www.linkedtv.eu
Method was described in [Moumtzidou et.
al., 2011]
346 concepts from TRECVID 2011 SIN task
Overall performance:
Correctly detected concepts > 64 %
About 25 % of them are characterized as
particularly useful mostly related to detecting
persons (e.g., person, face, adult)
Erroneous concepts vary between 22% -
42% and in many cases achieve high scores
(e.g., outdoor, amateur video)
Visit: http://mklab.iti.gr/eventdetection-linkedtv/
A. Moumtzidou, P. Sidiropoulos, S. Vrochidis, N. Gkalelis, S. Nikolopoulos, V. Mezaris, I. Kompatsiaris, I. Patras, "ITI-CERTH participation
to TRECVID 2011", Proc. TRECVID 2011 Workshop, December 2011, Gaithersburg, MD, USA.
9 Information Technologies Institute
Centre for Research and Technology Hellas](https://image.slidesharecdn.com/wp1presentationnem12-130131063803-phpapp02/85/Enrichment-of-News-Show-Videos-with-Multimodal-Semi-Automatic-Analysis-9-320.jpg)
![Automatic Speech Recognition
www.linkedtv.eu
Automatic speech recognition for German (using [Schneider08]):
segment of one news show WER notes
new airport 36.2 outdoor, spontaneous
soccer riot 44.2 tavern, dialect, background noise
various news I 9.5
murder case 24.0
boxing 50.6 dialect, very spontaneous
various news II 20.9
rbb game 39.1
weather report 46.7 spontaneous, casual
main obstacles: local dialect, spontaneous speech, background noise
Schneider, D., Schon, J., and Eickeler, S. (2008). Towards Large Scale Vocabulary Independent Spoken Term Detection: Advances
in the Fraunhofer IAIS Audiomining System. In Proc. SIGIR, Singapore.
10 Information Technologies Institute
Centre for Research and Technology Hellas](https://image.slidesharecdn.com/wp1presentationnem12-130131063803-phpapp02/85/Enrichment-of-News-Show-Videos-with-Multimodal-Semi-Automatic-Analysis-10-320.jpg)
Enrichment of News Show Videos with Multimodal Semi-Automatic Analysis
- 1. Television Linked To The Web Daniel Stein1, Evlampios Apostolidis2, Vasileios Mezaris2, Nicolas de Abreu Pereira3, Jennifer Müller3, Mathilde Sahuguet4, Benoit Huet4, Ivo Lašek5 Enrichment of News Show Videos with Multimodal Semi-Automatic Analysis 1 Fraunhofer Institute IAIS, Schloss Birlinghoven, Germany 2 Information Technologies Institute, CERTH, Greece 3 rbb - Rundfunk Berlin-Brandenburg, 14482 Potsdam, Germany 4 Eurecom, Sophia Antpolis, France 5 Czech Technical University and University of Economics, Prague, Czech Republic NEM Summit, Istanbul, www.linkedtv.eu October 2012
- 2. Synopsis www.linkedtv.eu Introduction: LinkedTV Project Use Cases Intelligent Video Analysis Results Conclusions & future plans 2 Information Technologies Institute Centre for Research and Technology Hellas
- 3. LinkedTV ― Television Linked To the Web www.linkedtv.eu Vision: 12 Excellent Partners hypervideo Fraunhofer Eurecom ubiquitously online cloud of STI GMBH Condat Networked Audio-Visual Content CERTH BEELD EN GELUID decoupled from place, device or UEP Noterik source UMONS U. ST GALLEN CWI RBB Aim: provide interactive multimedia service for non-professional end- users Focus on television broadcast content as seed videos Web: http://www.linkedtv.eu 3 Information Technologies Institute Centre for Research and Technology Hellas
- 4. LinkedTV Workflow www.linkedtv.eu Overall Architecture Use Case Scenarios Intelligent Video Analysis Linking Hypervideo to Web Content Contextualization and Personalization Interface and Presentation Engine 4 Information Technologies Institute Centre for Research and Technology Hellas
- 5. LinkedTV Workflow www.linkedtv.eu Overall Architecture Use Case Scenarios Intelligent Video Analysis Linking Hypervideo to Web Content Contextualization and Personalization Interface and Presentation Engine 5 Information Technologies Institute Centre for Research and Technology Hellas
- 6. Two Use Case Scenarios in LinkedTV www.linkedtv.eu Scenario 1 (this talk): Interactive News Show Professional news Due to legal constraints: whitelist Detailed scenario archetype description content produced by RBB News topic, Seed content: local people, news show "rbb locations, Aktuell" objects etc Scenario 2 (not covered here): Hyperlinked Documentary Cultural content from S&V (1700 hours of cultural heritage AV- content under CCL) Seed content: "Antique Roadshow" 6 6 Information Technologies Institute Centre for Research and Technology Hellas
- 7. Intelligent Video Analysis www.linkedtv.eu 7 Information Technologies Institute Centre for Research and Technology Hellas
- 8. Segmentation www.linkedtv.eu Shot segmentation technique Spatio-temporal Segmentation [Tsamoura et. al., 2008] [Mezaris et. al., 2004] News show video performance: News show performance: Good “remarkably well” False positives due to: Out of 269 shots detected: Camera movement or zoom in/out (~ 55 %) 2 had wrong starting points Gradual transition between frames (~ 10 %) Erroneous motion vectors (~ 35 %) 4 contained multiple shots 11 were too short to evaluate Unwanted effect: false recognition of properly moving banners which do not yield additional information V. Mezaris, I. Kompatsiaris, N. V. Boulgouris, and M. G. Strintzis, "Real-time compressed-domain spatiotemporal segmentation and ontologies for video indexing and retrieval", IEEE Transactions on Circuits and Systems for Video Technology, vol. 14, no. 5, pp. 606-621, May 2004. E. Tsamoura, V. Mezaris, I. Kompatsiaris, "Gradual transition detection using color coherence and other criteria in a video shot meta- segmentation framework", IEEE International Conference on Image Processing, Workshop on Multimedia Information Retrieval (ICIP-MIR 2008), San Diego, CA, USA, October 2008, pp. 45-48. 8 Information Technologies Institute Centre for Research and Technology Hellas
- 9. Concept Detection www.linkedtv.eu Method was described in [Moumtzidou et. al., 2011] 346 concepts from TRECVID 2011 SIN task Overall performance: Correctly detected concepts > 64 % About 25 % of them are characterized as particularly useful mostly related to detecting persons (e.g., person, face, adult) Erroneous concepts vary between 22% - 42% and in many cases achieve high scores (e.g., outdoor, amateur video) Visit: http://mklab.iti.gr/eventdetection-linkedtv/ A. Moumtzidou, P. Sidiropoulos, S. Vrochidis, N. Gkalelis, S. Nikolopoulos, V. Mezaris, I. Kompatsiaris, I. Patras, "ITI-CERTH participation to TRECVID 2011", Proc. TRECVID 2011 Workshop, December 2011, Gaithersburg, MD, USA. 9 Information Technologies Institute Centre for Research and Technology Hellas
- 10. Automatic Speech Recognition www.linkedtv.eu Automatic speech recognition for German (using [Schneider08]): segment of one news show WER notes new airport 36.2 outdoor, spontaneous soccer riot 44.2 tavern, dialect, background noise various news I 9.5 murder case 24.0 boxing 50.6 dialect, very spontaneous various news II 20.9 rbb game 39.1 weather report 46.7 spontaneous, casual main obstacles: local dialect, spontaneous speech, background noise Schneider, D., Schon, J., and Eickeler, S. (2008). Towards Large Scale Vocabulary Independent Spoken Term Detection: Advances in the Fraunhofer IAIS Audiomining System. In Proc. SIGIR, Singapore. 10 Information Technologies Institute Centre for Research and Technology Hellas
- 11. Person Detection www.linkedtv.eu Face clustering using the face.com api Speaker Identification using a GMM- Result: generally very good, some HMM model, with 253 German parliament erroneous clusters due to side-view speakers Result: 8.0% Equal Error Rate 11 Information Technologies Institute Centre for Research and Technology Hellas
- 12. Conclusions www.linkedtv.eu We have established: all the different video analysis techniques their exact functionality the connections among them Preliminary results work as a solid ground for further improvements Many challenges have been addressed but several aspects of the analysis techniques show much room for improvement, e.g., over-sensitivity of spatiotemporal segmentation algorithm to gradual transitions and camera’s movement adaptation of several TRECVID concepts to the needs of each specific multimedia content (news show, documentary, art show) over-sensitivity of speech recognizer to localized dialects and background noise 12 Information Technologies Institute Centre for Research and Technology Hellas
- 13. Future Plans www.linkedtv.eu Incorporate new methods: Near-duplicate Content Detection Goal: find parts that are already watched Optical Character Recognition Goal: exploit banner information to obtain a database for face and speaker recognition Topic Segmentation Goal: improve scene segmentation Find synergies between methods: ASR + Speaker Recognition + Face Detection Person Detection ASR + Topic Classification + Shot Segmentation Story Segmentation Concept Detection + Keywords Extraction + Topic Segmentation Video Similarity/Clustering 13 Information Technologies Institute Centre for Research and Technology Hellas
- 14. www.linkedtv.eu Questions ? More information: http://www.iti.gr/~bmezaris bmezaris@iti.gr http://www.linkedtv.eu 14 Information Technologies Institute Centre for Research and Technology Hellas