![IntroductionRelated worksOur approachExperimentsConclusion and perspectivesSpatial Pyramid Matching Kernel (SPM) & sparse coding Lazebnik et al. [CVPR06]Spatial Pyramid Matching Kernel (SPM): exploiting the spatial information of location regions.Yang et al. [CVPR09]SPM + sparse coding: replacing k-means in the SPM12](https://image.slidesharecdn.com/thesispresentation-13166064178274-phpapp01-110921070836-phpapp01/85/A-Higher-Level-Visual-Representation-For-Semantic-Learning-In-Image-Databases-14-320.jpg)
![IntroductionRelated worksOur approachExperimentsConclusion and perspectivesVisual phrase & descriptive visual phraseZheng and Gao [TOMCCAP08]Visual phrase: pair of spatially adjacent local image patchesZhang et al. [ACM MM09] Descriptive visual phrase: selected according to the frequencies of its constituent visual word pairs13](https://image.slidesharecdn.com/thesispresentation-13166064178274-phpapp01-110921070836-phpapp01/85/A-Higher-Level-Visual-Representation-For-Semantic-Learning-In-Image-Databases-15-320.jpg)
![IntroductionRelated worksOur approachExperimentsConclusion and perspectivesVisual phrase pattern & visual sysnetYuan et al. [CVPR07] Visual phrase pattern: spatially co-occurring group of visual wordsZheng et al. [CVPR08] Visual synset: relevance-consistent group of visual words or phrases in the spirit of the text synset14](https://image.slidesharecdn.com/thesispresentation-13166064178274-phpapp01-110921070836-phpapp01/85/A-Higher-Level-Visual-Representation-For-Semantic-Learning-In-Image-Databases-16-320.jpg)
![IntroductionRelated worksOur approachExperimentsConclusion and perspectivesMultilayer Semantically Significant Analysis (MSSA) model Parameter EstimationProbability distribution function :Log-likelihood function :Gaussier et al. [ ACM SIGIR05]: maximizing the likelihood can be seen as a Nonnegative Matrix Factorization (NMF) problem under the generalized KL divergenceObjective function:26](https://image.slidesharecdn.com/thesispresentation-13166064178274-phpapp01-110921070836-phpapp01/85/A-Higher-Level-Visual-Representation-For-Semantic-Learning-In-Image-Databases-29-320.jpg)
A Higher-Level Visual Representation For Semantic Learning In Image Databases
- 1. A Higher-Level Visual Representation For Semantic Learning In ImageDatabasesIsmail EL SAYAD18/07/2011
- 2. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesOverviewIntroductionRelated worksOur approachEnhanced Bag of Visual Words (E-BOW)Multilayer Semantically Significant Analysis Model (MSSA)Semantically Significant Invariant Visual Glossary (SSIVG) ExperimentsImage retrievalImage classificationObject RecognitionConclusion and perspectives2
- 3. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesMotivationDigital content grows rapidly Personal acquisition devicesBroadcast TV SurveillanceRelatively easy to store, but useless if no automatic processing, classification, and retrievingThe usual way to solve this problem is by describing images by keywords.This method suffers from subjectivity, text ambiguity and the lack of automatic annotation3
- 4. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesVisual representationsImage-based representations are based on global visual features extracted over the whole image like color, color moment, shape or texture Visual representationsImage-based representationsPart-based representations4
- 5. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesVisual representationsThe main drawbacks of Image-based representations:High sensitivity to :ScalePose Lighting condition changes OcclusionsCannot capture the local information of an imagePart-based representations: Based on the statistics of features extracted from segmented image regions5
- 6. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesVisual representationsPart-based representations (Bag of visual words)Feature spaceVisual word vocabulary VW1VW2VW3VW4...Compute local descriptorsFeature clusteringVW1VW2VW3VW1VW2VW3VW4...2111...VW4FrequencyVW16
- 7. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesVisual representations Bag of visual words (BOW) drawbacksSpatial information loss
- 8. Record number of occurrences
- 9. Ignore the positionUsing only keypoints-based Intensity descriptors:Neither shape nor color information is usedFeature quantization noisiness: Unnecessary and insignificant visual words are generated7
- 10. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesVisual representationsDrawbacks Bag of Visual words (BOW)Low discrimination power: Different image semantics are represented by the same visual wordsLow invariance for visual diversity: One image semantic is represented by different visual wordsVW1364VW1364VW330VW480VW263VW1488
- 11. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesObjectives Enhanced BOW representationDifferent local information (intensity, color, shape…)Spatial constitution of the imageEfficient visual word vocabulary structureHigher-level visual representationLess noisy More discriminativeMore invariant to the visual diversity9
- 12. MSSA modelIntroductionRelated worksOur approachExperimentsConclusion and perspectivesE-BOWOverview of the proposed higher-level visual representationSSVIWs & SSIVPsgenerationE-BOW representationSSIVG representationSSIVGLearning the MSSA model Visual word vocabulary buildingSet of images10
- 13. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesIntroductionRelated worksSpatial Pyramid Matching Kernel (SPM) & sparse coding Visual phrase & descriptive visual phraseVisual phrase pattern & visual synsetOur approachExperimentsConclusion and perspectives11
- 14. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesSpatial Pyramid Matching Kernel (SPM) & sparse coding Lazebnik et al. [CVPR06]Spatial Pyramid Matching Kernel (SPM): exploiting the spatial information of location regions.Yang et al. [CVPR09]SPM + sparse coding: replacing k-means in the SPM12
- 15. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesVisual phrase & descriptive visual phraseZheng and Gao [TOMCCAP08]Visual phrase: pair of spatially adjacent local image patchesZhang et al. [ACM MM09] Descriptive visual phrase: selected according to the frequencies of its constituent visual word pairs13
- 16. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesVisual phrase pattern & visual sysnetYuan et al. [CVPR07] Visual phrase pattern: spatially co-occurring group of visual wordsZheng et al. [CVPR08] Visual synset: relevance-consistent group of visual words or phrases in the spirit of the text synset14
- 17. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesComparison of the different enhancements of the BOW 15
- 18. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesIntroductionRelated worksOur approach Enhanced Bag of Visual Words (E-BOW)Multilayer Semantically Significant Analysis Model (MSSA)Semantically Significant Invariant Visual Glossary (SSIVG) ExperimentsConclusion and perspectives16
- 19. IntroductionRelated worksOur approachExperimentsConclusion and perspectives Enhanced Bag of Visual Words (E-BOW)Set of imagesE-BOWMSSA modelSSIVGSURF & Edge Context extractionFeatures fusion Hierarchal features quantizationE-BOW representation17
- 20. IntroductionRelated worksOur approachExperimentsConclusion and perspectives Enhanced Bag of Visual Words (E-BOW)Feature extraction Interest points detectionEdge points detectionColorfiltering using vector median filter (VMF ) SURF feature vector extraction at each interest point Colorfeature extraction at each interest and edge point Fusion of the SURF and edgecontextfeaturevectorsColor and position vector clustering using Gaussian mixture model Edge Context feature vector extraction at each interest point Collection of all vectors for the whole image set∑3 µ3Pi3∑2 µ2Pi2∑1 µ1Pi1HAC and Divisive Hierarchical K-Means clusteringVW vocabulary18
- 21. IntroductionRelated worksOur approachExperimentsConclusion and perspectives Enhanced Bag of Visual Words (E-BOW)Feature extraction (SURF)SURF is a low-level feature descriptorDescribes how the pixel intensities are distributed within a scale dependent neighborhood of each interest point. Good atHandling serious blurring Handling image rotation Poor atHandling illumination change Efficient19
- 22. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesEnhanced Bag of Visual Words (E-BOW)Feature extraction (Edge Context descriptor) Edge context descriptor is represented at each interest point as a histogram : 6 bins for the magnitude of the drawn vectors to the edge points 4 bins for the orientation angle20
- 23. IntroductionRelated worksOur approachExperimentsConclusion and perspectives Enhanced Bag of Visual Words (E-BOW)Feature extraction (Edge context descriptor) This descriptor is invariant to :Translation :The distribution of the edge points is measured with respect to fixed pointsScale:The radial distance is normalized by a mean distance between the whole set of points within the same GaussianRotation:All angles are measured relative to the tangent angle of each interest point21
- 24. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesEnhanced Bag of Visual Words (E-BOW)Hierarchalfeature quantization Visual word vocabulary is created by clustering the observed merged features (SURF + Edge context 88 D) in 2 clustering steps: Hierarchical Agglomerative Clustering (HAC) Divisive Hierarchical K-Means ClusteringStop clustering at desired level k k clusters from HAC…The tree is determined level by level, down to some maximum number of levels L, and each division into k parts. Merged feature in the feature spaceA cluster at k =422
- 25. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesMultilayer Semantically Significant Analysis (MSSA) model Set of imagesE-BOWMSSA modelSSIVGVWs semantic inference estimationSURF & Edge Context extractionNumber of latent topics EstimationFeatures fusion Parameters estimationHierarchal features quantizationGenerative processE-BOW representation23
- 26. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesMultilayer Semantically Significant Analysis (MSSA) model Generative ProcessDifferent VisualaspectsA topic model that considers this hierarchal structure is needed Higher-level aspect: People24
- 27. IntroductionRelated worksOur approachExperimentsConclusion and perspectives Multilayer Semantically Significant Analysis (MSSA) model Generative ProcessφΘΨIn the MSSA, there are two different latent (hidden) topics:High latent topic that represents the high aspects
- 28. Visual latent topic that represents the visual aspectsV W vhimMN25
- 29. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesMultilayer Semantically Significant Analysis (MSSA) model Parameter EstimationProbability distribution function :Log-likelihood function :Gaussier et al. [ ACM SIGIR05]: maximizing the likelihood can be seen as a Nonnegative Matrix Factorization (NMF) problem under the generalized KL divergenceObjective function:26
- 30. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesMultilayer Semantically Significant Analysis (MSSA) model Parameter EstimationKKT conditions are used to derive the multiplicative update rules for minimizing the objective function
- 31. This leads to the following multiplicative update rules : 27
- 32. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesMultilayer Semantically Significant Analysis (MSSA) modelNumber of Latent Topics EstimationMinimum Description Length (MDL) is used as a model selection criteria
- 33. Number of the high latent topics (L)
- 34. Number of the visual latent topics (K)
- 37. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesSemantically Significant Invariant Visual Glossary (SSIVG) representationSet of imagesE-BOWMSSA modelSSIVGVWs semantic inference estimationSURF & Edge Context extractionSSVP representationSSIVG representationNumber of latent topics EstimationFeatures fusion SSVPs generationSSIVP representationParameters estimationHierarchal features quantizationSSVW representationDivisivetheoretic clusteringSSIVW representationGenerative processE-BOW representationSSVWs selection29
- 38. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesSemantically Significant Invariant Visual Glossary (SSIVG) representationSemantically Significant Visual Word (SSVW)Set of relevant visual topics Set of VWsEstimating using MSSASet of SSVWsEstimating using MSSA30
- 39. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesSemantically Significant Invariant Visual Glossary (SSIVG) representationSemantically significant Visual Phrase (SSVP)SSVP: Higher-level and more discriminative representationSSVWs + their inter-relationships SSVPs are formed from SSVW sets that satisfy all the following conditions: Occur in the same spatial context
- 40. Involved in strong association rules
- 41. High support and confidence
- 42. Have the same semantic meaning
- 43. High probability related to at least one common visual latent topic31
- 44. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesSemantically Significant Invariant Visual Glossary (SSIVG) representationSemantically Significant Visual Phrase (SSVP)SSIVP126SSIVP126SSIVP326SSIVP326SSIVP304SSIVP30432
- 45. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesSemantically Significant Invariant Visual Glossary (SSIVG) representationInvariance Problem Studying the co-occurrence and spatial scatter information make the image representation more discriminative The invariance power of SSVWs and SSVPs is still low Text documentsSynonymous words can be clustered into one synonymy set to improve the document categorization performance33
- 46. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesSemantically Significant Invariant Visual Glossary (SSIVG) representationSSIVG : higher-level visual representation composed from two different layers of representationSemantically Significant Invariant Visual Word (SSIVW) Re-indexed SSVWs after a distributional clusteringSemantically Significant Invariant Visual Phrases (SSIVP) Re-indexed SSVPs after a distributional clusteringSet of relevant visual topics Set of SSVWs and SSVPsEstimating using MSSAEstimating using MSSADivisivetheoretic clusteringSet of SSIVGsSet of SSIVPsSet of SSIVWs34
- 47. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesExperimentsIntroductionRelated worksOur approachExperimentsImage retrievalImage classificationObject RecognitionConclusion and perspectives35
- 48. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesAssessment of the SSIVG representation performance in image retrievalEvaluation criteria :Mean Average Precision (MAP)
- 49. The traditional Vector Space Model of Information Retrieval is adapted
- 50. The weighting for the SSIVP
- 51. Spatial weighting for the SSIVW
- 52. The inverted file structure36
- 53. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesAssessment of the SSIVG representation Performance in image retrieval37
- 54. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesAssessment of the SSIVG representation performance in image retrieval3838
- 55. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesEvaluation of the SSIVG representation in image classificationEvaluation criteria :Classification Average Precision over each class
- 56. Classifiers :
- 57. SVM with linear kernelMulticlass Vote-Based Classifier (MVBC)39
- 58. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesEvaluation of the SSIVG representation in image classificationMulticlass Vote-Based Classifier (MVBC)For each , we detect the high latent topic that maximizes:isThe final voting score for a high latent topic : isEach image is categorized according to the dominant high latent40
- 59. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesEvaluation of the SSIVG representation performance in classification41
- 60. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesAssessment of the SSIVG representation Performance in object recognitionEach test image is recognized by predicting the object class using the SSIVG representation and the MVBCEvaluation criteria:Classification Average Precision (AP) over each object class42
- 61. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesAssessment of the SSIVG Representation Performance in Object Recognition43
- 62. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesExperimentsIntroductionRelated worksOur approachExperimentsConclusion and perspectives44
- 63. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesConclusionEnhanced BOW (E-BOW) representation Modeling the spatial-color image constitution using GMM
- 64. New local feature descriptor (Edge Context)
- 65. Efficient visual word vocabulary structure
- 66. New Multilayer Semantic Significance (MSSA) model
- 67. Semantic inferences of different layers of representation
- 68. Semantically Significant Visual Glossary (SSIVG)
- 70. More invariant to visual diversity
- 72. Outperform other sate of the art works45
- 73. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesPerspectivesMSSA Parameters update
- 74. On-line algorithms to continuously (re-)learn the parameters
- 75. Invariance issue
- 76. Context large-scale databases where large intra-class variations can occur
- 77. Cross-modalitily extension to video content
- 78. Cross-modal data (visual and textual closed captions contents)
- 79. New generic framework of video summarization
- 80. Study the semantic coherence between visual contents and textual captions46
- 81. Thank you for your attention !ismail.elsayad@lifl.frQuestions ?
- 82. IntroductionRelated worksOur approachExperimentsConclusion and perspectivesParameter Settings 48
Editor's Notes
- #3: Talk briefly about the introductionTalk about the different parts of the approach briefly
- #5: Nowadays, images can be described using their visual content.
- #9: Talk about analogy between text and imagesDifferent visual appearanceSAME VISUAL WORD APPEAR IN TWO DIFFERENT IMAGE DESCRIBING DIFFERENT SEMANTICS
- #10: This work aims at addressing these drawbacks with the following
- #12: All the related works are based on BOW representation, they propose different Higher-level representation
- #13: Spatial pyramid extended the BOW representationExample of spatial pyramid for three different spatial level and resolutions
- #14: enhanced this approach by selecting descriptive visual phrases from the constructed visual phrases according to the frequencies of their constituent visual word pairs.
- #16: EFFICIENT structure by other approaches are for visual word vocabulary but we used for word and phrase
- #21: Motivation of the edge context
- #26: Add training images
- #27: This generative process leads to the following conditional probability distribution:Following the maximum likelihood principle, one can estimate the parameters bymaximizing the log-likelihood function as follows:
- #29: The number of the high latent topics, L, and the number of the visual latent topics, K, is determined in advance for the model fitting based on the Minimum Description Length (MDL)principle
- #31: Check the size of the cylinders
- #34: See right part
- #35: Correct the animationMake boxes bigger
- #36: Global approach slide before this slide
- #37: Add parameter settings
- #39: Discriptive correct and upper casesAdd references
- #41: Add this slide at the end of the presentations and add sub points to slide before
- #42: Upper case corrections
- #45: Global approach slide before this slide
- #46: Add parameter settings
- #47: Parameters update: It will be essential to design on-line algorithms to continuously (re-)learn the parameters of the proposed MSSA model, as the content of digital databases is modified by the regular upload or deletion of images.Invariance issue: It will be interesting to investigate more on the invariance issue especially in the context large-scale databases where large intra-class variations can occur.Cross-modalitily extension: The proposed higher-level visual representation can be extended to video content. The extension can be based on cross-modal data (visual and textual closed captions contents). Video summarization: A new generic framework of video summarization based on the extended higher-level semantic representation of video content can be designed.Talk that this work is applied at the frame level
- #49: Parameters update: It will be essential to design on-line algorithms to continuously (re-)learn the parameters of the proposed MSSA model, as the content of digital databases is modified by the regular upload or deletion of images.Invariance issue: It will be interesting to investigate more on the invariance issue especially in the context large-scale databases where large intra-class variations can occur.Cross-modalitily extension: The proposed higher-level visual representation can be extended to video content. The extension can be based on cross-modal data (visual and textual closed captions contents). Video summarization: A new generic framework of video summarization based on the extended higher-level semantic representation of video content can be designed.Talk that this work is applied at the frame level