Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction
2 likes1,823 views
The document discusses the use of convolutional neural networks (CNNs) for predicting the sentiment evoked by images. It includes an introduction to the methodology, related work, experimental results, conclusions, and suggestions for future research. Key contributions involve fine-tuning CNNs and analyzing various datasets to improve sentiment prediction accuracy.
![22
ARCHITECTURE
CaffeNet
SOFTWARE
[Jia’14]
Experimental setup: CNN](https://image.slidesharecdn.com/asm2015divingdeepintosentiment-understandingfine-tunedcnnsforvisualsentimentprediction-151030073345-lva1-app6891/85/Diving-deep-into-sentiment-Understanding-fine-tuned-CNNs-for-visual-sentiment-prediction-22-320.jpg)
![Experimental setup: CNN
23
Pre-trained
Model
ARCHITECTURE
CaffeNet
SOFTWARE
[Jia’14]
DATASET
[Deng’09]](https://image.slidesharecdn.com/asm2015divingdeepintosentiment-understandingfine-tunedcnnsforvisualsentimentprediction-151030073345-lva1-app6891/85/Diving-deep-into-sentiment-Understanding-fine-tuned-CNNs-for-visual-sentiment-prediction-23-320.jpg)
![Experimental setup: CNN
24
Model
ARCHITECTURE
CaffeNet
SOFTWARE
[Jia’14]
DATASET
[Deng’09]
DATASET
[You’15]
Twitter 5-agree
+
Fine-tuning
Pre-training](https://image.slidesharecdn.com/asm2015divingdeepintosentiment-understandingfine-tunedcnnsforvisualsentimentprediction-151030073345-lva1-app6891/85/Diving-deep-into-sentiment-Understanding-fine-tuned-CNNs-for-visual-sentiment-prediction-24-320.jpg)
![49
Model
ARCHITECTURE
CaffeNet
SOFTWARE
[Jia’14]
DATASET
MVSO [Jou’15]
Future work](https://image.slidesharecdn.com/asm2015divingdeepintosentiment-understandingfine-tunedcnnsforvisualsentimentprediction-151030073345-lva1-app6891/85/Diving-deep-into-sentiment-Understanding-fine-tuned-CNNs-for-visual-sentiment-prediction-49-320.jpg)
Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction
- 1. DIVING DEEP INTO SENTIMENT: UNDERSTANDING FINE-TUNED CNNS FOR VISUAL SENTIMENT PREDICTION Víctor Campos Xavier Giró Amaia Salvador Brendan Jou
- 2. Outline 1. Introduction 2. Related work 3. Methodology and results 4. Conclusions 5. Future work 2
- 4. 4 Introduction: problem definition ▷ What? ▷ How?
- 5. ▷ What? Predict the sentiment that an image provokes to a human ▷ How? 5 Introduction: problem definition
- 6. ▷ What? Predict the sentiment that an image provokes to a human ▷ How? 6 Introduction: problem definition
- 7. ▷ What? Predict the sentiment that an image provokes to a human ▷ How? Using Convolutional Neural Networks (CNNs) 7 CNN Introduction: problem definition
- 10. Outline 1. Introduction 2. Related work 3. Methodology and results 4. Conclusions 5. Future work 10
- 11. Related work: low-level descriptors 11 Siersdorfer, S., Minack, E., Deng, F., & Hare, J. (2010, October). Analyzing and predicting sentiment of images on the social web. In Proceedings of the international conference on Multimedia (pp. 715-718). ACM. Machajdik, J., & Hanbury, A. (2010, October). Affective image classification using features inspired by psychology and art theory. In Proceedings of the international conference on Multimedia (pp. 83-92). ACM.
- 12. 12 Borth, D., Ji, R., Chen, T., Breuel, T., & Chang, S. F. (2013, October). Large-scale visual sentiment ontology and detectors using adjective noun pairs. In Proceedings of the 21st ACM international conference on Multimedia (pp. 223-232). ACM. Related work: SentiBank
- 13. Related work: CNNs for sentiment prediction 13 You, Q., Luo, J., Jin, H., & Yang, J. (2015). Robust image sentiment analysis using progressively trained and domain transferred deep networks. In The Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI).
- 14. Outline 1. Introduction 2. Related work 3. Methodology and results a. Convolutional Neural Networks b. Datasets c. Experimental setup and results 4. Conclusions 5. Future work 14
- 15. Convolutional Neural Networks 15 Krizhevsky, A.; Sutskever, I. & Hinton, G. E.: ImageNet Classification with Deep Convolutional Neural Networks. In: NIPS., 2012
- 16. Outline 1. Introduction 2. Related work 3. Methodology and results a. Convolutional Neural Networks b. Datasets c. Experimental setup and results 4. Conclusions 5. Future work 16
- 17. Datasets 17 Flickr Twitter Authors Borth et al. (2013) You et al. (2015) Size ~500k 1269 Annotation method Textual tags 5 human annotators
- 20. Outline 1. Introduction 2. Related work 3. Methodology and results a. Convolutional Neural Networks b. Datasets c. Experimental setup and results 4. Conclusions 5. Future work 20
- 24. Experimental setup: CNN 24 Model ARCHITECTURE CaffeNet SOFTWARE [Jia’14] DATASET [Deng’09] DATASET [You’15] Twitter 5-agree + Fine-tuning Pre-training
- 25. Experimental setup: outline 1. Fine-tuning CaffeNet 2. Layer by layer analysis 3. Layer ablation 4. Layer addition 25
- 31. Experimental setup: outline 1. Fine-tuning CaffeNet 2. Layer by layer analysis 3. Layer ablation 4. Layer addition 31
- 32. Layer by layer analysis 32
- 33. Layer by layer analysis 33
- 34. Experimental setup: outline 1. Fine-tuning CaffeNet 2. Layer by layer analysis 3. Layer ablation 4. Layer addition 34
- 35. Layer ablation 35 Raw ablation 2-neuron on top
- 39. Experimental setup: outline 1. Fine-tuning CaffeNet 2. Layer by layer analysis 3. Layer ablation 4. Layer addition 39
- 42. Outline 1. Introduction 2. Related work 3. Methodology and results 4. Conclusions 5. Future work 42
- 45. Conclusions 45
- 46. Outline 1. Introduction 2. Related work 3. Methodology and results 4. Conclusions 5. Future work 46
- 47. Future work 47 Size Flickr dataset Quality of the annotations Twitter dataset
- 48. Future work 48 Size Flickr dataset Quality of the annotations Twitter dataset MVSO dataset (†) B. Jou*, T. Chen*, N. Pappas*, M. Redi*, M. Topkara*, and S.-F. Chang. Visual Affect Around the World: A Large-scale Multilingual Visual Sentiment Ontology. ACM Int'l Conference on Multimedia (MM), 2015. †
- 50. Acknowledgements 50 Financial supportTechnical support Albert Gil Josep Pujal