Content Complexity, Similarity, and Consistency in Social Media: A Deep Learning Approach
Download as PPTX, PDF1 like390 views
This document discusses using deep learning techniques to analyze visual and textual content on social media platforms like Tumblr in order to understand what drives consumer engagement. It collected data on posts and engagements from 183 company Tumblr blogs. Using deep convolutional neural networks, it defined a measure of "semantic complexity" to capture the high-level meaning of images. It also analyzed other visual features like aesthetics and complexity, as well as textual features from posts. Finally, it used these content features to build a model for predicting consumer engagement with posts, as measured by likes and reblogs. The findings could help firms optimize their social media strategies and content design.
![Feature complexity (low level)
• Visual complexity theory [Donderi 2006a, Pieters et al. 2010]
• Visually complex (flashy) images (colors, luminance,
shape) gets more attention
• This feature complexity can be captured by the
image’s compressed file size [Donderi 2006a; Donderi
2006b; Machado et al. 2015; Forsythe et al. 2011]
• However, this complexity can only capture low-level
complexity based on “pixel” values
10](https://image.slidesharecdn.com/20161120-tumblr-170731215713/85/Content-Complexity-Similarity-and-Consistency-in-Social-Media-A-Deep-Learning-Approach-10-320.jpg)
![Semantic complexity (high level)
• Recognition-By-Components theory [Biederman 1987]
• Human object recognition is invariant to feature
factors (colors, brightness, edges, positions, etc.)
• Vessel and Rubin (2010) show that visual preferences
are influenced by semantic content in the image
• We posit that semantic complexity matters!
• Operational question: How do we calculate semantics
from unstructured images?
11](https://image.slidesharecdn.com/20161120-tumblr-170731215713/85/Content-Complexity-Similarity-and-Consistency-in-Social-Media-A-Deep-Learning-Approach-11-320.jpg)
![Semantic complexity via deep learning
• Deep convolutional neural network (CNN) [Jia et al. 2014]
• Model trained with 1.2 million images with tags (ImageNet, Flickr)
• Tested on 53,417 images from brand-generated Tumblr posts
• Each image is represented by a 1,700 dimensional vector, where each
value is the confidence score w.r.t. an object (tag)
• We define semantic complexity as the Shannon Diversity Index (entropy)
on the 1,700-dimensional vector
• max = log(d), if p is uniformly distributed
• min = 0, if p_i = 1 for some i
13](https://image.slidesharecdn.com/20161120-tumblr-170731215713/85/Content-Complexity-Similarity-and-Consistency-in-Social-Media-A-Deep-Learning-Approach-13-320.jpg)
![More visual features
• 7th-layer output = robust representation of the image for “computer vision” tasks
• Aesthetic/beauty score [Dhar et al. 2011 (CVPR, Vision)]
• Adult-content score [Sengamedu et al. 2011 (MM, Vision)]
• Celebrity (450 celebrities) [Parhki et al. 2015 (BMV, Vision)]
• Number of salient objects [Zhang et al. 2015 (CVPR, Vision)]
15](https://image.slidesharecdn.com/20161120-tumblr-170731215713/85/Content-Complexity-Similarity-and-Consistency-in-Social-Media-A-Deep-Learning-Approach-15-320.jpg)
Content Complexity, Similarity, and Consistency in Social Media: A Deep Learning Approach
- 1. Content Complexity, Similarity, and Consistency in Social Media: A Deep Learning Approach Gene Moo Lee University of Texas at Arlington Joint work with Donghyuk Shin (UT Austin/Amazon), Shu He (UConn), Andrew B. Whinston (UT Austin) DSI 2016, Austin TX
- 2. Social media: More users 2
- 3. Social media: More spending 3
- 4. Challenges and opportunities: 78% photos 4 Source: Chang et al. 2014
- 5. Research questions • How can firms optimize social media strategies by incorporating visual content? • Specifically, what are the determinants of consumer engagement in terms of “likes” and “reblogs” (sharing) actions? • How visual and textual contents play role? • Operationally, how to construct measures on these unstructured data sources? 5
- 6. Tumblr data • Tumblr: microblogging platform (acquired by Yahoo!) • 35,651 posts by 183 companies (May - Oct 2014) • Automobile, Entertainment, Food, Fashion, Finance, Leisure, Retail, Tech • 89.7% photo & text, 6.3% pure text, 4% videos • Collected “likes” and “reblogs” until Apr 2015 6
- 7. Company blogs in Tumblr 7 BMW USA Vogue IBM
- 8. Data: blog post and engagement 8 Post = Visual Info (Image) + Textual Info (Text, Tags) Customer engagement = Notes (Likes + Reblogs)
- 9. Visual features • Aesthetics (beautiful photos) • Adult-contents • Celebrity • Feature complexity (low-level, flashy images) • Semantic complexity (high-level, complex meaning) • Number of salient objects 9
- 10. Feature complexity (low level) • Visual complexity theory [Donderi 2006a, Pieters et al. 2010] • Visually complex (flashy) images (colors, luminance, shape) gets more attention • This feature complexity can be captured by the image’s compressed file size [Donderi 2006a; Donderi 2006b; Machado et al. 2015; Forsythe et al. 2011] • However, this complexity can only capture low-level complexity based on “pixel” values 10
- 11. Semantic complexity (high level) • Recognition-By-Components theory [Biederman 1987] • Human object recognition is invariant to feature factors (colors, brightness, edges, positions, etc.) • Vessel and Rubin (2010) show that visual preferences are influenced by semantic content in the image • We posit that semantic complexity matters! • Operational question: How do we calculate semantics from unstructured images? 11
- 12. Deep learning • A branch of machine learning, inspired by human brain • Algorithms to model high-level abstractions with multiple processing layers of non-linear transformations • (1) theoretical breakthroughs, (2) Big Data, (3) powerful computation • Successfully applied in image/video/voice recognition, AlphaGo, etc. 12
- 13. Semantic complexity via deep learning • Deep convolutional neural network (CNN) [Jia et al. 2014] • Model trained with 1.2 million images with tags (ImageNet, Flickr) • Tested on 53,417 images from brand-generated Tumblr posts • Each image is represented by a 1,700 dimensional vector, where each value is the confidence score w.r.t. an object (tag) • We define semantic complexity as the Shannon Diversity Index (entropy) on the 1,700-dimensional vector • max = log(d), if p is uniformly distributed • min = 0, if p_i = 1 for some i 13
- 14. ImageNet: Image DB with tree-structure tags 14 Source: ImageNet
- 15. More visual features • 7th-layer output = robust representation of the image for “computer vision” tasks • Aesthetic/beauty score [Dhar et al. 2011 (CVPR, Vision)] • Adult-content score [Sengamedu et al. 2011 (MM, Vision)] • Celebrity (450 celebrities) [Parhki et al. 2015 (BMV, Vision)] • Number of salient objects [Zhang et al. 2015 (CVPR, Vision)] 15
- 16. Examples: Visual features • Visual complexity theory (Attneave 1994, Donderi 2006, Pieters et al. 2010) • Visual stimuli are a composite of colors,luminance, shape, number of objects/patterns 16
- 17. Textual features • Two textual sources: text and tags • Length: # of words, # of tags • Topic complexity: LDA topic model (text, tags) • Order complexity: word2vec (for text only) 17
- 18. Examples: Textual features • Topics • Word clusters 18
- 19. Visual-Textual Content Similarity • Image: pixels, Text/Tags: characters — Need a common representation! 1. Represent each image as a collection of the predicted labels obtained from deep learning — “image corpus” 2. Train LDA with both image and text/tags corpora — topic distribution for images and text/tags 3. Cosine similarity between the two corresponding topic distribution 19
- 20. Examples: Content similarity • Topics • Word clusters 20
- 21. 21
- 22. Empirical Model • Linear fixed effects model • DV (likes/reblogs): take log transformation due to their skewed distributions • Capture blog (firm) heterogeneity • Capture time effects (day of week, month) • Other models • Identical results with random effects • Consistent results with negative binomial model 22
- 23. 23
- 24. Summary and implications 1. Large-scale analysis on visual content in social media 2. New visual semantic complexity via deep learning • Able to relate visual and textual content Visual content analysis can be used to optimize content design for social media marketing 24
- 25. Thank you! Contact Info: Gene Moo Lee gene.lee@uta.edu
Editor's Notes
- #24: Industry subsample analysis Long- and short-term customer engagement Categorize posts/blogs into ‘utilitarian’ vs ‘hedonic’ Examine non-linear effects