A Meteoroid on Steroids: Ranking Media Items Stemming from Multiple Social Networks
1 like751 views
The document presents a social media application called 'Social Media Illustrator' which ranks and classifies media items from multiple social networks based on user interactions and other criteria. It discusses the development of a ranking formula that incorporates factors such as likes, shares, comments, views, and recency for better media item clustering and ranking. Future enhancements for the application include improved visualization capabilities and further clustering options.
![A Meteoroid on Steroids: Ranking Media Items
Stemming from Multiple Social Networks
Thomas Steiner
Google Germany GmbH
ABC-Str. 19
20354 Hamburg, Germany
tomac@google.com
ABSTRACT 2. RELATED WORK
We have developed an application called Social Media In [6], San Pedro and Siersdorfer propse a methodology
Illustrator that allows for finding media items on multiple for automatically ranking and classifying photos from the
social networks, clustering them by visual similarity, ranking photo sharing platform Flickr according to their attractive-
them by different criteria, and finally arranging them in me- ness for Flickr members. They work with extracted user
dia galleries that were evaluated to be perceived as aestheti- feedback and annotations available on Flickr to train ma-
cally pleasing. In this paper, we focus on the ranking aspect chine learning models based on image features like sharp-
and show how, for a given set of media items, the most ade- ness and colorfulness. While their method is tailored to
quate ranking criterion combination can be found by inter- Flickr, our approach is based on a social network interaction
actively applying different criteria and seeing their effect on- abstraction layer on top of the social networks Facebook,
the-fly. This leads us to an empirically optimized media item Twitter, Google+, Instagram, YouTube, Flickr, MobyPic-
ranking formula, which takes social network interactions into ture, Twitpic, and Lockerz. Jaffe et al. describe [2] a ranking
account. While the ranking formula is not universally appli- and summary algorithm for geo-tagged photo sets based on
cable, it can serve as a good starting point for an individually spatial patterns as well as textual-topical patterns and pho-
adapted formula, all within the context of Social Media Illus- tographer identity cues. Their algorithm can be expanded
trator. A demo of the application is available publicly online to support social, temporal, and other factors. The shown
at the URL http://social-media-illustrator.herokuapp.com/. maps-based application necessarily requires geo-tagged me-
dia items, which is rarely the case with media items retrieved
from social networks due to privacy concerns. In [1], David-
Categories and Subject Descriptors son et al. describe the different criteria video quality, user
H.3.3 [Information Search and Retrieval]: Clustering specificity, and diversification that determine the video rank-
ing in the YouTube recommendation system. These criteria
General Terms include view count, the ratings of the video, commenting,
Algorithms favoriting, and sharing activity around the video. Finally,
Wiyartanti et al. introduce in [11] a ranking algorithm for
user-generated videos based on social activities.
Keywords
Ranking, Event Summarization, Social Networks 3. SOCIAL NETWORK INTERACTIONS
3.1 Abstraction Layer
1. INTRODUCTION Social networks have different paradigms of social interac-
When people witness events like concerts, sports matches, tions. In [5], we have introduced an abstraction layer on top
or meteoroid impacts, they more and more share media of the native data formats of all considered social networks
items like photos and videos that depict these events pub- in order to gain an agnostic view on them. Regardless of
licly on social networks. In the past, we have worked on the native data representation format of the social network
methods [3, 5, 9] for the automatic extraction, deduplica- of origin, the abstraction layer unifies and streamlines the
tion, and clustering of media items stemming from multiple available data for each media item to a greatest common
social networks. Up to now, we have ordered the retrieved divisor of all social networks. These interaction paradigms
media items chronologically, by social network, or by clus- must be exposed by the social networks via specific API
ter size, and thereby completely neglected social network calls in order to be considered. In Table 1, we detail how
interactions as ranking signals. Though truly added value we abstract the social interactions in question on each so-
lies in exploiting these social network interactions in order to cial network. We differentiate between unknown values that
obtain a more representative ranking of the potentially over- are returned as unknown, i.e., where the information is not
whelmingly many media items retrieved for a given event. exposed, and 0 values, where the value is known to be zero.
3.2 Merging Social Interactions
In the context of our previous research, we have devel-
Copyright is held by the author/owner(s).
WWW 2013 Companion, May 13–17, 2013, Rio de Janeiro, Brazil. oped a tile-wise histogram-based media item deduplication
ACM 9781450320382/13/05. algorithm with additional high-level semantic matching cri-](https://image.slidesharecdn.com/a-meteoroid-on-steroids-ranking-media-items-stemming-from-multiple-social-networks-www2013devtrack1-130411072559-phpapp02/85/A-Meteoroid-on-Steroids-Ranking-Media-Items-Stemming-from-Multiple-Social-Networks-1-320.jpg)
![Likes Shares Comments Views
Facebook Like Facebook Share Facebook Comments YouTube Views
Google+ +1 Google+ Share Google+ Comments Flickr Views
Instagram Like Twitter native ReTweet Instagram Comments Twitpic Views
Flickr Favorite Twitter manual RT, @Replies MobyPicture Views
YouTube Like Twitpic Comments
YouTube Favorite MobyPicture Comments
Twitter Favorite Flickr Comments
Table 1: Abstract social network interaction paradigms and their underlying native counterparts
teria that is tailored to photos and videos stemming from 5. IMPLEMENTATION DETAILS
multiple social networks. If a set of media items is visually The application has been implemented in Node.js, a server
similar enough to be clustered under the criteria detailed side JavaScript software system designed for writing scalable
in [5], we treat the whole of the cluster as if it were just one Internet applications. Programs are created using event-
media item. These criteria are pair-wise tile histogram sim- driven, asynchronous input/output operations to minimize
ilarity that does not exceed a given threshold and the same overhead and maximize scalability. The clustering and rank-
number of detected faces per media item. In consequence, ing logic is kept on the client side, while the media item
we specify a merging strategy for the associated social in- retrieval logic is kept on the server side. As the clustering
teractions of the individual media items in the particular logic needs read access to the pixel data of media items via
cluster. We treat unknown values as 0. The alternative to the canvas element’s getImageData function, all media items
this solution is to exclude unknown values from the merg- need to be proxied locally. Face detection works fully on the
ing step. However, as in practice a considerable amount client side based on a library made available by Liu [4]. The
of social interaction values are unknwon, we are forced to interface is fully interactive and event-driven. Figure 1 and
proceed with the abovementioned simplification. The algo- Figure 2 show screenshots of the deployed application.
rithm accumulates individual social interactions and assigns
the accumulated social interactions to the cluster.
6. CONCLUSIONS AND FUTURE WORK
In this paper, we have presented an application called
4. RANKING MEDIA ITEM CLUSTERS Social Media Illustrator with a special focus on its social
In this section, we describe a ranking formula to rank interactions abstraction layer and ranking capacities. The
a set of media clusters that match a given query. In the application has been successfully evaluated to produce both
ranking formula, we consider several well-defined ranking meaningful and beautiful visual and audial summaries for
criteria that were detailed in [10], namely these are visual, recent events. The majority of these summaries were made
audial, textual, temporal, social, and aesthetic. For a given available online.1 One example of such can be seen in Fig-
set of media item clusters, a ranking is calculated as follows. ure 2, which shows popular social media reactions for the
meteoroid impact event on 15 February 2013, when a small
asteroid entered the atmosphere of Earth, became visible as
α × likes + β × shares + γ × comments + δ × views+ a bright fireball and exploded in an air burst over Chelyabinsk.
× clusterSize + ζ × recency + η × quality (1) Future work will focus on adding more visualization for-
mats that will support text-to-speech once the text synthesis
The factors likes, shares, comments, and views stem from part of the Web Speech API [7] has landed in Web browsers.
the individual media items as described in Subsection 3.1 This will allow for true story-telling, where the associated
and Subsection 3.2. The factor clusterSize corresponds to microposts for a media item can be read as it is shown,
the size of the current cluster. The factor recency is calcu- potentially in an interactive slideshow format. Further, we
lated as follows. If the youngest media item in the cluster plan to add more clustering options that will allow for also
is less than or exactly one day old, the value is 8, for two clustering by extracted named entities [9] besides the cur-
days it is 4, for three days it is 2, and for each day more, rently visual clustering.
the value is 1. The factor quality is a representation of the Concluding, with our Social Media Illustrator application,
presence of faces and a media item’s photo or video quality. we have contributed an effective and efficient tool to deal
Empirically optimized default values that can be fine-tuned with social media overload and to identify the few needles
for a concrete media item set were determined as follows: in the social network haystack.
α = 2, β = 4 , γ = 8, δ = 1, = 32, ζ = 2, and η = 8.
Once a final ranking for all media items has been found,
the top-n media items are compiled to different kinds of 7. REFERENCES
media galleries that in two user studies were shown to be [1] J. Davidson, B. Liebald, J. Liu, P. Nandy,
perceived as aesthetically pleasing [8]. We differentiate be- T. Van Vleet, U. Gargi, S. Gupta, Y. He, M. Lambert,
tween the Loose Order, Varying Size style, where certain B. Livingston, and D. Sampath. The YouTube Video
media items can be featured more prominently by making Recommendation System. In Proceedings of the
them bigger at the cost of loosely disrespecting the ranking- Fourth ACM Conference on Recommender Systems,
implied order and the Strict Order, Equal Size style, which RecSys ’10, pages 293–296, 2010.
strictly respects the ranking-implied order [8].
1
http://twitpic.com/photos/tomayac, accessed 02/21/2013](https://image.slidesharecdn.com/a-meteoroid-on-steroids-ranking-media-items-stemming-from-multiple-social-networks-www2013devtrack1-130411072559-phpapp02/85/A-Meteoroid-on-Steroids-Ranking-Media-Items-Stemming-from-Multiple-Social-Networks-2-320.jpg)
![Figure 1: Media Item Clusters tab of the Social Media Illustrator application with individual and clustered (bottom middle)
media items from Facebook and YouTube, ranked by popularity for the Russian query Метеороид
Figure 2: Zoomed view of the Media Gallery tab of the application showing an automatically generated media gallery in loose
order, varying size style featuring ranked media items stemming from Facebook and YouTube for the query Метеороид
[2] A. Jaffe, M. Naaman, T. Tassa, and M. Davis. [7] G. Shires and H. Wennborg. Web Speech API
Generating Summaries and Visualization for Large Specification. Final Report, W3C, Oct. 2012.
Collections of Geo-referenced Photographs. In [8] T. Steiner and C. Chedeau. To Crop, Or Not to Crop:
Proceedings of the 8th ACM International Workshop Compiling Online Media Galleries. In Proceedings of
on Multimedia Information Retrieval, pages 89–98, the 22nd International Conference Companion on World
2006. Wide Web, WWW ’13, May 2013.
[3] H. Khrouf, G. Atemezing, G. Rizzo, R. Troncy, and [9] T. Steiner, R. Verborgh, J. Gabarro, and R. Van de
T. Steiner. Aggregating Social Media for Enhancing Walle. Adding Meaning to Facebook Microposts via a
Conference Experience. In Real-Time Analysis and Mash-up API and Tracking its Data Provenance. In
Mining of Social Streams, Papers from the 2012 Next Generation Web Services Practices (NWeSP), 2011
ICWSM Workshop, June 2012. 7th International Conference on, pages 342–345, Oct.
[4] L. Liu. JavaScript Face Detection Explained. 2011.
http://liuliu.me/eyes/javascript-face-detection-explained/, [10] T. Steiner, R. Verborgh, J. Gabarro, and R. Van de
2012. Accessed March 26, 2013. Walle. Defining Aesthetic Principles for Automatic
[5] G. Rizzo, T. Steiner, R. Troncy, R. Verborgh, J. L. Media Gallery Layout for Visual and Audial Event
Redondo Garc´ and R. Van de Walle. What Fresh
ıa, Summarization based on Social Networks. In Quality of
Media Are You Looking For?: Retrieving Media Items Multimedia Experience (QoMEX), 2012 Fourth
From Multiple Social Networks. In Proceedings of the International Workshop on, pages 27–28, July 2012.
2012 International Workshop on Socially-Aware [11] L. Wiyartanti, Y.-S. Han, and L. Kim. A Ranking
Multimedia, SAM ’12, pages 15–20. ACM, 2012. Algorithm for User-generated Video Contents Based
[6] J. San Pedro and S. Siersdorfer. Ranking and on Social Activities. In Digital Information Management,
Classifying Attractiveness of Photos in Folksonomies. 2008. ICDIM 2008. Third International Conference on,
In Proceedings of the 18th International Conference on pages 260–265, Nov. 2008.
World Wide Web, WWW ’09, pages 771–780, 2009.](https://image.slidesharecdn.com/a-meteoroid-on-steroids-ranking-media-items-stemming-from-multiple-social-networks-www2013devtrack1-130411072559-phpapp02/85/A-Meteoroid-on-Steroids-Ranking-Media-Items-Stemming-from-Multiple-Social-Networks-3-320.jpg)
A Meteoroid on Steroids: Ranking Media Items Stemming from Multiple Social Networks
- 1. A Meteoroid on Steroids: Ranking Media Items Stemming from Multiple Social Networks Thomas Steiner Google Germany GmbH ABC-Str. 19 20354 Hamburg, Germany tomac@google.com ABSTRACT 2. RELATED WORK We have developed an application called Social Media In [6], San Pedro and Siersdorfer propse a methodology Illustrator that allows for finding media items on multiple for automatically ranking and classifying photos from the social networks, clustering them by visual similarity, ranking photo sharing platform Flickr according to their attractive- them by different criteria, and finally arranging them in me- ness for Flickr members. They work with extracted user dia galleries that were evaluated to be perceived as aestheti- feedback and annotations available on Flickr to train ma- cally pleasing. In this paper, we focus on the ranking aspect chine learning models based on image features like sharp- and show how, for a given set of media items, the most ade- ness and colorfulness. While their method is tailored to quate ranking criterion combination can be found by inter- Flickr, our approach is based on a social network interaction actively applying different criteria and seeing their effect on- abstraction layer on top of the social networks Facebook, the-fly. This leads us to an empirically optimized media item Twitter, Google+, Instagram, YouTube, Flickr, MobyPic- ranking formula, which takes social network interactions into ture, Twitpic, and Lockerz. Jaffe et al. describe [2] a ranking account. While the ranking formula is not universally appli- and summary algorithm for geo-tagged photo sets based on cable, it can serve as a good starting point for an individually spatial patterns as well as textual-topical patterns and pho- adapted formula, all within the context of Social Media Illus- tographer identity cues. Their algorithm can be expanded trator. A demo of the application is available publicly online to support social, temporal, and other factors. The shown at the URL http://social-media-illustrator.herokuapp.com/. maps-based application necessarily requires geo-tagged me- dia items, which is rarely the case with media items retrieved from social networks due to privacy concerns. In [1], David- Categories and Subject Descriptors son et al. describe the different criteria video quality, user H.3.3 [Information Search and Retrieval]: Clustering specificity, and diversification that determine the video rank- ing in the YouTube recommendation system. These criteria General Terms include view count, the ratings of the video, commenting, Algorithms favoriting, and sharing activity around the video. Finally, Wiyartanti et al. introduce in [11] a ranking algorithm for user-generated videos based on social activities. Keywords Ranking, Event Summarization, Social Networks 3. SOCIAL NETWORK INTERACTIONS 3.1 Abstraction Layer 1. INTRODUCTION Social networks have different paradigms of social interac- When people witness events like concerts, sports matches, tions. In [5], we have introduced an abstraction layer on top or meteoroid impacts, they more and more share media of the native data formats of all considered social networks items like photos and videos that depict these events pub- in order to gain an agnostic view on them. Regardless of licly on social networks. In the past, we have worked on the native data representation format of the social network methods [3, 5, 9] for the automatic extraction, deduplica- of origin, the abstraction layer unifies and streamlines the tion, and clustering of media items stemming from multiple available data for each media item to a greatest common social networks. Up to now, we have ordered the retrieved divisor of all social networks. These interaction paradigms media items chronologically, by social network, or by clus- must be exposed by the social networks via specific API ter size, and thereby completely neglected social network calls in order to be considered. In Table 1, we detail how interactions as ranking signals. Though truly added value we abstract the social interactions in question on each so- lies in exploiting these social network interactions in order to cial network. We differentiate between unknown values that obtain a more representative ranking of the potentially over- are returned as unknown, i.e., where the information is not whelmingly many media items retrieved for a given event. exposed, and 0 values, where the value is known to be zero. 3.2 Merging Social Interactions In the context of our previous research, we have devel- Copyright is held by the author/owner(s). WWW 2013 Companion, May 13–17, 2013, Rio de Janeiro, Brazil. oped a tile-wise histogram-based media item deduplication ACM 9781450320382/13/05. algorithm with additional high-level semantic matching cri-
- 2. Likes Shares Comments Views Facebook Like Facebook Share Facebook Comments YouTube Views Google+ +1 Google+ Share Google+ Comments Flickr Views Instagram Like Twitter native ReTweet Instagram Comments Twitpic Views Flickr Favorite Twitter manual RT, @Replies MobyPicture Views YouTube Like Twitpic Comments YouTube Favorite MobyPicture Comments Twitter Favorite Flickr Comments Table 1: Abstract social network interaction paradigms and their underlying native counterparts teria that is tailored to photos and videos stemming from 5. IMPLEMENTATION DETAILS multiple social networks. If a set of media items is visually The application has been implemented in Node.js, a server similar enough to be clustered under the criteria detailed side JavaScript software system designed for writing scalable in [5], we treat the whole of the cluster as if it were just one Internet applications. Programs are created using event- media item. These criteria are pair-wise tile histogram sim- driven, asynchronous input/output operations to minimize ilarity that does not exceed a given threshold and the same overhead and maximize scalability. The clustering and rank- number of detected faces per media item. In consequence, ing logic is kept on the client side, while the media item we specify a merging strategy for the associated social in- retrieval logic is kept on the server side. As the clustering teractions of the individual media items in the particular logic needs read access to the pixel data of media items via cluster. We treat unknown values as 0. The alternative to the canvas element’s getImageData function, all media items this solution is to exclude unknown values from the merg- need to be proxied locally. Face detection works fully on the ing step. However, as in practice a considerable amount client side based on a library made available by Liu [4]. The of social interaction values are unknwon, we are forced to interface is fully interactive and event-driven. Figure 1 and proceed with the abovementioned simplification. The algo- Figure 2 show screenshots of the deployed application. rithm accumulates individual social interactions and assigns the accumulated social interactions to the cluster. 6. CONCLUSIONS AND FUTURE WORK In this paper, we have presented an application called 4. RANKING MEDIA ITEM CLUSTERS Social Media Illustrator with a special focus on its social In this section, we describe a ranking formula to rank interactions abstraction layer and ranking capacities. The a set of media clusters that match a given query. In the application has been successfully evaluated to produce both ranking formula, we consider several well-defined ranking meaningful and beautiful visual and audial summaries for criteria that were detailed in [10], namely these are visual, recent events. The majority of these summaries were made audial, textual, temporal, social, and aesthetic. For a given available online.1 One example of such can be seen in Fig- set of media item clusters, a ranking is calculated as follows. ure 2, which shows popular social media reactions for the meteoroid impact event on 15 February 2013, when a small asteroid entered the atmosphere of Earth, became visible as α × likes + β × shares + γ × comments + δ × views+ a bright fireball and exploded in an air burst over Chelyabinsk. × clusterSize + ζ × recency + η × quality (1) Future work will focus on adding more visualization for- mats that will support text-to-speech once the text synthesis The factors likes, shares, comments, and views stem from part of the Web Speech API [7] has landed in Web browsers. the individual media items as described in Subsection 3.1 This will allow for true story-telling, where the associated and Subsection 3.2. The factor clusterSize corresponds to microposts for a media item can be read as it is shown, the size of the current cluster. The factor recency is calcu- potentially in an interactive slideshow format. Further, we lated as follows. If the youngest media item in the cluster plan to add more clustering options that will allow for also is less than or exactly one day old, the value is 8, for two clustering by extracted named entities [9] besides the cur- days it is 4, for three days it is 2, and for each day more, rently visual clustering. the value is 1. The factor quality is a representation of the Concluding, with our Social Media Illustrator application, presence of faces and a media item’s photo or video quality. we have contributed an effective and efficient tool to deal Empirically optimized default values that can be fine-tuned with social media overload and to identify the few needles for a concrete media item set were determined as follows: in the social network haystack. α = 2, β = 4 , γ = 8, δ = 1, = 32, ζ = 2, and η = 8. Once a final ranking for all media items has been found, the top-n media items are compiled to different kinds of 7. REFERENCES media galleries that in two user studies were shown to be [1] J. Davidson, B. Liebald, J. Liu, P. Nandy, perceived as aesthetically pleasing [8]. We differentiate be- T. Van Vleet, U. Gargi, S. Gupta, Y. He, M. Lambert, tween the Loose Order, Varying Size style, where certain B. Livingston, and D. Sampath. The YouTube Video media items can be featured more prominently by making Recommendation System. In Proceedings of the them bigger at the cost of loosely disrespecting the ranking- Fourth ACM Conference on Recommender Systems, implied order and the Strict Order, Equal Size style, which RecSys ’10, pages 293–296, 2010. strictly respects the ranking-implied order [8]. 1 http://twitpic.com/photos/tomayac, accessed 02/21/2013
- 3. Figure 1: Media Item Clusters tab of the Social Media Illustrator application with individual and clustered (bottom middle) media items from Facebook and YouTube, ranked by popularity for the Russian query Метеороид Figure 2: Zoomed view of the Media Gallery tab of the application showing an automatically generated media gallery in loose order, varying size style featuring ranked media items stemming from Facebook and YouTube for the query Метеороид [2] A. Jaffe, M. Naaman, T. Tassa, and M. Davis. [7] G. Shires and H. Wennborg. Web Speech API Generating Summaries and Visualization for Large Specification. Final Report, W3C, Oct. 2012. Collections of Geo-referenced Photographs. In [8] T. Steiner and C. Chedeau. To Crop, Or Not to Crop: Proceedings of the 8th ACM International Workshop Compiling Online Media Galleries. In Proceedings of on Multimedia Information Retrieval, pages 89–98, the 22nd International Conference Companion on World 2006. Wide Web, WWW ’13, May 2013. [3] H. Khrouf, G. Atemezing, G. Rizzo, R. Troncy, and [9] T. Steiner, R. Verborgh, J. Gabarro, and R. Van de T. Steiner. Aggregating Social Media for Enhancing Walle. Adding Meaning to Facebook Microposts via a Conference Experience. In Real-Time Analysis and Mash-up API and Tracking its Data Provenance. In Mining of Social Streams, Papers from the 2012 Next Generation Web Services Practices (NWeSP), 2011 ICWSM Workshop, June 2012. 7th International Conference on, pages 342–345, Oct. [4] L. Liu. JavaScript Face Detection Explained. 2011. http://liuliu.me/eyes/javascript-face-detection-explained/, [10] T. Steiner, R. Verborgh, J. Gabarro, and R. Van de 2012. Accessed March 26, 2013. Walle. Defining Aesthetic Principles for Automatic [5] G. Rizzo, T. Steiner, R. Troncy, R. Verborgh, J. L. Media Gallery Layout for Visual and Audial Event Redondo Garc´ and R. Van de Walle. What Fresh ıa, Summarization based on Social Networks. In Quality of Media Are You Looking For?: Retrieving Media Items Multimedia Experience (QoMEX), 2012 Fourth From Multiple Social Networks. In Proceedings of the International Workshop on, pages 27–28, July 2012. 2012 International Workshop on Socially-Aware [11] L. Wiyartanti, Y.-S. Han, and L. Kim. A Ranking Multimedia, SAM ’12, pages 15–20. ACM, 2012. Algorithm for User-generated Video Contents Based [6] J. San Pedro and S. Siersdorfer. Ranking and on Social Activities. In Digital Information Management, Classifying Attractiveness of Photos in Folksonomies. 2008. ICDIM 2008. Third International Conference on, In Proceedings of the 18th International Conference on pages 260–265, Nov. 2008. World Wide Web, WWW ’09, pages 771–780, 2009.