IUI 2016 talk on predicting attitude and actions
- 1. Predicting Attitude and Actions of Twitter Users Jalal Mahmud – IBM Research, Almaden Geli Fei – University of Illinois, Chicago Anbang Xu – IBM Research, Almaden Aditya Pal – Facebook Michelle Zhou – Juji Inc
- 2. Motivation & Background Social media such as Twitter has emerged, and different brands have social media presence to attract their potential customers People express various opinions about such brands in social media - Some people may like a brand (e.g., Delta Airlines, Fitbit), some may show neutral attitude, and others may dislike the brand. - Some may have formed an attitude towards a brand very recently, and others may have an attitude for quite a long time. - Some people have attitude with higher confidence than others. - Some may remember their attitude well . - Some are more likely to change their attitudes. People also take different actions (e.g., buy a product/service corresponding to that brand, recommend others to buy) based on their attitude 2
- 3. Motivation & Background Prior works on sentiment/opinion analysis can be useful to know whether a user may like/dislike a brand Prior works do not address whether attitudes are persistent (e.g., an individual formed an attitude for a long time) or temporary (e.g., an individual formed an attitude recently) They also do not provide the strength of attitude (e.g., whether the individual has attitude with high/low confidence). Furthermore, they do not address how well a user remembers his/her attitude or whether the user is likely to change the attitude. 3
- 4. Our Work We present computational models to predict a Twitter user's attitude in terms of a number of characteristics such as: - attitude favorability (How much a consumer likes or dislikes an attitude object) - attitude persistence (whether an attitude is persistent) - attitude confidence (strength of attitude) - attitude accessibility (How well a consumer remembers her attitude) - attitude resistance (How likely a consumer keeps the present attitude). Our work is inspired by marketing literature where attitude is described in terms of such characteristics (Schiman et. Al.) We have also developed statistical models to infer likelihood of different action intention (e.g., intention to buy a product) based on one's attitude. 4 Schiman, L., and Kanuk, L. 2010. Consumer Behavior, 10th edition. Prentice Hall.
- 5. Approach We have collected ground truth data of attitude characteristics and action intentions using self-report surveys conducted among Twitter users. Using the ground-truth data, we developed statistical models to predict users’ attitude. We have also developed statistical models to infer likelihood of different action intention (e.g., intention to buy a product). We performed experiments demonstrating the effectiveness of our models. We developed a visual analytic system that integrates our models and allows customer intervention. 5
- 6. Survey Study - Methodology Self reported ground truth data of attitude and action intentions from Twitter users for two brands – Delta Airlines and Fitbit - 7534 Twitter users who tweeted about Delta (using @Delta) and 5361 Twitter users who tweeted about Fitbit (using @Fitbit) were identified and invited. - 823 users responded to Delta survey (10.9% response rate) and 507 users responded to Fitbit survey (9.6% response rate) . - We offered $50 Amazon gift card to 1 out of every 100 participants. - After removing incomplete and inconsistent responses, we had 751 survey responses for Delta and 447 survey responses for Fitbit. - For each user, we collected their most recent 3200 tweets (max limit). - If they had less than 3200 tweets, then we collected all their tweets. We also checked their tweets and they were all English. - Anonymized ground truth data so that participants were not identifiable afterwards. 6
- 7. Example Survey Questions Attitude Characteristics Questions Favorability How much have you liked your travel experience with Delta Airlines? Persistence How often have you used Delta Airlines for your travel? Confidence Based on your answers, how certain are you about your answers? Accessibility How well do you remember your attitude about Delta Airlines? Resistance How likely will you switch to another airlines if Delta reduces efficiency of service? Action Intentions Questions Buy How likely are you going to buy ticket of your next trip from Delta airlines? Recommend How likely are you going to recommend others to fly by Delta airlines? Prohibit How likely are you going to tell others not to fly by Delta airlines? • Response to each question was on a 5 point Likert scale performance.
- 8. Classification Approaches Statistical models to classify each attitude characteristics and action intention Features • Unigram features • Unigram extracted from tweets • Sentiment features • Computed using a sentiment/opinion dictionary • Context-based Sentiment/Opinion feature • Similar to sentiment features, but looks for sentiment words that appear in the surrounding area of the brand name • Domain-specific sentiment feature • Similar to context-based sentiment feature but uses a domain specific sentiment dictionary • Length of use feature • timestamp difference of a user’s latest and oldest mention of the brand • Frequency feature • This feature represents how often the user mentions the brand. Statistical Models • We tried a number of classifiers such as Naive Bayes, SMO (SVM), Random Forest from WEKA and performed 5-fold cross validation. • SMO and Random Forest based classifier achieved comparable performance. 8
- 9. Experimental Results F1 ROC Area (AUC) Delta Fitbit Delta Fitbit Favorability 0.71 0.68 0.69 0.65 Persistence 0.61 0.57 0.59 0.57 Confidence 0.58 0.56 0.56 0.55 Accessibility 0.55 0.53 0.53 0.52 Resistance 0.53 0.53 0.52 0.52 F1 ROC Area (AUC) Delta Fitbit Delta Fitbit Buy 0.68 0.65 0.67 0.63 Recommend 0.65 0.60 0.64 0.58 Prohibit 0.60 0.58 0.58 0.56 We report experimental result for SMO classifier.
- 10. System for Customer Intervention We integrated our models with a visual analytics system The system uses simple keyword filtering to identify a set of Twitter users who have recently discussed the brand in their tweets - Their attitudes and intentions are computed based on their tweets and presented to agents - Agents can create visual filters on any individual bar chart in the attitude & intention component by selecting a range on the axis, and customers within the selected range would then be shown in the detailed view component (left in Figure 1) to reflect the selection. The filtering feature helps agents identify target customers for intervention.
- 11. Conclusion & Future Work This work is a first exploration to predict one's attitude towards a brand in terms of a set of characteristics, and likelihood to take different actions based on attitudes. Future Work - Explore other forms of ground truth collection - Experiment with more features and their relationships - Explore how to construct attitude models which are easily scalable across multiple brands - Apply our models in real-world scenarios. Questions?
Editor's Notes
- #3: Such as celebrating birthdays, attending weddings, and becoming parents. Just by the sheer volume of activities on social media, we can certainly feel the heat and power of social media. 1 min.
- #4: Such as celebrating birthdays, attending weddings, and becoming parents. Just by the sheer volume of activities on social media, we can certainly feel the heat and power of social media. 1 min.
- #5: Such as celebrating birthdays, attending weddings, and becoming parents. Just by the sheer volume of activities on social media, we can certainly feel the heat and power of social media. 1 min.
- #6: Such as celebrating birthdays, attending weddings, and becoming parents. Just by the sheer volume of activities on social media, we can certainly feel the heat and power of social media. 1 min.
- #8: 30 sec.
- #9: Such as celebrating birthdays, attending weddings, and becoming parents. Just by the sheer volume of activities on social media, we can certainly feel the heat and power of social media. 1 min.
- #10: We observe mixed result for attitude characteristics prediction. Attitude favorability can be predicted with reasonable accuracy (65%-69% AUC). However for other attitude characteristics, we found moderate to low prediction accuracy. It is more intuitive that textual and sentiment based feature extracted from users' historical tweets contain predictive information to predict attitude favorability (i.e., how much they like/dislike a brand). Attitude accessibility and resistance were more difficult to predict which could be either due to the reason that our survey users did not provide very reliable response for those dimensions or users' historical tweets did not contain enough predictive feature for predicting them. Table 6 shows the result of action intention prediction for both datasets in terms of F1 and AUC. Overall, action intentions can be predicted within reasonable accuracy (56% to 67% AUC). In each case, buy action achieves best performance, while prohibit action seems harder to predict. We think one of the difficulties of predicting prohibit action comes from the particularly unbalanced training data, as prohibit action is an extreme action that not many users would do.