A Multimodal Ensemble Model for Detecting Unreliable Information on Vietnamese SNS
0 likes339 views
This document proposes a multimodal ensemble model for detecting unreliable information on Vietnamese social media. It uses text, image, and metadata features as inputs to three deep learning models - BERT+CNN, and two variants with additional CNN layers. An attention mechanism is applied to learn which image parts to focus on for each text. The models are ensembled by averaging their prediction probabilities. Evaluation on a private test set shows the ensemble model achieves an AUC of 0.945, outperforming the individual models. Future work could involve comparing posts to external sources to find evidence of fakes.
![Text pre-processing
9
n Convert emojis such as =]], :( into sentiment
words "happy" or "sad" in Vietnamese.
n Converted words and tokens that have been
lengthened into short form.
¨ “coool” to “cool”
n Changed different terms about COVID-19 into
one term for consistency.
¨ “covid”, “ncov”](https://image.slidesharecdn.com/vlsp2020reintelslides-201218145855/85/A-Multimodal-Ensemble-Model-for-Detecting-Unreliable-Information-on-Vietnamese-SNS-9-320.jpg)
A Multimodal Ensemble Model for Detecting Unreliable Information on Vietnamese SNS
- 1. A Multimodal Ensemble Model for Detecting Unreliable Information on Vietnamese SNS Phạm Quang Nhật Minh AImesoft JSC, Vietnam minhpham0902@gmail.com December 18, 2020 Nguyễn Mạnh Đức Tuân Toyo University, Japan ductuan024@gmail.com 7th International Workshop on Vietnamese Language and Speech Processing (VLSP 2020)
- 2. What is Fake News? 2 n “Fake news is a news article that is intentionally and veritably false.” (Shu et al., 2017)
- 3. Why Fake News Detection? 3 n Fake news negatively affects to society n Fake news spreads like a real virus, especially via social medias ¨ https://engineering.stanford.edu/magazine/article/how- fake-news-spreads-real-virus n Fake news detection is useful to increase the credibility of information of medias, and prevent spreading of fake contents
- 4. Why Multimodal is Important? 4 n In a addition to texts, images and videos are popular in social medias ¨ Visual information is helpful in detecting rumors n Other metadata information is useful: number of likes, shares, retweets, time stamps, etc
- 5. Our Approach 5 Text contentsImages Metadata features VGG 19 Fully-connected layer BERT + CNN Making Classification
- 6. Main Findings 6 n The proposed attention mechanism used to get the representation of images is useful n Adding residual connections in blocks leads to performance improvement n System accuracy is improved with our proposed ensemble model
- 7. Proposed Method in Detail 7 n Data processing n Model architecture n Experiments and results
- 8. Data Format 8 n Each piece of information includes 6 main attributes: ¨ The anonymized id of the owner ¨ Text contents ¨ Timestamp ¨ Number of likes ¨ Number of comments ¨ Number of shares n Each news may contain zero or more than one image
- 9. Text pre-processing 9 n Convert emojis such as =]], :( into sentiment words "happy" or "sad" in Vietnamese. n Converted words and tokens that have been lengthened into short form. ¨ “coool” to “cool” n Changed different terms about COVID-19 into one term for consistency. ¨ “covid”, “ncov”
- 10. Data Imputation 10 n Mean values to fill missing values. n For the timestamp, we applied the MICE imputation method (Azuret al., 2011)
- 11. General Model 11 Given the representation of an image and a text, we learn which parts of the impage we should give more attention
- 12. Model 1 12 1D-CNN layers with filter sizes 2, 3, 4, 5 follow the BERT module, and then a fully connected layer with Batch Normalization follow 1D-CNN layers
- 13. Model 2&3 13 Model 2&3 used three additional 1D-CNN layers Model 3 used residual connections for additional 1D-CNN layers
- 14. Feature Design (1) 14 n Timestamp feature is converted into: ¨ Day ¨ Month ¨ Year ¨ Hour ¨ Weekday n Text-based features: ¨ Number of hashtags ¨ Number of URLs ¨ Number of characters ¨ Number of words ¨ Number of question-marks ¨ Number of exclaim-marks ¨ A Boolean variable to indicate that post contains images or not
- 15. Feature Design (2) 15 n User-based features: ¨ Number of unreliable news ¨ Number of reliable news ¨ Ratio between two numbers, to indicate the sharing behavior n All the above features will be standardized by subtracting the mean and scaling to unit variance, except for the Boolean feature.
- 16. Multi Image Posts 16 n Some posts contain more than one image n Two strategies: ¨ Use one image as input ¨ Multiple images (4 images at most) as input.
- 17. Proposed Ensemble Model 17 n Choose two best models among three models n Calculate averages of probabilities returned by two models
- 18. Experiments & Results 18 n Evaluation measure: ROC AUC n We conducted experiments in order to evaluate ¨ The effect of pre-trained BERT models ¨ Text preprocessing strategies ¨ The effectiveness of the attention mechanism
- 19. PhoBERT vs NlpHUST/vibert4news 19 n Bert4news uses syllable-based tokenization ¨ Trained on 20GB of news texts n PhoBERT uses word-level/subword tokenization ¨ Trained on 20GB of texts including Wikipedia and news Pre-trained model Result on private test (AUC) PhoBERT 0.921 bert4news 0.928
- 20. Effectiveness of Attention Mechanism 20 n Using attention mechanism significantly improved the result n Images and texts are co-related. ¨ Images and texts of reliable news are often related ¨ Someone may use images that do not relate to the content of the news for the click-bait purpose Models Result on private test (AUC) w/o attention 0.928 attention 0.940
- 21. Incorrect vs correct form words 21 n “sá.thại” vs “sát hại” ¨ Contain violent contents or ex-treme words. ¨ Can bypass the social media’s filtering function. n Keeping is better! ¨ Partly reflect the sentiment of the text. ¨ Unreliable contents tend to use more subjective or extreme words to convey a particular perspective. Models (PhoBERT) Result on private test (AUC) Words in correct form 0.918 Words in incorrect form 0.921
- 22. Results 22 Run Result on private test (AUC) Model 1 0.939 Model 2 0.919 Model 3 0.940 Ensemble 0.945 n Results on the private test
- 23. Future work 23 n Use external data for fake news detection n The natural way to make a judgement in fake news detection task is to compare with different information sources to find out relevant evidences of fake news.
- 24. Thank you very much for listening! 24