Forensic linguistics with Apache Spark
4 likes227 views
This document discusses using Apache Spark for forensic linguistics analysis of tweets from Greece during the 2015 financial crisis. It introduces key concepts like idiolect, sociolect and intertextuality. It then explores the dataset of 190k Greek Twitter users from June-August 2015 using Spark and techniques like word2vec, LDA and clustering to analyze writing styles, identify similar users and detect communities. Future directions discussed include deep learning approaches for tasks like author profiling.
![Forensic Linguistics
"Forensic linguistics, legal linguistics, or language and the law, is the application of
linguistic knowledge, methods and insights to the forensic context of law,
language, crime investigation, trial, and judicial procedure. It is a branch of applied
linguistics.” [Wikipedia]
- Authorship Attribution
- Authorship Identification
- Gender/Age classification etc](https://image.slidesharecdn.com/forensiclinguisticswithapachespark-160601130747/85/Forensic-linguistics-with-Apache-Spark-3-320.jpg)
![Similarities & interactions graph [Gephi]](https://image.slidesharecdn.com/forensiclinguisticswithapachespark-160601130747/85/Forensic-linguistics-with-Apache-Spark-10-320.jpg)
![Similarities & interactions graph [Gephi]
Gephi : Modularity analysis, 9 communities detected
Communities:
- “Yes”, black
- “No”, magenta
- media, red
- celebrities, dark green
- “Romantic twitter”, orange
- ....](https://image.slidesharecdn.com/forensiclinguisticswithapachespark-160601130747/85/Forensic-linguistics-with-Apache-Spark-11-320.jpg)
![- Choose top N most frequent words [1]
- Build frequency vectors for all users
- Compare user signatures [eg Cosine Similarity]
- Identified double-account user among 180K candidates (so much for anonymity)
[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3694980/
2. Idiolect : Style signatures](https://image.slidesharecdn.com/forensiclinguisticswithapachespark-160601130747/85/Forensic-linguistics-with-Apache-Spark-12-320.jpg)
![- User generates texts by sampling a number of topics
- “Similar” users will tend to have similar topic distributions
- Given a subset of similar users, identify the most influential, eg the user who enforces writing style. [But that’s another presentation :)]
Challenges
Noise
“Random events”
Opinion shifting: People change their opinions and their writing styles accordingly. Social media tends to amplify this behaviour [one
more presentation :) ]
Intertextuality: LDA + signatures](https://image.slidesharecdn.com/forensiclinguisticswithapachespark-160601130747/85/Forensic-linguistics-with-Apache-Spark-15-320.jpg)
Forensic linguistics with Apache Spark
- 1. Forensic Linguistics with Apache Spark Kostas Perifanos @k_perifanos
- 2. Idiolect, sociolect, intertextuality What? - Idiolect: individual’s distinctive and unique use of language - Sociolect : variety of language associated with a social group (socioeconomic, ethnic, age) - Intertextuality: the shaping of a text’s meaning by another text
- 3. Forensic Linguistics "Forensic linguistics, legal linguistics, or language and the law, is the application of linguistic knowledge, methods and insights to the forensic context of law, language, crime investigation, trial, and judicial procedure. It is a branch of applied linguistics.” [Wikipedia] - Authorship Attribution - Authorship Identification - Gender/Age classification etc
- 4. Dataset - 8m tweets between 18/06/2015 - 06/08/2015 - 92m words (white space tokenized) - 190K users - Key events during this period - Referendum Announcement - Capital Controls - Referendum voting
- 5. Toolset - Apache Spark 1.6.1 - RDD - DataFrames / Spark SQL - Word2vec, KMeans - Apache Zeppelin - Gephi
- 6. Basic Data Exploration - Counting Check for trends: - Lowercase vs Uppercase ratios - Relative frequencies of important (propaganda) words - Average text length (per day) - Average word length (per day)
- 7. Counting - lowercase / uppercase ratio
- 9. - Build a word2vec model, treat @mentions as vocabulary words - Find top-N “synonyms” using seed accounts, keep all starting with “@” - @handle1: @handle2, @handle3, ... - @handle32: @handle5, @handle3, ... - Visualize the graph Similarities & user interactions
- 10. Similarities & interactions graph [Gephi]
- 11. Similarities & interactions graph [Gephi] Gephi : Modularity analysis, 9 communities detected Communities: - “Yes”, black - “No”, magenta - media, red - celebrities, dark green - “Romantic twitter”, orange - ....
- 12. - Choose top N most frequent words [1] - Build frequency vectors for all users - Compare user signatures [eg Cosine Similarity] - Identified double-account user among 180K candidates (so much for anonymity) [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3694980/ 2. Idiolect : Style signatures
- 13. 2. Idiolect : Style signatures
- 14. - Apply clustering on signature vectors - KMeans on signatures - KMeans on word2vec vectors: - Transform words to vectors, sum and average - Also works very well for metaphor detection Sociolect: Clustering
- 15. - User generates texts by sampling a number of topics - “Similar” users will tend to have similar topic distributions - Given a subset of similar users, identify the most influential, eg the user who enforces writing style. [But that’s another presentation :)] Challenges Noise “Random events” Opinion shifting: People change their opinions and their writing styles accordingly. Social media tends to amplify this behaviour [one more presentation :) ] Intertextuality: LDA + signatures
- 16. - User - Topic Classification - Gender classification - Age - Personality, stress, anxiety etc - Try Deep Learning approaches Next steps
- 17. Thank you! Questions? @k_perifanos - http://github.com/kperi