NEWORDER Project - Science in the online knowledge order
0 likes12 views
The New Order project, led by Stefan Dietze and collaborators, focuses on scientific discourse in online environments, particularly on Twitter, and its impact on trust and credibility. It uses a large dataset to analyze patterns in the sharing of scientific information, highlighting challenges such as the prevalence of non-peer-reviewed works and the difficulty of linking to primary scientific sources. The project employs advanced AI techniques to detect and classify scientific discourse, aiming to improve the quality and reliability of science communication in digital media.
NEWORDER Project - Science in the online knowledge order
- 1. NEWORDER – Science in the Online Knowledge Order Stefan Dietze, 13.10.2023
- 2. Discourse Interactions Algorithms/AI Motivation: science online discourse vs offline society & policies Society, Media, Politics & Policies (Offline & Online) Science discourse online (NEWORDER focus: news & social media) 2
- 3. 3 ▪ Percentage of tweets containing links to scientific articles (journals, publishers, science blogs etc) ▪ Uses list of > 17 K science web domains (URLs) ▪ Data source: 1% sample of Twitter (https://data.gesis.org/tweetskb/), (> 14 bn tweets archived since 2013) Motivation: scientific online discourse is on the rise Example: Twitter / X
- 4. 4 NEWORDER project Interdisciplinary approach, team and objectives ▪ Perception of roles, sources, and authority; impact on trust- worthiness assessment (Cognitive Psychology) ▪ Dissolution of phases, hierarchies and contexts in the scientific process (Social Sciences, Media & Communication Studies) ▪ Computational methods for collecting, detecting and classifying scientific online discourse (Computer Science/AI & Computational Linguistics) Cress, Utz (IWM & Uni Tübingen) Marcinkowski, Koss (HHU) Dietze, Boland, Jabeen (GESIS), Kallmeyer (HHU)
- 5. How can „scientific discourse“ be defined? Example: Twitter / X 5 Science claim Science reference Science relevance No science Science reference Hafid, S., Schellhammer, S., Bringay, S., Todorov, K., Dietze, S., SciTweets - A Dataset and Annotation Framework for Detecting Scientific Online Discourse, CIKM2022
- 6. Training AI to detect science discourse: SciTweets dataset & classifier 6 ▪ Manual annotation of ground truth dataset for testing models (heuristics-based sampling, annotation framework, > 1K annotated tweets) ▪ Training AI models to detect science discourse in large-scale discourse data (e.g. Web archives) ▪ Reasonable classification performance using fine- tuned language model (SciBERT) applied to TweetsKB data Hafid, S., Schellhammer, S., Bringay, S., Todorov, K., Dietze, S., SciTweets - A Dataset and Annotation Framework for Detecting Scientific Online Discourse, CIKM2022 https://github.com/AI-4-Sci/SciTweets
- 7. What is science discourse and how does it evolve? Increasing amount and proportion of not peer-reviewed science works 7 Absolute amount of tweets sharing preprints Proportion of preprints among shared science URLs
- 8. How is public attention distributed? Power law distribution 8 • 10% of studies receive > 75% of all Twitter mentions • Long tail of studies with few mentions • Data source: 1.67 M tweets mentioning at least one of the primary science studies in the „Altmetrics“ corpus Top x (%) of mentioned science studies Share of twitter mentions (%)
- 9. Challenge: online science discourse is not well-informed Links to actual scientific studies/context missing in news & social media 9
- 10. ▪ NLP models able to predict missing primary science reference (e.g. DOI or journal paper link) for given informal reference (e.g. “Heinsberg Studie”) or secondary reference (news article) Challenge: online science discourse is not well-informed Links to actual scientific studies/context missing in news & social media 10
- 11. ▪ NLP models able to predict missing primary science reference (e.g. DOI or journal paper link) for given informal reference (e.g. “Heinsberg Studie”) or secondary reference (news article) Challenge: online science discourse is not well-informed Links to actual scientific studies/context missing in news & social media 11 ▪ Supervised & unsupervised approaches using DL language models
- 12. Science discourse is „different“ 12 Examples from http://snopes.com Non-science claim Science claim
- 13. Computational (AI) challenge NLP methods (e.g. for fact-checking) perform worse on science discourse 13 ▪ Take-away: AI-based methods geared towards scientific discourse required Performance of state-of-the-art AI/deep learning using standard benchmark datasets Claim Check-Worthiness Detection Fake News Detection Claim Verification
- 14. Wrap-Up & Outlook: Interdisciplinary Work Plan Media & Communication Studies (spreading pattern & societal impact) WP5 Longitudinal online discourse analysis WP2 Dissolution of phases & contexts WP3 Perception of roles, credibility & trust WP1 Data collection & study preparation Cognitive & Social Psychology (effects on individuals) Computer & Information Science (understanding online discourse) WP4 NLP for classifying sources & roles