Detecting and Visualizing Twitter Sentiment during COVID-19 Pandemic using AWS Comprehend and Quicksight
Download as PPTX, PDF0 likes167 views
The document outlines a methodology for detecting and visualizing public sentiment related to the COVID-19 pandemic using AWS services like Comprehend and QuickSight. It details the process of data collection from various cities in India, sentiment extraction using Lambda functions, and visualization in QuickSight. Additionally, it suggests enhancements for real-time data streaming and dashboard hosting.
![Export the DynamoDB data
Configure your AWS CLI and then execute the following command to export the
Dynamo DB. Replace the table name as per your config.
$ aws dynamodb scan — table-name covidsentiment — select
ALL_ATTRIBUTES — page-size 500 — max-items 100000 — output json |
jq -r “.Items” | jq -r “(.[0] | keys_unsorted) as $keys | $keys, map([.[
$keys[] ].S])[] | @csv” > table.csv](https://image.slidesharecdn.com/vivekdeckawscdanz-210715181017/85/Detecting-and-Visualizing-Twitter-Sentiment-during-COVID-19-Pandemic-using-AWS-Comprehend-and-Quicksight-32-320.jpg)
Detecting and Visualizing Twitter Sentiment during COVID-19 Pandemic using AWS Comprehend and Quicksight
- 1. Australia & New Zealand
- 2. Detecting and Visualizing COVID-19 Pandemic Public Sentiment using AWS Comprehend and QuickSight Vivek Raja P S | 2020
- 3. A big salute to all COVID-19 Front-line workers
- 4. COVID-19, Lockdown & Psychology Orchestration of AWS Services Data Collection Setting up the environment Extracting Sentiment Visualizing the results Q&A Agenda of the talk
- 6. COVID-19 Coronavirus disease (COVID-19) is an infectious disease caused by a newly discovered coronavirus. Most people infected with the COVID-19 virus will experience mild to moderate respiratory illness and recover without requiring special treatment. Older people, and those with underlying medical problems like cardiovascular disease, diabetes, chronic respiratory disease, and cancer are more likely to develop serious illness. Source: WHO
- 7. Lockdown & Quarantine Taking India as my case-study, we are going to see the measures taken by the government First COVID-19 Positive Recorded Jan 31 Mar 25 Lockdown 1.0 Apr 15 May 04 May 18 Jun 01 Jul 01 Lockdown 2.0 Lockdown 3.0 Lockdown 4.0 Unlock 1.0 Unlock 2.0 Jan 01 2020 Present
- 10. Orchestration of AWS Services
- 12. Data Collection
- 13. Tools Data Source Data Collection Tool Data Preprocessing Tool Twitter GetOldTweets3 Library Numpy, Pandas, JSON library
- 14. Tweet Collection Criterias 1. Language : English 2. Cities 3. Date Ranges 4. Keywords
- 15. Tweet Collection Criteria - Cities Six Major Cities New Delhi Mumbai Kolkata Hyderabad Bengaluru Chennai
- 16. Tweet Collection Criteria - Cities 100km Radius per each city
- 17. Tweet Collection Criteria- Date Ranges
- 18. Tweet Collection Criteria- Keywords 15 Top Trending Keywords/ Hashtags
- 19. Setting up the environment
- 20. Create AWS Account Create a new AWS Account if you don’t have *Note: Keep an eye on incurring additional costs
- 21. Create IAM Role Create an IAM Role with the following roles attached
- 22. S3 Bucket Set up the two Storage tools DynamoDB Table Set “text” as partition key
- 24. Lambda Function Setup The lambda function must be triggered whenever there is a new object created in S3 bucket. The data is sent to AWS Comprehend to extract the sentiment. The results are updated in DynamoDB table
- 25. Lambda Function Runtime: Python 3.8 Import necessary libraries and initialize AWS Comprehend and DynamoDB
- 27. Lambda Function Define the Lambda function where the event is the S3 object creation trigger The lambda function takes the S3 object that was created as event param
- 29. Lambda Function For every record of the S3 object, the sentiment is extracted from AWS Comprehend and the result is updated in DynamoDB
- 32. Export the DynamoDB data Configure your AWS CLI and then execute the following command to export the Dynamo DB. Replace the table name as per your config. $ aws dynamodb scan — table-name covidsentiment — select ALL_ATTRIBUTES — page-size 500 — max-items 100000 — output json | jq -r “.Items” | jq -r “(.[0] | keys_unsorted) as $keys | $keys, map([.[ $keys[] ].S])[] | @csv” > table.csv
- 33. Upload the data to AWS QuickSight Navigate to AWS QuickSight. Under the Analysis tab, click New Analysis -> New Dataset -> Upload a file. Click and upload the exported DynamoDB data CSV file.
- 34. Visualize the data using AWS QuickSight
- 35. Visualize the data using AWS QuickSight
- 37. How this can be enhanced? 1. You can use AWS Firehose to stream real-time data from Twitter to S3. 2. You can build a data pipeline between DynamoDB / or any other DB to Quicksight Data source. 3. Host the dashboard as a web app using Elastic Beanstalk
- 38. Resources
- 39. How this can be enhanced? COVID-19 Twitter Dataset: https://github.com/Vivek0712/covidtwitterdataset Step by step tutorial blog: https://vivekraja98.medium.com/detecting-and- visualizing-twitter-sentiment-during-covid-19-pandemic-using-aws-
- 40. Let’s flatten the curve
- 41. Open for Q&A Vivek Raja P S /vivek0712 /Vivek0712 @Vivekraja007