Text Mining & Sentiment Analysis with Power BI & Azure
Download as PPTX, PDF1 like3,040 views
This document discusses using Azure and Power BI to perform qualitative text analysis and gain insights from unstructured data. It describes using Azure Cognitive Services' Text Analytics API to analyze employee survey responses and assign sentiment scores. The responses are then loaded into Power BI for visualization and dashboards. Key techniques shown include combining qualitative and quantitative analysis, custom functions in Power BI, and different analytics tools for text mining.
Text Mining & Sentiment Analysis with Power BI & Azure
- 1. Qualitative Text Analysis Using Azure and Power BI Sanil Mhatre @sqlsuperguru https://sqlwithsanil.wordpress.com/ https://github.com/SQLSuperGuru
- 2. About me • Senior Developer – Data Analytics @ WWT • DBA, Developer, Architect, Amateur Data Scientist • Oracle, SQL Server, My SQL, DB2, Mongo, Vertica • Cloud, Big Data & Hadoop, Data Visualization • Speaking, Blogging & volunteering
- 3. Agenda • From Qualitative to Quantitative • Employee Survey for Acme Corp. • AI + Cognitive Services – Text Analytics APIs • Power BI • ETL with Power BI & Azure • Visual Analytics • Dashboards & Power BI Service • Gaining useful Insights • Q&A
- 4. Qualitative vs Quantitative Data & Analysis • Qualitative • Data consists of Words & Narratives • Analysis includes Key Words, Themes & Concepts • Understand perceptions, look for themes • Examples : Transcripts, Comments, Free form text responses to surveys • Quantitative • Data is numeric • Summarize/Aggregate, slice & dice • Use statistical techniques to draw conclusions • Examples : Anything that can be counted and measured • Number of responses to a survey, Number of participants in a study • Shoe size, temperature
- 5. Combining Techniques • Perform Qualitative Analysis on Text • Translate the text to some Numeric measure (sentiment score) • Perform Quantitative Analysis on the score • Descriptive Statistics • Range, Min, Max, Frequency • Measures of Central Tendency • Mean – The Average • Median – The middle score after values are numerically ordered • Mode – The most frequently occurring value • More Insights by combining Qualitative & Quantitative Analysis
- 6. Use Case • Employee Survey at Acme Corp. • 2 Questions • 1. Is there anything unique or unusual about this company that makes it a great place to work? • 2. If you could change one thing about this company to make it a better place to work, what would it be? • Free form Text • 252 replies • 995 lines • 12,368 words • Lets take a look at the Excel document
- 7. AI + Cognitive Services • Microsoft AI + Cognitive Services • A collection of machine learning & AI algorithms in the Azure cloud • Artificial Intelligence • Machine Learning services • Azure bot service • Azure Batch AI • Cognitive Services • Vision • Speech • Language (Text Analytics is a part of this group) • Knowledge • Search
- 8. Text Analytics API • Cloud based service - natural language processing over raw text • 3 types of analysis https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/overview
- 9. Take it for a spin • Try out the Text Analytics API - https://azure.microsoft.com/en- us/services/cognitive-services/text-analytics/ • Create Cognitive Services API Account in Azure • Demo • Location – Not available everywhere • Pricing – Free Tier • Endpoints • Access keys
- 10. Power BI Desktop • Desktop Product for Data Analytics & Visualization • Connects to variety of Data sources • ETL – Prepare and Model Data • Advanced Analytics • Interactive reports and visualizations • Share • Quick tour
- 11. Power BI Desktop in action • Demo • Import the Excel file • Filter the responses to remove NULL • See the Relationships Pane • Create Custom Functions • Invoke the Custom Functions to add columns for KeyPhrases & SentimentScore • Adjust Data Source Permission settings if needed • Round the score to 2 decimals • Create Bins for Sentiment Score
- 12. Visualizations in Power BI Desktop • Demo • Summary Page • Text box • Cards for # of Questions & # Responses • Word Cloud (Qualitative) • Details Page (Descriptive statistics) • Slicer for filtering by Question • Score Gauge • Box & Whiskers Plot – Mean, Median, Max, Min • Histogram – Frequency • Responses Page (Curated Raw text) • Slice by question & Response Score Bin • See all the associated response texts
- 13. Power BI Service • Share this report with .pbxi file • Publish to Power BI service • Dashboards, Reports, workbooks & datasets • Organized into workspaces • Dashboard • A canvas that uses visualizations to tell a story • Tiles and Pages can be pinned from reports to one or more dashboards • More information about Dashboards : https://docs.microsoft.com/en-us/power-bi/service-dashboards • Demo – Build the Dashboard • Insights (The bulb on your Data sets) • Natural Language Queries (Ask) for Dashboards
- 14. Gaining Useful Insights • How many employees responded • Word Cloud – Trends/what are people talking about ? • Descriptive statistics on Sentiment Scores • What is the overall perception/feeling • Range, Mean, Median, Outliers • Distribution – Responses seem polarizing • How do these numbers compare between the 2 questions • See the raw response text, sliced by • Question • Response Score Bin • Share, Collaborate
- 15. Tableau vs Power BI • Power BI • Easier ETL and invoking APIs • Easier to visualizations • Tiles are easier to move and size to page/canvas grid • Slices and Page/Report level filters can create some confusion • Tableau • Creating some visualizations like Gauge or WordCloud requires several manual steps • Filters are easier to work with at both worksheet and Dashboard level • Moving & sizing various worksheets on a Dashboard manually can get tedious
- 16. Other Analytics Tools • Vertica • Distributed columnar Database & Big Data/Advanced Analytics Platform • Pulse Package for Text Analysis functions • Easy to use SQL Interface • Expensive • R • Popular for data analytics • Extensive libraries and capabilities • Easy to use , powerful , creative • Requires familiarity with R programming language • Free Desktop • Limited by memory of the machine • Demo
- 17. Resources • Qualitative or Quantitative Data - https://cyfar.org/qualitative-or-quantitative-data • Quantitative Data Analysis - https://cyfar.org/analysis-quantitative-data • Azure Cognitive Services Text Analytics - https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/overview • Power BI Desktop - https://powerbi.microsoft.com/en-us/desktop/ • Text Analytics with Power BI - https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/tutorials/tutorial-power-bi-key-phrases • Power BI Service - https://docs.microsoft.com/en-us/power-bi/service-basic-concepts • Dashboards in Power BI Service - https://docs.microsoft.com/en-us/power-bi/service-dashboards • Tableau - https://www.tableau.com/ • Vertica Pulse - https://my.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/Pulse/PulseCookbook.htm%3FTocPath%3DVertica%2520Pulse%7CPulse%2520Cookbook%7C_____0 • R Tidytext package - https://cran.r-project.org/web/packages/tidytext/vignettes/tidytext.html • R syuzhet package - https://cran.r-project.org/web/packages/syuzhet/vignettes/syuzhet-vignette.html • My GitHub for the R code use in Demos - https://github.com/SQLSuperGuru