Big Data, Machine Learning till Business Intelligence
- 1. Big Data and Business Intelligence Dinesh Kumar P Mallieswari D “Without data you are just another person with opinion” - W. Edwards Deming.
- 3. Pre-requisites for workshop Visual Studio 2017 15.9.12 or later - For ML.Net Auto ML with Model Builder Bold BI
- 4. Setup to Bold BI – www.boldbi.com Create your own tenant. • Option -1 Free Single User plan. • Option -2 Choose any of the paid plans. (Free trail – 15 days). Just work for 1 day in my tenant. • I will add you as a user via your email. • Will be available only for today.
- 5. For what basic skill, different people get different salaries? $10,000 $5000 $1000 $500 $50
- 7. Big Data Hype • Companies were selling Big Data solutions as a silver bullet solution to all the enterprises’ problems. • For a use case using Hive+Oozie didn’t provided expected result. Butworked better with MySQL+Cron. • There was no ideal approach like below; • Client - “I want to implement a Big Data solution” • Consultant- “That could be the solution, but let us discuss the problem first”.
- 8. Kafka and Spark • Streaming platform’s necessity. • Hadoop made fast.
- 12. Cloud Burst Allowed to be conservative in investments. Crawl before you walk and walk before you run. Elasticity both for Storage and Processing.
- 13. Cloud services Storage – HDFS –> Azure Data Lake... Processing – Map Reduce –> Azure Kubernetes cluster, Azure Synapse Analytics, Azure Databricks... Resources elasticity -> Create as you go via Automation scripts (Powershell, Ansible). That too Synapse take care of resource computation by itself “server as service”.
- 14. Review the customer feedbacks with 3 rating out of 5 Requirement • Build a cost-effective data pipeline system that analyzes customer feedbacks provided in a shopping website. • Prepare a Machine Learning model to know the sentiment of comments mentioned in feedbacks. • Also prepare a visualization that helps stake holders for decision-making.
- 17. Azure Synapse Analytics • Simplifies your data lake and data warehousing solutions – Datalakehouse. • Reduces project development time for machine learning, BI, and AI.
- 19. Sentiment Analysis • What drive’s business? – Its “customer satisfaction”. • Online Surveys, Social media posts-tweets- comments, Support requests,… • Sentiment Analysis - the quickest way for decision makers to ask a team to drill down further about feedbacks based on the ratio they get for more data or over a period of time (more bad / more good). • Almost in any domain – Healthcare, Supply Chain apart from just Retail.
- 20. Importance of Data Analytics and BI • Analytics everywhere on few clicks - Excel – Ideas, Outlook – Insights, PowerPoint – Design Ideas, Word – CV Assistant, Read aloud, Spelling & Grammar.
- 22. Borrowed from - https://www.youtube.com/watch?v=4QoCbhsQeyE
- 23. What is Machine Learning ? Looking for patterns in Data Generate Code Recognize those pattern in new Data
- 24. Enter Text What Machine Learning does ? ➢Predicting future ➢Organizing or Grouping
- 25. Enter Text Features vs Labels Features Input Labels Output Name, Amount, Where Issued, Age of Cardholder Genuine/Fraud
- 26. Enter Text Types of algorithms
- 27. Enter Text Five Questions 1. Is this A or B ? 2. Is this Weird ? 3. How much or many ? 4. How this is organized ? 5. What should I do next ?
- 28. Enter Text Role of Data Scientist ➢Which raw data to use? ➢How should that data be processed to create prepared data? ➢Identify Combinations of prepared data and machine learning algorithms should you use to create the best model?
- 29. ML.Net Open source cross-platform machine learning framework. Custom ML made like a play with AutoML – GUI. Not required much expertise in ML.
- 30. ML.Net - cmd tool https://visualstudiomagazine.com/articles/2019/09/30/automl-mlnet.aspx
- 32. Bold BI Business Intelligence tool. 80+ connectors. (SQL, NoSQL, File, Web) Easy drag and drop designer. Dashboards renderable in Mobile, TV, Large screen displays. Available as SaaS, On-premise and Embedded forms. Drilled down maps, Forecast feature.
- 33. Prototype using Azure Cognitive Dashboard for insights Create own model Standardize the pipeline
- 36. Resources
- 39. Enter Text Books & Movies ➢Cartoon Introduction to Statistics ➢Thinking Fast & Slow by Daniel Kaheman ➢Art of thinking clearly – Rolf Dobelli ➢I-Robot ➢Terminator 2
Editor's Notes
- #3: Please register yourself. In the meantime, can we get introduced? Just let us know NAME, WHERE ARE WE FROM and importantly WHAT YOU EXPECT OUT OF TODAY’s EVENT.
- #4: (or) Visual Studio or VS Code for Sentiment Analysis (or) ML.Net CLI
- #6: Let me ask a simple question. For what basic skill, different people get different salary? Perhaps that is the same basic thing which all the ML or AI projects are solving and make our life easier.
- #7: Its DECISION MAKING. And that’s what AI is doing for Humans right? I can say many real time examples for this. For now, lets take - the Google Maps. Before 5 years, if you are riding to a destination for which you don’t know the route, you will constantly ask the persons who are on your path way, think over the route and go on. But now, Google Maps in our mobile phone, decides and tells this is the right route. And we are travelling just by looking into that Map path. Notice that here decision making work is partially taken over by the software comparing to the same situation before 5 years. Software are no more just software. As engineers, we are making them as intelligent agents and make them evolve as more human in nature. Simply say, future machines with software will slowly rise to the conscious level of humans even!
- #8: My opinion - Ecosystem projects are not growing to be compatible with Hadoop. E.g. Pig, Sqoop, Oozie,…
- #12: https://trends.google.com/trends/explore?date=2012-01-14%202020-02-14&q=Apache%20Hadoop,Apache%20Spark,Apache%20Kafka
- #15: Details - An online shopping site store their feedback data in Azure SQL Data Warehouse. The data, in short, is about the rating(1 to 5) and comments by the buyer for each delivered order. The management would like to extract insights about this feedbacks and improve its business. Analysis of bad feedbacks with 1,2 rating and good feedbacks that are with 4,5 rating is straight forward. Feedbacks with 3-rating expresses the biased feeling of customers with both good and bad comments. So apart from the overall rating, the comments in 3-rating should be segregated as good and bad comments using Machine Learning. Finally need to prepare a business dashboard to showcase the feedbacks for the sake of decision making.
- #19: Synapse Analytics combines Big Data and Relational Data to combine as one.
- #22: AI is everywhere. As you see in the slide, ‘Design Ideas’ in PowerPoint suggests you good designs and icon sets for the content, in a fraction of second than a human can think. ‘Insights’ in outlook lets you know, if anything you have promised is getting missed out without addressing it. ‘Ideas’ in Excel, automatically decides the suitable chart based on the data you have and populates it right inside of the Excel app automatically.
- #23: Nowadays, a bank itself is software and takes the decision. Same case earlier humans took decisions using that software. A perfect example would be of sanctioning a personal loan. Things like the CIBIL score, a person’s authenticity available as up-to-date data has provided the power to decide to sanction a loan to machines themselves. Earlier, 1 or more bank employees should do the entire work of analyzing all these. Finally I would like to end with this quote from Confluent.IO. “Every Company is Becoming software, which were previously just Using the software.”
- #27: Error from sensitivity to small fluctuations(Including anomaly) . Overfitting
- #29: data scientist is a specialist in solving problems like the ones that arise in machine learning. People in this role typically have a wide range of skills. They’re comfortable working with complex machine learning algorithms, for example, and they also have a strong sense of which of these algorithms are likely to work best in different situations. A data scientist might also need to have software development skills, as they’re often called upon to write code.
- #32: Dataset of 200 items had 180 low-satisfaction items, 10 medium-satisfaction items and 10 high-satisfaction items. A model could just predict low-satisfaction for all items and score 180 / 200 = 0.9000 accuracy for the MicroAccuracy metric. But the MacroAccurcy would be (0.9000 + 0.0500 + 0.0500) / 3 = 0.3333.
- #38: Good book -- and free! Another recommended book on Azure Machine is Learning is "Predictive Analytics with Microsoft Azure Machine Learning " (https://www.amazon.com/Predictive-Analytics-Microsoft-Machine-Learning/dp/1484212010).
- #40: I-Robot Terminator 2