AI-900: Microsoft Azure AI Fundamentals 2021
4 likes8,316 views
The document provides a comprehensive overview of the AI-900 Microsoft Azure AI Fundamentals course, covering essential skills and concepts related to artificial intelligence, machine learning, and Azure services. It outlines various types of AI workloads, including computer vision, natural language processing, and conversational AI, while emphasizing responsible AI principles. Additionally, it describes core machine learning tasks and the Azure ML service, enabling students to create, manage, and publish machine learning models.
![©Sean Xie
ML Solution Core Tasks – Data Preparation Common Tasks Cont.
Separate data into training and validation datasets
Examples:
• Split Rows - [0-1, default 0.5] the fraction of rows in the first output
dataset
For example, if the value is 0.7, then the dataset is divided 70/30
Data
Transformation
Select Columns in
Dataset
Add Columns
Join Data
Remove Duplicate
Rows
Normalize Data
Clean Missing Data
Split Data
Change the values of numeric columns to be on a common scale
Examples:
• MinMax - Rescaling to the [0,1]
Ignore, replace, or infer missing values
Examples:
• Minimum missing value ratio - [0-1, default 0] the minimum portion
of missing values to be operated
• Maximum missing value ratio - [0-1, default 1] the maximum portion
of missing values to be operated
• Custom substitution value
• Replace with mean / median / mode
• Remove entire row / column
Significantly larger values
may lead to higher bias](https://image.slidesharecdn.com/ai-900courseslidesseanxiev2-210906062143/85/AI-900-Microsoft-Azure-AI-Fundamentals-2021-53-320.jpg)
![©Sean Xie
Face Service – Face Detection - Attribute
# Attributes Outputs
1 Age Estimated age in years
2 Gender 'male’ or 'female'
3 Smile Smile intensity (a number between [0,1])
4 Facial Hair moustache, beard, sideburns + confidences
5 Glasses 'noGlasses', 'readingGlasses', 'sunglasses', 'swimmingGoggles'
6 Head Pose 'roll', 'yaw', 'pitch’ with degrees
7 Emotion 'anger’, ‘contempt’, ‘disgust’, ‘fear’, ‘happiness’, ‘neutral’, ‘sadness’, ‘surprise’ + confidences
8 Hair ‘bald’ with confidence
Invisible (TRUE/FALSE)
Hair Color: 'unknown', 'white', 'gray', 'blond', 'brown', 'red', 'black', 'other’ + confidence
9 Makeup Eye Makeup (TRUE/FALSE); LIP Makeup (TRUE/FALSE)
10 Occlusion Forehead occluded (TRUE/FALSE)
Eye occluded (TRUE/FALSE)
Mouth occluded (TRUE/FALSE)
11 Accessories 'headWear', 'glasses', 'mask’ + confidence
12 Blur Blur Level: 'Low', 'Medium', 'High’ + a number indicating blurring level (0 to 1)
13 Exposure Exposure Level: ‘UnderExposure', 'GoodExposure', 'OverExposure’ + a number indicating exposure level (0 to 1)
- [0, 0.25) is under exposure; [0.25, 0.75) is good exposure; [0.75, 1] is over exposure
14 Noise Noise Level: 'Low', 'Medium', 'High’ + a number indicating noise level (0 to 1) - [0, 0.3) is low noise level. [0.3,
0.7) is medium noise level. [0.7, 1] is high noise level](https://image.slidesharecdn.com/ai-900courseslidesseanxiev2-210906062143/85/AI-900-Microsoft-Azure-AI-Fundamentals-2021-105-320.jpg)
![©Sean Xie
Language Understanding Service – Resource & App Creation
Custom Vision Resource
Azure Portal
https://portal.azure.com
Language Understanding
• Key
• Endpoint (URL)
Prediction
Model
Training
Model
• Key
• Endpoint (URL)
Cognitive-Services Resource
• Key
• Endpoint (URL)
Prediction
Model
Language
Understanding Portal
https://www.luis.ai/
Custom Vision Resource
Resource Group
Custom Vision Resource
Entities Creation Intents Creation
Authoring
Prediction
Option 1
Option 2
Application
Creation
Resource
Creation
Types
• Machine learned - learned from context
• List - a fixed, closed set of related words along with their
synonyms, e.g. door and gate, light and lamp
• Regex - extract an entity based on a regular expression
pattern, e.g. [0-9]{4}-[0-9]{3}-[0-9]{3} for 1234-567-890
• Pattern.any - variable-length placeholders used only in a
pattern’s template utterance to mark where entities begin
and end, e.g. Have you passed the {AI-900} exam[?]
• Intent
• NONE intent (handle utterances
that do not map any of the
utterances have been entered)
• Related Utterances (Examples)
• Labelling - Entities](https://image.slidesharecdn.com/ai-900courseslidesseanxiev2-210906062143/85/AI-900-Microsoft-Azure-AI-Fundamentals-2021-148-320.jpg)
AI-900: Microsoft Azure AI Fundamentals 2021
- 1. ©Sean Xie AI-900: Microsoft Azure AI Fundamentals Sean Xie 2021
- 2. ©Sean Xie Section 1 Course & Exam Introduction
- 3. ©Sean Xie Course Introduction • Help you prepare for the AI-900 MS Azure AI Fundamentals • Go through all the required skills and knowledge for the exam • Get you 100% confident to sit in the exam Rich Diagrams & Pictures Real Samples & Lab Results • Illustrate core concepts & features with multiple diagrams & samples • Demonstrate how to create intelligent applications with Azure services
- 4. ©Sean Xie Exam Guide – Skills Measured Skills/Topics Percentage Describe Artificial Intelligence workloads and considerations - Common AI workloads - Guiding principles for responsible AI 15-20% Describe fundamental principles of machine learning on Azure - Common machine learning types - Core machine learning concepts - Core tasks in creating a machine learning solution - Azure machine learning studio 30-35% Describe features of computer vision workloads on Azure - common types of computer vision solution - Azure tools and services 15-20% Describe features of Natural Language Processing (NLP) workloads on Azure - Common NLP workload - Azure tools and services 15-20% Describe features of conversational AI workloads on Azure - Common use cases - Azure tools and services 15-20% Official website: https://docs.microsoft.com/en-us/learn/certifications/exams/ai-900
- 5. ©Sean Xie Section 2 Introduction to Artificial Intelligence
- 6. ©Sean Xie “The rise of powerful AI will be either the best or the worst thing ever to happen to humanity.” - Prof Stephen Hawking
- 7. ©Sean Xie What’s Artificial Intelligence (AI) – Definition from dictionaries • “The theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages.” – Oxford Dictionary • “The study of how to produce machines that have some of the qualities that the human mind has, such as the ability to understand language, recognize pictures, solve problems, and learn” – Cambridge Dictionary • “The study of how to make computers do intelligent things that people can do, such as think and make decisions” – Longman Dictionary
- 8. ©Sean Xie What’s Artificial Intelligence (AI) – Definition in AI-900 • Artificial intelligence (AI) is the capability of a computer to imitate intelligent human behavior. Analyze Images Comprehend Speech Interact in Natural Ways Detect Anomalies Make Predictions Vision Hearing Cognition Language
- 9. ©Sean Xie Introduction to Machine Learning • Machine Learning A data science technique that allows computing resources to use historical data to create a model to forecast future behaviours, outcomes, and trends. • Purpose - Forecasts / Predictions Register Data Build & Train Model Evaluate Model Deploy Model
- 10. ©Sean Xie Machine Learning (ML) is the foundation of an AI system Artificial intelligence Machine Learning Deep Learning Prediction/Forecasting Anomaly Detection Computer Vision Natural Language Processing Conversational AI Build, Train, Deploy Models Neural Network
- 12. ©Sean Xie Prediction/Forecasting in AI • Workloads • Prediction of the global growth and trend of COVID-19 pandemic • Prediction of a disease diagnosis based on symptoms and medical history • Prediction of the future levels of carbon dioxide (CO2) in the atmosphere • Weather prediction/forecasting • Predict application workload and performance for resource management • Predict sales of cars next month based on historic data • Predict home prices next year • Predict the travel time/fastest route using a navigation app • Azure Services • Machine Learning
- 13. ©Sean Xie Anomaly Detection in AI • Workloads • Identifying suspicious logins by looking for deviations from normal activities • Identifying suspicious behaviors by detecting too many failed login attempts • Monitoring machines’ temperature in the factories • Detecting irregular heart beats using a health monitoring system in the hospital • Detecting fraud in credit card transactions in the Banking systems • Azure Services Decision Anomaly Detector Cognitive Services
- 14. ©Sean Xie Computer Vision in AI – Overview Computer Vision Image Classification Face Detection, Analysis & Recognition Object Detection Semantic Segmentation Image Analysis Optical Character Recognition (OCR) Cognitive Services Vision • Azure Services
- 15. ©Sean Xie Computer Vision in AI – Image Classification & Object Detection Image Classification Object Detection • Classify images based on contents • Classify individual objects • Identify object’s location • Indicate a bounding box
- 16. ©Sean Xie Computer Vision in AI – Image Analysis & Semantic Segmentation Image Analysis Semantic Segmentation • Extract information from images • Return ‘tags’ with ‘confidence’ • Summarize the scene with a descriptive caption • An advanced ML technique – label every pixel • Locate objects and shapes • Return a mask layer of objects
- 17. ©Sean Xie Computer Vision in AI – Face Detection & OCR Face Detection, Analysis & Recognition Optical Character Recognition (OCR) • Specialized form of object detection • Locate human faces with a bounding box • Suggest age and emotional state • Identify or verify the identity of an individual • Recognise text in images (photos or scanned documents) • Convert printed or handwritten text characters into text data
- 18. ©Sean Xie Natural Language Processing (NLP) in AI Natural Language Understanding (NLU) Natural Language Generation (NLG) Natural Language Processing (NLP) Language Speech Cognitive Services • Azure Services • Read • Interpret • Write • Generate
- 19. ©Sean Xie Key Phrases Natural Language Processing (NLP) in AI – NLU Extract Known Entities Determine Sentiment Text Interpretation Speech Recognition • Analyze and interpret text in documents • Interpret spoken language “Hello everyone.” Audio Text “To be or not to be? That is the question. Whether TIS nobler in the mind to suffer the slings and arrows of outrageous fortune, or to take arms against a sea of troubles, and by opposing end them.” Fantastic castle and beautiful gardens Larnach Castle, Dunedin, New Zealand 16/12/2015 This is a unique place with more than a hundred and forty years of history. It's called New Zealand's only Castle. It's a must-see place in Dunedin. We stayed a night. We really enjoyed our stay at Larnach Castle. Clean rooms, good service, amazing location with incredible views of the Otago Harbour. The dinner was fantastic. We also really enjoyed exploring the gardens and flowers. It's an unforgettable visit.
- 20. ©Sean Xie Natural Language Processing (NLP) in AI – Intent Recognition Commands Interpretation / Intent Recognition • Interpret commands and determine appropriate actions * Turn on the light please * Please switch the lamp on * Turn off the light please * Please turn the lamp off * Switch on the fan please * Please turn the AC on * Switch off the fan please * Please turn the AC off * Please close the gate * Could you close the door * Please open the door * Could you open the gate
- 21. ©Sean Xie Natural Language Processing (NLP) in AI - NLG Speech Synthesis Translation • Synthesize speech responses • Translate spoken or written texts Text Translation Speech Translation English to French: -> Hello -> Bonjour “Turning the light off.” Audio Text “To be or not to be? That is the question. Whether tis nobler in the mind to suffer The slings and arrows of outrageous fortune, Or to take arms against a sea of troubles And by opposing end them.” Text to Text Text to Speech Speech to Speech Speech to Text Hello -> Bonjour Hello -> ->
- 22. ©Sean Xie Conversational AI • Azure Services • QnA Maker • Bot Services • AI agents participate in conversations with humans • Conversation AI Scenarios • Chatbots • IBM Watson Assistant • Home automation / Smart home • Smart personal assistant / Virtual assistants • Siri, Alexa, Google Assistant, Cortana, etc. • Turing Test Machine or Human?
- 24. ©Sean Xie Six Ethics Principles of Microsoft Responsible AI Fairness Reliability & Safety Privacy & Security Inclusiveness Transparency Accountability Responsible AI To build a trust, reliable, safe and secure AI
- 25. ©Sean Xie Ethics Principles of Responsible AI
- 26. ©Sean Xie Ethics Principles of Responsible AI
- 27. ©Sean Xie Ethics Principles of Responsible AI
- 28. ©Sean Xie Ethics Principles of Responsible AI
- 29. ©Sean Xie Ethics Principles of Responsible AI
- 30. ©Sean Xie Ethics Principles of Responsible AI
- 32. ©Sean Xie Azure AI + ML Services in AI-900 Vision Language Speech Decision Computer Vision Custom Vision Face Form Recognizer Text Analytics Language Understanding QnA Maker Speech to Text Text to Speech Speech Translation Anomaly Detector AI-900 Translator Content Moderator Cognitive Services Bot Services Responsible AI Fairness Reliability & Safety Privacy & Security Inclusiveness Transparency Accountability ML Studio Machine Learning AutoML ML Designer ML Workspace
- 33. ©Sean Xie Section 3 Introduction to Azure Machine Learning
- 34. ©Sean Xie Azure Machine Learning – Azure ML Service • Azure Machine Learning service - A cloud-based platform for creating, managing, and publishing machine learning models. ML Studio Azure ML ML Workspace Notebooks Code with Python SDK Automated ML ML Designer Data & Compute Management Pipelines ML Types ML Core Concepts ML Core Tasks No-code ML No-code ML
- 36. ©Sean Xie Machine Learning Types – Supervised ML Models • Supervised ML Model: training data includes known label values Classification predicting one of several categories or classes • Whether a patient is at risk of diabetes • whether tumor is benign or malignant • Categorize loan applications as low-risk or high-risk • Whether a loan will be repaid • Fraud detection • Object detection Regression predicting numeric values • The number of cars that will be sold next month • The price of a pre-owned car based on its maker, model, engine size, and mileage • The number of future car rentals • Predicting how many minutes a flight will be delayed based on weather information • The Carbon Dioxide Emissions by Energy Consumption Use in an area Time-series forecasting predicting numeric values based on time • Heights of ocean tides • Weather forecasting • Global temperature • Monthly counts of sunspot • Pollution levels • Blood pressure tracking • Heart rate monitoring • Population • Unemployment rates
- 37. ©Sean Xie Machine Learning Types – Unsupervised ML Models • Unsupervised ML Model: to separate inputs into clusters based purely on their characteristics, or features without a known label value Clustering group inputs into clusters based on the similarity of their features • Segment customers into different groups • Group cars of similar size or weight • Group penguins based their similarities • Image classification - group images that have similar characteristics • Face grouping – divide faces into groups Feature: Shape The minimum number of features: 1
- 38. ©Sean Xie Machine Learning Types – Summary Machine Learning Types Classification Regression Time-series Forecasting Reinforcement Learning Supervised Learning Unsupervised Learning Clustering Robotics IoT
- 40. ©Sean Xie • Feature: Data values that influence the prediction of a model • Labelling: Process of tagging training data with known values Core ML Concepts – Features & Labels Feature (Input) x Historical Data Label (Output) y Future Outcomes f (x) = y Function (f) Algorithms
- 41. ©Sean Xie Core ML Concepts – Training and Validation Datasets • Split Data Split Rows (one of the common techniques) • randomly split the data into rows for training and rows for evaluation/testing Training Model Training Data Evaluation Data Split data • Cross-validation • Iteratively test the trained model with data it wasn't trained with and compare the predicted value with the actual known value Data Train Model Evaluate Model Training Model Training Data Subsets Evaluation Data Split data Data Train Model Evaluate Model Should NOT evaluate the model using the same data that was used for training
- 42. ©Sean Xie Core ML Concepts – Machine Learning Algorithms • Microsoft Azure Machine Learning Algorithm Cheat Sheet https://download.microsoft.com/download/3/5/b/35bb997f-a8c7-485d-8c56-19444dafd757/azure-machine-learning-algorithm-cheat-sheet-nov2019.pdf?WT.mc_id=docs-article-lazzeri Machine Learning Algorithms Multiclass Classification Regression Two-Class Classification Clustering Predict Values Predict Categories Group Data Two-Class Averaged Perceptron Two-Class Decision Forest Multiclass Boosted Decision Tree Multiclass Logistic Regression K-Means Image Classification Text Analytics Anomaly Detection Recommenders Additional Requirements Accuracy Training Time Linearity Number of Parameters Number of Features Boosted Decision Tree Regression Decision Forest Regression Fast Forest Quantile Regression Linear Regression Neural Network Regression Poisson Regression Bayesian Linear Regression One vs. All Multiclass Two-Class Neural Network Two-Class Boosted Decision Tree Two-Class Logistic Regression Two Class Support Vector Machine Multiclass Decision Forest Multiclass Neural Network One vs. One Multiclass Generate Recommendations Detect Unusual Occurrences Extract Information Classify Images
- 43. ©Sean Xie Core ML Concepts – Evaluation Metrics for Classification True Positives False Positives False Negatives True Negatives Actual Predicted 1 0 1 0 Confusion Matrix Accuracy !" + !$ !" + !$ + %" + %$ Precision !" !" + %" Recall !" !" + %$ What’s the proportion of correctly classified instances? What’s the percentage of the class predictions made by the model were correct? What’s the percentage of a certain class correctly identified? The weighted average of precision and recall F1 Score 2 ∗ !" 2 ∗ !" + %" + %$ 2* ()*+,-,./ ∗ 0*+122 ()*+,-,./30*+122 = Closer to 1 the better
- 44. ©Sean Xie Core ML Concepts – Evaluation Metrics for Classification Example YES NO YES NO Apple Classification Predicted Actual TP FP FN TN
- 45. ©Sean Xie Core ML Concepts – Evaluation Metrics for Classification – ROC Curve & AUC True Positives False Positives False Negatives True Negatives Actual Predicted 1 0 1 0 Confusion Matrix False Positive Rate (FPR) !" !" + $% True Positive Rate (TPR) $" $" + !% Recall 0 1 0 1 TPR FPR ROC Curve (receiver operating characteristic curve) Ideal Bad Model Example Random Classifier ROC Curve Example AUC (area under the curve) Ideal Random guessing (Y = X) Worse than random guessing AUC = 1 AUC = 0.5 AUC < 0.5
- 46. ©Sean Xie Core ML Concepts – Evaluation Metrics - Predicted vs. True • Residuals (Errors) - the difference between the predicted and actual value (label) • Residual Histogram – a diagram shows the frequency of residual value ranges Predicted vs. True Residuals Bad Model Bad Model Example Residual Histogram The closer of frequently occurring residual values to 0, more accurate the ML model. Bad Model Shaded Aare: Variance of predictions around that mean predicted value
- 47. ©Sean Xie Core ML Concepts – Evaluation Metrics for Regression Mean Absolute Error (MAE) Root Mean Squared Error (RMSE) Relative Absolute Error (RAE) Relative Squared Error (RSE) Coefficient of Determination (R2) The average of the absolute errors (difference between predicted values and true values) !"# = ∑&'( ) | +& − -&| . The square root of the mean squared errors (difference between predicted and true values) The relative absolute difference between predicted and true values • compare models where the labels are in different units The relative squared difference between predicted and true values • compare models where the labels are in different units Represents the predictive power of the model as a value between 0 and 1 RMSE = ∑/01 2 3/45/ 6 ) - = ∑&'( ) -& . 7"# = ∑&'( ) | +& − -&| ∑&'( ) |- − -&| 78# = ∑&'( ) +& − -& 9 ∑&'( ) - − -& 9 79 = 1 − ∑&'( ) +& − -& 9 ∑&'( ) - − -& 9 = 1 − 78#
- 48. ©Sean Xie Core ML Concepts – Evaluation Metrics for Regression Example Lower the value the better Closer to 1 the better Closer to 0 the better
- 50. ©Sean Xie ML Solution Core Tasks Overview Core Tasks Data Ingestion & Preparation Feature Engineering & Selection Model Training & Evaluation Model Deployment & Management Step 1 Step 2 Step 3 Step 4 ML Pipelines
- 51. ©Sean Xie ML Solution Core Tasks – Data Ingestion • Create Datasets in Azure Machine Learning Designer (Recommended) • Import Data module – import data from Azure cloud storages or public URLs • Enter Data Manually module – create a small dataset by manually entering values Datasets Azure cloud storages • Azure Blob Container • Azure File Share • Azure Data Lake • Azure Data Lake Gen2 • Azure SQL Database • Azure Database for PostgreSQL • Databricks File System • Azure Database for MySQL Datastore Local File Azure Open Datasets Public URL
- 52. ©Sean Xie ML Solution Core Tasks – Data Preparation Common Tasks Data Transformation Select Columns in Dataset Add Columns Join Data Remove Duplicate Rows Normalize Data Clean Missing Data Split Data Exclude unwanted features (columns) from the dataset Combine all columns from two or more datasets Merge two datasets using a database-style join operation based on the key column, or Composite keys using multiple columns. Examples: • Inner Join – combined rows when the values of key column match • Left Outer Join – keep left table Remove potential duplicates from a dataset ID A B 01 XXX XXX 02 XXX XXX 03 XXX XXX ID C 01 XXX 02 XXX 04 XXX ID A B C 01 XXX XXX XXX 02 XXX XXX XXX 03 XXX XXX ID A B 01 XXX XXX 02 XXX XXX 03 XXX XXX ID C 01 XXX 02 XXX 04 XXX ID A B C 01 XXX XXX XXX 02 XXX XXX XXX Missing value/ Replacement value Key Column
- 53. ©Sean Xie ML Solution Core Tasks – Data Preparation Common Tasks Cont. Separate data into training and validation datasets Examples: • Split Rows - [0-1, default 0.5] the fraction of rows in the first output dataset For example, if the value is 0.7, then the dataset is divided 70/30 Data Transformation Select Columns in Dataset Add Columns Join Data Remove Duplicate Rows Normalize Data Clean Missing Data Split Data Change the values of numeric columns to be on a common scale Examples: • MinMax - Rescaling to the [0,1] Ignore, replace, or infer missing values Examples: • Minimum missing value ratio - [0-1, default 0] the minimum portion of missing values to be operated • Maximum missing value ratio - [0-1, default 1] the maximum portion of missing values to be operated • Custom substitution value • Replace with mean / median / mode • Remove entire row / column Significantly larger values may lead to higher bias
- 54. ©Sean Xie ML Solution Core Tasks – Feature Engineering & Selection • Feature Engineering • The process of creating new features from original data to help machine learning algorithms to learn better • Automation – Featurization • Automatically pre-process the features before training • Feature Type: DateTime • Engineer Features: • Year • Month • Day Features (Columns) Observation / Record (Rows) • Day of week • Day of year • Quarter • Week of the year • Hour • Minute • Second • Feature Selection • The purpose is to narrow the features by selecting a subset of relevant, useful features of original data to reduce noise and improve training performance • The process of applying statistical tests to the input datasets to determine which columns are more predictive of the output
- 55. ©Sean Xie Purpose Azure ML Module Input / Output ML Solution Core Tasks – Model Training & Evaluation Evaluate Model Measures the accuracy of a trained model • Evaluate Model module – for a regression or classification or clustering model • Output: evaluation metrics for comparison of the performance of multiple training models Score Model Generate predictions on new actual data • Score Model module – for a regression or classification model • Assign Data to Clusters module – for a clustering model • Input: dataset for validation (Split Data) Train Model Train a ML model • Train Model module – for a regression or classification model • Train Clustering Model module – for a clustering model • Input: dataset for training and algorithm • K-means clustering Number of centroids parameter – the number of clusters for the model
- 56. ©Sean Xie ML Solution Core Tasks – Model Deployment & Management Azure Kubernetes Service (AKS) Azure Container Instance (ACI) Development and Testing Production (Real-time Inference) Inference Clusters Compute Target Trained Model New Data Predict Outcomes
- 57. ©Sean Xie ML Solution Core Tasks – Building Azure ML Pipelines • Azure Machine Learning Pipelines • Orchestrate data preparation, model training, deployment, and tasks management • Help automate machine learning workflows Data Ingestion & Preparation Feature Engineering & Selection Model Training & Evaluation Model Deployment & Management Training Pipeline Inference Pipeline
- 58. ©Sean Xie ML Solution Core Tasks – Creating Pipelines – Regression Model Trained Model Asset Reused Inference Pipeline Training Pipeline Algorithm Selection Data Preparation Training Scoring Evaluation
- 59. ©Sean Xie ML Solution Core Tasks – Creating Pipelines – Classification Model Trained Model Asset Reused Inference Pipeline Training Pipeline Algorithm Selection Data Preparation Training Scoring Evaluation
- 61. ©Sean Xie Azure Machine Learning – Workspace • Azure Machine Learning Workspace A centralized place to manage data, compute resources, code, models, and other artefacts related to your machine learning workloads. • Workspace Management Task Azure Portal Azure ML Studio Python SDK / R SDK Azure ML CLI Visual Studio Code Create a workspace X X X X Manage workspace access X X Create and manage compute resources X X X X Create a Notebook VM X No Code Coding Business/Data Experts Programmer
- 62. ©Sean Xie Azure Machine Learning – Workspace Architecture Azure Portal https://portal.azure.com Azure ML Studio https://ml.azure.com Create Resources Custom Vision Resource Associated Azure Resources Storage Account Container Registry Key Vault Application Insights Custom Vision Resource Azure ML Workspace Environments Experiments Pipelines Datasets Models Endpoints Compute Instances Compute Clusters • Python packages • Environment variables • Software settings • Groupings of many runs (execution of a training script) • Workflows of machine learning tasks • Reference to the data source location and its metadata • Trained files to recognize certain types of patterns • Instantiation of ML models • Stores artefacts & logs • Default datastore • Registers docker containers • Lazy-loaded - ACR is only created when deployment images are created • Stores secrets • Stores monitoring information about the models • Fully managed VMs • Development workstations • Scalable clusters of VMs • Compute targets for on- demand processing of experiment code Inference Clusters • Deploy models to Azure Kubernetes Service for real- time inferencing Attached Compute • Links to existing Azure compute resources (VMs or Azure Databricks clusters) Dependencies Managed Resources Assets Manage Assets and Resources Azure Blob Storage Azure File Share • Stores Jupyter notebooks Linked Services Datastore Compute Targets Create Workspace
- 63. ©Sean Xie Azure Machine Learning – Compute Resource Compute • Fully managed VMs • Development workstations • Run Jupyter Notebooks • Scalable clusters of VMs • Compute targets for on-demand processing of experiment code • Run Training Pipelines • Run Inference Pipelines • Deploy models to Azure Kubernetes Service for real- time inferencing (Production) • Links to existing Azure compute resources (VMs or Azure Databricks clusters)
- 64. ©Sean Xie Demo Creating an Azure ML Workspace in Azure Portal
- 65. ©Sean Xie No-code Machine Learning with Azure ML Studio #1 AutoML
- 66. ©Sean Xie Azure Machine Learning – Automated ML (AutoML) • AutoML - the process of automating the time-consuming, iterative tasks of machine learning model development A B C D Trained Model Feature Selection Parameter Tuning 62% 75% 56% 84% 91% 86% 78% 93% 83% 59% 43% 86% 69% 75% 82% 79% 81% 70% 95% 84% Algorithm Selection Model Optimization Best Model • Classification • Regression • Time-series Forecasting
- 67. ©Sean Xie Azure Machine Learning – AutoML Model Training Steps Create Dataset Create Experiment Train Model Evaluate Model Deploy Model Automated Training Model Selection • Classification • Regression • Time series forecasting Upload dataset Feature Selection Settings • Algorithm selection • Parameter setting • Featurization Azure Kubernetes Service (AKS) Azure Container Instance (ACI) Production (Real-time Inference) Development & Testing
- 68. ©Sean Xie Hands-on Lab Training a Regression Model with Azure Automated ML
- 69. ©Sean Xie No-code Machine Learning with Azure ML Studio #2 Designer
- 70. ©Sean Xie Azure Machine Learning – ML Designer • Azure ML designer provides a drag-and-drop visual canvas to build, test, and deploy ML models. • Datasets • Modules • Pipelines • Supported open-source frameworks • MLflow • Kubeflow • ONNX (Open Neural Network Exchange) • PyTorch • TensorFlow • Supported programming languages • Python and R
- 71. ©Sean Xie Azure Machine Learning – ML Designer Model Training Steps Create Dataset Create Training Pipeline Evaluate Model Upload dataset Feature Selection Create Inference Pipeline Deploy Model Azure Kubernetes Service (AKS) Azure Container Instance (ACI) Production (Real-time Inference) Development & Testing Feature Engineering & Selection Model Selection Algorithm selection Run the training Pipeline Run the Inference Pipeline
- 72. ©Sean Xie Hands-on Lab Training a Classification Model with Azure ML Designer
- 74. ©Sean Xie Azure Machine Learning Summary ML Studio ML Workspace Automated ML ML Designer Data & Compute Management Pipelines ML Types ML Core Concepts ML Core Tasks Data Ingestion & Preparation Feature Engineering & Selection Model Training & Evaluation Model Deployment & Management Regression Classification Time-series Forecasting Reinforcement Learning Supervised Learning Unsupervised Learning Clustering Machine Learning Azure ML Features & Labels Datasets Algorithms Evaluations Metrics Classification Model Regression Model Confusion Matrix Accuracy Precision F1 Score ROC Cure & AUC MAE RMSE RAE RSE R2 Select Columns in Dataset Add Columns Join Data Remove Duplicate Rows Normalize Data Clean Missing Data Split Data Train Model Score Model Evaluate Model Training Pipeline Inference Pipeline
- 75. ©Sean Xie Section 4 Computer Vision in Microsoft Azure
- 76. ©Sean Xie Overview of Computer Vision Services in Azure • Azure Cognitive Services Services Pre-trained Models Custom Models Description Computer Vision X Analyze content in images and videos Custom Vision X Customize image recognition to fit your business needs Face X X Detect and identify people and emotions in images Form Recognizer X X Extract text, key/value pairs, and tables from documents
- 78. ©Sean Xie Azure Cognitive Computer Vision Service - Overview Image Description Image Tags Object Detection Color Scheme Image Type Image Category Brand Detection Face Detection Domain-Specific Content Detection Adult Content Detection Thumbnails Generation Optical Character Recognition
- 79. ©Sean Xie Image Tags • Tags based on thousands of recognizable objects, living beings, scenery, and actions • Tag’s confidence Image Description • A caption or a list of captions ordered from highest to lowest confidence • based on a collection of content tags Computer Vision Service – Image Analysis Object Detection • Location of the object – bounding box’s coordinates
- 80. ©Sean Xie Computer Vision Service – Color Scheme Image Color Scheme • Black & White: (Boolean value) • 1 – True • 0 – False • Dominant background color • Dominant foreground color • Dominant color of the image • Set of dominant color (12): black, blue, brown, gray, green, orange, pink, purple, red, teal, white, and yellow • Accent color: Hexadecimal code for HTML coding (combination of dominant colors and saturation)
- 81. ©Sean Xie Computer Vision Service – Image Type Image Type • Likelihood of clip art: • 0 – Non-clip-art • 1 – Ambiguous • 2 – Normal-clip-art • 3 – Good-clip-art • Line Drawing: (Boolean value) • 1 – True • 0 – False
- 82. ©Sean Xie Computer Vision Service – Image Category
- 83. ©Sean Xie Computer Vision Service – Image Category & Taxonomy Category Taxonomy • 86 categories • parent/child hereditary hierarchy Image Category • Categories • Confidences Source: https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/concept-categorizing-images
- 84. ©Sean Xie Computer Vision Service – Brands Detection Brands Detection • Brand name • Confidence score • Location - coordinates of a bounding box around the logo • Logo image and brand name can be recognized as two separate logos
- 85. ©Sean Xie Computer Vision Service – Face Detection Face Detection • Age • Gender • Location - coordinates of a bounding box
- 86. ©Sean Xie Computer Vision Service – Domain-Specific Content Domain-specific Content Celebrities Landmarks • Name • Confidence score • Coordinates of a bounding box around the logo Two ways to use: • Scoped analysis (standard) • Enhanced categorization analysis • Celebrities: people_ category • Landmarks: outdoor_ or building_ categories
- 87. ©Sean Xie Computer Vision Service – Optical Character Recognition (OCR) – OCR API OCR API • Read a small amount of text • Have false positive with large amount of text • Multiple languages (27) • Detects Synchronously (Quick) • Returns bounding box coordinates • Regions that contain text • Lines of text in each region • Words in each line of text
- 88. ©Sean Xie Computer Vision Service – Optical Character Recognition (OCR) – Read API Read API • Optimized for larger documents • Printed and handwritten text • Supported Language: Dutch, English, French, German, Italian, Portuguese and Spanish • Detects asynchronously (wait until the analysis to complete) • coordinates • Pages - One for each page of text (page size & orientation) • Lines of text on a page • Words in each line of text
- 89. ©Sean Xie Computer Vision Service – Adult Content Detection Adult Content Detection - Basic • Categories: • Adult – explicitly sexual in nature (e.g., nudity and sexual acts) • Racy – sexually suggestive in nature (less sexually explicit content) • Gory – blood/gore • Outputs • Three Boolean value • Confidence score Azure Content Moderator - Advanced • Analyze text, image, and videos • Moderation Outputs • Classification Label • Classification Score • Operation-specific insights • Review APIs • Review Tools – web-based • integrate with human reviewers A bleeding finger protected by a band-aid
- 90. ©Sean Xie Computer Vision Service – Thumbnails Generation Area of Interest • Coordinates of area of interest / region of interest (ROI) • Bounding box Thumbnails Generation (with Smart Cropping Enabled) • Area of interest / region of interest (ROI) identification • Cropped image based on ROI • Small version of image • 50 x 50 pixels Smart Cropping Enabled Smart Cropping Disabled Use the center area of image to generate thumbnails Use the area / region of interest of image to generate thumbnails Center area Area of Interest
- 91. ©Sean Xie Computer Vision Service – Image Criteria • General Image Criteria: • Supported format: JPEG, PNG, GIF, BMP • File size: < 4 MB • Dimensions: > 50 x 50 pixels • OCR API: • Dimensions: 50 x 50 to 4200 x 4200 pixels • Read API: • Supported format: JPEG, PNG, BMP, PDF, TIFF • PDF and TIFF documents: • For the free tier, only the first 2 pages are processed. • For the paid tier, up to 2,000 pages are processed. • Image file size: < 50 MB (4 MB for the free tier) • Dimensions: 50 x 50 to 10000 x 10000 pixels • PDF File Dimensions: < 17 x 17 inches (A3)
- 92. ©Sean Xie Computer Vision Service – Create Resource Custom Vision Resource Azure Portal https://portal.azure.com Computer Vision Resource • Key • Endpoint (HTTP address) Model Cognitive-Services Resource • Key • Endpoint (HTTP address) Models Option 1 Option 2 Resource Group Options Purpose Resource Key & Endpoint Cognitive-Services Resource • Use other cognitive services together • Simplify administration and development For all Cognitive services models Computer Vision Resource • Track utilization and costs of Computer Vision service separately For Computer Vision model only
- 93. ©Sean Xie Hands-on Lab Analyzing Image with Azure Computer Vision Service
- 94. ©Sean Xie Hands-on Lab Analyzing Text with Azure Computer Vision Service
- 96. ©Sean Xie Custom Vision Service – Introduction Pre-trained Models Training Model Predication Model Custom Vision Resource Computer Vision Custom Vision Organic Recyclable Gardening Hazardous Others Waste Classification 1 2 Image Analysis Image Classification
- 97. ©Sean Xie Custom Vision Resource Custom Vision Service – Resource & Project Creation Azure Portal https://portal.azure.com Custom Vision Resource • Key • Endpoint (URL) Training Model Predication Model • Key • Endpoint (URL) Cognitive-Services Resource • Key • Endpoint (URL) Models (Training & Predication) Option 1 Option 2 Custom Vision Portal https://customvision.ai Custom Vision Resource Project Creation Resource Group Custom Vision Resource Image Classification Object Detection Option 1 Option 2 Option 3 Both Training & Predication Resource Creation
- 98. ©Sean Xie Custom Vision Service – Image Classification Steps Upload & Tag Images Train Model Evaluate Model Publish Model Consume Model Tag: kangaroo Tag: giant panda Tag: puppy Measure Metric: • Precision • Recall • Average Precision (AP): recall & precision • Project ID • Model name • Predication Key • Predication Endpoint Quick Test Predication Model Training Model
- 99. ©Sean Xie Custom Vision Service – Object Detection Steps Upload & Tag Images Train Model Evaluate Model Publish Model Consume Model Measure Metric: • Precision • Recall • Mean Average Precision (mAP): recall & precision • Project ID • Model name • Predication Key • Predication Endpoint Predication Model Training Model Tag: kangaroo Tag: bird Quick Test
- 100. ©Sean Xie Use Cases Image Classification Use Case Description Product Identification Identification of items/products in the supermarkets Animals/Flowers Identification Identification of known animals/flowers in the wild Disaster Investigation Evaluation of key infrastructure (bridges, railways, etc.) in aerial surveillance images Medical Imaging Diagnosis Evaluation of X-ray or Magnetic resonance imaging (MRI) images • Detection of diabetic retinopathy stages Object Detection Use Case Description Personal Protective Equipment (PPE) Detection Detection of safety helmet, glasses, goggles, gloves, face masks, face shield, high visibility and protective workwear, etc. Building Safety Evaluation Detection of fire extinguishers, safety signs, etc. Self-driving Cars Lane Keeping Assist (LKA) - road markings detection Medical Imaging Diagnosis Evaluation of X-ray or Magnetic resonance imaging (MRI) images • Unknown object detection
- 101. ©Sean Xie Hands-on Lab Classifying Images with Azure Custom Vision Service
- 102. ©Sean Xie Hands-on Lab Detecting Object with Azure Custom Vision Service
- 104. ©Sean Xie Face Service – Face Detection Face ID • A unique identifier string Face Attribute • 14 attributes • Head Pose (3D)
- 105. ©Sean Xie Face Service – Face Detection - Attribute # Attributes Outputs 1 Age Estimated age in years 2 Gender 'male’ or 'female' 3 Smile Smile intensity (a number between [0,1]) 4 Facial Hair moustache, beard, sideburns + confidences 5 Glasses 'noGlasses', 'readingGlasses', 'sunglasses', 'swimmingGoggles' 6 Head Pose 'roll', 'yaw', 'pitch’ with degrees 7 Emotion 'anger’, ‘contempt’, ‘disgust’, ‘fear’, ‘happiness’, ‘neutral’, ‘sadness’, ‘surprise’ + confidences 8 Hair ‘bald’ with confidence Invisible (TRUE/FALSE) Hair Color: 'unknown', 'white', 'gray', 'blond', 'brown', 'red', 'black', 'other’ + confidence 9 Makeup Eye Makeup (TRUE/FALSE); LIP Makeup (TRUE/FALSE) 10 Occlusion Forehead occluded (TRUE/FALSE) Eye occluded (TRUE/FALSE) Mouth occluded (TRUE/FALSE) 11 Accessories 'headWear', 'glasses', 'mask’ + confidence 12 Blur Blur Level: 'Low', 'Medium', 'High’ + a number indicating blurring level (0 to 1) 13 Exposure Exposure Level: ‘UnderExposure', 'GoodExposure', 'OverExposure’ + a number indicating exposure level (0 to 1) - [0, 0.25) is under exposure; [0.25, 0.75) is good exposure; [0.75, 1] is over exposure 14 Noise Noise Level: 'Low', 'Medium', 'High’ + a number indicating noise level (0 to 1) - [0, 0.3) is low noise level. [0.3, 0.7) is medium noise level. [0.7, 1] is high noise level
- 106. ©Sean Xie Face Service – Face Detection - Landmarks Face Landmarks • 27 landmark points • Eyebrow • Eye • Nose • Mouth • Lip • Locations of the points Nose Root Left Nose Root Right Eyebrow Left Inner Eyebrow Left Outer Eye Left Top Eye Left Outer Eye Left Bottom Eye Left Inner Nose Tip Nose Left Alar Top Nose Left Out Tip Upper Lip Top Upper Lip Bottom Mouth Left Eyebrow Right Inner Eyebrow Right Outer Eye Right Top Eye Right Outer Eye Right Bottom Eye Right Inner Nose Right Alar Top Nose Right Out Tip Under Lip Top Under Lip Bottom Mouth Right
- 107. ©Sean Xie Face Service – Face Recognition Verification Identification Similarity Grouping • Analyze the likelihood that two faces belong to the same person ? Do two faces belong to the same person? • Search and identify faces (one to many) Person 1 Person 2 Known Faces DB ? Who is this person • Find similar faces ? Does this person look like other people? • Divide faces into groups Person 1 Person 2 Do all of these faces belong together? ? Similar faces
- 108. ©Sean Xie Face Service – Face Recognition – Verification Image Input Image Analysis Face Service Analysis Output Result of finding similarity Do two faces belong to the same person? ?
- 109. ©Sean Xie • Sean Person Group Face Service – Face Recognition – Identification Image Input Training Model Face Service Analysis Output Prediction Face Service Model Label: Sean Label: Sean Label: Sean • Sean Person Group Who is this person ? • Sean Person Group
- 110. ©Sean Xie Face Service – Face Detection - Image Criteria and Performance • Image Criteria • Image type: JPEG, PNG, GIF, BMP • File size: 6 MB or less • Face size: 36 x 36 to 1920 x 1080 pixels • Maximum detectable face size: 4096 x 4096 pixels • Limitation - faces might not be detected • Face size between 1920 x 1080 pixels to 4096 x 4096 pixels • Extreme face angles (head pose) – Best results: Frontal and near-frontal faces • Face occlusion (objects such as sunglasses that block part of the face) • Performance Improvement • Smoothing – turn smoothing feature off as it creates a blur between frames • Shutter Speed – recommend shutter speeds: 1/60 second or faster to create clearer video frames • Shutter Angle – a lower shutter angle will create a clearer video frame
- 111. ©Sean Xie Hands-on Lab Analyzing Face with Azure Face Service
- 113. ©Sean Xie Form Recognizer Service – Introduction • Features • Identifying key/value pairs • Extracting text • Processing tables of data • Identifying specific types of fields • Receipts • Invoices • Business cards • Layout extraction • Identity Document (ID) • Models • Pre-built Model • Custom Model • Tailored to identify specific forms • Known key/value pairs • Known table data
- 114. ©Sean Xie Form Recognizer Service – Prebuilt Model – Receipt Receipt Type • Restaurants, gas stations, retail, grocery store, etc. key information • Time and date of the transaction, merchant information, amounts of taxes, line items, transaction totals, etc. • 15 receipt fields • Confidence value • Bounding box • OCR raw text • All text on the receipt Supported Locales • Australia (EN-AU), Canada (EN-CA), Great Britain (EN-GB), India (EN-IN), and United States (EN-US) Customer scenarios • Business expense reporting • Auditing and accounting • Consumer behavior
- 115. ©Sean Xie Form Recognizer Service – Prebuilt Model – Invoice Extract text, key/value pairs, and table data key information • customer, vendor, invoice ID, invoice due date, total, invoice amount due, tax amount, ship to, bill to, etc. • 26 invoice fields • Confidence value • OCR raw text • Bounding box • All text on the invoice
- 116. ©Sean Xie Form Recognizer Service – Prebuilt Model – Business Card No longer used Details no longer available For Demo / Training purpose key information • Personal contact info, company name, department, job title, email, mobile, phone numbers, address, website, etc. • 13 fields • Confidence value • OCR raw text • Bounding box • All text on the invoice Business card sample
- 117. ©Sean Xie Form Recognizer Service – Prebuilt Model – Layout Tables Structures Selection Marks
- 118. ©Sean Xie Form Recognizer Service – Image Criteria & Sample Tool • Image Criteria • Image type: JPEG, PNG, PDF, and TIFF • File size: < 50 MB • Face size: 50 x 50 to 10000 x 10000 pixels • PDF and TIFF documents: • For the free tier, only the first 2 pages are processed. • For the paid tier, up to 2,000 pages are processed. • PDF File Dimensions: < 17 x 17 inches (A3) • Text: English • Online Tools • Invoice/Receipt/Business card/ID: https://fott-preview.azurewebsites.net/prebuilts-analyze • Layout: https://fott-preview.azurewebsites.net/layout-analyze
- 119. ©Sean Xie Hands-on Lab Analyzing Receipt with Azure Form Recognizer Service
- 121. ©Sean Xie Computer Vision Summary Computer Vision Custom Vision Face Form Recognizer Cognitive Services Vision Image Description Image Tags Object Detection Color Scheme Image Type Image Category Brand Detection Face Detection Domain-Specific Content Detection Adult Content Detection Thumbnails Generation Optical Character Recognition Image Classification Object Detection Face Detection Face Recognition Face Attribute Face Landmarks Verification Identification Similarity Grouping Receipts Invoices Business Cards Layout ID
- 122. ©Sean Xie Section 5 Natural Language Processing in Microsoft Azure
- 123. ©Sean Xie Overview of NLP Services in Azure • Azure Cognitive Services Services Pre-trained Models Custom Models Description Text Analytics X Detect sentiment, key phrases, and named entities Speech – Speech-to-Text (Speech Recognition) X X Transcribe audible speech into readable, searchable text Speech – Text-to-Speech (Speech Synthesis) X X Convert text to lifelike speech for more natural interfaces Speech – Speech-Translation X Integrate real-time speech translation into your apps Translator X X Detect and translate more than 90 supported languages Language Understanding X X Build natural language understanding into apps, bots, and IoT devices
- 125. ©Sean Xie Text Analytics Service – Language Detection Review #1 Fantastic castle and beautiful gardens Larnach Castle, Dunedin, New Zealand 16/12/2015 This is a unique place with more than a hundred and forty years of history. It's called New Zealand's only Castle. It's a must-see place in Dunedin. We stayed a night. We really enjoyed our stay at Larnach Castle. Clean rooms, good service, amazing location with incredible views of the Otago Harbour. The dinner was fantastic. We also really enjoyed exploring the gardens and flowers. It's an unforgettable visit. Review #2 16/12/2015 Language Detection • Language name • IETF BCP-47 language tag • ISO 639-1 language code - a two-letter lowercase abbreviation Examples: • en – English • es – Spanish • de – German • fr – French • hi – Hindi • ja – Japanese • ko – Korean • ru - Russian • zh - Chinese • Confidence score (0-1) Language Support: https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/language-support?tabs=language-detection
- 126. ©Sean Xie Text Analytics Service – Language Detection Review #3 C'est incroyable - Fantastic castle and beautiful gardens Larnach Castle, Dunedin, New Zealand 16/12/2015 This is a unique place with more than a hundred and forty years of history. It's called New Zealand's only Castle. C'est un endroit incontournable. We stayed a night. We really enjoyed our stay at Larnach Castle. Clean rooms, good service, amazing location with incredible views of the Otago Harbour. The dinner was fantastic. We also really enjoyed exploring the gardens and flowers. It's an unforgettable visit. Mixed-language content • Returns predominant language Ambiguous content • Limited text or only punctuation • countryHint parameter • Default countryHint – ‘US’ United States • Language can’t be detected • Name/code: unknown • Score: NaN (Not a number) French words commonly used in both English & French aviation country_hint = 'fr' country_hint = ’us' illusion country_hint = 'fr' country_hint = ’us' Mixture of English & French Language: unknown with confidence: NaN Emoticons / Kaomoji faces :-) :) :-3 :-> 8-) :-} (>_<) (^_^;) ^_^ (?_?) *<|:-)
- 127. ©Sean Xie Text Analytics Service – Sentiment Analysis Sentiment Analysis • Label Positive: ≥ one positive sentence + reset neutral Negative: ≥ one negative sentence + reset neutral Neutral: All neutral sentences Mixed: ≥ one positive + ≥ one negative sentence (Document Level) • Confidence score • Indeterminate Sentiment • No sufficient context or insufficient phrasing • Wrong language code Review #1 Fantastic castle and beautiful gardens Larnach Castle, Dunedin, New Zealand 16/12/2015 This is a unique place with more than a hundred and forty years of history. It's called New Zealand's only Castle. It's a must-see place in Dunedin. We stayed a night. We really enjoyed our stay at Larnach Castle. Clean rooms, good service, amazing location with incredible views of the Otago Harbour. The dinner was fantastic. We also really enjoyed exploring the gardens and flowers. It's an unforgettable visit. Review #2 The Worst Hotel Ever The XXX Hotel 17/12/2015 The room was dirty and the service was terrible. Our experience at this hotel was really bad. We were shocked by the staff's unprofessional services. I won't recommend it to anyone. Don't make the same mistake like we did. Language Support: https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/language-support?tabs=sentiment-analysis 0 0.5 1 Positive Negative Neutral/ Indeterminate
- 128. ©Sean Xie Text Analytics Service – Sentiment Analysis Text Sample: “To be, or not to be, that is the question. The hotel is amazing, but the location is too far away from the town. We still really enjoyed our stay at this hotel.” Sentence #1 Sentence #2 Sentence #3 Document/Whole Text
- 129. ©Sean Xie Text Analytics Service – Key Phrase Extraction Key Phrase Extraction • List of main/key points Review #1 Fantastic castle and beautiful gardens Larnach Castle, Dunedin, New Zealand 16/12/2015 This is a unique place with more than a hundred and forty years of history. It's called New Zealand's only Castle. It's a must-see place in Dunedin. We stayed a night. We really enjoyed our stay at Larnach Castle. Clean rooms, good service, amazing location with incredible views of the Otago Harbour. The dinner was fantastic. We also really enjoyed exploring the gardens and flowers. It's an unforgettable visit. Review #1 Fantastic castle and beautiful gardens Larnach Castle, Dunedin, New Zealand 16/12/2015 This is a unique place with more than a hundred and forty years of history. It's called New Zealand's only Castle. It's a must-see place in Dunedin. We stayed a night. We really enjoyed our stay at Larnach Castle. Clean rooms, good service, amazing location with incredible views of the Otago Harbour. The dinner was fantastic. We also really enjoyed exploring the gardens and flowers. It's an unforgettable visit. Language Support: https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/language-support?tabs=key-phrase-extraction
- 130. ©Sean Xie Review #1 Fantastic castle and beautiful gardens Larnach Castle, Dunedin, New Zealand 16/12/2015 This is a unique place with more than a hundred and forty years of history. It's called New Zealand's only Castle. It's a must-see place in Dunedin. We stayed a night. We really enjoyed our stay at Larnach Castle. Clean rooms, good service, amazing location with incredible views of the Otago Harbour. The dinner was fantastic. We also really enjoyed exploring the gardens and flowers. It's an unforgettable visit. Text Analytics Service – Entity Recognition Review #1 Fantastic castle and beautiful gardens Larnach Castle, Dunedin, New Zealand 16/12/2015 This is a unique place with more than a hundred and forty years of history. It's called New Zealand's only Castle. It's a must-see place in Dunedin. We stayed a night. We really enjoyed our stay at Larnach Castle. Clean rooms, good service, amazing location with incredible views of the Otago Harbour. The dinner was fantastic. We also really enjoyed exploring the gardens and flowers. It's an unforgettable visit. Entity Recognition • Named Entity Recognition Category & Subcategories • Person: Names • PersonType: Job types or roles • Location • Geopolitical Entity (GPE) / Structural / Geographical • Organization • Medical / Stock exchange / Sports • Event • Cultural / Natural / Sports • Product • Computing products • Skill • Address • Phone number (US and EU phone numbers only) • Email • URL • IP • DateTime • Date / Time / DateRange / TimeRange / Duration / Set • Quantity • Number / Percentage / Ordinal numbers / Age / Currency / Dimensions / Temperature • Entity Linking - Wikipedia • Personally Identifiable Information (PII) • Health (PHI) (Preview) Language Support: https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/language-support?tabs=named-entity-recognition https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/language-support?tabs=entity-linking
- 131. ©Sean Xie Review #1 Fantastic castle and beautiful gardens Larnach Castle, Dunedin, New Zealand 16/12/2015 This is a unique place with more than a hundred and forty years of history. It's called New Zealand's only Castle. It's a must-see place in Dunedin. We stayed a night. We really enjoyed our stay at Larnach Castle. Clean rooms, good service, amazing location with incredible views of the Otago Harbour. The dinner was fantastic. We also really enjoyed exploring the gardens and flowers. It's an unforgettable visit. Text Analytics Service – Entity Recognition Sample Text Analytics API (v3.1)
- 132. ©Sean Xie Hands-on Lab Analyzing Text with Azure Text Analytics Service
- 134. ©Sean Xie Speech Recognition vs. Speech Synthesis • Detect and interpret spoken input • Generate spoken output • Recorded voice -- audio file • Live audio -- microphone Input Process & Output Acoustic Model audio signal phonemes Language Model words • Text to be spoken • Voice to be used to vocalize the speech • Generating captions for recorded or live videos • Creating transcripts of phone calls or meetings • Automating note dictation • Determining user’s input for further processing Use Cases Feature Speech Recognition Speech Synthesis • Generating spoken responses to user’s input • Creating telephone voice menus • Reading email or text messages • Broadcasting announcements in public areas • Creating voices for extinct languages phonetic sounds Language Model Acoustic Model individual words prosodic units (phrases, clauses, sentences) phonetic transcription Voice (pitch, timbre) audio signal
- 135. ©Sean Xie Speech Recognition – Speech-to-Text Service Overview Audio files Microphone Custom Model Universal Language Model Azure Blob Storage Real-time Transcription Batch Transcription (asynchronous) Automatic Detection Specify Language Source Language Gstreamer Handling compressed audio: MP3, OPUS/OGG, etc. Default: WAV Dictation Mode Interpreting word descriptions of sentence structures, e.g. punctuation • question mark à ? Transcription data Pronunciation Assessment Evaluating the accuracy and fluency of the audio input Conversational Scenarios Multi-speaker Multi-device Advanced Scenario Language Support: https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/language-support#speech-to-text
- 136. ©Sean Xie Speech Synthesis – Text-to-Speech Service Overview Text data Real-time Synthesis Asynchronous Synthesis (Long Audio API) Books / Lectures (> 10 minutes) Speech Synthesis Markup Language (SSML) Adjust speaking styles (pronunciation, volume, pitch, speaking rate, etc.) Custom Standard Voice Statistical Parametric Synthesis Concatenation Synthesis Neural Voices Deep Neural Networks Human-like natural prosody Custom Neural Voice Text Analyzer Neural Acoustic Model Neural Vocoder Providing phoneme sequence Predicting acoustic features Converting the acoustic features Audio files Speaker Audio files Language Support: https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/language-support#text-to-speech Stream Standard Voices More than 75 standard voices Over 45 languages and locales Being sunset Viseme en-US English (United States)
- 137. ©Sean Xie Speech Service Samples To be or not to be? That is the question. Whether TIS nobler in the mind to suffer the slings and arrows of outrageous fortune, or to take arms against a sea of troubles, and by opposing end them. Speech-to-Text Input Output To be, or not to be, that is the question: Whether tis nobler in the mind to suffer The slings and arrows of outrageous fortune, Or to take arms against a sea of troubles And by opposing end them. Text-to-Speech Output Input Standard Voice
- 138. ©Sean Xie Hands-on Lab Recognizing & Synthesizing Text with Azure Speech Service
- 140. ©Sean Xie Literal Translation vs. Semantic Translation • Literal Translation • word-for-word translation • Semantic Translation • using semantic context Chinese light open English Open the light Close the light Turn on the light Turn off the light Results of semantic translation light close Results of literal translation
- 141. ©Sean Xie Speech Service Translation Services Overview Speech Translation Text Translation Speech-to-Text Text-to-Speech Translator Service Speech Translation Translation Text to Text Text to Speech Speech to Speech Speech to Text Use Cases: • Document translation • Email translation • Webpages translation • Customer service translation at kiosks Use Cases: • Real-time closed captioning for recorded or live videos • Simultaneous two-way translation of a spoken conversation • Creating a transcript of a telephone call or meeting Azure Services
- 142. ©Sean Xie Translation Services – Translator Service Overview • Text-to-text translation • Supports more than 90 languages and dialects • Neural Machine Translation (NMT) replaces the legacy Statistical Machine Translation (SMT) • Supporting Custom Translator to improve translations, e.g. industry terminology • Multiple target languages translation • specify one from language • multiple to languages • Optional Configurations • Profanity Filtering - omitting profane text translation (default is disabled) • Selective Translation - content can be tagged so that it isn't translated • For example, brand names Language Support: https://www.microsoft.com/en-us/translator/business/languages/#list
- 143. ©Sean Xie Translation Services – Speech Translation Service Overview • Speech-to-text translation • Speech-to-speech translation • Translating speech to multiple target languages • Supports more than 60 languages and dialects in real-time • Speech recognition occurs before speech translation Speech-to-Text Text-to-Speech Text-to-text translation Speech to Speech Translation Speech to Text Translation Speech-to-Text Text-to-text translation Language Support: https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/language-support#speech-translation
- 144. ©Sean Xie Translation Services – Translator Service Samples Option 1: Specify source • ‘from’: 'en-US’ Option 1: Specify one target each time, translate twice • ‘to’: ‘zh-CN’ • ‘to’: ‘fr’ Source Language Target Language Option 2: Language Detection Option 2: Specify multiple targets with a comma between the language codes, translate simultaneously • ‘to’: ‘zh-CN’, ‘fr’
- 145. ©Sean Xie Hands-on Lab Translating Text & Speech with Azure Translator & Speech Services
- 147. ©Sean Xie Please open the window. Language Understanding Service – Language Understanding • Utterances – what sentences a user might say • Entities – an item to which an utterance refers • Intents – purpose, goal, or task expressed in an utterance Please open the window. Utterance Intent Entity What does the user wants? What are they talking about?
- 148. ©Sean Xie Language Understanding Service – Resource & App Creation Custom Vision Resource Azure Portal https://portal.azure.com Language Understanding • Key • Endpoint (URL) Prediction Model Training Model • Key • Endpoint (URL) Cognitive-Services Resource • Key • Endpoint (URL) Prediction Model Language Understanding Portal https://www.luis.ai/ Custom Vision Resource Resource Group Custom Vision Resource Entities Creation Intents Creation Authoring Prediction Option 1 Option 2 Application Creation Resource Creation Types • Machine learned - learned from context • List - a fixed, closed set of related words along with their synonyms, e.g. door and gate, light and lamp • Regex - extract an entity based on a regular expression pattern, e.g. [0-9]{4}-[0-9]{3}-[0-9]{3} for 1234-567-890 • Pattern.any - variable-length placeholders used only in a pattern’s template utterance to mark where entities begin and end, e.g. Have you passed the {AI-900} exam[?] • Intent • NONE intent (handle utterances that do not map any of the utterances have been entered) • Related Utterances (Examples) • Labelling - Entities
- 149. ©Sean Xie Language Understanding Service – LUIS Model Building Workflow Create Entities & Intents Train & Test Model Publish Model Consume Model • Project ID • Predication Key • Predication Endpoint • Predication Location Predication Model • Device (Type: List) Normalized value • door • light • fan Synonyms • gate • lamp • AC • None • close • open • switch_off • switch_on Training Model Creating Intents Creating Entities Test Model * Turn on the light please * Please switch the lamp on * Turn off the light please * Please turn the lamp off * Switch on the fan please * Please turn the AC on * Switch off the fan please * Please turn the AC off * Please close the gate * Could you close the door * Please open the door * Could you open the gate
- 150. ©Sean Xie Hands-on Lab Understanding Language with Azure Language Understanding Services
- 152. ©Sean Xie NLP Summary Language Speech Text Analytics Language Understanding Speech to Text Text to Speech Speech Translation Translator Cognitive Services Useful Reference: Language Support: https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/language-support Language Detection Sentiment Analysis Key Phrase Extraction Entity Recognition Real-time Transcription Batch Transcription Real-time Synthesis Asynchronous Synthesis Speech-to-Text Translation Speech-to-Speech Translation Text-to-Text Translation Utterances Entities Intents
- 153. ©Sean Xie Section 6 Conversational AI in Microsoft Azure
- 154. ©Sean Xie Overview of Conversational AI Services in Azure Services Pre-trained Models Custom Models Description QnA Maker X X Create a knowledge base of conversational question and answer pairs Azure Bot Services N/A N/A Intelligent, serverless bot services that scale on demand Telephone Voice Menus Virtual Assistants Webchat Bots Knowledge Base Step one: Build a KB Step two: Build a Bot Participate in a conversation Use Cases • Webchat Bots answering common questions on a website to provide customer services • Telephone voice menus to reduce the load on call center • Virtual assistants to provide home automation
- 156. ©Sean Xie Conversational AI – QnA Maker Service • Knowledge Base: Question and Answers Pairs • Data Source Locations & Type • Existing FAQ document or web page • Entered and edited manually • Question-answer pairs from semi-structured content, including SharePoint documents, etc. • Pre-defined chit-chat (small talk) data source Sample Question: When is your birthday? Source Type Content Type URL FAQs Support pages PDF / DOC FAQs, Product Manual, Brochures, Paper, Flyer Policy, Support guide, Structured QnA, etc. Excel Structured QnA file TXT / TSV Structured QnA file • Azure SQL database query is not supported
- 157. ©Sean Xie Conversational AI – QnA Maker Service – Multi-turn Conversations • Multi-turn prompts Question 1 Answer 1 - Prompt #1 - Prompt #2 Answer 2 - Prompt #1 - Prompt #2 - Prompt #3 Question 2 Could
- 158. ©Sean Xie Conversational AI – QnA Maker Service – Resource Setup Custom Vision Resource Azure Portal https://portal.azure.com QnA Maker • KB ID • QnA Auth Key • Endpoint (URL) Production KB QnA Maker Portal https://qnamaker.ai Custom Vision Resource Resource Group Knowledge Base Creation KB Creation Resource Creation Azure Cognitive Search • Store QnA pairs • Initial ranking of QnA pairs Publish • Enable multi-turn extraction from URLs, .pdf or .docx files • Add URL • Add File • Chi-Chat data source Test KB Train
- 159. ©Sean Xie Conversational AI – QnA Maker Service – KB Creation Steps Create Knowledge Base Edit Knowledge Base Train & Test KB Publish KB Predication Model Training Model Test Model Define question-and-answer pairs • Existing FAQ document • Existing web page (URL) • Pre-defined chit-chat data source • Manual QnA entries Add new Questions & Answers • KB ID • KB endpoint • KB authorization key QnA Maker doesn’t store customer data (Q&A pairs and chatlogs). All customer data is stored in the region of Cognitive Search service deployed • Alternative phrasing • Multi-turn prompts • Metadata tags used to filter answer choices
- 161. ©Sean Xie Conversational AI – Azure Bot Services • Azure Bot • Conversationally engages with customers • Web application with conversational interfaces • Three ways to create a bot: • Bot Framework SDK - coding • Bot Framework Composer - visual canvas • QnA Maker automatic bot creation feature
- 162. ©Sean Xie Conversational AI – Azure Bot Services – Channels Standard channel: Integrate a bot into a mobile app, web page, or other applications Automatically configured when you create a bot with the Bot Framework Service • Channel - connection between communication applications and a bot Knowledge Base (QnA Pairs) Bot (Office 365) WeChat Webex Additional Adapters Channels Users
- 163. ©Sean Xie Conversational AI – Azure Bot Services – Resource & Bot Setup Custom Vision Resource Azure Portal https://portal.azure.com QnA service • KB ID • QnA Auth Key • Endpoint (URL) Production KB QnA Maker Portal https://qnamaker.ai Custom Vision Resource Resource Group Knowledge Base Creation KB Creation Resource Creation Azure Cognitive Search Publish • Enable multi-turn extraction from URLs, .pdf or .docx files • Add URL • Add File • Chi-Chat data source Test KB Train Azure Bot Service (Web App Bot) Web App Bot Custom Vision Resource Publishing KB Connect Channels 1 2 3 Create Bot Config Bot Creating Bot
- 164. ©Sean Xie Hands-on Lab Building a QnA Bot with Azure QnA Maker and Bot Service
- 166. ©Sean Xie Conversational AI Summary Language QnA Maker Cognitive Services Knowledge Base Chi-Chat Multi-turn Prompts Channels Azure Bot Services Web App Bot
- 167. ©Sean Xie Section 7 AI-900 Course Summary
- 168. ©Sean Xie Resources • AI-900 Exam Official Website https://docs.microsoft.com/en-us/learn/certifications/exams/ai-900 • Azure Documentation https://docs.microsoft.com/en-us/azure/?product=all • Azure SDK for Python https://docs.microsoft.com/en- us/python/api/overview/azure/?view=azure-python
- 169. ©Sean Xie Lab Details Lab 01 – Creating an Azure ML Workspace • Azure portal: https://portal.azure.com/ • Please try creating a new ML workspace in a different region if you couldn’t create a compute instance with a Free Trial subscription. That’s why I created a second ML workspace in the region of Asia East for the following demonstrations. Lab 02-03 – Training a Regression/Classification Model • Azure Machine Learning Studio: https://ml.azure.com/ • Link of the lab02 exercise: https://docs.microsoft.com/en-us/learn/modules/use-automated-machine-learning/ • Link of the lab03 exercise: https://docs.microsoft.com/en-us/learn/modules/create-classification-model-azure-machine-learning-designer/ Lab 04-14 – Building Computer Vision, NLP, and Conversational AI Models • Link of the lab exercises repository: https://github.com/MicrosoftDocs/ai-fundamentals • Command to download the repository: !git clone https://github.com/MicrosoftDocs/ai-fundamentals • Custom Vision Portal: https://customvision.ai • Language Understanding Portal: https://www.luis.ai/ • QnA Maker Portal: https://qnamaker.ai Clean-up After you have finished the learning, please remember to delete all the resources created during the labs and cancel the Free-trial subscription within 30 days. Otherwise, they can cost your money.
- 170. ©Sean Xie Thank you • sean.xht@gmail.com • www.linkedin.com/in/haitaoxie It always seems impossible until it's done. – Nelson Mandela