AUSOUG - NZOUG-GroundBreakers-Jun 2019 - AI and Machine Learning

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Autonomous Database
Cloud Diagnosability using
Machine Learning
ANZ Workshop – June 2019
Sandesh Rao
VP AIOps , Autonomous Database
@sandeshr
https://www.linkedin.com/in/raosandesh/
1

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, timing, and pricing of any
features or functionality described for Oracle’s products may change and remains at the
sole discretion of Oracle Corporation.
2

whoami
Real
Application
Clusters - HA
DataGuard-
DR
Machine
Learning-
AIOps
Enterprise
Management
Sharding
Big Data
Operational
Management
Home
Automation
Geek
@sandeshr
https://www.linkedin.com/in/raosandesh/
3

Agenda
• Architecture for the AIOps platform for the
Autonomous Database
• Which algorithms, tools & technologies are
used?
• Oracle use cases for – AIOps in
Autonomous Database
• Questions and Open Talk
4

Why Machine Learning for us and why now?
• Lots of Data generated as exhaust from systems
– Cloud , different formats and interfaces , frameworks
• Machine Learning has become accessible
– Anyone can be a Data Scientist
– Algorithms are accessible as libraries aka scikit , keras ,
tensorflow ..
– Sandbox to get started as easy as a docker init
• Business use cases
• How to find value from the data , fewer guesses to make decisions
5

ML Project Workflow
• Set Business Objectives
• Gather , Prepare and Cleanse Data
• Model Data
– Feature Extraction , Test , Train ,
Optimizer
– Loss Function , effectiveness
– Framework and Library to use
• Apply the Model as an inference
engine
– Decision making using the Model’s
output
– Tune Model till outcome is closer to
Business Objective
Set Business
Objectives
Understand Use
case
Create Pseudo
Code
Synthetic Data
Generation
Pick Tools and
Frameworks
Train Test Model
Deploy Model
Measure Results
and Feedback
6

Types of Machine Learning
Supervised Learning
Predict future outcomes with the help of
training data provided by human experts
Semi-Supervised Learning
Discover patterns within raw data and make
predictions, which are then reviewed by human
experts, who provide feedback which is used to
improve the model accuracy
Unsupervised Learning
Find patterns without any external input other
than the raw data
Reinforcement Learning
Take decisions based on past rewards for this
type of action
7

• Hierarchical k-means, Orthogonal
Partitioning Clustering, Expectation-
Maximization
Clustering
Feature Extraction/Attribute
Importance / Component Analysis
• Decision Tree, Naive Bayes, Random
Forest, Logistic Regression, Support
Vector Machine
Classification
Machine Learning Algorithms
• Multiple Regression, Support Vector
Machine, Linear Model, LASSO, Random
Forest, Ridge Regression, Generalized
Linear Model, Stepwise Linear Regression
Regression
Association & Collaborative Filtering
Reinforcement Learning - brute force,
Monte Carlo, temporal difference....
• Many different use cases
Neural network & deep Learning with
Deep Neural Network
8

Modeling Phase – AutoML to the rescue
Provide Dataset to
AutoML
Configuration parameters
for model picked
Dataset is divided into
training set & testing set
Actual Training
Evaluate performance of
trained model
Tweak model parameters,
change predictors change
test/train data splits and
change algorithms
Pick model plus
parameters depending on
outcome and measure , F1
, Precision , Recall , MSE
Document all runs and
apply A/B testing to see
what the variations
produce
9

Tools & Libraries Assisting ML projects
10

What is Oracle Doing Around Machine Learning?
• Big Data Appliance
• Big Data Discovery , Big Data Preparation Data Visualization Cloud
• Analytics Cloud
– Sales, Marketing, HCM on top of SaaS
• DaaS – Oracle Data Cloud , Eloqua ..
• Oracle Labs (labs.oracle.com)
– Machine Learning Research Group
• Autonomous Database
– Zeppelin Notebooks preloaded with use cases
– Applied Machine Learning used for Implementing AIOps
11

Oracle AI Platform Cloud Service – Coming Soon…
• Collaborative end-to-end machine learning in the cloud
• Enables data science teams to
– Organize their work
– Access data and computing resources
– Build , Train , Deploy
– Manage models
• Collaborative , Self-Service , Integrated
• https://cloud.oracle.com/en_US/ai-platform
12

Oracle Autonomous Data Warehouse Cloud Key Features
Highly Elastic
Independently scale compute and
storage, without having to overpay for
fixed blocks of resources
Built-in Web-Based SQL ML Tool
Apache Zeppelin Oracle Machine Learning
notebooks ready to run ML from browser
Database migration utility
Dedicated cloud-ready migration tools
for easy migration from Amazon
Redshift, SQL Server and other databases
Enterprise Grade Security
Data is encrypted by default in the cloud,
as well as in transit and at rest
High-Performance Queries
and Concurrent Workloads
Optimized query performance with
preconfigured resource profiles for different
types of users
Oracle SQL
Autonomous DW Cloud is compatible with
all business analytics tools that support
Oracle Database
Self Driving
Fully automated database for self-tuning
patching and upgrading itself while the
system is running
Cloud-Based Data Loading
Fast, scalable data-loading from Oracle
Object Store, AWS S3, or on-premises
Oracle Machine Learning
13

Oracle Machine Learning and Advanced Analytics
• Support multiple data platforms, analytical engines, languages, UIs and
deployment strategies
Strategy and Road Map
Big Data / Big Data Cloud Relational
ML Algorithms
Common core, parallel, distributed
SQL R, Python, etc.GUI
Data Miner, RStudio
Notebooks
Advanced Analytics
Oracle Database Cloud DWCS
14

CLASSIFICATION
– Naïve Bayes
– Logistic Regression (GLM)
– Decision Tree
– Random Forest
– Neural Network
– Support Vector Machine
– Explicit Semantic Analysis
CLUSTERING
– Hierarchical K-Means
– Hierarchical O-Cluster
– Expectation Maximization (EM)
ANOMALY DETECTION
– One-Class SVM
TIME SERIES
– Holt-Winters, Regular & Irregular,
with and w/o trends & seasonal
– Single, Double Exp Smoothing
REGRESSION
– Linear Model
– Generalized Linear Model
– Support Vector Machine (SVM)
– Stepwise Linear regression
– Neural Network
– LASSO
ATTRIBUTE IMPORTANCE
– Minimum Description Length
– Principal Comp Analysis (PCA)
– Unsupervised Pair-wise KL Div
– CUR decomposition for row & AI
ASSOCIATION RULES
– A priori/ market basket
PREDICTIVE QUERIES
– Predict, cluster, detect, features
SQL ANALYTICS
– SQL Windows, SQL Patterns,
SQL Aggregates
A1 A2 A3 A4 A5 A6 A7
• OAA (Oracle Data Mining + Oracle R Enterprise) and ORAAH combined
• OAA includes support for Partitioned Models, Transactional, Unstructured, Geo-spatial, Graph data. etc,
Oracle’s Machine Learning & Adv. Analytics Algorithms
FEATURE EXTRACTION
– Principal Comp Analysis (PCA)
– Non-negative Matrix Factorization
– Singular Value Decomposition (SVD)
– Explicit Semantic Analysis (ESA)
TEXT MINING SUPPORT
– Algorithms support text type
– Tokenization and theme extraction
– Explicit Semantic Analysis (ESA) for
document similarity
STATISTICAL FUNCTIONS
– Basic statistics: min, max,
median, stdev, t-test, F-test,
Pearson’s, Chi-Sq, ANOVA, etc.
R PACKAGES
– CRAN R Algorithm Packages
through Embedded R Execution
– Spark MLlib algorithm integration
EXPORTABLE ML MODELS
– C and Java code for deployment
15

Oracle Machine Learning
Key Features
• Collaborative UI for data scientists
– Packaged with Autonomous Data
Warehouse Cloud (V1)
– Easy access to shared notebooks,
templates, permissions, scheduler, etc.
– SQL ML algorithms API (V1)
– Supports deployment of ML analytics
Machine Learning Notebook for Autonomous Data Warehouse Cloud
16

Oracle Machine Learning UI in ADW
17

PaaS Resource Lifecycle
Management
Bare-Metal thru Installation
Upgrade
Patching
Dependency Resolution
Prerequisites Resolution
Required Capabilities
Automatable
Scalable
Online (if possible)
PaaS Application Lifecycle
Management
Installation
Upgrade
Patching
Dependency Resolution
Prerequisites Resolution
Workload Profile Identification
Placement determination
SLA management
Automatable
Provider Interoperable
Cloud Operations Early Warning
Response System
Detect degradations and faults
Pinpoint root cause & component
Push warnings and alerts
Push targeted corrective actions
SLA – based resource management
Real-time Health Dashboard
Continuous and frequent
Autonomous Action Enabled
OSS Integration Enabled
Management Interoperable
AIOps Cloud Operations – 3 Strategic Pillars
Resource Lifecycle Management Database Lifecycle Management Database Autonomous Self-Repair
20

Autonomous Health
Cloud Platform
MachinesSmart Collectors
SRs
Expert Input
Feedback &
Improvement
Bugs
1
SRs
Logs
Model
Generation
Model
Knowledge
Extraction
Applied Machine Learning
Cloud Ops
Object Store
Admin UI in Control Plane
Oracle Support
Bug DB
SE UI in Support
Tenant (CNS)
Cleansing,
metadata creation
& clustering
5 Model generation
with expert scrubbing
6
Deployed as
part of cloud
image, running
from the start
1 Proactive regular health checking, real-
time fault detection, automatic
incident analysis, diagnostic collection
& masking of sensitive data
2
Use real-time health dashboards for anomaly
detection, root cause analysis & push of
proactive, preventative & corrective actions.
Auto bug search & auto bug & SR creation.
3
Auto SR analysis, diagnosis assistance via
automatic anomaly detection, collaboration
and one click bug creation
4
Message
Broker
21

DomU
Machine View
Alert
logs
Health
Data
Availability
Data
Performance
Data
Capacity
DataOracle Stack
Control Plane
Diagnostic
Collection
Object Store
TFA Service
TFA Agents detect issues & create
telemetry JSON
1
Uploads
telemetry to
Object Store
Telemetry
JSON
2TFA agent collects
diagnostics then
uploads to Object
store 3
TFA Service reads telemetry from Object
Store and pushes metrics to T2 and then
processed diagnostic collection
3
TFA /
EXAchk
Compliance
Data
22

SRDCs (Service Request Diagnostic Collection)
Oracle Grid Infrastructure
& Databases
TFAML
1
TFAML detects a
fault
2Diagnostics
are collected
3
Distributed diagnostics
are consolidated and
packaged
4
Notification of fault is sent
5 Diagnostic collection is
uploaded to Oracle
Storage Service for later
analysis
Object Store
23

Autonomous Database Health -
Anomaly Timeline
Remove clutter from log files to
find the most important events to
enable root cause analysis
24

Anomaly Detection – High Level
Known normal log entry (discard)
Probable anomalous Line (collect)
Log
Collection
File
Type
1
File
Type
2
File
Type
n..
Log File
Anomaly Timeline
Probable
Anomalies
25

Trace File Analyzer – High Level Anomaly Detection Flow
Log
Cleansing
1 2 3 4 5 6
Entry Feature
Creation
Entry
Clustering
Model
Generation
Expert
Input
Knowledge Base
Creation
Knowledge
Base Indexing
Feedback
Training
Real-time
Log File Processing
Timestamp Correlation & Ranking
8 9
7
Batch
Feedback
26

Knowledge
Base Indexing
Entry
Clustering
Model
Generation
Entry Feature
Creation
Log
Cleansing
1 2 3 4 5 6
Expert
Input
Knowledge Base
Creation
FeedbackTraining Real-time
Log File
Processing
Timestamp
Correlation &
Ranking
8 97 Batch
Feedback
Log File
Collection
Data
Cleansing &
Reduction
waited for 'ASM file metadata operation', seq_num: 29
2016-10-20 02:12:56.937 : OCRRAW:1: kgfo_kge2slos error
stack at kgfoAl06: ORA-29701: unable to connect to Cluster
Synchronization Service
waited for [STR] seq_num:
[NSTR]
[NSTR] [NSTR] : [NSTR] [NSTR]
unable to connect to Cluster
27

.. Seen
in
Bugs
Total
Bugs
Seen
Seen
in
Files
Total
Files
Seen
Total
Count
..
.. 13 40 1440 5088 2890 ..
Feature
Extraction
waited for [STR] seq_num: [NSTR]
[NSTR] [NSTR] : [NSTR] [NSTR]
unable to connect to Cluster
Knowledge
Base Indexing
Entry
Clustering
Model
Generation
Log
Cleansing
1 3 4 5 6
Expert
Input
Knowledge Base
Creation
Log File
Processing
Timestamp
Correlation &
Ranking
8 97 Batch
Feedback
Entry Feature
Creation
2
28

Data
Clustering
Record Merging and feature
aggregation for records belonging
to same log signature
Knowledge
Base Indexing
Entry
Clustering
Model
Generation
Log
Cleansing
1 3 4 5 6
Expert
Input
Knowledge Base
Creation
Log File
Processing
Timestamp
Correlation &
Ranking
8 97 Batch
Feedback
2
Entry Feature
Creation
29

Model
Generation
Data
Clustering
Expert
Input
Decision Tree
Classifier
First time labelling through
functional rules
Labelled dataset
Result
EvaluationUpdate
Labelling
3
4
5
76
Entry
Clustering
Log
Cleansing
1 3
Training Real-time
Log File
Processing
Timestamp
Correlation &
Ranking
8 9
Batch
Feedback
2
Knowledge
Base Indexing
Model
Generation
4 5 6
Expert
Input
Knowledge Base
Creation
Feedback
7
Entry Feature
Creation
30

Expert Input - Model Evaluation
• After expert input the model is automatically evaluated for:
– Accuracy
– Precision
– Recall
– F1 score (harmonic mean of precision and recall)
Classification
Model
Prediction
for entry
Expert
Review of
Entry
True Positive (TP) ü ü
True Negative (TN) X X
False Positive (FP) ü X
False Negative (FN) X ü
• Accuracy = (TP+TN) / (TP + FP + FN + TN)
• Precision = TP / (TP + FP)
• Recall = TP / (TP + FN)
• F1 Score = 2*(Recall * Precision) / (Recall + Precision)
31

Autonomous Database Health - Maintenance Slot
Identification
Find the next best window when
maintenance can be performed with
minimal service impact
32

Identification
• Identify Relevant Workload Metrics
– Ex: Average Active Sessions, CPU/Mem/IO Utilization
• Time Series Decomposition
– Trend
– Seasonality
– Residual
• Workload Seasonality Determination Locating Minimas
• Optimum Window Identification and Validation
Model Generation and Training Flow
33

START_TIME CNT
2018-04-11 15:00:00 290
2018-04-11 16:00:00 31120
2018-04-11 17:00:00 21530
2018-04-11 18:00:00 26240
2018-04-11 19:00:00 40520
2018-04-11 20:00:00 54270
2018-04-11 21:00:00 51460
2018-04-11 22:00:00 44310
2018-04-11 23:00:00 25690
START_TIME
2018-04-11 15:00:00 -0.226098
2018-04-11 16:00:00 -0.069821
2018-04-11 17:00:00 -0.350088
2018-04-11 18:00:00 -0.187483
2018-04-11 19:00:00 -0.513240
2018-04-11 20:00:00 0.019737
2018-04-11 21:00:00 0.059213
2018-04-11 22:00:00 -0.011312
2018-04-11 23:00:00 -0.179156
START_TIME
2018-04-11 15:00:00 5.669881
2018-04-11 16:00:00 10.345606
2018-04-11 17:00:00 9.977203
2018-04-11 18:00:00 10.175040
2018-04-11 19:00:00 10.609551
2018-04-11 20:00:00 10.901727
2018-04-11 21:00:00 10.848560
2018-04-11 22:00:00 10.698966
2018-04-11 23:00:00 10.153857
Current Date : 2018-05-12 15:00:00
Current Position in Seasonality : -0.22609829742533585
Best Maintenance Period in next Cycle : 2018-05-12 19:00:00
Worst Maintenance Period in next Cycle : 2018-05-13 08:00:00
Original observation data1 Apply convolution filter & average2 Calculate seasonality3
Use seasonality to
predict best
maintenance window
4
Identification
Seasonality Determination to Window Identification Flow
34

Identification
Validating Performance Against Random or Periodic Window Selection
35

Detect Metric Anomalies
Find combinations of unusual OS
metrics to enable root cause
analysis
36

Use of Z-Score
• Z-Score gives us a measurement of standard deviation from the mean
• Allows us to compare the relative “unusualness” of different types of
incomparable metrics like CPU usage vs IO waittime
• We multiple the Z-Score by a common factor, for ease of graphing and
zooming
37

Original metric values are
not comparable
Z-Score factored values are now comparable
Larger spikes show more unusual values
38

Identifying time periods with high z-score events across multiple metrics 42

Autonomous Health - Bug Duplicate Identification
Discovers Duplicate Bugs,
Correlated Issues and Prioritizes
Based Upon Customer Impact
43

Maintenance Slot Identification
44

BUG
DB
Adaptive Bug Search – Applied Machine Learning
• Bugs are submitted from over 400 Oracle
products
• Performs ML Logistic Regression on
training set of bugs to generate model
• Displays up to 8 possible duplicates per
bug or SR
• Feedback improves model accuracy
– Direct from developers
– Indirect from bug updates
Discovers Duplicate Bugs and Correlated Issues
ABS Dev TeamBugs
Bugs
DupBugs
ML Logistic
Regression
Model
Generation
Expert
Supervision
ABS
Runtime
Model
Dev
Feedback
Bug
Submission Bug and
Duplicates
Together
ABS
Service
Feedback
Scrub Data
45

Autonomous Database Health – Adaptive Bug Search (ABS)
• Issues parsed into different features
– Error stack, Trace data, Problem description, etc.
• Issues represented as a cluster of features
– i.e. All bugs in a bug tree contribute towards the feature set
• Logistic Regression applied to build a model
– Model defines the significance of each feature
• Similarity between issues computed using the model
– Identifies the root of the cluster (aka bug tree)
• Feedback used to improve the model
– Feedback is automatically derived based on how the bug gets closed
High Level Flow
46

Autonomous Database Health - Anomaly Analysis
Identify a series of events as
connected and representing the
signature of a problem
47

Longest Common Subsequence of Anomalous Entries
1. Start by classifying a problem such as an important
ORA or CRS error
2. Find occurrences of the problem across many different
log files
3. Identify anomalous entries and lifecycle events in
chronological order within a predefined time window
around the occurrence of the problem in all the logs
4. Compare the repeating anomalous / lifecycle entries
to identify the longest common subsequence of
anomalous entries
Find the Finite State Automata(FSA)
48

Example signatures and their analysis
Sample Central Event : 2017-01-19 16:51:20.562 [OCSSD(24862)]CRS-1656: The CSS daemon is terminating due
to a fatal error; Details at (:CSSSC00012:) in /tools/list/grid/orabase/diag/crs/ur102ora3502c/crs/trace/ocssd.trc
Knowledge Id Sample Line (States in FSA for central event)
52CC1E8631FC2674E053B580E80AB08D 2016-10-16 21:22:36.520+CRS-5008: Invalid attribute value: en4 for the network interface
52CC1E8632082674E053B580E80AB08D
2016-10-16 21:25:11.516 [OCSSD(6816354)]CRS-1608: This node was evicted by node 3, rwsbs03; details at (:CSSNM00005:) in
/u01/app/crsusr/diag/crs/rwsbs02/crs/trace/ocssd.trc.
52CC1E8632212674E053B580E80AB08D 2016-10-16 21:25:17.927 [OCSSD(18219406)]CRS-1654: Clean up of CRSD resources finished successfully.
52CC1E8631EC2674E053B580E80AB08D 2016-10-16 21:25:17.927 [OCSSD(18219406)]CRS-1655: CSSD on node rwsbs01 detected a problem and started to shutdown.
52CC1E8632272674E053B580E80AB08D
2016-10-16 21:25:19.431 [OCSSD(18219406)]CRS-8503: Oracle Clusterware process OCSSD with operating system process ID 18219406 experienced fatal
signal or exception code 6.
52CC1E8632202674E053B580E80AB08D
2016-10-16 21:25:21.788 [CRSD(44696012)]CRS-0805: Cluster Ready Service aborted due to failure to communicate with Cluster Synchronization Service with
error [3]. Details at (:CRSD00109:) in /u01/app/crsusr/diag/crs/rwsbs01/crs/trace/crsd.trc.
52CC1E86208C2674E053B580E80AB08D 2016-10-18 02:02:00.835 : CSSD:6684: (:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
52CC1E861F132674E053B580E80AB08D
CLSB:6684: Oracle Clusterware infrastructure error in OCSSD (OS PID 12452524): Fatal signal 6 has occurred in program ocssd thread 6684; nested signal count
is 1
52CC1E861E552674E053B580E80AB08D Incident 393 created, dump file: /u01/app/crsusr/diag/crs/rwsbs02/crs/incident/incdir_393/ocssd_i393.trc
52CC1E861F332674E053B580E80AB08D 2016-10-18 02:02:07.113 : SKGFD:5655: ERROR: -9(Error 27041, OS Error (IBM AIX RISC System/6000 Error: 47: Write-protected media
52CC1E86207C2674E053B580E80AB08D
2016-10-18 02:02:07.774 : CSSD:5655: clssnmvDiskCreate: Cluster guid ea34893b9442ef79ff642d70699aff9d found in voting disk /dev/rbs01_100G_asm1
does not match with the cluster guid 7b63590c34fa5f44bf6944aefa4ee85d obtained from the GPnP profile
52CC1E863DB82674E053B580E80AB08D
2017-01-19 16:48:01.057 [OCSSD(24862)]CRS-1649: An I/O error occurred for voting file: /dev/rdsk/c1d16; details at (:CSSNM00059:) in
/tools/list/grid/orabase/diag/crs/ur102ora3502c/crs/trace/ocssd.trc.
52CC1E863DBC2674E053B580E80AB08D
2017-01-19 16:49:40.550 [OCSSD(24862)]CRS-1615: No I/O has completed after 50% of the maximum interval. Voting file /dev/rdsk/c1d16 will be considered
not functional in 99508 milliseconds
49

Example signatures and their analysis
5 Mins
before
Central
Event
5 Minute
After
Central
Event
Central
Event
52CC1E863
1FC2674E0
53B580E80
AB08D
52CC1E862
07C2674E0
53B580E80
AB08D
52CC1E861
F332674E0
53B580E80
AB08D
52CC1E861
E552674E0
53B580E80
AB08D
52CC1E861
F132674E0
53B580E80
AB08D
52CC1E862
08C2674E0
53B580E80
AB08D
52CC1E863
2202674E0
53B580E80
AB08D
52CC1E863
2272674E0
53B580E80
AB08D
52CC1E863
1EC2674E0
53B580E80
AB08D
52CC1E863
2212674E0
53B580E80
AB08D
52CC1E863
2082674E0
53B580E80
AB08D
52CC1E863
DBC2674E0
53B580E80
AB08D
52CC1E863
DB82674E0
53B580E80
AB08D
52CC1E867
22C2674E0
53B580E80
AB08D
50

Autonomous Database Health - Anomaly Analysis
Generating Event Signatures
Event Signature 35
Event Signature
3435
Event Signature
494
Event Signature
3948
Event Signature
292
Event Signature
434933
Node Eviction 1
Timeline
Event Signature
3434
Event Signature
3435
Event Signature
4344
Event Signature
3048
Event Signature
202
Event Signature
434983
Node Eviction 2
Timeline
Event Signature 35
Event Signature
3435
Event Signature
3048
Event Signature
3948
Event Signature
292
Event Signature
434933
New Signature
Check for weighted
probabilistic match
ProblemSignatureRepository 51

Autonomous Database Health - Database Performance
Preserving instance performance
when database resources are
constrained
52

Database Data Flow Overview
Autonomous Database Health – Database Performance
OS DataDB Data
Database Prognostics Engine
Alert &
Preventive
Action
• Reads OS and DB Performance data
directly from memory
• Uses Machine Learning models and data
to perform prognostics
• Detects common RAC database problems
• Performs root cause analysis
• Sends alerts and preventative actions to
Cloud Ops per target
53

Data Sources and Data Points
Time CPU ASM
IOPS
Network
% util
Network_
Packets
Dropped
Log
file
sync
Log file
parallel
write
GC CR
request
GC current
request
GC current
block 2-way
GC current
block busy
Enq: CF
-
conten
tion
…
15:16:00 0.90 4100 13% 0 2 ms 600 us 0 0 300 us 1.5 ms 0
A Data Point contains > 150 signals (statistics and events) from multiple sources
OS, ASM , Network DB ( SH, AWR session, system and PDB statistics )
Statistics are collected at a 1 second internal sampling rate , synchronized,
smoothed and aggregated to a Data Point every 5 seconds
54

Data Flow Overview
55

Inline and Immediate Fault Detection and Diagnostic Inference
Machine Learning,
Pattern Recognition,
& BN Engines
Time CPU ASM
IOPS
Network
% util
Network_
Packets
Dropped
Log
file
sync
Log file
parallel
write
GC CR
request
GC current
request
GC current
block 2-way
GC current
block busy
Enq: CF
-
conten
tion
…
15:16:00 0.90 4100 88% 105 2 ms 600 us 504 ms 513 ms 2 ms 5.9 ms 0
15:16:00
OK OK HIGH
1
HIGH
2
OK OK HIGH
3
HIGH
3
HIGH
4
HIGH
4
OK
Input : Data Point at Time t
Fault Detection and Classification
Diagnostic Inference
15:16:00
Symptoms
1. Network Bandwidth Utilization
2. Network Packet Loss
3. Global Cache Requests Incomplete
4. Global Cache Message Latency
Root Cause
(Target of Corrective Action)
Network Bandwidth Utilization
Diagnostic
Inference
Engine
56

Real-time Prevention
• Data Ingestion
– Kernel Smoothing and Moving Average
– Interpolation and Imputation
• Prediction and Pattern Recognition
– Multivariate and Auto-Associative Regression
– Clustering, Similarity Operators and Bayes Networks
• Fault and Anomaly Detection
– Sequential Probability Ratio Tests
– Conditional Probability Filters & Hidden Markov
Models
• Prognosis and Diagnosis
– Bayesian Belief Networks and Probabilistic Inference
– Remaining Useful Life Regression and GPM Models
Rapid Recovery
Autonomous Database Health Platform ML Technologies
• Data Ingestion
– ELK
– Lucene
• Prediction and Pattern Recognition
– TF-IDF and Bag-of-Words modelling
– Sequence Matcher
– K-nearest Neighbour
• Fault and Anomaly Detection
– Decision Trees and Random Forest
– Sequential Pattern Mining
• Prognosis and Diagnosis
– Recurrent neural Network
– Long short-term memory Predictive Analysis
57

Autonomous Database Health - Database Performance
Workload Determination and
deviation and when to scale the
load or look for problems
58

What is Workload
Automatically
check
workload for
past x mins
Decide if
workload is
abnormally
high
Highlight any
abnormal
workload
issues
Optionally run on demand
Optionally snooze checking
of a component
Calculated via machine learning
59

Adaptive Learning
Workload Process
Captures metrics for key performance
dimensions across 5 X 1 minute time
windows
CAPTURE1
Using semi-supervised learning via SME
threshold rules, the following models are
retrained :
• Isolation Forest
• One-Class Support Vector Machine
• Local Outlier Factor
Model with highest confidence becomes
the primary, if confidence is high enough
TRAIN2
Straight after capture, the primary model is
used to predict anomalies.
Where anomalies are identified, metrics are
compared to SME threshold rules to identify
the type of anomaly
PREDICT3
Every
5 Mins
Every
Week
Every
5 Mins
60

Capture
• Initial one-time setup defines configuration for scope of CDBs,
PDBs & Services
• Every 5 minutes capture metrics for key performance
dimensions:
• Other performance related dimensions can be used in the future
• Capture gets ASH data for later analysis
61

Train
• The following models are retrained to identify anomalies in the metrics
1. Isolation Forest
2. One-Class Support Vector Machine
3. Local Outlier Factor
• Each model is evaluated using 5 test accuracy scores
• Model with the highest confidence becomes the primary
and is used for prediction until next training iteration,
as long as confidence is > 95%
• Testing has shown minimum of 7 days data collection is required
• Maintain a rolling window of 30 days of data to account for seasonality
within a month & provide better predictability
63

Train the model using
normal workload data
Model determines how to
classify normal
observations based on the
combination of
performance metrics
across key dimensions
New observations can be
classified as anomalies if
combination of the metrics
fall out of normal
classification
One-Class Support Vector Machine
1 32
68

Local Outlier Factor
Anomalous data points are
further away from the center
of all data points &
more isolated than the other
data points
The distance between a
single data point and it’s
closest neighbours can be
measured
Anomalous data points will
have greater distance to their
closest neighbours than
other data points
Data points that have
significantly greater distances
than other data points can be
identified as anomalous
1 2
43
69

Semi-Supervised Learning
• SME (Subject Mater Expert) anomaly threshold
rules are shipped with the product
• Models are trained in loco by evaluating their predictions
of real world data against the SME rules
• Design allows rules used during retraining to be tuned for
sensitivity based on production feedback
• Models are compared to determine the primary
– Only models with >95% confidence are considered
– The model with the highest confidence is declared the primary
70

Prediction (Every 5 minutes)
5 X 1 min metrics captured for
each dimension & ASH report
captured for later analysis
Metrics evaluated by the primary model to
determine if there are anomalies
If there is no primary model
(i.e. <7 days of data or <=95% model confidence)
then SME rules are used for anomaly detection
Each anomaly is compared against
the SME rules to determine which
dimension it applies to
Any anomalies are
raised along with
recently captured ASH
report
72

Key Feature Values
Feature Value
Number of performance metric dimensions 3
Granularity of time window sampling 1 minute
Frequency of data capture & prediction 5 minutes
Initial data required for ML model usage 7 days
Confidence threshold for model to be used 95%
Rolling window of data used for ML model retraining 30 days
73

• An expert system that implements
indexes based on what a
performance engineer skilled in index
tuning would do
• It identifies candidate indexes and
validates them before implementing
• The entire process is full automatic
• Transparency is equally important as
sophisticated automation
– All tuning activities are auditable via
reporting
Identify the best indexes
Capture
Identify
VerifyDecide
Monitor
75

Conclusions
• ML is here to stay and is just getting started
• The last 2.5 years of advances in this field dwarfs the previous 50 years of
growth
• We need to identify use cases to make the business better
• Modeling and ML infrastructure will become standard aka AutoML
• Getting the right data to train matters to have a successful outcome
• Models will get better with sparse data
• Most enterprise applications are already using embedded ML
76

AUSOUG - NZOUG-GroundBreakers-Jun 2019 - AI and Machine Learning

More Related Content

What's hot (20)

Similar to AUSOUG - NZOUG-GroundBreakers-Jun 2019 - AI and Machine Learning (20)

More from Sandesh Rao (20)

Recently uploaded (20)

AUSOUG - NZOUG-GroundBreakers-Jun 2019 - AI and Machine Learning