Anomaly Detection - Discover unknown Frauds and Anomalies using Machine Learning
2 likes1,746 views
The document presents various use cases for anomaly detection using machine learning, including fraud detection and predictive maintenance. It discusses a specific dataset of credit card transactions and describes the application of autoencoders for detecting anomalies, as well as techniques for managing time series data from sensors in predictive maintenance. Key findings and methodologies are outlined, including model training and deployment via KNIME server.
![© 2019 KNIME AG. All Rights Reserved.
The Data
• 28 time series from 28 sensors attached to 8 different part of a rotor
• Time Series are FFT-derived Spectral Amplitudes
[date, time, FFT frequency, FFT amplitude]
• After Preprocessing: 313 time series for different frequency bands (100
Hz-wide) falling between 0 Hz and 1200 Hz and different sensors
• One breakdown, which is visible only from some sensors and only in
some frequency bands
16](https://image.slidesharecdn.com/frauddetection-190410055909/85/Anomaly-Detection-Discover-unknown-Frauds-and-Anomalies-using-Machine-Learning-12-320.jpg)
![© 2019 KNIME AG. All Rights Reserved.
A1-SV3 [500, 600] Hz
New motor piece
Old motor piece
Breaking point
July 21, 2008
Only some Spectral Time Series shows the break down
A1-SV3 [0, 100] Hz
Data Visualization: Time Plots by Frequency Bands
18
White paper: https://files.knime.com/sites/default/files/inline-images/knime_anomaly_detection_visualization.pdf
Community Hub: https://hub.knime.com/knime/workflows/*Ra9zhH3q0zKo-tUu](https://image.slidesharecdn.com/frauddetection-190410055909/85/Anomaly-Detection-Discover-unknown-Frauds-and-Anomalies-using-Machine-Learning-13-320.jpg)
![© 2019 KNIME AG. All Rights Reserved.
A1-SV3 [0, 100] Hz
A1-SV3 [500, 600] Hz
Breaking point
July 21, 2008
31 August 2007
Training Set
Predictive Maintenance
Learn “normal”: Training Set
21
Idea:
1. Train Auto-Regressive (AR)
Model for each time series
on “ normal” data
2. Apply model and calculate
distance between predicted
and real values
3. Define alarm levels based
on distance statistics](https://image.slidesharecdn.com/frauddetection-190410055909/85/Anomaly-Detection-Discover-unknown-Frauds-and-Anomalies-using-Machine-Learning-15-320.jpg)
Anomaly Detection - Discover unknown Frauds and Anomalies using Machine Learning
- 1. © 2019 KNIME AG. All Right Reserved. Discover Unknown Frauds and Anomalies using Machine Learning Kathrin Melcher kathrin.melcher@knime.com
- 2. © 2019 KNIME AG. All Rights Reserved. Anomaly Detection: Use Cases 3 Fault Detection Fraud Detection Predictive Maintenance Intrusion Medicine Heart Beat Sensor Data AssemblingDetails Transactions Networks Finance IoT Weather Information Fraud Detection Predictive Maintenance System Health Monitoring
- 3. © 2019 KNIME AG. All Rights Reserved. Anomaly Detection What have all those use cases in common? • Discover rare events that shouldn’t happen => often no labeled data • Find a problem before other people see it => anomaly is unknown Example with 2 dimensions, e.g. vibration and heat of an engine: 4 Historical data of normal New data samples How can we detect the anomalies?
- 4. © 2019 KNIME AG. All Rights Reserved. Use Case 1: Fraud Detection 6
- 5. © 2019 KNIME AG. All Rights Reserved. The Dataset • Kaggle Dataset https://www.kaggle.com/mlg-ulb/creditcardfraud • 284 807 credit card transactions performed in September 2013 by European cardholders • 492 (0.2 %) transactions in the dataset are fraudulent • Features: – 28 principal components – Time from the first transaction – Amount of money 7
- 6. © 2019 KNIME AG. All Rights Reserved. The KNIME Community Workflow Hub 8 https://hub.knime.com
- 7. © 2019 KNIME AG. All Rights Reserved. Fraud Detection with Labeled Data Model Training Model Deployment
- 8. © 2019 KNIME AG. All Rights Reserved. Fraud Detection using Autoencoder 11 Input Layer Hidden Layers Output Layer Input ! Output !‘ !#$% − !#$% ' ( > * ⇒ anomaly Each layer: 23(!356) = 9(: !356 ) = 9(;356 !356)) f is a non-linear activation function, e.g. tanh, relu. min ;= 3∈(6,(,@,A) B(!, !') with B !, !’ = 1 E F GH6 I ! − !' ( Execution of the Network: The network structure on the left: ! ∈ ℝK ℎ6 ! = 96 ;6! ∈ ℝ@ ℎ( ! = 9((;(26) ∈ ℝ( ℎ@ ! = 9@ ;@2( ∈ ℝ@ !′ = 9A ;A2@ ∈ ℝK Training of the network:
- 9. © 2019 KNIME AG. All Rights Reserved. Fraud Detection using Autoencoder 13
- 10. © 2019 KNIME AG. All Rights Reserved. Deployment via REST on KNIME Server 14 Workflow deployed as (REST) web service on KNIME Server Workflow calling another workflow on KNIME Server
- 11. © 2019 KNIME AG. All Rights Reserved. Use Case II Anomaly Detection in Predictive Maintenance (IoT) 15
- 12. © 2019 KNIME AG. All Rights Reserved. The Data • 28 time series from 28 sensors attached to 8 different part of a rotor • Time Series are FFT-derived Spectral Amplitudes [date, time, FFT frequency, FFT amplitude] • After Preprocessing: 313 time series for different frequency bands (100 Hz-wide) falling between 0 Hz and 1200 Hz and different sensors • One breakdown, which is visible only from some sensors and only in some frequency bands 16
- 13. © 2019 KNIME AG. All Rights Reserved. A1-SV3 [500, 600] Hz New motor piece Old motor piece Breaking point July 21, 2008 Only some Spectral Time Series shows the break down A1-SV3 [0, 100] Hz Data Visualization: Time Plots by Frequency Bands 18 White paper: https://files.knime.com/sites/default/files/inline-images/knime_anomaly_detection_visualization.pdf Community Hub: https://hub.knime.com/knime/workflows/*Ra9zhH3q0zKo-tUu
- 14. © 2019 KNIME AG. All Rights Reserved. Control Chart (Rule Based) • Define signal boundaries based on anomaly-free time windows !"# = %&' + 2 ∗ +,--.& /.%+01. > !"# ⇒ 4.&.4 1 %4%1/ #"# = %&' − 2 ∗ +,--.& /.%+01. < #"# ⇒ 4.&.4 1 %4%1/ • Sequence of level 1 alarms across many time series => level 2 alarm 19 White paper: https://files.knime.com/sites/default/files/181212_Whitepaper_Anomaly_Detection_Predictive_Maintenance_KNIME.pdf Community Hub: https://hub.knime.com/knime/workflows/*OwayKRE08PXVWiqx
- 15. © 2019 KNIME AG. All Rights Reserved. A1-SV3 [0, 100] Hz A1-SV3 [500, 600] Hz Breaking point July 21, 2008 31 August 2007 Training Set Predictive Maintenance Learn “normal”: Training Set 21 Idea: 1. Train Auto-Regressive (AR) Model for each time series on “ normal” data 2. Apply model and calculate distance between predicted and real values 3. Define alarm levels based on distance statistics
- 16. © 2019 KNIME AG. All Rights Reserved. Training of an Auto-Regressive Model 22 Community Hub: https://hub.knime.com/knime/workflows/*OwayKRE08PXVWiqx
- 17. © 2019 KNIME AG. All Rights Reserved. Deployment of an Auto-Regressive Model 23 Community Hub: https://hub.knime.com/knime/workflows/*OwayKRE08PXVWiqx
- 18. © 2019 KNIME AG. All Rights Reserved. Time Series Production: Use KNIME to Send Alarms !!! 25
- 19. © 2019 KNIME AG. All Rights Reserved. Free Copy Practicing Data Science 26 • 2nd Edition: 22 Case Studies! • Available at KNIME Press: https://www.knime.com/knimepress • Select book “Practicing Data Science” • For free copy, use promotion code: MEETUP-VIENNA-PDS
- 20. © 2019 KNIME AG. All Rights Reserved. The KNIME® trademark and logo and OPEN FOR INNOVATION® trademark are used by KNIME AG under license from KNIME GmbH, and are registered in the United States. KNIME® is also registered in Germany. 27