Sap Hana and Virtustream for Predictive Maintenance and Big Data
Download as PPTX, PDF1 like193 views
This document discusses predictive maintenance solutions using EMC and Virtustream technologies. It describes how raw sensor data is acquired from sources like railway sensors and processed. This involves data preparation, model training and testing. The data is then stored in various types and locations depending on how often it needs to be accessed. This includes hot/real-time data stored in SAP HANA, warm historical data stored in SAP IQ, and cold archived data stored in Hadoop. Combining SAP, EMC storage, and Virtustream cloud services can provide solutions for predictive analytics use cases involving large volumes of both structured and unstructured data.
Sap Hana and Virtustream for Predictive Maintenance and Big Data
- 1. © Copyright 2015 EMC Corporation. All rights reserved. 1© Copyright 2015 EMC Corporation. All rights reserved. PREDICTIVE MAINTENANCE EMC/VIRTUSTREAM SOLUTION RICCARDO ROMANI
- 2. © Copyright 2015 EMC Corporation. All rights reserved. PREDICTIVE MAINTENANCE PROCESS 1. Data acquisition and processing: A. Data acquisition from raw data sources ( i.e. railways sensors ) B. Long times for data preparation and transformation into data set be used as structured data model and historical data base C. Long times for model training and testing iterations D. Some batch calculations required 1. Data Storage : Huge amount of data generated at both acquisition as well as historical stages MOST COMMON ISSUES
- 3. © Copyright 2015 EMC Corporation. All rights reserved. DIFFERENT TYPES OF DATA IN PREDICTIVE ANALYTICS FROM RAW DATA SOURCE… • Example of Data Sets created starting from the raw data • Structured Data • Training data: It is the engine run-to-failure data. • Testing data: It is the engine operating data without failure events recorded. • Ground truth data: It contains the information of true remaining cycles for each engine in the testing data. • Predictive data used to predict when an in-service machine will fail, so that maintenance can be planned in advance. • Responds to the question :” Given these aircraft engine operation and failure events history, can we predict when an in-service engine will fail?” • Regression: Predict the Remaining Useful Life (RUL), or Time to Failure (TTF). • Example of raw data gathered from a Stream Processing System. • Millions of txt files • Unstructured/semi-structured data …TO DATASET
- 4. © Copyright 2015 EMC Corporation. All rights reserved. SAP HANA Sybase IQ Sybase IQ Hot Data Warm Data Cold Data • Analytics run on Data Model − Modern in-memory platform − Transact/analyze in real- time − Native predictive, text, and spatial algorithms • Warm Historical Data Model − Disk backed, smart column store ( HANA on disk or IQ ) − It helps in offloading HANA from huge amount of data that are dinamically stored on disk instead of in-memory − Excels at queries on structured data from terabyte to petabyte scale • Cold Historical Data Model − Less frequently accessed data is archived in time partitions on IQ − The data is static and used primarily for read access − data resides in cost- efficient storage with fewer backups to reduce operational costs − Lower SLA requirements HOW COMBINING SAP AND EMC/VIRTUSTREAM CAN HELP HADOOP Raw data • Data acquired from Sensors − New type of User Defined Function for data federation − Direct access to HDFS without need for the package, mapper, and reducer specification − Invoke custom Map Reduce jobs Solution: EMC with Isilon, Hadoop-native storage combined with Virtustream Cloud Storage (EMC ECS Object Storage ) DIFFERENT TYPES OF DATA IN PREDICTIVE ANALYTICS Solution : Virtustream with SAP HANA Cloud IaaS/PaaS
- 5. © Copyright 2015 EMC Corporation. All rights reserved. SCENARIO #1 SAP HANA, SAP IQ @ VIRTUSTREAM , DATA ACQUISITION ON ISILON @ HYPERCED HISTORIC/COLD Gateway Rolling Stock Raw data RAW DATA 1. Raw Data acquisition done at gateway level– serialization could be a bottleneck as well as raw data history growth 2. Predictive model computation done at Hot Data level, in Hana 3. Warm data on SAP IQ 4. Cold data archived ( could be SAP IQ ) for compliance and outlier data from faulty sensors and for statistical purposes • EMC can control rolling stock data growth during acquisition with Isilon storage certified by SAP and sitting at gateway layer HISTORIC/ WARM HOT DATA Datacenter Almaviva “Hyperced” Datacenter
- 6. © Copyright 2015 EMC Corporation. All rights reserved. SCENARIO #2 ADDING HADOOP COMBINED WITH ISILON @ HYPERCED 1. Raw Data acquistion done in parallel streams. Hadoop speed up parsing, and creation of reduced dataset to be loaded and historicized 2. Predictive model computation can be done accessing a wider dataset accessing Warm data on IQ and Hadoop 3. Cold data stored on hadoop filesystem as “intelligent” archiving for statistical analysis and retrieval • EMC can control and improve Data growth management at both Hadoop and Storage level − Hadoop for unlimited capacity for raw data processing − Apache Kafka or Storm for distributed stream processing Gateway/Data Lake Rolling Stock Raw data INTELLIGENT ARCHIVING FOR RAW DATA AND HSISTORIC HISTORIC/ WARM HOT DATA Almaviva “Hyperced” Datacenter
- 7. © Copyright 2015 EMC Corporation. All rights reserved. HADOOP AND SAP 1/3 HADOOP IN ACTION @ SAP PREDICTIVE MAINTENANCE BLUEPRINTS Where SAP and EMC combined technologies fits in a predictive maintenance project
- 8. © Copyright 2015 EMC Corporation. All rights reserved. HADOOP AND SAP 2/3 Data acquisition • Real time events streams coming in at a rate of thousands of raw events per seconds. • The stream processing system should be able to process those events in a fault tolerant and distributed manner and with parallel processing • Streaming processing systems should also keep record of old data for some reasonable amount of time before they are archived or destroyed in order to : A. build pattern recognition and statistical model building methods B. Cope with local country laws. C. to identify any potential outliers in the streamed-in data from the sensors. While monitoring for the faults in the assets it is possible that the sensor that is taking the readings, being a machine itself, could fail and start sending faulty records. Intelligent CBM management systems capable of detecting such outliers will try to isolate these faulty sensors and notify We have evaluated two popular open source technologies; Apache Kafka , which is the distributed messaging system, and Storm which is a distributed stream processing engine. Both having HADOOP as repository for managing data growth. SAP provides a product called “ESP – Event Stream Processor” that integrates HANA ad Hadoop HADOOP IN ACTION @ SAP PREDICTIVE MAINTENANCE BLUEPRINTS
- 9. © Copyright 2015 EMC Corporation. All rights reserved. Build, train and validate the model – creating Data Sets • To detect failure in a given stream of sensor data, we need to first define normal behavior. • For this we need to build model around the historical sensor data. • Predictive models analyze current and historical data on individuals to produce metrics. • A model is reusable and is created by training an algorithm using historical data and saving the model for reuse purpose to share the common business rules which can be applied to similar data, in order to analyze results without the historical data, by using the trained algorithm • The process involve running one or more algorithms on the data set where prediction is going to be carried out. This is an iterative processing and often involves training the model, using multiple models on the same data set and finally arriving on the best fit model based on the business data understanding. • Raw data, once prepared, are organized in Data Sets, needed to be stored to be used by different type of models and algorithms HADOOP AND SAP 3/3 HADOOP IN ACTION @ SAP PREDICTIVE MAINTENANCE BLUEPRINTS
- 10. © Copyright 2015 EMC Corporation. All rights reserved. VIRTUSTREAM SAP HANA DT ALMAVIVA HYPERCED Servers & Network DATA STORAGE powered by EMC HDFS or NFS EMC ISILON , VNX FibreChannel or NFS EMC VMAX, VNX or XtremIO EMC/VIRTUSTREAM HYBRID ARCHITECTURE HOT Extended Storage IQ NLS Backup / Archive Best in class Cloud IaaS/PaaS for SAP workloads , through Almaviva White Label Option EMC Scale Out Unstructured Storage with de-duplication features and native HDFS support From 15% to 35 % of data storage efficiency depending on data specific nature GATEWAY Unstructured Storage Raw Data
- 11. © Copyright 2015 EMC Corporation. All rights reserved.© Copyright 2015 EMC Corporation. All rights reserved.
- 12. © Copyright 2015 EMC Corporation. All rights reserved. 1. Acquire OT data in parallel streams 2. Clean data and import only the validated data into SAP HANA 3. Historic OT data in IQ and/or Hadoop 4. Use Hive to build a relational/historical view of the OT data also at Hadoop level 5. Unify OT view with IT data, if needed 6. Save HANA resources for realtime predictions only 7. De-duplicate and store raw OT data for future use or statistic purposes Gateway/Data Lake HADOOP AND SAP PDMS BLUEPRINT
- 13. © Copyright 2015 EMC Corporation. All rights reserved. HADOOP AND SAP GATHERING REAL-TIME EVENTS AT SCALE