Operationalizing the IoT with Mtell and Apache Spark
0 likes453 views
The document discusses operationalizing Apache Spark for the Internet of Things (IoT), emphasizing its role in big data analytics and maintenance through advanced analytics techniques. It highlights the integration of sensor data, predictive models, and automated failure detection agents that enhance machine reliability and reduce maintenance costs. Additionally, it outlines the use of resilient distributed datasets (RDDs) to optimize data processing and improve decision-making in industrial applications.
Operationalizing the IoT with Mtell and Apache Spark
- 1. Bigdataeverywhere 2016 Alex Bates, CTO, Mtell Operationalizing Apache Spark for the IoT
- 3. image purchased from Alamy, NY for use in commercial presentation
- 6. The Collection of: Sensors Sensor Networks Smart Machines Computer Power Analytics People One element of the Industrial Internet of Things Solving problems that were previously unsolvable TheInternetofThingsTheInternetofThings
- 7. Network of physical objects or “things” embedded with electronics, software, sensors, and network connectivity, which enables these objects to collect and exchange data. 4 BILLION Connected People 25+ MILLION Apps 25+ BILLION Devices 50 TRILLION GBs of DATA
- 8. Use Cases remote monitoring SME’s manage many machines condition monitoring detect early when problems are small share learning learn on one transfer to many
- 9. maintenance sensor streams leverages existing infrastructure: plant historian sensor data streams, and EAM system connect import combine analyze present prescriptive action Information Flow
- 10. Library Benchmark Statistics Equipment Metadata Raw Data Failure Signatures, PM Repository, Failure Code Hierarchy Failure Rates, MTBF/MTTF/MTBR/MTBM, Average Repair Cost, Performance Data Equipment Type, ISO 14224 Structure, Sensor Templates Sensor Data, Maintenance Work Orders, Operational Events, Crowdsourced Info Levels of Data
- 11. global equipment knowledge base Machine Genome Project
- 12. WAN Evolution to Tier 2
- 13. IoT and Big Data Comet 67P vs Los Angeles Scale Data Points/Yr Single Rig 315 B 100 rigs 31.5 T Scale Data Points/Yr
- 14. Mtell and OpenTSDB Optimized for Time Series Data Access Querying sensor data by date range plus other filters
- 15. MapR OpenTSDB Optimization • Blob Ingestion – 100x faster – Instead of inserting each point, buffer data in memory and insert a blob containing batch – Move blob maker upstream of insertion into the storage tier – Less writes to disk (once / blob instead of once/point) and reduced data size (blob compresses raw data) – 10 node MapR cluster achieved 100 million points/sec ingestion (10 million points/sec/node)
- 16. What has happened? What will happen? What should we do? DESCRIPTIVE PREDICTIVE PRESCRIPTIVE Analytics in Maintenance
- 18. Types of Agents Learns precise specific failure signature & performs live monitoring, providing early warnings of recurrences Failure Agent Learns baseline normal & performs live monitoring to expose abnormal operations – updates as conditions change Anomaly Agent Finds unrecorded failure patterns in training data & excludes suspect data from baseline normal conditions Hidden Failure Agent
- 19. anomaly agent knows all learned patterns matching all normal operating states failure agent 001 knows precise signature of patterns leading to bearing failure failure agent 002 knows precise signature of patterns leading to drive coupling failure failure agents 003+ many other agents each assigned to detect exact failure patterns Many Agents Per Asset
- 20. …each one holds the precise multi- dimensional/temporal data pattern of a machine in a specific operating mode capture worker experiences & actual measured sensor data Mtell uses agents for individual machines created by you …in minutes one single job …constantly monitor for that exact pattern & sound off “alarm bells” Agents – Retained Knowledge
- 21. Find Degradation Earlier Multi-variate Temporal & multi-variate Detect complex failure patterns that cannot be detected by humans, or other technologies, or seen in any single variable trend
- 22. Platform for IoT Analytics Equipment Sets & Taxonomy make & model operating context population analysis Equipment asset hierarchy sync from EAM Sensor Mapping data streams Live Agents rules maintenance scheduling machine learning M2M population learning transfer learning Performance usage, states wear, fatigue efficacy benchmarking
- 23. advanced analytics analyst automation population learning deep learning fleet bench- marking reservoir signature library sensor data store cloud sync fault/eff. signatures signature search operating center mgmt. incident response immersive visualization adaptive feedback knowledge capture intelligent signal processing audit automation instrument reliability derived signal mgt. interpola- tion global equipment taxonomy eqmt model catalog Industry op. context fleet sensor templates sensor groups Platform Functions
- 24. Transfer Learning Signature Library Template Signature Pump 02 Pump 01 Library of Known Failure Signatures Time-Series Sensor Data A B C A B C
- 26. Mtell and Spark RDD – Resilient Distributed DataSet OpenTSDB
- 27. Spark RDD RDD – Resilient Distributed DataSet • Read-only collection partitioned across a set of machines • Can be rebuilt if partition lost • Enables spark to outperform Hadoop 10x on iterative machine learning jobs
- 28. Query data via HTTP Over time builds RDD Data-frame; distributed across nodes Aim: query database only once Distributed data storage system Stores high-precision data points Scales almost linearly But lacks analytics Mtell REST API Any request Any client Flexible/scalable link to any Python-based machine learning libraries Human friendly Spark Integration
- 29. Automated & Self-Improving Previously Learned Normal
- 30. Automated & Self-Improving Known Failure Signature prescriptive maintenance well in advance fixing a small problem before it’s a big one
- 31. Automated & Self-Improving Learn New Operating State
- 32. Automated & Self-Improving Learn New Failure Signature search deeper & improve 7-day anomaly alert 30+ day failure signature alert
- 33. Predictive / Prescriptive Analytics Maintenance costs decrease dramatically Machines last longer Net output increases dramatically Critical Assets stop breaking down