The key to unlocking the Value in the IoT? Managing the Data!
Download as PPTX, PDF2 likes661 views
The document discusses how managing data is key to unlocking value from the Internet of Things. It emphasizes that variety, not size, is most important with big data. Example use cases mentioned include predictive maintenance, search and root cause analysis. The technology landscape is changing with new architectures like data lakes and new patterns such as event histories and timelines. Managing data is also changing with schema on read, loosely coupled schemas, and increased importance of metadata. The document concludes that data management patterns and practices are foundational to effective analytics with IoT data.
![12
Actor id Segment
s
Ev1:id Ev1:fact Ev1:dims Ev1:ext Ev2:id …
uid1 [1, 3, 7] 123 1/1/15
13:16:11
… { “TstA” : 1
…}
789 …
uid2 [2, 3] 456 1/1/15
13:16:14
… { “TstB” : 1
…}
0ab …
• Pivot on event history: table of actors with events over time
– Device history, usage in consumer journey
– Enable support/analysis on specific items, long-lived analysis
• May have hierarchy of actors (e.g., cluster, device, component)
• May be array of events, many columns or subsorted (cluster key)
• Also stored in columnar format, may be partitioned
• May be updated in near real-time AND batch
• Often holds cached algorithm values (combined Profile)
Timeline Pattern
4/25/2016© 2015 Think Big, a Teradata Company](https://image.slidesharecdn.com/april131750thinkbigbodkinv2-160425200302/85/The-key-to-unlocking-the-Value-in-the-IoT-Managing-the-Data-12-320.jpg)
The key to unlocking the Value in the IoT? Managing the Data!
- 1. MAKING BIG DATA COME ALIVE The key to unlocking the Value in the Internet of Things? Managing the Data!
- 2. 2 For Big Data the Key is Variety! 4/25/2016© 2015 Think Big, a Teradata Company Definition: Datasets so complex and large that they are awkward to work with using standard tools and techniques Location Social Images Weblogs Videos Text Audio Sensor Size is not what is most important; it’s variety
- 3. 3 Example Use Cases • Predictive Maintenance • Search and view detail on issue on the fly • Identify critical alerts • Root cause analysis • Understanding usage • And many more!
- 5. 5 © 2015 Teradata AccessPreparationAcquisition Data Lake Architecture Math and Stats Data Mining Business Intelligence Applications Languages Marketing ANALYTIC TOOLS & APPS USERS Marketing Executives Operational Systems Frontline Workers Customers Partners Engineers Data Scientists Business Analysts Streams SearchAggregations Security, Metadata/Lineage, Administration Distributed Storage Msg. queues Cleansing Access ExperimentsGovernanceFeeds SOURCES Sensors email Social Telemetry Mobile Tabular Data Machine logs C
- 6. 6 © 2015 Teradata REFERENCE INFORMATION ARCHITECTURE New with Big Data Security, Workload ManagementPublishingPreparation SecuredLanding Acquisition SharedViews&Obfuscation OptimizedStructures CommonKeys DerivedValues,SensitiveDataProtection CommonSummaries UserDefinedDataSets Validation&KeyResolution ERP SCM CRM Images Audio and Video Machine Logs Text Web and Social SOURCES Business Analysts Math and Stats Data Mining Business Intelligence Applications Marketing ANALYTIC TOOLS & APPS Search Profiling,Masking,Obfuscation Data Scientists Business Analysts Data Modelers IT
- 7. 7 How is Data Management Changing? • Schema on Read? – Yes… as step one – But data still has underlying structure – It’s more like agile modeling – reflect as much structure as needed • Loosely coupled schemas loses platform guarantees but gains more application flexibility • Data Modeling isn’t dead! • Metadata is more important than ever 4/25/2016© 2015 Think Big, a Teradata Company
- 8. 8 Changes in Logical Modeling • JSON-like structures – Complex collections of relations, arrays, map of items • Graphs – Storing complex, dynamically changing not static relationships • Binary/CLOB/specialized data – Ability to execute specialized programs to interpret and process 4/25/2016© 2015 Think Big, a Teradata Company
- 10. 10 Important New Patterns • Denormalized Fact • Profile • Event History • Timeline • Network • Distributed Sources • Late Data • Deep Aggregates • Recovery • Multiple Active Clusters 4/25/2016© 2015 Think Big, a Teradata Company
- 11. 11 Event id Actor id Time Event col’s Dim id’s Dim col’s Ext. Data 123 uid1 1/1/15 13:16:11 … … … { “TstA” : 1 …} 456 uid2 1/1/15 13:16:14 … … … { “TstB” : 1 …} • Fact table about common events to allow e.g., analytics in context – E.g., wearable device, telematics • Stored in columnar format (e.g., Parquet, ORCfile) • Join as was value of slowly changing dimensions • Often “extension” column of unparsed/not modeled JSON-like data • Partitioned by event time buckets, perhaps also by other dimension(s) Event History Pattern 4/25/2016© 2015 Think Big, a Teradata Company
- 12. 12 Actor id Segment s Ev1:id Ev1:fact Ev1:dims Ev1:ext Ev2:id … uid1 [1, 3, 7] 123 1/1/15 13:16:11 … { “TstA” : 1 …} 789 … uid2 [2, 3] 456 1/1/15 13:16:14 … { “TstB” : 1 …} 0ab … • Pivot on event history: table of actors with events over time – Device history, usage in consumer journey – Enable support/analysis on specific items, long-lived analysis • May have hierarchy of actors (e.g., cluster, device, component) • May be array of events, many columns or subsorted (cluster key) • Also stored in columnar format, may be partitioned • May be updated in near real-time AND batch • Often holds cached algorithm values (combined Profile) Timeline Pattern 4/25/2016© 2015 Think Big, a Teradata Company
- 13. 13 • Ongoing status of configuration – Parts in assembly – Related items (versions) – Peer groups • For physical configuration and/or software components • Maintain links in graph structure – May be current or historical • Use links to pull full context from Event History or Timeline • Search -> simple query -> complex analytics – E.g., transitive closure, impact analysis • Technologies – BlazeDB, TitanDB, Neo4j – Spark GraphX & GraphFrame, Giraph Network 4/25/2016© 2015 Think Big, a Teradata Company
- 14. 14 Late Data • Delays from intermittent connectivity, upstream failures • Lineage tracking is critical • Watermarks to identify when sufficient data has arrived (based on statistics, upstream) • May trigger early, on time & late • Report on how much data has arrived late 4/25/2016© 2015 Think Big, a Teradata Company Zipfian Distribution
- 16. 16 • Global manufacturer of storage devices: hard-drives, SSDs, object storage • Produces 100’s of millions of devices annually • Each device contains multiple complex components – Manufacturing sites are geographically dispersed – Some components are sourced from suppliers – Each device generates ~100-1000MB of data during its lifecycle Case Study: Overview Confidential
- 17. 17 Business Challenges Need to speed cycle time for new product development Customer’s demanding faster Failure Analysis Engineer’s wasting time playing “where’s Waldo” with the data Confidential
- 18. 18 Technical Challenges Difficulty storing & exposing binary and other data types Current DW’s Unable to Keep Pace with the Volume No platform for large-scale analytics Data silos across manufacturing facilities Confidential
- 19. 19 Goal: Expose the entire “DNA” of the device—from development, manufacturing, to reliability testing and “living behavior” of device for live behavior —to increase operational efficiency and quality -- Chief Information Officer Confidential
- 20. 20 Platform Overview Site 1 Site 2 Site 3 Site 4 Final Assembly Customer Data Supplier Shop Floor Data Shipment Data ... Data Sources End-to-End Integrated Data Big Data Platform Consumers Ad hoc Analysis Defect Pattern Recognition Enterprise DW Batch Analytics Parallelized batch analytics App-Specific Views New High-Value Parameters raw extracts Enriched data End-to-End Traceability Tester Failure Analytics Failure Analysis Customer data lookup ... Applications Confidential
- 21. 21 • Large volumes of Binary Data: – Require 5 years for warranty reasons, leading to PB’s of binary objects • Schema on read: – Development/Process Engineering teams change the manufacturing/test data very frequently; thus, the decoding of the binary data changes very frequently. – It is very difficult to keep pace with these changes with a traditional RDBMS, often leading to time-consuming data purging and reloading Use Case 1: Binary Data…with daily decoding changes Parsing Confidential
- 22. 22 Use Case 2: Wide Structures (Timeline) 10’s of thousands of parameters collected over the course of 6-8 months for a single device…a wide, de-normalized structure reduces the complexity for end-user analysis Confidential
- 23. 23 Solve new problems - exposing previously “untapped” data sources at a scale that allowed for identification of patterns causing the issues, E.g., scan 380 billion test points for 8 million products. Several irregular distributions were found, which allowed the team to identify a code-level bug that was causing the failures (and therefore scrapped drives).” Use Case 3: “Un-Paralleled” Parallel Analysis Confidential
- 25. 25 Conclusions • IoT is about blending data • Data management patterns & practices are foundational • Lead to effective analytics • Reach me at @ronbodkin, ron.bodkin@thinkbiganalytics.com 4/25/2016© 2015 Think Big, a Teradata Company
- 26. 26 Big Data Strategy & Roadmap Analytics & Data Science Training & Managed Services How can my Organization Get Value from Big Data? How Do We Reap Value from Our Big Data Investment? How Do We Keep Our People and Environment Operating At A High Level? How Do We Build a Best Practices Big Data Environment That Will Meet our Needs? Data Lake Implementation Hadoop, Spark Solutions since 2010. We’re Hiring
- 27. 27 • Incorporate all data from all touch points to understand true customers’ behavior • Leverage multi-genre advanced analytics techniques to generate behavior-based insights • Available NOW Customer Satisfaction Index Analytic Solution Announcement
Editor's Notes
- #6: This illustrates the fundamental processing in iconic form.
- #7: This slide shows how security and governance would be managed across the different phases of ingest, data preparation and Publishing. It is the heart of the Goldilocks governance This slide builds and starts with Data Scientist that need early access, Modelers next where they expect a lot of data science work already occurred, then jumps up to Business Analysts, the three major user groups, Business Analyst would need access to materialized models and common summaries, but will be mostly accessing through view heavy interfaces to enforce security and help with further materialization or at least representation (views) of the specific models. IT is last and it needs access to everthing
- #18: Images courtesy of: http://hybridclaims.com
- #19: Images courtesy of: http://neaglobal.com http://eweek.com
- #20: Increase customer satisfaction: Commitment to quality Improve customer service and access to data (internally & externally) Increase operational efficiency: improved yield & time-to-market By having end to end visibility to: every test, every diagnostic and all info from all components of a product Enable the business to extract new insights (never-before possible)
- #27: This slide represents Think Big’s end-to-end big data services portfolio. We have services that span from big data strategy and roadmap all the way to training and managed services. Today we’re going to talk about data lake implementation best practices.