The Evolution of Data Architecture
Download as PPTX, PDF7 likes1,938 views
The document discusses the evolution of data architecture, emphasizing the rise of distributed systems and big data technologies due to stagnant hardware improvements and accelerating data generation. It explores the development of platforms like Hadoop, open source software's significance in the data economy, and various applications of big data across industries, including predictive maintenance and smart city management. Key takeaways include the importance of treating data as a vital resource and the shift towards open source solutions in technology infrastructure.
The Evolution of Data Architecture
- 1. The Evolution of Data Architecture Wei-Chiu Chuang 2017. 10 @ NCKU 1
- 3. Data Value Chain AI Machine Learning Data Science Analytics Big Data Decision making Insight Automated Decision making Hype (?) 3
- 4. Data is the new Oil https://www.economist.com/news/leaders/2172165 6-data-economy-demands-new-approach-antitrust- rules-worlds-most-valuable-resource 4
- 5. Fastest way to transmit 5MB of data in 1956
- 6. 6 Fast forward 60 years… transmit 100PB of data in 2016
- 7. Once upon a time, processors double in speed every 18 months … The “Moore’s Law” stopped 10 years ago. CPU, RAM and disk almost stopped improving in speed ever since. 7
- 8. Processor speed has been stagnant But data is being generated at ever increasing speed. Hardware improvement cannot keep up with data generation. Multi-threaded systems, distributed systems are the must. 8
- 9. Distributed Systems are hard Programmability Scalability Consistency Availability Partition Tolerance Fault Tolerance 9
- 10. Big Data/Parallel Computing/Distributed Sys. D HPCBig DataCloud Distributed Systems 10
- 11. Scale out 11
- 12. Modern Data Architecture How do you: transmit collect store compute Petabyte+ storage on 1000+ compute nodes? 12
- 13. Modern Data Center DataCenter ToR Server1 Server10 ToR Server1 Server10 ToR Server1 Server10 ToR Server1 Server10 Aggr Aggr Aggr Core Core Internet AR AR 10Gbps 10Gbps 1Gbps 13
- 14. GFS Master – slave architecture Separation of control plane and data plane Low cost, commodity hardware Failures are norm, rather than exceptions Balance availability and network partition tolerance Control messages Data messages GFS Master GFS chunkservers /foo/bar GFS client 14
- 15. MapReduce A very simple yet powerful distributed programming model Share-nothing architecture Programmability Data-locality: ship compute to data, rather than shipping data to compute Fault tolerance: Intermediate state is stored in storage. Failed tasks can be restarted easily. Split 0 Split 1 Split 2 worker worker worker Input files Map phase worker worker Intermediate files Reduce phase Output 0 Output files Output 1 master assign map assign reduce 15
- 16. Hadoop 16
- 17. Hadoop GFS, MapReduce inspired Hadoop Initially developed by Yahoo! Released in 2006. Used by most large enterprises Hadoop 3.0 beta 1! 17
- 18. 2006 2008 2009 2010 2011 2012 2013 Core Hadoop (HDFS, MapReduce) HBase ZooKeeper Solr Pig Core Hadoop Hive Mahout HBase ZooKeeper Solr Pig Core Hadoop Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig Core Hadoop Flume Bigtop Oozie HCatalog Hue Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig YARN Core Hadoop Spark Tez Impala Kafka Drill Flume Bigtop Oozie HCatalog Hue Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig YARN Core Hadoop Parquet Sentry Spark Tez Impala Kafka Drill Flume Bigtop Oozie HCatalog Hue Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig YARN Core Hadoop 2007 Solr Pig Core Hadoop Knox Flink Parquet Sentry Spark Tez Impala Kafka Drill Flume Bigtop Oozie HCatalog Hue Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig YARN Core Hadoop 2014 2015 Kudu RecordService Ibis Falcon Knox Flink Parquet Sentry Spark Tez Impala Kafka Drill Flume Bigtop Oozie HCatalog Hue Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig YARN Core Hadoop Evolution of the Hadoop Platform The stack is continually evolving and growing! 18
- 19. Mix and match Resource Management YARN Mesos Kubernetes Storage HDFS HBase Kudu S3 ADLS Compute MapReduce Hive Impala Spark Presto Pig Drill Solr Storm Ingest Kafka Flume Beam Samza 19
- 20. Open source in infra & platform 20
- 21. Why open source? It’s free ($$$) No vendor lock-in. Faster development and faster adoption. A new approach to foster collaboration. Open source software is becoming the standard. 21
- 22. Sell open source software, really? Water is free, but bottled water is not. Cloudera sells the “bottle” Cloudera’s Distribution of Hadoop. The integration of software. The support and services. The management software is proprietary. The OSS is free of charge. 22
- 23. Market for open source software? 23 0 50 100 150 200 250 300 350 400 FY2015 FY2016 FY2017 FY2018 (f) Revenue (million USD) Hortonworks Cloudera MongoDB
- 24. Open Source Business Model • MySQL Dual licensing • RedHat, Hortonworks Support + services • Java EE, Qt Open core • DataBricks, Amazon AWS, Microsoft Azure Software as a Service • Google Chrome, Android Advertising-supported • Cloudera, Confluent, MongoDB Hybrid Open Source Software 24
- 25. Use Cases 25
- 26. “Big Data” finds many applications across many industries IT Healthcare Transportation Retail Utilities Telecomm Public sector Manufactring 27
- 27. Applications and Use cases Realtime database for serving internet traffic Internet services (Facebook messenger), Twitter, Uber, Airbnb … Data analytics Assist in the development of new drugs by analyzing millions of medical records Data science / Machine learning Fraud detection Anti-money laundry Cybersecurity 28
- 28. Fraud Detection System using Hadoop
- 29. The Cloudera Platform for IoT – Data Mgmt. Value Chain Data Sources Data Ingest Data Storage & Processing Serving, Analytics & Machine Learning ENTERPRISE DATA HUB Apache Kafka Stream or batch ingestion of IoT data Apache Sqoop Ingestion of data from relational sources Apache Hadoop Storage (HDFS) & deep batch processing Apache Kudu Storage & serving for fast changing data Apache HBase NoSQL data store for real time applications Apache Impala MPP SQL for fast analytics Cloudera Search Real time searchConnected Things/ Data Sources Other Data Sources Security, Scalability & Easy Management Deployment Flexibility: Datacenter Cloud Apache Spark Stream & iterative processing, ML
- 30. IoT Use Case 1: Predictive Maintenance
- 31. Predictive Maintenance on Thousands of Industrial Machinery in Real- Time Challenge: • Collect and analyze data from thousands of diverse manufacturing systems in real-time Solution: • iTrak application using Cloudera in the Cloud to monitor the performance of individual manufacturing systems in real-time • Predictive Maintenance - Proactively identifying & fixing issues before they break MANUFACTURING » INDUSTRIAL IoT » PREDICTIVE MAINTENANCE » IMPROVED EFFICIENCIES Industrial IoT – Predictive Maintenance DATA-DRIVEN PROCESS CASE STUDY DATA-DRIVEN PRODUCTS
- 32. Use Case 2: Connected Vehicles
- 33. Using Predictive Maintenance to Improve Performance and Reduce Fleet Downtime Challenge: • Monitor the health of 180,000+ trucks in real-time in order to minimize downtime Solution: • OnCommand Connection collecting telematics and geolocation data across thousands of trucks • Identify and correct engine problems early, and increase fleet uptime • Reduced maintenance costs to $.03 per mile from $.12-$.15 per mile Connected Vehicles & Telematics DATA-DRIVEN PROCESS CASE STUDY DATA-DRIVEN PRODUCTS TRANSPORTATION » PREDICTIVE MAINTENANCE » TELEMETRY » LOWER TCO
- 34. Use Case 3: Smart Cities & Smart Infrastructure
- 35. Enabling the State of Kentucky manage snow and ice events in real time Challenge: • Kentucky Transportation Cabinet (KYTC) oversees the state’s transportation system, which includes 27,000 miles of highways, 230 airports and heliports, and more than three million drivers. • Needed more efficient approach to inclement weather road management Solution: • KYTC has built a real-time weather response system that incorporates real-time data from Waze, HERE, ESRI’s GeoEvent processor, and Automatic Vehicle Locations (providing sensor data from salt trucks). • KYTC aggregates 15-20 million records every day and process more than a million records per second. Data Driven Dept. of Transportation Source: http://www.routefifty.com/2016/09/data-drives-government/131821/ 2016 Data Impact Award Winner State of Kentucky Department of Transportation
- 36. Use Case 4: Connected Healthcare
- 37. Improve Parkinson's Disease Monitoring and Treatment through IoT Challenge: • Collect and analyze data from wearables (more than 300 readings per second) from thousands of patients in real-time Solution: • Cloudera on Intel architecture to detect patterns in patient data streaming from wearables • Continuously monitor the patients and symptoms to understand the progression of the disease objectively HEALTHCARE » WEARABLES » PREDICTIVE ANALYTICS » IMPROVED CARE Connected Healthcare DATA-DRIVEN PROCESS CASE STUDY DATA-DRIVEN PRODUCTS
- 38. Building a Holistic Picture of the US Securities Market From 50 Billion Daily Events • Saving $10-20M in operational efficiencies annually • 90-minute queries run in 10 seconds • Supporting future market growth and a dynamic regulatory environment. CUSTOMER 360
- 39. Using Big Data to Help Consumers Save Hundreds of Millions in Utility Bills • Relevant insight into household energy use improves energy consciousness • 2.7+ TWH (terawatt hours) saved to date • Motivated consumers to save enough energy to power every household in Salt Lake City and St. Louis for a year CUSTOMER 360 ENERGY & UTILITIES » PRODUCT INNOVATION » SERVICE IMPROVEMENT » IOT
- 40. Saving Lives by Detecting Sepsis Early Enough for Successful Treatment • Builds a more complete picture of patients, conditions, and trends • Has saved 100’s of lives already • Reduces hospital readmissions • 2PB+ in multi-tenant environment supporting 100s of clients • Secure yet explorable HEALTHCARE » 360° CUSTOMER VIEW » PREDICTIVE ANALYTICS » IMPROVED SERVICE
- 41. Improving Pediatric Care and Outcomes • Quantifying effect of ambient noise on children’s vital signs • Identifying cancerous genome variants in 20 minutes (vs. days before) • Performing fewer CT scans and higher quality surgeries CUSTOMER 360 HEALTHCARE » MACHINE LEARNING » IOT » 360o CUSTOMER VIEW
- 42. Government Revenue Service Increasing Customer Convenience • Provides view of the complete taxpayer journey • Creates ability to pre-populate tax returns for increased ease of use • Supports move to near-real-time oversight of operations and faster response CUSTOMER 360 GOVERNMENT » SERVICE IMPROVEMENT » PROCESS IMPROVEMENT » 360° CUSTOMER VIEW
- 43. Driving Growth and Innovation • Combines 80+ years’ data spanning all business units and 50 states • Expedites holistic analysis and reports by 500X • Enables more accurate and detailed predictive models to customize offers, optimizing pricing, and minimize risk CUSTOMER 360 INSURANCE » 360° CUSTOMER VIEW » FRAUD DETECTION » PREDICTIVE ANALYTICS
- 44. Re-Platformed 1,600 Operational Databases & Systems onto a Cloudera EDH • Business & consumer data was spread over a dozen different customer databases • One daily ETL job (processing 1 billion customer records) used to take 24 hours • Increased data velocity by 15x (5 times the data in 1/3 of the time) Now completes in 1 ½ hours • BT now has access to the most up-to- date and centralized data for all their customers CUSTOMER 360 TELECOMMUNICATIONS » IMPROVED SERVICE » PROCESS IMPROVEMENT » IT COST REDUCTION
- 45. Future 48
- 46. Future Hardware evolution: Cloud 40Gbps, 100Gbps networks GPU, TPU Flash disk Application-driven: Machine learning, deep learning Realtime data stream processing (IoT) 49
- 47. Future How to scale by an order of magnitude in 5 years? We are here today In 10 years? 50
- 48. 台灣資料工程協會 Click to enter confidentiality information
- 49. 台灣人參與Apache Click to enter confidentiality information 葉祐欣 謝良奇、蔡東邦 陳恩平 戴資力 莊偉赳 蔡嘉平
- 50. Apache Contributor 育才賽 Click to enter confidentiality information
- 51. Takeaway If you only remember 3 things from this talk: 1.Data is the new Oil 2.Open source is the standard 3.Think big! Remember GFS: failures are the norm rather than the exception! 54
- 52. Thank you jojochuang@gmail.com / weichiu@apache.org / weichiu@cloudera.com 55