When Databases Meet Big data and Hadoop - Uni of Tromso Online Lecture
0 likes33 views
This document discusses the rise of big data and how it has disrupted traditional databases. It covers key drivers like decreasing costs of computing resources and the explosion of data volume, variety and velocity. It introduces Hadoop as an open source framework that addressed scalability challenges through its distributed, horizontally scalable architecture. The document also discusses trends like commercialization of Hadoop and using it for data warehousing and analytics use cases. It highlights both challenges and opportunities in bringing data warehousing capabilities to big data platforms.
When Databases Meet Big data and Hadoop - Uni of Tromso Online Lecture
- 1. Memory Speed Big Data Analytics: Alluxio vs Apache IgniteIrfan Elahi - Deloitte 1
- 2. Data Analytics Explained MeetupIrfan Elahi - Deloitte • Working as a Senior Consultant in Deloitte (Analytics Service Line) • Trainer of Deloitte’s Data Science Training • Speaker at DataWorks Summit, Sydney (2017) and AWS meetup, Melbourne • Premium Udemy Instructor with 18,000+ students from 145 countries • Author of upcoming book about Scala Programming for Big Data Analytics • Technical Reviewer of an upcoming book on Hadoop published by APress About Me
- 3. Irfan Elahi - Deloitte When Databases Meet Big Data Traditional Databases Drivers of Disruption The Advent & Rise of Big Data Data Warehousing in Hadoop Big Data Adoption in Enterprises Above and Beyond Overview Agenda
- 4. Irfan Elahi - Deloitte When Databases Meet Big Data
- 5. Irfan Elahi - Deloitte When Databases Meet Big Data Core Capabilities Store: Store data in relational models/objects Process: Perform relational processing via SQL Language Value & Use-Cases Traditional Databases Use Cases • Backend Persistence Layer of applications (OLTP -> OLAP) • Batch Analysis • Reporting • Business Intelligence • Data Warehousing TightlyCoupled
- 6. Irfan Elahi - Deloitte When Databases Meet Big Data Functional Capabilities Traditional Databases ACID
- 7. Irfan Elahi - Deloitte When Databases Meet Big Data Functional Capabilities Traditional Databases Indexing
- 8. Irfan Elahi - Deloitte When Databases Meet Big Data Functional Capabilities Traditional Databases Mutability
- 9. Irfan Elahi - Deloitte When Databases Meet Big Data Capabilities Traditional Databases Unsurpassed SQL Support
- 10. Irfan Elahi - Deloitte When Databases Meet Big Data
- 11. Irfan Elahi - Deloitte When Databases Meet Big Data Compute and Storage continued to become cheaper Decreasing Cost of Resources Volume, variety and veracity of data continued to increase Data Explosion Propositions like elasticity, less CAPEX and low time to value accelerated adoption of the Cloud Increased Adoption of the Cloud Changing Trends Drivers of Disruption
- 12. Irfan Elahi - Deloitte When Databases Meet Big Data Databases and Scalability Drivers of Disruption Approaches to scale: • Denormalization • Caching • Sharding • Materialized Views But… Efficiency and operational cost to operate at scale? Ability to store massive volume of unstructured data at high velocity?
- 13. Irfan Elahi - Deloitte When Databases Meet Big Data
- 14. Data Analytics Explained MeetupIrfan Elahi - Deloitte Leveraged commodity hardware and open source licensing resulting in lower operational costs Segregated computing and storage layers which enabled innovative capabilities Decoupled Storage and Compute Provided fault-tolerance and high availability to optimize operations Fault Tolerance Low CAPEX/OPEX and TCO Aced the notion of bringing compute to data Addressed scalability challenges by introducing horizontal scaling model resulting in distributed storage and compute Horizontal Scaling Was devised to enable large scale processing of unstructured data like logs and social network Originally for Analytics Use-Cases Optimal Data Locality Key Propositions The Advent & Rise of Big Data
- 15. Irfan Elahi - Deloitte When Databases Meet Big Data Nuances of Scalability Drivers of Disruption source: ResearchGate
- 16. Data Analytics Explained MeetupIrfan Elahi - Deloitte Hadoop Ecosystem Growth Commercialization of Hadoop Embodiment of Enterprise Appealing Capabilities Wide Spectrum of Use-cases Data Warehousing in Hadoop? SQL on Hadoop Technologies IoT, Machine Learning, NLP …For instance Cloudera, Hortonworks, MapR, Pivotal, EMR, HDInsight… Boulevard to Data Warehousing in Hadoop The Advent & Rise of Big Data
- 17. Irfan Elahi - Deloitte When Databases Meet Big Data
- 18. Irfan Elahi - Deloitte When Databases Meet Big Data Traditional use- case; New Platform Same expectations and mind-set may result in conflict of expectations Challenges x No In-place Mutation* x NoSQL not fit for relational processing x Table Locking x ACID x No PK/FK constraints* x No Indexing* x Limited Data Models support (Multi-Dim vs Tabular) x Limited SQL coverage Opportunities New Products supporting in-place mutation (Kudu, Apache Ignite) New Products enabling MOLAP on Hadoop (Apache Kylin) Data Virtualization enabling enterprise-wide integration (Denodo, atScale) Enhancement in existing products (Hive ACID) SQL Interface to everything (Kafka, Spark, Ignite) Shift to Cloud-Native technology stack (Big Data (Altus, HDInsight, Databricks), Storage (Blob, ADLS, S3)) Above and Beyond Data Warehousing on Hadoop trend will continue to grow Success will lie in the sweet spot of enhanced capabilities of the stack and flexible mind-set Panoramic View Data Warehousing on Big Data
- 19. Irfan Elahi - Deloitte When Databases Meet Big Data Elasticity …Questions? If interested: Develop skill-set in Big Data and Analytics from my resources: Apache Spark Hands-on Specialization for Big Data Analytics Udemy Course R Programming Hands-on Specialization for Data Science Udemy Course Scala Programming for Big Data Analytics E-book