Why Data Lake should be the foundation of Enterprise Data Architecture
Download as PPTX, PDF4 likes391 views
The document discusses the importance of the Enterprise Data Lake (EDL) as a foundational component of enterprise data strategy, emphasizing its advantages over traditional Enterprise Data Warehouses (EDW). It outlines the impact of digital transformation on data management, promoting self-service analytics and agile data access. The EDL addresses challenges such as data latency and the complexities of traditional BI systems, positioning itself as a solution for modern analytics needs.
Why Data Lake should be the foundation of Enterprise Data Architecture
- 1. Why Data Lake Should be the Foundation of Enterprise Data Architecture?
- 2. © 2017 Agilisium Consulting LLC All rights reserved. Confidential Information 2 Raj has about 20+ years of technical implementation and domain expertise in BI & Analytics solutions. Very passionate about applying Design Thinking & Digital Transformation principles for Analytics, BI & Insights. Agilisium ( Agile + Elysium ) is a Design Thinking applied Big Data, BI & Analytics services company. Our services are designed to accelerate our customers Digital Transformation’s Data Strategy in Analytics, BI & Insights Raj Babu Founder & CEO, Agilisium Consulting WWW.AGILISIUM.COM About the Speaker
- 3. Key topic covered Digital Transformation & Design Thinking in Analytics Why Enterprise Data Lake (EDL ) EDW vs EDL vs EDWL EDL key value proposition & consumers M&E Lake on AWS Cloud Ref Arch Lake on Cloud Get in Touch
- 4. © 2017 Agilisium Consulting LLC All rights reserved. Confidential Information 4 • Digital Transformation is transforming EDW to EDL & EDWL • EDL (Enterprise Data Lake ) becoming de facto foundation for Enterprise Data Strategy • Apply Design Thinking for Data, Analytics, Insights • Think Outside – In, When you think of data • Data Monetization should be at the heart of Data Science or Analytics project. Think beyond business operations • Smart Automation – Building enablers with Smart BOTS & Widgets for smart automation. How is Digital Transformation impacting Enterprise Data thinking – Key industry trends in Data Insights, Analytics & BI
- 5. © 2017 Agilisium Consulting LLC All rights reserved. Confidential Information 5 Controlled & Structured vs. Self-Service & Exploratory EDW to EDL – What is causing the Shift? – Digital Transformation to Enterprise Data …. Digital Age Approach Iterative & Exploratory Analysis Legacy Approach Structured & Repeatable Analysis IT & IS Role • Build and closely administer a BI system to answer pre determined questions Business Users • Focus was mostly on internal business operations related data • Need to determine and lay out what question to ask Monthly Operational reports Enterprise Analytics Customer surveys ? IT & IS Roles Delivers a self-service platform to business to access data at various level Business Users • Focus now is now on social, market and external data. Not just on internal data • They don’t know yet what all they could ask and it constantly evolves Brand sentiment Enterprise Data strategy Maximum asset utilization ?
- 6. © 2017 Agilisium Consulting LLC All rights reserved. Confidential Information 6 Our BI cant do anything ad-hoc, they need requirements, design, architecture, ETL for everything & it never gets done after all Our BI is Always incomplete, it never has all the data we need Our BI team and system can’t implement changes fast Our BI is not suitable for ad-hoc Analytics BI is too Expensive to Build and Manage and never on the schedule that Business wants Over complicated Architecture… Not flexible and it takes too long to get anything changed Challenges raised by Business on BI or EDW solutions?
- 7. © 2017 Agilisium Consulting LLC All rights reserved. Confidential Information 7 Building Lake on Cloud – Build your EDL, EDW & EDWL on AWS Key AWS technology stack for Enterprise Data solution. • AWS S3 Data Lake • Redshift Spectrum • AWS Athena • AWS Glu ( ETL & Catalog ) • AWS Quicksight
- 8. © 2017 Agilisium Consulting LLC All rights reserved. Confidential Information 8 Why Data Lake Platform should be the foundation of Enterprise Data Strategy ? EDL solves EDW’s Data Latency & ETL challenge. • An EDL is designed to “RETAIN ALL DATASETS“ and all data is available for use as they are needed. • With EDL data management is Agile, nimble, economical & valuable • EDL once and for all solves Dark Data problem & Data Ceralization problem. ? EDL ( Enterprise Data Lake ) key value proposition
- 9. © 2017 Agilisium Consulting LLC All rights reserved. Confidential Information 9 Your EDL foundation can be following A warm and Active Data Archive /Vault A central Enterprise Data Repository ODS Data Hub & Staging source for all systems Data Science, Analytics & Hadoop Data Warehouse EDL Foundation & Consumption
- 10. © 2017 Agilisium Consulting LLC All rights reserved. Confidential Information 10 In EDL, no ETL or database needed for reporting or analytics Service ad-hoc requests with no latency & no development No more waiting….Perfect to offload all new & ad-hoc requests Minimal development team involvement, unless data is needed in Data MartInexpensive and low maintenance cost to manage as there is no or very minimal Build effort EDL Key benefits ?
- 11. © 2017 Agilisium Consulting LLC All rights reserved. Confidential Information 11 Sources Ingest Prepare Discover Process Users OLTP, ERP, CRM Systems Document s & Emails Web Logs, Click Streams Social Networks Machine Generated Geo-location Data STREAMS BATCH WRANGLE CLEANSE GOVERN SEARCH ASSESS ANALYSE Applications Business Intelligence Data Mining Data Science Reporting Biz Ops Business Analysts Data Scientist BIG DATA PLATFORM Security, Metadata, Operations VALUE Data flow through EDL / EDWL Value Chain
- 12. © 2017 Agilisium Consulting LLC All rights reserved. Confidential Information 12 Consumers of EDL or EDWL Anyone one and everyone who is impatient about getting their hands on data The ones that cant give requirement but wanted reports yesterday The ones that have no patience for ETL or Report development Analytics, Data Science team ETL team for staging By not having to buy DB capacity to store all data in BI database When volume of data too high to process through a regular DB
- 13. © 2017 Agilisium Consulting LLC All rights reserved. Confidential Information 13 With Redshift Spectrum we can have users access Redshift DB & S3 seamlessly EDWL stands for Enterprise Data Lake & Data Warehouse. • EDWL almost gives best of both worlds EDL & EDW • AWS Redshift Spectrum is a perfect EDWL solution • With Redshift Spectrum, we can have MPP DB also access Data Lake • AWS Athena offers a server less Query As Service on S3 Data Lake Is there anything better than EDL? Yes, It is EDWL – AWS Redshift Spectrum
- 14. © 2017 Agilisium Consulting LLC All rights reserved. Confidential Information 14 Questions? CONTACT US – We are also Hiring Address 2629, Townsgate Road, Suite 235, Westlake Village, CA 91361 Phone 661-645-2189 E-mail Raj@Agilisium.com Website www.Agilisium.com Tweet me @BigDataRajBabu For exciting careers in Big Data and Data lake projects, send us your profile to careers@agilisium.com For details on our ‘Data lake offerings and capabilities’, reach out to contact@agilisium.com
- 15. Thank You Agilisium is a Los Angeles based Systems Integrator that delivers business agility through optimized, cost-effective cloud & data engineering services, industry domain expertise and technology innovation. Agilisium’s big data, analytics and cloud solutions empower companies to improve their business productivity and customer experience. Being an AWS, Google Cloud & Microsoft Azure certified consulting partner, we help organizations design, architect, build, migrate and manage web & mobile applications on the cloud. Agilisium’s next-generation managed services are designed to align IT operations to business process performance. We provide a secure, flexible and resilient cloud environment for responsive web development and ERP implementations. Agilisium works with leading businesses in media & entertainment, healthcare, electronics & consumer goods and empowers them to make faster business decisions.