BI on Big Data with instant response times at Verizon
Download as PPTX, PDF3 likes498 views
The document discusses Verizon's strategies and technologies for managing big data, specifically focusing on the challenges faced, such as increased data volumes and the need for real-time access. It emphasizes the benefits of modern OLAP on Hadoop, which provides flexible, scalable, and performant solutions for data analysis. Key features include self-service capabilities, faster insights, and the consolidation of multiple data sources into a single cube.
BI on Big Data with instant response times at Verizon
- 1. Arun Jinde Technical Architect Verizon Ajay Anand Vice President, Products and Marketing Kyvos Insights Inc. BI on Big Data with Instant Response Times at Verizon © Verizon 2018, All Rights Reserved. Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners.
- 2. Employees 154.7K worldwide Network 98% US wireless coverage Broadband 100% fiber optic network Video & Advertising 200M hours of video streamed monthly Internet of Things 14K+ developers hosted on ThingSpace Security 25+ years of experience and expertise *Source: Verizon.com/About
- 3. © Verizon 2018, All Rights reserved. Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners. • Big data business challenges • Verizon’s big data architecture • What is OLAP and why is it needed? • Traditional vs Modern OLAP • How modern OLAP achieves interactive BI Agenda
- 4. © Verizon 2018, All Rights reserved. Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners. Increased data volume Varied data sources Real-time aspect of data Data access and reuse Self-service needs Cost and hidden system debt A challenging big data landscape
- 5. © Verizon 2018, All Rights reserved. Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners. Challenges Technology Solution Increased data volumes Hadoop distribution Varied data sources Unified Data Models and standardized KPI stores Real time aspect of data Kafka/Storm/Nifi/StreamAnalytix Data access and reuse Hybrid architecture Self-service needs Pre-Connected data models/OLAP on Hadoop Cost and hidden system debt Data locality and compute Dealing with the complexity of Big Data
- 6. © Verizon 2018, All Rights reserved. Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners. • 12K Files Per Day in Microbatch mode. • 300 Messages/Sec in Real-Time. • Dynamic Metadata management. • Cluster Size: 4 PB • # of Cluster Nodes: 130 • Kyvos Memory: 24 GB/Node • # of Kyvos Query Engine: 10 • # of Kyvos OLAP Engines: 2 Verizon’s Video Analytics Platform
- 7. © Verizon 2018, All Rights reserved. Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners. OLAP is a hierarchical multidimensional data model • Analyze multidimensional data interactively from multiple perspectives • Dimensions are qualitatively represented by Measures • Cube Operations: Rolling up, Drilling down, Slice & Dice and Pivot • Cube Query Language: Multi Dimensional expressions (MDX) TIME PRODUCT 365 269 295 377 1306 234 465 255 678 1632 164 135 153 145 597 132 144 111 555 942 895 1013 814 1755 4477 Jan05 Feb05 Mar05 Apr05 Yr05 LCD Monitor Digital Camera 40G Drive Game Console All Products Account #1 Account #2 Account #3 Account #4 All Accounts TIME PRODUCT ACCOUNT What is OLAP
- 8. © Verizon 2018, All Rights reserved. Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners. Why the need for OLAP? • Speed of responses for multi-dimensional queries • Iterative queries for data discovery • The need to drill down for more details • Access to historical side-by-side comparative analysis
- 9. © Verizon 2018, All Rights reserved. Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners. • Very rigid and not scalable • Massive increase in data volumes (size of cubes) • Explosion of cardinality (granularity) from added dimensions • Processing is not linearly scalable forcing towards fixed column reporting • Data movement and frequency of updates impacting Service Level Agreements • Requires expert developers TRADITIONAL OLAP ON SQL BI Tools ODBC SQL MDX Query Engine Cube Processing Data Movement Star Schema Data Repository OLAP Cube Limitations of traditional OLAP
- 10. © Verizon 2018, All Rights reserved. Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners. More flexible, scalable, and performant • Convenient and easy access to data • Single source of truth (one cube vs many) • More flexibility to model data • Empower users to explore data at scale • Scalable with Hadoop ecosystem • No data movement • No new infrastructure Need for modern OLAP
- 11. © Verizon 2018, All Rights reserved. Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners. MODERN OLAP ON HADOOP REST Server Query Engine Cube Build Engine (Map Reduce) BI Tools Rest API/MDX/XMLA/ODBC Star Schema OLAP Cube HADOOP Modern OLAP / Kyvos Architecture
- 12. © Verizon 2018, All Rights reserved. Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners. Cubes Row Count HDFS File Size Cube Size # of Dimensions Daily Processing Time Query Performance Live TV 78 B 12 TB 70 TB 38 20 mins < 12 secs DVR 25 B 4 TB 30 TB 46 10 mins < 5 secs VOD 1 B 2 TB 5 TB 56 5 mins < 5 secs Search 0.5 B 1 TB 1.5 TB 32 5 mins < 5 secs Results attained with Kyvos
- 13. © Verizon 2018, All Rights reserved. Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners. 1 Faster time to insights from faster processing 2 Interactive access to big data for analysis without interruption 3 Self service model to empower users 4 Turn any BI tool to a native on Hadoop big data tool 5 Consolidate multiple cubes to one cube for single source of truth 6 No data movement to access all the data at every granular level 7 No new infrastructure to control technical costs Benefits of achieving interactive BI
- 14. © Verizon 2018, All Rights reserved. Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners. 1 Registering files. 2 Creating Datasets. 3 Building Relationship. 4 Building Cube. 5 Processing Cube. 6 Creating Visualizations. 7 Creating Dashboard. Self service using Kyvos
- 15. © Verizon 2018, All Rights reserved. Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners. Arun.Jinde@Verizon.com Thank you.