The State of the Data Warehouse in 2017 and Beyond
2 likes1,127 views
The document provides an overview of the changing analytic environment and the evolution of the data warehouse. It discusses how new requirements like performance, usability, optimization, and ecosystem integration are driving the adoption of a real-time data warehouse approach. A real-time data warehouse is described as having low latency ingestion, in-memory and disk-optimized storage, and the ability to power both operational and machine learning applications. Examples are given of companies using a real-time data warehouse to enable real-time analytics and improve business processes.
The State of the Data Warehouse in 2017 and Beyond
- 1. The State of the Data Warehouse in 2017 and Beyond Presented by
- 2. Copyright (C) 2017 451 Research LLC The Changing Analytic Environment James Curtis, Senior Analyst, Data Platforms & Analytics
- 3. Copyright (C) 2017 451 Research LLC 33 451 Research is a leading IT research & advisory company Founded in 2000 300+ employees, including over 120 analysts 2,000+ clients: Technology & Service providers, corporate advisory, finance, professional services, and IT decision makers 50,000+ IT professionals, business users and consumers in our research community Over 52 million data points published each quarter and 4,500+ reports published each year 3,000+ technology & service providers under coverage 451 Research and its sister company, Uptime Institute, are the two divisions of The 451 Group Headquartered in New York City, with offices in London, Boston, San Francisco, Washington DC, Mexico, Costa Rica, Brazil, Spain, UAE, Russia, Taiwan, Singapore and Malaysia Research & Data Advisory Events Go 2 Market
- 4. Copyright (C) 2017 451 Research LLC 4 A combination of research & data is delivered across fifteen channels aligned to the prevailing topics and technologies of digital infrastructure… from the datacenter core to the mobile edge.
- 5. Copyright (C) 2017 451 Research LLC 5 • Data Platforms & Analytics • Some Trends • The Evolving Data Warehouse • The Evolution of Analytics • Key Takeaways 5 Agenda
- 6. Copyright (C) 2017 451 Research LLC 66 Data Platforms & Analytics § Technologies to store, process and analyze data § Collect/analyze data to identify potential opportunities for improvement § Includes: • Operational, analytic databases, Hadoop, data grid/cache, event/stream processing • Data management/integration technologies to prepare data for analysis, and analytics tools
- 7. Copyright (C) 2017 451 Research LLC 7 Some Trends The lines will continue to blur between operational and analytical databases. Machine learning and deep learning will enter a new phase of strategic adoption for predictive analytics. Stream processing adoption will accelerate as companies grapple with fast data. • Avoid seeing transactional and analytic databases as two totally different systems • Think about existing database admin and BI skills • Don’t ignore the demand for predictive analytics using machine learning • Balance user-friendliness against complexity • Does your existing data processing and analytics infrastructure handle streaming data? • Understand the business use case for streaming data TREND RECOMMENDATION Source: 451 Research. 2016/2017 Trends in Data Platforms and Analytics. Oct 2015/2016.
- 8. Copyright (C) 2017 451 Research LLC 88 A Growing Market Source: 451 Research Market Monitor. Total Data: Platforms & Analytics. May 2017.
- 9. Copyright (C) 2017 451 Research LLC 9 “He that will that not apply new remedies must expect new evils.” −Francis Bacon
- 10. Copyright (C) 2017 451 Research LLC 10 DECISION MAKERS DATA ANALYSTS IT PROSENTERPRISE APPLICATIONS DATA WAREHOUSE Enterprise Data Warehouse: Common characteristics
- 11. Copyright (C) 2017 451 Research LLC What’s driving the change? 11 COMPUTE OPTIONS STORAGE CHOICES ORGANIZATIONAL EXPECTATIONS OPEN SOURCE SOFTWARE DATA, DATA, AND MORE DATA
- 12. Copyright (C) 2017 451 Research LLC 12 ENTERPRISE APPLICATIONS DECISION MAKERS DATA ANALYSTS IT PROSDATA WAREHOUSE Adapt and Expand Our Field of Vision
- 13. Copyright (C) 2017 451 Research LLC 13 ENTERPRISE APPLICATIONS CLOUD STORAGE DECISION MAKERS HADOOP SPARK AI+ML DATA ANALYSTS IT PROSDATA WAREHOUSE Expanded Processing Choices
- 14. Copyright (C) 2017 451 Research LLC 14 ENTERPRISE APPLICATIONS CLOUD STORAGE MOBILE APPS BOTS IOT DEVICES AND SENSORS SOCIAL MEDIA DECISION MAKERS HADOOP SPARK AI+ML DATA ANALYSTS IT PROS LOG AND CLICKSTREAM DATA DATA WAREHOUSE Leads to Expansion of Data Sources
- 15. Copyright (C) 2017 451 Research LLC 15 ENTERPRISE APPLICATIONS CLOUD STORAGE MOBILE APPS BOTS IOT DEVICES AND SENSORS SOCIAL MEDIA BUSINESS USERS DATA-DRIVEN APPLICATIONS DATA SCIENTISTS DECISION MAKERS HADOOP SPARK AI+ML DATA ANALYSTS IT PROS LOG AND CLICKSTREAM DATA OT USERS DATA WAREHOUSE Which leads to More Advanced Decision- Making Processes
- 16. Copyright (C) 2017 451 Research LLC The evolution of analytics 16
- 17. Copyright (C) 2017 451 Research LLC The evolution of analytics 17
- 18. Copyright (C) 2017 451 Research LLC The evolution of analytics 18
- 19. Copyright (C) 2017 451 Research LLC 19 Key takeaways
- 20. Copyright (C) 2017 451 Research LLC 20 Thank you james.curtis@451research.com @jmscrts www.451research.com
- 21. New Data Warehouse Architectures Mike Boyarski
- 23. New Data Warehouse Requirements Performance / Intraday results on fast growing data Usability / Easier to setup, tune, and scale Optimization / Address scale and performance at a lower cost Ecosystem / Drive operational and machine learning applications 23
- 24. 24 The Modern Approach to Addressing New Requirements A Real-Time Data Warehouse
- 25. Real-Time Data Warehouse Explained § Low latency between data generation and analysis § Micro batching or stream ingestion § Sub-second views on operational data and applications § Transaction processing for accelerated transformation § Extensibility functions for ML applications § Durable for operational readiness
- 26. MemSQL: A Real-Time Data Warehouse Easy to setup real-time data pipelines with exactly-once semantics Streaming Data Ingest Memory optimized tables for analyzing real-time events Live Data Disk optimized tables with up to 10x compression and vectorized queries for fast analytics Historical Data 26
- 27. Real-Time Data Warehouse Ecosystem 27 Streaming Ingest Live Data Historical Data Real-Time Data Pipelines Memory Optimized Tables Disk Optimized Tables Real-Time Data Messaging and Transforms Historical Data Real-Time Application Analytics Business Intelligence Dashboards Bare Metal, Virtual Machines, Containers On-Premises, Cloud, As a Service Kafka Spark Relational Hadoop Amazon S3
- 29. Data Lake Acceleration Application Reference Store New Data Input Transformation Structured Real-Time Data Platform
- 30. Application Reference Store New Data MemSQL Spark Connector Real-Time Data Platform Spark Data Lake Acceleration with Spark
- 31. Application Reference Store Exactly-once Semantics Real-Time Data Platform New Data Data Lake Acceleration with Kafka
- 32. Application Historical Data Analysis Legacy Data Warehouse Live Data Analysis Real-Time Data Platform New Data Legacy Data Warehouse Acceleration Stream Batch
- 33. Application Historical Data Analysis Legacy Data Warehouse Live Data Analysis Real-Time Data Platform New Data Legacy Data Warehouse Acceleration Stream Batch
- 34. Real-Time Application New Data Real-Time Scoring Reference Store High Speed ML in SQL Operationalizing Machine Learning Applications
- 35. 35 http://blog.memsql.com/image-recognition-at-the-speed-of-memory-bandwidth/ § Dot product implementation for image recognition § Similarity search § Massive performance improvement
- 36. Companies are changing how they interact with their data and customers. 36
- 37. This image cannot currently be displayed. Real-time data with high concurrency tracking millions of cars, drivers, and riders to optimize fleet operations + 37
- 38. Uber Real-Time Analytics Architecture
- 39. ++ BUSINESS BENEFITS • Real-time data with massive concurrency across millions of drivers, riders, and employees accessing the database concurrently • Enables real-time indicators to understand operating performance • Geospatial indexing for live location-based analysis • Company-wide dashboard for global trends 39 TECHNICAL BENEFITS • Analyze millions of rows/second • Analyze historical and live data simultaneously • Massive concurrency: Hundred of users query reporting databases
- 40. 40 Real-time analytics transformed profitability analysis of customer logistics from weekly to daily, and reduced latency from days to minutes +
- 41. + BUSINESS BENEFITS • Real-time analytics transformed profitability analysis of customer logistics data • Reduced data latency from hours to minutes giving business users access to the most recent data 41 TECHNICAL BENEFITS • Reduced 22 hour ETL to minutes • Increased query response time by 80x over mySQL