Southwest Power Pool big data case study
1 like1,341 views
The document discusses Southwest Power Pool's initial steps towards creating a data lake. It describes: - Storing historical and real-time data that exceeded initial expectations, with around 50% being less frequently used - Conducting a proof-of-concept evaluation of three vendors to offload less frequently used data and allow SQL query access with minimal changes to existing queries - Choosing BigInsights based on its ability to do this along with supporting existing Netezza functions and allowing federated queries between Netezza and BigInsights - The multi-phase vision to eventually incorporate more data types and workloads while improving performance, security, and governance
Southwest Power Pool big data case study
- 1. First Steps Towards a Data Lake: Insight from Southwest Power Pool Srinivas Kolluru and Russell Mason 10/26/2016
- 2. Background
- 3. • Data stored in historical data store for data analysis and reporting • Near real-time data feeds • ETL data feeds • More data than initial expectations • Almost 50% of data is less frequently used 3 Background 9/23/2016World of Watson 2016
- 4. Near-term requirements – Offload less frequently used data • SQL query access to the data • Minimal changes to the business queries Long-term vision – Data Lake infrastructure • ETL and real-time data feeds • Transactional and analytics workloads • Cost economical model • Should be able to add compute and storage incrementally • Long-term partnership 4 Proof-of-Concept (PoC) Evaluation Criteria 9/23/2016World of Watson 2016
- 5. • Evaluated three different vendor technologies • On-site PoC with three months of data and with a sample of business queries • Six week evaluation of each vendor technology • Chose BigInsights product • Minimal business query modifications • Support for Netezza functions • Federated query capabilities between Netezza and BigInsights • Partnership 5 Proof-of-Concept (PoC) Evaluation 9/23/2016World of Watson 2016
- 6. Data Lake Vision and Phase 1 Implementation
- 7. 7 Data Lake Vision 9/23/2016World of Watson 2016 Data Zone ETL Data DB Data Streaming API Data BLOB Data Metadata SecurityQuality GovernanceMonitoring Landing Zone Data Extraction Data Refinement Data Discovery Data Analytics Business Users IT Users
- 8. • Phase 1 – Offload less frequently used data, provide SQL query, and federated query capabilities. • Phase 2 – Query performance improvements, transactional capabilities, and security controls • Phase 3 – ETL data, streaming data, and governance • Phase 4 – Real-time and BI analytics 8 Data Lake Implementation 9/23/2016World of Watson 2016
- 9. 9 Phase1 - Data Offload Process 9/23/2016World of Watson 2016 Landing Zone ORCORC Data Zone Data export & CRC check validation ORC table import and CRC check validation Metadata Security Monitoring Source System :
- 10. • Custom export process instead of using Fluid query or Scoop for data transfer • Fluid query • Additional burden on source system • Scoop • Performance challenges with JDBC approach • Limitations with incremental data pulls with external table approach • Limitations with using database views • Difficulty with data validation checks • Difficulty with partitioning data based on record’s last update date • Operational controls to minimize the impact on the source system 10 Architecture and Design Decisions 9/23/2016World of Watson 2016
- 11. • Data Integrity • Row level CRC checks with data export and import processes • ORC format • Monthly data partitioning • Seen better compression with ORC format compared to Parquet (Haven’t compared the query performance with large sets of data) • Separate HDFS storage cluster • Scale storage independent of compute • Align with the IT strategy and processes • Expose the same set of files through multiple file system protocols 11 Architecture and Design Decisions 9/23/2016World of Watson 2016
- 12. 12 Phase 1 Deployment Model 9/23/2016World of Watson 2016 Node #1 Node #2 Node #3 Node #4 Node #5 Hadoop Postgres Metadata Database IBM BigInsights Cluster Node #1 Node #2 Node #3 HDFS Storage Cluster BigInsights Component Deployment (Tentative) Node #1 Zookeeper Server, Hive Metastore, HiveServer2, Knox Gateway, WebHCat Server, BigInsights Home Server, BigSheets Master, BigSQL worker, Metrics Monitoring, and Node Manager. Node #2 Zookeeper Server, History Server, Hive Metastore, HiveServer2, Metrics Collector, Active Resource Manager, WebHCat Server, Big SQL Worker, Metrics Collector, and Node Manager. Node #3 Zookeeper Server, App Timeline Server, Active Hbase Master, Hive Metastore, HiveServer2, Oozie Server, Standby Resource Manager, WebHCat Server, Big SQL Worker, Metrics Monitor, and Node Manager. Node #4 Big SQL Head, Data Server Manager, Hive Metastore, HiveServer2, WebHCat Server, Metrics Monitor, and Node Manager. Node #5 BigSQL Secondary Head, Zookeeper Server, Hive Metastore, HiveServer2, WebHCat Server, Region Server, HBaseREST Server, Metrics Monitor, and Node Manager.
- 13. 13 Phase 1 Production Deployment Model 9/23/2016World of Watson 2016 Node #1 Node #2 Node #3 Node #4 Node #5 IBM BigInsights Cluster Hadoop Postgres Metadata Database Node #1 Node #2 Node #3 HDFS Storage Cluster Node #1 Node #2 Node #3 Node #4 Node #5 IBM BigInsights Cluster Hadoop Postgres Metadata Database Node #1 Node #2 Node #3 HDFS Storage Cluster Data Center 1 Data Center 2
- 14. • Active-Active deployment across the data centers • Evaluating IBM Big Replicate product • Training for support teams • Day light savings challenges 14 Challenges 9/23/2016World of Watson 2016
- 15. Thank You