Big Data i CSC's optik, CSC Representative
Download as PPTX, PDF1 like1,321 views
This document provides an overview of CSC's Netezza case study for a client in Zurich. It discusses how the client was struggling with performance issues on their DB2 database. CSC conducted a proof of concept that showed Netezza and Teradata providing significant performance improvements over DB2. Netezza was ultimately chosen due to cost and compatibility factors. The implementation of Netezza reduced the client's month-end processing time from 9 days to 3 days and improved query performance dramatically. Future plans include migrating more systems to Netezza and taking advantage of upcoming Netezza upgrades.
Big Data i CSC's optik, CSC Representative
- 1. CSC REFERENCE CASE IBM NETEZZA John V Fabienke Chief Architect, Nordic Architecture Practice
- 2. Agenda CSC Enterprise Business Intelligence I. capability overview 3 II. Zürich Netezza case study 7 III. CSC Nordics Netezza PoC capability 22
- 4. CSC Enterprise Business Intelligence Framework CSC Proprietary and Confidential 9/18/2012 9:02 AM 4
- 5. CSC Enterprise Performance and Information Management Practice (EPIM) EPIM Practice (40+ SMEs) BI Capability Data Management Capability GLOBAL GOVERNANCE PRACTICE Strategy BI SOLUTION and SERVICE DATA SOLUTION and SERVICES Tower of Competency Tower of Competency Planning & Knowledge Management Data Solutions GLOBAL BI Solution Integration OFFERINGS Analytics Solutions Architecture & Disruptive Technologies Solutions BI Services Data Services Analytics Services Integration Services GLOBAL EMEA Enterprise Information Capability Group (120+ SMEs) DELIVERY BI Capability Data Management BI Services Data Services Analytics Services Integration Services Offshore Capability (800+ SMEs) BI Capability Data Management BI Services Data Services Analytics Services Integration Services CSC Proprietary and Confidential 9/18/2012 9:02 AM 5
- 6. CSC’s Global Business Intelligence Alliances CSC Proprietary and Confidential 9/18/2012 9:02 AM 6
- 7. Zürich Case Study IBM Netezza solution
- 8. Netezza Overview Business problem to solve • Client‟s current mainframe and DB2 based infrastructure that their financial close processing, data warehouse processing, regulatory reporting and Business Intelligence reporting runs on was over 4 years old • Client was at full capacity on multiple fronts and the DB2 environment required a significant investment to upgrade • With the introduction of the U2Z program data into client‟s month-end close the system was no longer able to meet SLAs for the month-end close process • Even after 9 months of tuning the DB2 environment SLAs were still unable to be met CSC Proprietary and Confidential 9/18/2012 9:02 AM 8
- 9. Netezza Overview What scenarios were considered? • Netezza was suggested as a simple, high-performance solution • Teradata was included due to their being the long-term industry leader • IBM‟s traditional offering was considered as well • Oracle was dismissed due to TCO, lack of existing footprint and skill sets CSC Proprietary and Confidential 9/18/2012 9:02 AM 9
- 10. Netezza Overview PoC performance results • Netezza and Teradata reporting consistently in the 10x – 200x performance improvement range across the board • Analytic run queries ran 150 to 18,000 times faster than the current DB2 platform on both Netezza and Teradata • Compared to DB2 performance differences between Teradata and Netezza were negligible CSC Proprietary and Confidential 9/18/2012 9:02 AM 10
- 11. Netezza Overview Reference calls • Teradata is usually used as an Enterprise Data Warehouse “workhorse” • Netezza is usually used as a reporting engine • Cost was the major factor in choosing Netezza over Teradata ‒ A few references had purchased both platforms for contract negotiations • Both platforms returned over 95% of user queries within 60 seconds CSC Proprietary and Confidential 9/18/2012 9:02 AM 11
- 12. Netezza Overview Netezza was chosen • Teradata‟s version 13 was not compatible with the client‟s version of Business Objects and Informatica • Teradata ran out of time to provide data points used to measure the month-end processing performance • The Netezza platform was a less complex solution than Teradata ‒ Better fit for client‟s out-sourced model CSC Proprietary and Confidential 9/18/2012 9:02 AM 12
- 13. Netezza Overview Slice of User Data Disk Enclosures Swap and Mirror partitions High speed data streaming SQL Compiler Query Plan SMP Hosts Optimize Admin Snippet Blades™ (S-Blades™) High-performance database engine streaming joins, aggregations, sorts, etc. CSC Proprietary and Confidential 9/18/2012 9:02 AM 13
- 14. EDW Architecture Overview Application Architecture: EDW INPUT OUTPUT DMS Mainframe CESAR CIID EDW on DB2 COS Informatica RADIUS/PALM EDW BO ZNAW ZDW DICE ZEA C Class Work ZDW Bench/EDB Netezza CORE WORKBENCH BIW Farmers nzload CSC Proprietary and Confidential 9/18/2012 9:02 AM 14
- 15. Netezza Overview PoC Query Timings Business Objects Reports & SAS queries DB2 Timings (sec) NZ Timing ROWS Comm Analysis Template – Dec 2 „08 - 23 113.516 Comm Analysis Template – Dec ‟08 - AON 227 60 56.134 HPY AY Loss Ratio Analysis 1.380 51 190.688 Policy Effective year Template 2.460 14 888.324 Comm Analysis Template – Dec 2 „08 7.166 41 341.234 Comm Analysis Template – Dec 2 „08 - Marsh 2.150 15 44.308 Direct Written Premium - 2006 300 118 1.322.590 Construction Expiration List – Date Promt 2.024 22 29.510 Domestic Linked to Customer with Foreign 2009 1.800 17 20.597 International Report – Final – Dec Results 2.100 36 29.755 NSBs Large Account Renewals 660 40 3.567 Comm Analysis Template – Dec 2 „08 - AJG 7.826 15 33.263 Comm Analysis Template – Dec 2 „08 - Wells Fargo 6.854 13 25.290 HPA Comm Analysis 5.971 16 4.182 Comm Analysis Template – Dec 2 „08 - HUB 2.840 14 15.508 Producer Code to Dist ID - 17 29.780 Query 1 38.317 43 1.480.812 Query 2 54.707 5 4.169 CSC Proprietary and Confidential 9/18/2012 9:02 AM 15
- 16. Netezza Overview DB2 and Netezza landscape • DB2 has not gone away, a few applications still live on DB2 • DB2 environment was moved to InfoSphere Change Data Capture (CDC) • Informatica environment is completely on DB2 • Working on shrinking the DB2 environment • Netezza replication to Reston can be accomplished via Netezza utilities ‒ NZ_MIGRATE ‒ Data is sent compressed ‒ 2 TB per hour throughput CSC Proprietary and Confidential 9/18/2012 9:02 AM 16
- 17. Netezza Overview How have things improved with Netezza? • Month-end processing is down from 9 days to 3 days ‒ Within 6 months client anticipate to have this down to 2 days • ADS processing does not stop for month-end (no catch-up time) • Queries can run on the Netezza platform during month-end processing • SAP feed down from day 3 to day 1 of Month-end processing • Netezza load and unload speed of 2 TB per hour • Developer mindset change from sequential to set processing • Business Object query time has dropped significantly ‒ Average BO query on DB2 took over 3,100 seconds ‒ Average BO query on Netezza now runs in 6.5 seconds CSC Proprietary and Confidential 9/18/2012 9:02 AM 17
- 18. Netezza Overview How have things improved with Netezza? • On November 18, 2011 16.172 queries ran on the TF12 ‒ Largest number of queries run on a single day ‒ Average query time was 0.83 seconds ‒ Longest query time was 47 minutes (Informatica) ‒ Peek of 8,243 queries completed per hour ‒ Host CPU utilization never went above 10% ‒ SPU (blade) utilization never went above 17% • EDW conversion PoC ‒ 4.1 million updates ran in 18 seconds ‒ Executed on TF6 ‒ Informatica & post session task CSC Proprietary and Confidential 9/18/2012 9:02 AM 18
- 19. Netezza Overview Bumps along the way • Netezza and Linux upgrades have caused issues • Netezza off-hours support is poor • Technical Account Manager (TAM) knowledge is lacking • Integration error debugging knowledge is lacking • 48 concurrent query limit on each Netezza appliance have caused issues • Service provider requirements not accommodated for ‒ Features lacking for guaranteeing SLA‟s ‒ Limited multi-tenancy features • Netezza documentation is lacking or sometimes incorrect Light at the end of the tunnel • IBM will bring stability to these issues • Client is member of the Netezza Advisory Board to escalate issues CSC Proprietary and Confidential 9/18/2012 9:02 AM 19
- 20. Netezza Overview What’s Next? • More projects being developed on Netezza • Plan Migration of ZEA to Netezza • Look at other systems to migrate (EDW + CDW) • Netezza upgraded in 2012 ‒ 2x performance ‒ 5x or better compression ‒ 4TB/hour load and unload speeds ‒ Concurrent Query limit raised to 100 ‒ Smart Caching ‒ Query Mix: large percentage of short, tactical queries ‒ Granular Zone Maps (potential 24x reduction in reads) CSC Proprietary and Confidential 9/18/2012 9:02 AM 20
- 21. Netezza Overview What’s Next? • Informatica Pushdown Optimization (PDO) ‒ Informatica mapping converted to SQL ‒ SQL is submitted to Netezza ‒ All data stays on Netezza ‒ All processing happens within Netezza • PDO example – Creation of Combined Mart ‒ Before: DB2 only method – 18 to 20 hours ‒ After: Netezza conversion – 6 to 8 hours ‒ Future: Netezza with PDO – expected 1 to 2 hours CSC Proprietary and Confidential 9/18/2012 9:02 AM 21
- 22. Netezza PoC capability CSC Nordics
- 23. Scandihealth POC Setup Selected Near Real Tables Push Time Xfer Reports Changes Mainframe + DB2 CDC server Netezza Cognos + Targit CSC Proprietary and Confidential 9/18/2012 9:02 AM 23
- 24. Business case Optimize hospital beds • Calculate availability of hospital beds more frequently in order to optimize utilization • Utilization of beds is an important KPI for a hospital • Before – At best the customers would get calculations once a day • After – Calculations performed every 15 minutes CSC Proprietary and Confidential 9/18/2012 9:02 AM 24
- 25. Findings All business requirements met • Data is updated in near real-time • Queries can now be very frequent • No incremental MIPS cost for queries and reports • The load CDC imposed on mainframe MIPS was insignificant CSC Proprietary and Confidential 9/18/2012 9:02 AM 25
- 26. Thank You Q&A