Reconciling your Enterprise Data Warehouse to Source Systems
- 1. 1 © 2014 Method360, Inc. All rights reserved. Reconciling Your Enterprise Data Warehouse to Source Systems PRESENT ED BY: | | Santosh Pappachan Scott Hertel Sunny Hiran James Hanck
- 2. ©2014 Method360, Inc. All rightsreserved. 2 Method360 Method360 RECONCILING YOUR EDW Method360 Presenters JAMES HANCK Vice President Enterprise Information Management SANTOSH PAPPACHAN Director Business Intelligence Solutions SUNNY HIRAN Director Quality Assurance SCOTT HERTEL Senior Lead Enterprise Information Management
- 3. 3 © 2014 Method360, Inc. All rights reserved. RECONCILING YOUR EDW MMeetthhoodd336600 • What does it means to reconcile? Why do we need to reconcile? • Traditional method of reconciling between ECC and BW • SAP Information Steward Data Insight to reconcile between any source system and any data warehouse • Leveraging Automation to implement data reconciliation as part of a quality assurance program • Key Learnings and Q&A Agenda
- 4. ©2014 Method360, Inc. All rightsreserved. 4 Method360 Method360 RECONCILING YOUR EDW Reconciliation = Consistency Between Source and Target Reconciling your EDW is the process of proving your EDW data is consistent with its source of record. What Does It Mean To Reconcile? =
- 5. ©2014 Method360, Inc. All rightsreserved. 5 Method360 Method360 RECONCILING YOUR EDW Foundational Point Why do we want to reconcile between our operational and BI systems? Investment in BI Organizations invest large sums of time and money in BI systems with the expectation that after deployment the systems will be adopted and project objectives to enable a system to provide actionable answers with enterprise data will be achieved Mistrust and Poor Adoption Unfortunately, when inconsistent data shows up between your operational and BI systems any trust by your user base is put at risk and provides a the potential of users continuing to rely on legacy reporting systems or worse start creating new tactical – “one off” reporting solutions resulting in more cost, duplication of data, and risk of more inconsistent data Why Reconcile?
- 6. ©2014 Method360, Inc. All rightsreserved. 6 Method360 Method360 RECONCILING YOUR EDW Some Common Causes of Data Warehouse Inconsistencies •Source systems change and impact to extraction logic from down stream integration overlooked •Out of date reference data causing data to be excluded or included incorrectly •Aggregation logic using the incorrect fields to aggregate on •Invalid transformation within design of integration logic •Data load exceptions going undetected and/or nor be correctly resolved Root Cause?
- 7. 7 © 2014 Method360, Inc. All rights reserved. RECONCILING YOUR EDW MMeetthhoodd336600 • What does it means to reconcile? Why do we need to reconcile? • Traditional method of reconciling between ECC and BW • SAP Information Steward Data Insight to reconcile between any source system and any data warehouse • Leveraging Automation to implement data reconciliation as part of a quality assurance program • Key Learnings and Q&A Agenda
- 8. ©2014 Method360, Inc. All rightsreserved. 8 Method360 Method360 RECONCILING YOUR EDW Type I: Summary Reconciliation Data Source Validation Method Type II:Source to Target Record Counts Validation Type III: Using LSA++ Architecture to Reconcile BW Data BW Data Reconciliation Using BW Tools
- 9. ©2014 Method360, Inc. All rightsreserved. 9 Method360 Method360 RECONCILING YOUR EDW Data Reconciliation Process Reconciliation DataSource Method Type I Recon Process Acquisition/ Harmonization Layer Reporting Layer Propagation Layer Control Database Summary ECC Table Existing ECC Source Foundation DSO (Or Virtual Provider) Foundation DSO Summary Recon ODS or Virtual Provider Master Data Reportable ODS Reportable ODS BW Control Detail Write Record Count/Cutoff Datetimeto Control Table Error Reports Recon Reports Summary or Reconciliation Data Source Existing Data Source ECC Application Specific Logic
- 10. ©2014 Method360, Inc. All rightsreserved. 10 Method360 Method360 6:30 AM Extract ECC Staging of Logistics Documents from ECC to BW Time = 6:30 AM Request = REQ1 Record Count = 50 Update Control Table Update Control Table N1 Records Extracted Request Number REQ1 N2 Records Extracted Request Number REQ2 N3 Records Extracted Request Number REQ3 2:30 PM Extract BW 4:30 PM End of Day Extract Time = 2:30 PM Request = REQ2 Record Count = 100 Time = 4:30 PM Request = REQ3 Record Count = 200 Update Control Table Acquisition/ Harmonization Layer BW Control Table Error Log Reporting Layer Error Reports Compare The record Counts in Foundation Layer DSO With the Control Table for each Request id Propagation Layer Information about N1 + N2 + N3 records with respective request numbers Foundation Layer DSO with request numbers in each record RECONCILING YOUR EDW Type II Model for BW BC – Record Count (Cont.)
- 11. 11 © 2014 Method360, Inc. All rights reserved. RECONCILING YOUR EDW MMeetthhoodd336600 Recon Process Acquisition Layer Propagation Layer Reporting Layer Control Database Foundation Layer DSO Architected Data Marts Architected Data Marts Error Reports Recon Reports BW Data Source ECC SAP BW using LSA++ (With HANA) ECC Source OPEN ODS Composite /Hybrid Provider Master Data SAP ECC Source (Tables, Function Module, ABAP Query etc) Reconciliation using LSA++ Logic (With HANA)
- 12. ©2014 Method360, Inc. All rightsreserved. 12 Method360 Method360 RECONCILING YOUR EDW HANA Optimized BI Content Type III: Using LSA++ Architecture to Reconcile BW Data •SAP HANA-optimized BI Content combines enterprise data warehouse capabilities of SAP NetWeaverBW running on SAP HANA with explorative and interactive real-time analytics using the SAP HANA in- memory database. The new data model follows the recommendations for the layered scalable architecture LSA++, which is optimized for BW running on SAP HANA. •The HANA Optimized BI Content comes packaged with Reconciliation Queries which can be activated to deliver quick Reconciliation reports.
- 13. 13 © 2014 Method360, Inc. All rights reserved. RECONCILING YOUR EDW MMeetthhoodd336600 • What does it means to reconcile? Why do we need to reconcile? • Traditional method of reconciling between ECC and BW • SAP Information Steward Data Insight to reconcile between any source system and any data warehouse • Leveraging Automation to implement data reconciliation as part of a quality assurance program • Key Learnings and Q&A Agenda
- 14. ©2014 Method360, Inc. All rightsreserved. 14 Method360 Method360 RECONCILING YOUR EDW Key Capabilities: •Empowers business users to define, measure, and improve data quality throughout the enterprise •Central location to identify, track, and remediate data quality issues •Integrate a wide variety of enterprise data sourcesSAP Applications (ECC, BW, etc.) Relational databases (including SAP HANA) ODBCFlat files •Overcomes traditional obstacles of relying on highly manual efforts and the resulting data latency What is Information Steward?
- 15. ©2014 Method360, Inc. All rightsreserved. 15 Method360 Method360 RECONCILING YOUR EDW SAP Information Steward’s Data Insight BUILDING BLOCKS OF DATA INSIGHT Centrally monitor and reconcile data across your heterogeneous enterprise systems Create custom views built across source system and data warehouse tables Create validation rules to identify data reconciliation exceptions Assign Validation Rules for re-use across multiple source tables and systems Configure data quality scorecards to present graphical interactive dashboards of data quality exceptions Schedule and automate the execution of Validation Rules at appropriate intervals Save records that fail Validation Rules to database tables Validation Rules Validation Rule Bindings Data Quality Scorecards Custom Views Score Tasks Failed Data Tables Source Data Tables
- 16. ©2014 Method360, Inc. All rightsreserved. 16 Method360 Method360 RECONCILING YOUR EDW Data Insight Workspace •Central location for users to monitor and reconcile enterprise data •Intuitive graphical navigation
- 17. ©2014 Method360, Inc. All rightsreserved. 17 Method360 Method360 RECONCILING YOUR EDW Data Quality Scorecards •Dashboard layout with traffic light coloring for scores •Intuitive hierarchical navigation across data domains, quality dimensions, validation rules, and down to exception records SAP Information Steward’s Data Insight
- 18. ©2014 Method360, Inc. All rightsreserved. 18 Method360 Method360 RECONCILING YOUR EDW Validation Rules •Define rules for reconciling data between source and DW •Save data exceptions to database tables for remediation processes SAP Information Steward’s Data Insight
- 19. ©2014 Method360, Inc. All rightsreserved. 19 Method360 Method360 RECONCILING YOUR EDW Data Remediation with Data Insight •Information Steward Score Task runs: ‒Pulls data from source ‒Applies Validation Rule to data ‒Writes failed data records to database tables •Data Remediation Processes use failed data tables: ‒Automated and Manual data cleanup processes
- 20. 20 © 2014 Method360, Inc. All rights reserved. RECONCILING YOUR EDW MMeetthhoodd336600 • What does it means to reconcile? Why do we need to reconcile? • Traditional method of reconciling between ECC and BW • SAP Information Steward Data Insight to reconcile between any source system and any data warehouse • Leveraging Automation to implement data reconciliation as part of a quality assurance program • Key Learnings and Q&A Agenda
- 21. ©2014 Method360, Inc. All rightsreserved. 21 Method360 Challenges RECONCILING YOUR EDW There are inherently complex challenges associated with validation of your EDW given the need for accurate and meaningful data, as well as enforcing data quality standards that impact key systems. Challenges include: •The need to satisfy overarching Data Governance and Data Quality goals and objectives •Extensive manual test labor required to perform complex validations at significant cost and different levels of testing ―Data quality ―Data reconciliation ―Data security •ETL Processes •Data Warehouses (Data marts, OLAP, OLTP and Predictive Analysis) •Understanding Reporting and Data Analysis needs •Test Data and Environment Management •Performance and Response-time validation •Dashboards and Scorecards
- 22. 22 © 2014 Method360, Inc. All rights reserved. Best Practices EDW QA Strategy Method360 RECONCILING YOUR EDW Data Governance & Data Quality Alignment Environment & Data Planning Execution Planning Strategy Building a foundational strategy that will allow your organization to be agile, scalable and repeatable is critical to success Ongoing Monitoring & Support Manual, Automated, Security, Performance Data Quality Reviews & Reporting QA Framework & Scalable Processes Data Quality - Risk Assessment & Review Scope Finalization / Prioritization/ Resourcing Planning EDW Execution Strategy Building a foundational strategy that will allow your organization to be agile, scalable and repeatable is critical to success Tools & Validation Requirements
- 23. ©2014 Method360, Inc. All rightsreserved. 23 Method360 Performance Testing Services RECONCILING YOUR EDW
- 24. 24 © 2014 Method360, Inc. All rights reserved. RECONCILING YOUR EDW MMeetthhoodd336600 • What does it means to Reconcile? Why we need to Reconcile?? • Traditional method of reconciling between ECC and BW • SAP Information Steward Data Insight to reconcile between any source system and any data warehouse • Leveraging Automation to implement data reconciliation as part of a quality assurance program • Key Learnings and Q&A Agenda
- 25. 25 © 2014 Method360, Inc. All rights reserved. Key Learnings and Q&A RECONCILING YOUR EDW MMeetthhoodd336600 • Why reconcile? To have a positive return on investment you need accurate data. • Traditional method of reconciling between ECC and BW: Data Reconciliation needs to be incorporated as a Design Principle and part of the original deliverable and should not be an after thought when inconsistencies start showing. • SAP Information Steward Data Insight to reconcile between any source system and any data warehouse: Information Steward can connect to any data source, monitor data consistency on a scheduled basis and alert groups that an inconsistency exists. • Leveraging Automation to implement data reconciliation as part of a quality assurance program: Adopt a risk based approach to ensure alignment on scope and prioritization of validation areas. Any Questions
- 26. ©2014 Method360, Inc. All rightsreserved. 26 Method360 RECONCILING YOUR EDW Contact Info BUSINESS DEVELOPMENT KATIE COBURN 214.506.6046 kcoburn@method360.com SALES PRACTICES MANAGED SERVICES, PMO AND DELIVERY ERIK AXTON 925.389.6708 eaxton@method360.com BUSINESS INTELLIGENCE BLAKE DUGGAN 415.548.0762 bduggan@method360.com ENTERPRISE INFORMATION MANAGEMENT JIM HANCK 415.694.8229 jhanck@method360.com QUALITY ASSURANCE SUNNY HIRAN 415.535.5453 shiran@method360.com FINANCIAL MANAGEMENT GREG STIMPSON 925.640.8785 gstimpson@method360.com
- 27. ©2014 Method360, Inc. All rightsreserved. 27 Method360 Thank You RECONCILING YOUR EDW END-TO-END SERVICES. HIGH VALUE SOLUTIONS.