Test Automation for Data Warehouses
2 likes4,963 views
This document discusses challenges and opportunities in automating testing for data warehouses and BI systems. It notes that while BI projects have adopted agile methodologies, testing has not. Large and diverse data volumes make testing nearly infinite test cases difficult. It proposes a testing lifecycle and V-model for BI systems. Automating complex functional tests, SQL validation, reconciliation, and test data generation can help address challenges by shortening regression cycles and enabling continuous testing. Various automation tools are discussed, including how they can validate ETL processes and reporting integrity. Automation can help complete testing and ensure data quality, compliance, and performance.
Test Automation for Data Warehouses
- 1. Automade Test Automation Data Vault and Data Warehouse Automation 9th of December Stefaan De Vos
- 3. Challenges • Lead times of BI / Analytical projects are ever decreasing • Tools • Automation (Wherescape...) • Virtualization (Denodo, CIS...) • Appliances (Netezza, DATAllegro...) • Methodologies • Data Vault • Agile DWH BI went Agile, but Testing didn’t
- 4. Challenges • Data • Larger data volumes • IoT • Unstructured data / poor data quality • Social networks • No relevant test data sets available • The number of possible test cases are near infinite Data Volume & Quality
- 5. Challenges Review requirements Review requirements Define testing Strategy Define testing Strategy Prepare Test DataPrepare Test DataEntry / Exit criteria Design Test Cases & Scripts Design Test Cases & Scripts Configure the test environment Configure the test environment Prepare & Design Agree on Entry / Exit test criteria Agree on Entry / Exit test criteria Integration testing Integration testing Performance testingPerformance testing Acceptance testingAcceptance testing - Basic testing - DI jobs accessible - Reports accessible - Cleansed - Complete - Correct - Integrated - Valid reports - Relevant data - Available data - Consistent data - Accessible - DI & BI integration - Full test cycles - Check NFRs - Scalable - Check SLAs - Peak user - Peak loads - Functional - User - Production - SME - End user Construct Defect metrics review Defect metrics review Performance statistics Performance statistics Lessons learntLessons learnt Process & Quality Improvement Accept Data Completeness testing Data Completeness testing BI & Analytical Testing BI & Analytical Testing Smoke test Unit test Smoke test Unit test BI Testing Lifecycle
- 6. Challenges ** Regulatory compliance might require to use the v-model, e.g. validated environments. Functional analysis Requirements functional Non-functional Technical Design Construction Unit Smoke Test Data Completeness BI & Analytical Integration testing Performance User Production AcceptanceValidation Verification BI Testing V-Model
- 7. Challenge • Lots of data • Lots of testing to be done • Little time to do it Summary
- 9. Complete DWH Testing • Complete testing is a must • It requires: • Testing methodology • Right project culture/mindset and organization • Tools Our View
- 10. Complete DWH Testing Database integrity checking. Risk based testing Effective defect management and collaboration End to End Performance testing Adherence to compliance and regulatory standards and…AUTOMATE Critical Success Factors
- 11. Test Automation Leverage social development principles to deepen functionalities Testers - functional testing - regression testing - result analysis Developers / DBAs - unit testing - result analysis Data Analysts - review, analyze data - verify mapping failures Operations teams - monitoring - result analysis Collaboration/WorkflowTestManagement Rational Quality Manager JIRA Team Foundation ServerQuality Center
- 12. Test Automation
- 14. Test Automation Basic functionality • Auto detection of anomalies (or at least prior to being detected by a user) • Targeted regression testing for planned changes • Data error identification errors by comparing (huge) data result sets • Data error detection via a rules engine • In between data layer reconciliation and auditing • Test data generation Additional functionality • No programming or coding • Heterogeneous connectivity • Collaboration and workflow capabilities • Visually attractive development and monitoring environment • Intuitive reports & dashboards Functionality
- 15. Test Automation • Shorten regression cycles • Save report developers time • Test the same data set in less time • Test more • Faster deployment of defect resolution cycle • Faster deployment of enhancements • Less cumbersome upgrades/migrations • Enabler for • Implementing continuous testing • operationalization of testing Benefits
- 16. Test Automation Vendors - tools • ICEDQ • RTTS - QuerySurge Unconnected ETL Source Data Layer Target Data Layer ETL - ELT 1 3 2 Load test data Execute Job externally Extract result 4 Compare & report • Validation of the business rules implemented in ETL processes are assessed by comparing the results against a ground truth. • The ETL processes are executed separated from the test automation tool • Less adequate for multi-step ETL processes. Approach
- 17. Test Automation Vendors - tools • Zuzena Connected ETL • The business rules present in the ETL tool are analyzed and the test results are assessed against the anticipated results. • The test automation tool executes the ETL processes. Approach Source Data Layer Target Data Layer 2 4Load test data Extract result 5 Compare & report 1 Analyze logic 3 Run job
- 18. Test Automation Vendors - tools • Report Valid8tor (BO) • 360Bind (BO) • Integrity Manager (MSTR) • Report Validator - BSP Software (Cognos) Report Integrity validator • Parallel testing of 2 live systems • Comparing against a (historical) ground truth • Comparing against known good baselines Approach Reports Reporting data Layer Load test data Scrape data 4 Compare & report 1 3 Run report 2
- 19. Test automation
- 20. QuerySurge • Integrates with HP QC / IBM RQM / MSFT TFS • Provides collaboration features pulls data from data sources pulls data from target data store compares data quickly generates reports, audit trails reports SQL Design Tests Scheduling Reporting Run Dashboard Wizards Data Health Dashboard • a SQL execution and data comparison tool working against heterogeneous datasources. • No programming needed
- 21. Automade introduction • Automade • Provides an agile answer to the ever-increasing information appetite • By automating the mind-numbing aspects of constructing, maintaining and testing of data warehouses. • Automade is a spinoff of MindThegap and has established partnerships with
- 22. Test automation Thank you for listening, Any questions? Feel free to send questions & feedback to stefaan.devos@automade.be