Leveraging HPE ALM & QuerySurge to test HPE Vertica
1 like35,790 views
The document discusses a webinar on leveraging Application Lifecycle Management (ALM) and QuerySurge for testing data warehouses, particularly focusing on Vertica. Key points include the integration of ALM with QuerySurge for data validation, improving data quality, and optimizing costs and risks involved in data processes. It outlines the features of QuerySurge, such as automation of the testing cycle, support for various data sources, and aids in efficient data management and reporting.
Leveraging HPE ALM & QuerySurge to test HPE Vertica
- 1. WEBINAR Leveraging ALM & QuerySurge to test Vertica Bill Hayduk Founder & CEO RTTS Chris Thompson Senior Domain Expert data quality & testing QuerySurge™ a software division of
- 2. • a custom view of ALM to store source-to-target mappings • data validation tests in QuerySurge • The execution of QuerySurge tests from ALM • The results of data validation tests stored in ALM • custom ALM reports that show data validation coverage of Vertica • how we improve your data quality while reducing your costs & risks QuerySurge™ a software division of WEBINAR Here is what you will see:
- 3. QuerySurge™ About FACTS Founded: 1996 Location: New York, NY Software Division: QuerySurge Customer profile: • Fortune 1000 • 700+ customers Alliance Partners: IBM, Microsoft, HPE, Oracle, Teradata, HortonWorks, Cloudera, MongoDB RTTS is the parent company of QuerySurge and is the premier software & services firm in the QA & Testing space that specializes in test automation a software division of
- 4. Testing a Data Warehouse QuerySurge™ a software division of
- 5. Business Intelligence & Analytics CxOs are using Business Intelligence & Analytics to make critical business decisions – with the assumption that the underlying data is fine. “The average organization loses $14.2 million annually through poor Data Quality.” - Gartner ETL Data Architecture BUSINESS CASE: Executive Office & Critical Data potential problem areas Mainframe ETL
- 6. Extract Data Warehouse & the ETL process Source Data ETL Process Target Data Warehouse/ Data Mart Transform Load Mainframe QuerySurge™ a software division of
- 7. Which Data Goes Where? Source-to-Target Mapping doc Elements of Mapping Document • Attributes of original source data (table name, column name, source type) • Attributes of the target data store (table name, column name, source type) • Transformation rules, if any (aggregation, combining records, etc.) Definition: Source-to-Target Mapping • Specification or requirements doc that shows how data in one system is mapped to another system • Typically created and stored in Word or Excel QuerySurge™ a software division of Issues with Mapping Document & Testing • No easy way to link mappings to tests for tracability • No easy way to store test results • No easy way to report on data testing coverage
- 8. Data Process: Developer & Tester Developer: Codes data movement based on Mapping Requirements Tester: Tests data movement based on Mapping Requirements Data Mart ETL Source Data ETL Process Target DWH Mainframe ETL ETL Process
- 10. About HPEVertica HPE Vertica is one of the most advanced SQL database analytics portfolio built from the very first line of code to address the most demanding Big Data analytics initiatives. Vertica's column-oriented storage approach significantly increases query performance in data warehouses About HPE ALM Application Lifecycle Management (ALM) is the industry standard for modern quality management. It is a centralized management system that provides reporting and traceability throughout the application delivery lifecycle.
- 11. is the leading Data Testing solution for automated testing & validation of Big Data and Data Warehouses QuerySurge Use Cases About QuerySurge
- 12. QuerySurge™ Automate the entire testing cycle Automate the launch, tests, comparison, auto-emailed results Query Wizards - no coding needed Query Wizards can automate about 80% of your tests Test across different platforms EDWH, Hadoop, NoSQL, database, flat file, XML, mainframe Analyze & Collaborate Data Health dashboard, shared tests & auto-emailed reports Verify more data & do it quickly verifies up to 100% of all data up to 1,000 x faster Integrate for Continuous Delivery (DevOps) Integrates with most Build, ETL & QA management software a software division of the QuerySurge advantage
- 13. Web-based… Installs on... Linux Connects to… …and any JDBC compliant data source QuerySurge™ QuerySurge Controller QuerySurge Server QuerySurge Agents Flat Files a software division of QuerySurge Architecture
- 14. SQL HQL SQL HQL SQL SQL QS pulls data from data sources QS pulls data from target data store QS compares data quickly QS generates reports, auto-emails Reports, Data Health Dashboard, auto emails Source Data • Databases • Data Warehouses • Data Marts Flat Files • Fixed Width • Delimited • Excel Mainframe Target Data Big Data stores • Hadoop • NoSQL Data Warehouses How QuerySurge Works
- 15. QuerySurge™ QuerySurge Modules Design Library SchedulingRun Dashboard Deep-Dive Reporting Data Health Dashboard Query Wizards a software division of
- 16. Fast and Easy. No programming needed. QuerySurge™ • Perform 80% of all data tests with no SQL coding • Opens up testing to novices & non-technical members • Speeds up testing for skilled coders • provides a huge Return-On-Investment a software division of QuerySurge Modules
- 17. QuerySurge™ a software division of QuerySurge Modules
- 18. Design Library • Create custom Query Pairs (source & target SQLs for tests that have transformations) Scheduling Build groups of Query Pairs Schedule Test Runs • Run immediately • Run at set date/time • Have event kick it off ™ a software division of QuerySurge Modules
- 19. Deep-Dive Reporting Examine and automatically email test results Run Dashboard View real-time execution Analyze real-time results ™ a software division of QuerySurge Modules
- 20. QuerySurge™ • view data reliability & pass rate • add, move, filter, zoom-in on any data widget & underlying data • verify build success or failure a software division of QuerySurge Modules
- 21. QuerySurge™ Drive QuerySurge execution from your Test Management Solution See QuerySurge Pass/Fail results in your Test Management solution Click link to drill into detailed results in QuerySurge Integration with leading Test Management Solutions • HPE ALM (Quality Center) • Microsoft Team Foundation Server • IBM Rational Quality Manager a software division of QuerySurge Test Management Connectors
- 22. QuerySurge & DevOps: Continuous Integration QuerySurge™ Automated Testing Automated Reporting Automated Launch Data Integration/ ETL solutions QuerySurge™ and many others… email report and many others… QuerySurge™ Build/Configuration solutions email report Test Management solutions QuerySurge™ email report a software division of
- 23. USE CASE QuerySurge™ Testing VERTICA using ALM & QuerySurge a software division of
- 24. USE CASE: Data Process for Developer & Tester Developer: Codes data movement based on Mapping Requirements Tester: Tests data movement based on Mapping Requirements ETL Source Data ETL Process Target DWH
- 25. Target Data ALM Custom-built Mapping Requirements SQL SQL SQL SQL SQL SQL Schedule ALM to kick off QuerySurge through our API QuerySurge automatically reports back all data failures ALM Custom-built Issue Tracking & Coverage reports QuerySurge runs unattended • executing all tests • comparing all data • compiling reports Testing VERTICA using ALM & QuerySurge Source Data
- 26. Custom-built Source-to-Target Data Mapping report Target HP VerticaSource MySQL
- 27. Custom-built Data Mappings Module
- 28. Custom-built Data Mapping Info tab: Data Specification details Source MySQL Target HP Vertica
- 29. Custom-built Mapping Summary Coverage by Table
- 30. 2/9/2017 30 Custom-built Mapping Summary Coverage by Status
- 31. Demonstration built by QuerySurge™ Chris Thompson Senior Domain Expert data quality & testing a software division of Want to see the video of the webinar? Go to: https://youtu.be/bKtUPtva9cE QuerySurge™