State of the Market - Data Quality in 2023
0 likes62 views
The document discusses the challenges organizations face regarding data quality in 2023, emphasizing the issues of data fragmentation and complexity. It highlights the significant costs associated with poor data quality and the necessity for effective data validation to enhance decision-making and customer satisfaction. Additionally, it covers the growth of the DevOps market, its goals, and the importance of continuous testing to optimize data management practices.
State of the Market - Data Quality in 2023
- 1. Bill Hayduk CEO State of the Market: Data Quality in 2023
- 2. a software division of Statistics & Expert Opinions State of the Market: Data Quality
- 3. a software division of Study Driving Business Value from Data in the Face of Fragmentation and Complexity Organizations Face Data Fragmentation and Complexity More volume in more sources, spread across more clouds makes it difficult to discover, manage, and control data. State of the Market: Data Quality
- 4. a software division of 79% of organizations have >100 data sources and 30% have >1,000” survey of global Chief Data Officers (Dec. 2021) Study State of the Market: Data Quality
- 5. a software division of Study of organizations have been able to standardize across all data management functions State of the Market: Data Quality
- 6. a software division of Report: How to Improve Your Data Quality Poor data quality costs organizations an average $12.9 million, increases the complexity of data ecosystems and leads to poor decision-making. Good quality data provides better leads, better understanding of customers and better customer relationships. Data quality is a competitive advantage that Data & Analytics leaders need to improve upon continuously.” By the end of 2022, 70% of organizations will rigorously track data quality levels via metrics, improving it by 60% to significantly reduce operational risks and costs.” State of the Market: Data Quality
- 7. a software division of State of the Market: Data Quality World Quality Report 2022 – 2023 • from • WQR’s the industry’s largest research study • Provides the most comprehensive assessment of the current state of quality engineering practices o from around the world o across different industries • Survey population includes: o > 1,750 senior executives globally o across multiple sectors o from 32 countries
- 8. a software division of 89% agree that a robust data validation capability will not only improve efficiencies in terms of time and resources but, more importantly, will help to improve business decision-making.” 88% agree that a robust data validation capability has a direct impact on customer satisfaction and accuracy of insights and that it will help to boost business profitability” Executive Survey State of the Market: Data Quality
- 9. a software division of World Quality Report Survey State of the Market: Data Quality
- 10. a software division of World Quality Report Survey State of the Market: Data Quality
- 11. a software division of Key Observations • An increasing number of organizations see data validation as an important task to be carried out for better decision-making. • Most organizations agree that effective data validation has a direct impact on customer experience and business outcomes. Key Recommendation • Invest in skills and the right level of thought leadership to get the best results and outcomes from data analytics. Key Observations & Recommendation State of the Market: Data Quality
- 12. a software division of QuerySurge Prospect Survey Survey population includes: • 1,090 managers & practitioners • across 13 industries • from 45 countries • looking for an automated Data Validation solution State of the Market: Data Quality
- 13. Are you currently performing data validation & ETL testing? How are you currently performing it? Which Use Case are you interested in? a software division of Manual Data Sampling: Excel / Utility: Minus Queries: ETL Testing solution: DevOps for Data: State of the Market: Data Quality
- 14. a software division of State of the Market: Data Quality
- 15. The Key Focus of Data Quality in 2022 - 2023: DEV OPS DevOps for Data & Continuous Testing State of the Market: Data Quality
- 16. Some Descriptions: State of the Market: Data Quality
- 17. William Blair: Equity Research The most comprehensive report on DevOps • Update on DevOps Market Consolidation • Where Infrastructure and DevOps Meet • Revisiting the CD Bottleneck: Balancing Speed and Risk With Progressive Delivery • Software Testing Coming Into Its Own • Preventing the Next Sunburst Attack: Shifting Security Further Left • Dawn of a New Category: DevOps Management • You Can Be a Developer Too: The Low Code/No Code Boom5 • Key Players of Scale in the DevOps Market State of the Market: Data Quality
- 18. Source: Markets And Markets, William Blair What % of your apps are developed & deployed using DevOps? Most enterprises have deployed < 50% of their apps using DevOps, as companies struggle to scale their DevOps models. State of the Market: Data Quality
- 19. Sources: IDC U.S. DevOps Survey and William Blair Equity Research What are the core challenges to building DevOps capabilities at your organization? State of the Market: Data Quality
- 20. Source: Markets And Markets, William Blair Optimizing Test Automation should = DevOps There are 6 primary goals of DevOps: 1) Increase speed of development and release processes 2) Make builds more reliable 3) Reduce turnaround for new features and bug fixes 4) Provide greater scalability of applications and infrastructure 5) Increase security by automating compliance practices 6) Improve collaboration throughout the development lifecycle State of the Market: Data Quality
- 21. State of the Market: Data Quality “The DevOps Market Remains Highly Fragmented and Relatively Immature” • $8.5 billion DevOps market (IDC) • Immense 28% growth rate • IDC estimates a $28 billion market in 2024 • Continuous testing & DevOps will remove testing bottlenecks & clear path for Continuous Delivery (CD) - Source: Agility Now 2.0 - Key Trends Shaping the DevOps Landscape, William Blair Equity Research QuerySurge seamlessly integrates with all other solutions in the DevOps marketplace Key Players of Scale now in DevOps Market
- 22. a software division of ETL Test Tool Vendors and DevOps Tools in the market that are fully DevOps-enabled with a robust API State of the Market: Data Quality
- 23. a software division of Patterned after CMMI Source: QuerySurge Data Maturity Model for Validation & Testing Data Testing & Commercial Solution Level 4 Fully repeatable test automation, centralized reporting, training, knowledge base, professional support, partner ecosystem. DevOps-for-Data Testing Solution Level 5 CI/CD pipeline test automation, RESTful API, minimum human interaction, predictive data issues, auditable history & automated reports. Business value is fully supported by management. 1% of companies 9% of companies Data Sampling & Excel Level 1 Using Excel spreadsheets to compare a small % of data sets visually. Ad-hoc reporting. Doesn’t scale & isn’t repeatable. Data Compares & Minus Queries Level 2 Utilizing SQL editor & minus queries to test data. Ad hoc reporting. Data Testing & Home-Grown Framework Level 3 Built internally or by a system integrator. Expensive upfront costs to build and maintain. No documentation, limited or custom reporting. 90% of companies on the Data Maturity Level
- 24. In most instances, QuerySurge is the only data validation & ETL testing tool mentioned… State of the Market: Data Quality
- 25. QuerySurge’s DevOps for Data features provides testers with the capability to dynamically generate, execute, and update tests and data stores utilizing API calls • It’s the first full DevOps for Data solution • improve their data quality at speed QuerySurge DevOps for Data integrates with: • Continuous integration/ETL solutions • Automated build/release/deployment solutions • Operations and DevOps monitoring solutions • Test management/issue tracking solutions • Scheduling and workload automation solutions 60+ API calls with almost 100 different properties that users can utilize to retrieve, edit, update, or delete information. QuerySurge DevOps for Data
- 26. a software division of QuerySurge provides two different ways to trigger API calls: Command Line RESTful – Documented with Swagger QuerySurge DevOps for Data
- 27. a software division of How to implement… The QuerySurge team has expanded our REST API. Almost all UI commands can now be executed through API request: QuerySurge Focus for 2023
- 28. Test Suite Results Individual Test Results Source and Target Data Failed Record Data Test Suite Execution Status Retrieve QueryPairs Create / Modify / Delete Datastore Connections Test Suites Staging Tables Query Snippets Staging Queries Design, execute & analyze QuerySurge operations externally from the user interface QuerySurge Server Run Test Scenario Kill Test Scenario Execution Run QueryPair QuerySurge DevOps for Data
- 29. Unlimited Use Cases…. Tests are automatically initiated after ETL execution completes and conditional logic is applied based on specific results When new environments are deployed, new connections are automatically created, and tests are duplicated & executed Tests automatically analyze and detect changes to the data store schemas and those changes are reflected in the tests. QuerySurge DevOps for Data
- 30. a software division of The QuerySurge team will be releasing new training courses and new certifications that have a specific focus on DevOps How we are going to help you get there… Use discount code TESTGUILD2023 Learn. Earn. Inform. • Learn DevOps for Data Testing • Earn your digital badge • Inform your social network of your new skills • Receive access to free self-paced training • take free certification exams • receive Labor Market insights on jobs, locations, salaries FREE Training & Certs QuerySurge Focus for 2023
- 31. We’re seeing a huge culture shift…in business, people want their code to be delivered immediately, and good-to-go. This is why we’re seeing a move toward continuous testing, where we’re aiming to prevent defects, rather than find them” - The World Quality Report, 2021-2022 State of the Market: Data Quality
- 32. 32 a software division of State of the Market: Data Quality www.QuerySurge.com