Principles of effective software quality management
- 1. WHITEPAPER Principles of Effective Software Quality Management Author: Neeraj Tripathi
- 2. Table of contents: 4 5 5 6 7 8 9 10 11 12 Customer Experience Enabling Environment Establish Vision and Direction Active QA Involvement Raise Innovation and Collaboration Nurture Empowerment Repeatable & Reusable Time to Market Summary Author Bio.
- 3. Managing large quality assurance (QA) teams requires a focused, effective strategy to ensure your customers are best-served by the software you’re building for them. Having a well-defined process framework in place will give you a starting point for a planning roadmap, action items to implement, and metrics to measure your progress. This article lays out one such framework I’ve seen success with over my career: The CERT framework. CERT is four distinct themes designed to guide you to the most effective quality process for you and your customers. The themes are: Using these themes will help your organization improve its quality effectiveness and your customers’ satisfaction. This culture change isn’t just for your software delivery teams—it’s needed across your entire business. Customer Experience Enabling Environment - Active QA Involvement - Innovate & Collaborate - Nurture Empowerment Repeatable and Reusable Time to Market 3
- 4. Quality software doesn’t just focus on “meeting specifications.” Jerry Weinberg, a longtime recognized thought leader in the quality consulting space, simply puts it as “quality is something of value to someone.” Weinberg’s succinct phrasing may seem too vague for some, but the concept’s solid: Users have different features that are important to them, and their idea of quality can vary immensely between them. Instead, Customer Experience means understanding the customers’ business needs, success criteria, their competitors, and even going so far as taking part in their social media activities. Are your customers’ referrals increasing? Is their own customer retention improving? In this world of the Internet Of Things (IoT), usability, security & performance are key to attract & retain customers. You’ll need to work with your customers to ensure you’re meeting their expectations on those key areas. Both topics are especially critical as end users are harsh in their social media channels for applications that are insecure or slow. These are strategic, high-level actions but there are plenty of things at the tactical level for organizations to focus on as well. Quality/Testing teams should be taking proactive steps such as engaging customers in testing strategy and plans. Customers ought to be part of building entry and exit criteria for testing, plus they should be part of the review and sign off teams for acceptance. Tight feedback loops are critical for any software delivery process. Shortening those loops by actively engaging your customers can sometimes be difficult. It’s critical to avoid customer involvement becoming a bottleneck for your delivery. Part of making this effective is avoiding waste or unnecessary churn. Don’t load up the customer with needless activities or information. Instead, focus on high-value activities such as defining acceptance criteria at an appropriate time, keeping your regression suite as automated as possible, provisioning beta/customer test environments and using them to gain customer insight, and working to ensure the entire team is focused on delivering quality together. You’ve got to establish a regular cadence of customer collaboration if you’re going to build that great experience for your customer. Get to know them personally, not just through sterile conference calls and emails! Customer Experience “Your most unhappy customers are your greater source of learning.” - Bill Gates Customer An unwavering focus on Client Satisfaction People Focus Igniting Minds to enable knowledge based eco system Relationship Focus Proactive & healthy client relationships for enduring process Technology Focus World’s best technology @ the desktop of the client Process Focus Global standards on process management 4
- 5. Enabling Environments Incubation ImmersionEngagement Application Verification “Quality means doing it right when no one is looking.” - Henry Ford Enabling Environment Any significant software project involves a wide range of people moving in a number of different directions. QA leaders should establish the purpose and direction of the organization. A fundamental aspect of this involves creating and maintaining the environment in which people can become fully involved in achieving the organization’s objectives. Meeting those objectives and creating an enabling environment requires leaders to think about several different supporting themes: • Establish Vision and Direction • Active QA Involvement • Raise Innovation and Collaboration • Nurture Empowerment Establishing a common vision is critical to focus your organization on your clients’ needs. Vision isn’t about the number of test cases, detailed specifications, or test coverage. Vision needs to define success for the organization—and that success needs to closely align with business problems at a strategic level. A broad vision isn’t enough. You also need to look to short- and long-term goals that establish a path to your vision and build enabling environment. Moreover, you’ll need to look to reasonable measurements to ensure the organization is aligning with goals. For example, if you’re emphasizing shifting quality “to the left” then you’ll need to look at suitable early stage metrics such as the number of requirement changes you’re making in planning meetings with your stakeholders. (Early conversations clarifying requirements are one of the best ways of preventing versus finding bugs!) Organizations are finally getting rid of old mindsets around testing as a late event versus early, left-shifted activity. They’re also finally getting rid of bad practices that put walls between development and testing. It’s not enough for executives to simply lay out vision. To build enabling environment, something I’ve found very helpful is to identify champions for your vision in all strategic areas. It’s an issue of scale and engagement: champions help spread the word, practices, and effectiveness across the entire organization much faster! Leaders must provide their team member the required resources and training. Leaders also need to provide the freedom to act with responsibility and accountability. It’s also very critical to define challenging but achievable milestones and celebrate success. Establish Vision and Direction 5
- 6. Requirement Phase Testing Considerations & Customer’s Expectation Design Phase Define Test Components & Build Requirements Traceability Development Phase Build Test Components, Test Cases, Data & Continuous Integration (CI) Testing Phase E2E & Business Process Testing, Robust Regression Thankfully our QA industry is shedding ourselves of the “testing is a one-time event late in the delivery cycle.” You need to approach testing as an activity, not an event. Moreover, you need to start testing at the same time as other work starts - as early in the project as possible! Imagine having an impact on customers’ business decisions at product envisioning stages; testers’ input on complexity, risk, and work effort could help stakeholders make more informed decisions. Stakeholders might choose entirely different products or features based on that input. At the bare minimum, testers need to be involved at the requirements phase. That involvement should carry all the way through design, development, and, of course, testing phases. Quality should be measured like cost and schedule throughout the project, using Quality Indicators (QI). QIs should be defined early on to track and take corrective actions. Finding good metrics is crucial to organizations adopting the right practices and behaviors. Here are a few useful metrics that can help ensure your QA is getting the right level of involvement. • Requirements ambiguities: Measures how well your testers are heading off downstream issues. The best bug is one we prevent instead of find! • Requirements changes: Stakeholders are allowed to change their minds—they’re writing the checks, after all! Measuring the number of changes helps teams understand if they’re missing early elaboration conversations. • Defects, reopened defects, amount of rework: These three metrics really help teams understand how well they’re building the software. Rework and re-opened defects are high-impact issues that need to be addressed by the entire team. Bugs are bad enough! Additional rework caused by reopened bugs is even worse and has to be addressed immediately. Tracking these metrics helps emphasize the value of QA involvement through the project lifecycle. It’s a great way to ensure your stakeholders understand the value of skilled testers—and these metrics also help motivate testers by recognizing their value as professional members of the organization. Active QA Involvement “The time to repair the roof is when the sun is shining.” - John F. Kennedy 6
- 7. We constantly hear about innovations in technology and development methodologies. It’s important to remember that QA can’t stand still, either! Constant innovation in QA/testing is necessary to ensure you’re continuing to improve the value you add to your teams and customers. QA resources commonly face challenges around one-dimensional thinking, such as using same set of test sets and environments across the project’s entire lifecycle. These challenges are also due to many QA/testers working on the same product on an ongoing basis. This allows complacency to set in and impede any possible innovation on the job. It is important to break out of these comfort zones periodically and challenge team members to think differently and build better, more efficient processes and tools. Innovation in the testing industry is evolving on several interesting fronts. We’re seeing an explosion of interest in testing methodologies like Exploratory Testing, Session-Based Testing, and new development methodologies like Test Driven Development (TDD) and Behavior Driven Development (BDD). Wonderful advances have been made in tooling and device clouds that help testers speed through accurate, thorough coverage matrices of mobile devices. And of course we’re seeing innovation in how we’re changing our views of metrics to focus on usable information instead of masses of numbers—Quality Indicators (QI) are a perfect example of this as they’re helping project managers and team members focus on eliminating defects early in the lifecycle. One of the best innovations in QA has been the focus on great collaboration, especially early in the lifecycle. The mindset change from testing as a late event to testing as an early activity means testers and developers working together clarifying requirements ambiguities before software’s written. I’m seeing faster defect triage cycles—in some cases immediate fixes!—which means fewer delays and less rework. Although “Agile” is the hot term, and has been for some time, many large organizations still use waterfall, especially for their larger projects. The great thing is, due to QA’s cross-cutting involvement, QA can often be a change agent for improving existing cultures and processes. QAs in a waterfall organization should look to align with the organization’s Software Delivery Lifecycle (SDLC); however, they can work for the active, early collaboration we’ve spoken of earlier in this paper. QA can also help bridge the gap between waterfall and agile with a rapid, robust regression approach to ensure both environments frequently deploy and deliver. Raise Innovation and Collaboration “Our industry does not respect tradition – it only respects innovation.” - Satya Nadella, CEO Microsoft Innovate Embrace Collaborate 7
- 8. Our ultimate asset is Human Resources. Too often we forget humans need recognition and acknowledgement. You need to make sure you’re recognizing accomplishments of your QA team members.. First, recognition is an important reward to those hard-working individuals. Secondly, recognition encourages and reinforces positive behaviors and culture you’re trying to build up in your organizations. Just as importantly, and perhaps more so from the organizational level, you need to share the actual effort and behavior out to the rest of the organization. Promoting effective, desired practices, ideas, and behaviors out helps nurture the broader changes you’re looking to spread in your quest for the most effective quality program you can build. Nurture Empowerment “Great things in business are never done by one person, they’re done by a team of people.” - Steve Jobs Build Pool of Software QA Professionals Reward & Recognition Attract & Retain Talent Culture of QA vs. Testing Accountability & Empowerment 8
- 9. Making processes and activities repeatable and reusable is a critical piece to any effective organization’s culture. Repeatability ensures consistent results in your activities. Repeatability isn’t just about test execution; it’s important across many other areas such as developing effective build pipelines, regular regression cadence, etc., Reusability helps your teams understand the importance of decomposing larger work into smaller pieces—looking for small, granular tasks you can re-use when building larger efforts. Looking at activities and workflows with an eye to reusability has another great outcome: you’re able to identify wasteful processes. Duplicated signoffs, unnecessary coordination steps, even excessive testing suites or cases can be ferreted out as long as teams focus on efficiency. Testers draw this concept directly from the software they’re building in their systems and the code they write to test those systems. Object oriented design is a fundamental approach for modularizing functionality in small, easily maintained, reusable test components. Automating acceptance checks give organizations ready-made regression suites that can be applied as part of a more extensive delivery pipeline. A subset of those suites can run automatically after each build or deployment, giving organizations an immediate “smoke check” test for immediate feedback! It’s not just automated acceptance tests: each task you do in delivering software can likely be broken down in to small tasks that can be scripted or otherwise automated. You can use those reusable blocks in perhaps the most important value proposition for your organization: continuous integration/delivery pipelines. Those pipelines are based off numerous small, reusable and reputable tasks such as • Building the latest version directly from source control • Running automated tests against the build • Deploying the build to an environment—potentially even production! • Running a set of smoke tests • Running the full set of acceptance and regressions suites in a live environment Getting to this point isn’t an easy task—it requires vision, support, and discipline. The payoff is dramatic coverage, efficiency, and value back to the customer! Repeatable & Reusable “We are what we repeatedly do. Excellence then, is not an act, but a habit.” - Aristotle 9
- 10. I’m often asked “When will we be done with testing?” My response is usually “You tell me! We can keep testing forever!” Of course you want great quality for your customers. However, you have to balance driving out quality with production value. Ensure you have a clear picture of your customers’ risk/value concerns. Phase containment is key to reduce the rework, cost and improve quality. It is worth spending time and money to protect and preserve the brand image. One of the biggest blocks to fast delivery is the amount of time spent regression testing. Testing professionals are getting much better about helping teams and customers understand information about a system’s quality state rather than just quality metrics. Developing an effective regression testing strategy ensures high-value features work as expected and high-risk concerns are inspected enough to give the stakeholders confidence to decide whether to release or hold back. Effective regression testing also involves n appropriate amount of automated testing at all levels: unit, integration/API, functional, acceptance, performance, and security. It requires delivery teams to have the skills to create useful, maintainable automation suites that run smoothly in CI/CD pipelines. High performing teams do this by blending roles to create the automation: developers and testers pairing up to create lean, powerful automation that’s focused on covering the right areas. Automation isn’t a final answer to good regression testing, however. Fast, effective regression testing involves skilled testers who understand how to complement automated testing with exploratory, session-based, or other expert manual testing that ensure stakeholders’ needs are met in a rapid fashion. Often organizations want to speed up delivery methods by outsourcing testing. That doesn’t always work. It’s important to not jump into sourcing testing simply because other companies are doing it—that’s fraught with peril! Successful organizations look to long-term goals for sourcing, and take the time to mature their own processes before trying to integrate sourced teams into their delivery pipelines. Time to Market “It takes 20 years to build a reputation and five minutes to ruin it. If you think about that, you’ll do things differently.” - Warren Buffett 10
- 11. Every QA leader and organization needs to focus on being the most effective group delivering the highest value possible. Attempting to do so without a clear roadmap often results in an extraordinary amount of thrashing and wasted effort. Using CERT can help you identify where your broader organization could benefit from improvements. Utilize the four themes as guideposts to narrow your view into specific areas: I hope you’ll find my CERT framework useful for starting your own journey to improving your organizations and teams! Summary Customer Experience Enabling Environment Repeatable and Reusable Time to Market 11
- 12. Author Bio. Information technology leader with significant experience in design/delivery of cost-effective, high quality solutions that generate business value. Key capabilities include strategic/long term planning, establishing business partnership, building and managing large IT program & QA CoEs Unique ability to envision, develop and execute organizational strategies that drives growth, productivity, cost reduction and advance technology capabilities. Customer-centric with the ability to initiate profitable alliances with global vendors and suppliers. Specialties: • Organization Transformation, • Building and Leading Global COEs • Program/Project Management, • Business Development & Relationship Management, • IT Governance & Vendor Management, • Team Building & Leadership Neeraj Tripathi 12