![Measurement of Software Quality*
The way we choose to measure quality depends on the
view point we take…
•User View: Quality concept can be broken down into
directly measurable attributes
•Manufacturing View: measuring defect counts and
rework costs (cost of defect correction)
•Business View: How the product enables a business to
expand and gain more market share
* Kitchenham, B.; Pfleeger, S.L.; , "Software quality: the elusive target [special issues section]," Software, IEEE ,
vol.13, no.1, pp.12-21, Jan 1996; doi: 10.1109/52.476281
Software Quality - An Elusive Target 15](https://image.slidesharecdn.com/softwarequality-120901204840-phpapp01/85/Software-Quality-15-320.jpg)
Software Quality
- 1. Software Quality: The Elusive Target? Syed Sohail Javaad 28th July 2011
- 2. Presentation Theme • Introduction to Quality • Software Quality • Addressing Software Quality • Final Discussion • Theme for the Term Paper Software Quality - An Elusive Target 2
- 4. Introduction to Quality* • What is Quality? • How do writers define Quality? – Conformance to requirements – Customers’ reactions – Notions of completeness – Notions of cost effectiveness • Defining Quality is a problem. Why? – Ambiguous and subjective. • …lies in the eye of beholder • Depends on who is the respondent? • Depends on requirements and psychology • Quality Vs. Grade * Most of the material in section is from lecture slides of Dr. Darrell Raymond from his course on Quality Systems (Winter Quality at UW). Used with permission Software 2011 - An Elusive Target 4
- 5. Introduction to Quality* • Measurement and Scope: – How much Quality is good enough? – Where to start and where to stop? • Cost-Benefit trade-off – Example: Five 9s • How to achieve Quality? – Is quality about documentation only or standards or processes or everything? • Who is responsible for Quality? * Most of the material in section is from lecture slides of Dr. Darrell Raymond from his course on Quality Systems (Winter Quality at UW). Used with permission Software 2011 - An Elusive Target 5
- 6. Introduction to Quality • Misconceptions* – Quality is only about testing and audits – Quality is the job of Quality department – Even with all these, you may not have actual quality or a quality system * Most of the material in section is from lecture slides of Dr. Darrell Raymond from his course on Quality Systems (Winter Quality at UW). Used with permission Software 2011 - An Elusive Target 6
- 7. Cost of Quality • Cost of providing acceptable quality • Two types of costs: – Conformance costs: defect prevention and inspection – Non-conformance costs: losses from defects (reworks) and product failures • Improvements in quality, therefore leads to lowered cost because of reduced defects, less process failures and opportunity costs Software Quality - An Elusive Target 7
- 8. Returns to Quality • Focusing on quality by just reducing costs may not result in better performance. • Focus on revenue generation is also important Software Quality - An Elusive Target 8
- 9. PMBOK’s Approach Software Quality - An Elusive Target 9
- 10. Software Quality
- 11. Characteristics of Software • Deming: “If you do not have a competitor today, you will get one tomorrow” • Appreciating that software is different because it is characterised by volatility and rapid change • Low barriers to entry in the market • Internet enabled market place provides variety and product knowledge to customers • Producing a product without defects is not sufficient • Knowing the customer and the environment is really important • Cooperation and Collaboration are the keys to success Software Quality - An Elusive Target 11
- 12. What is Software Quality? • A recent survey of Chief Information Officers (CIOs)indicates that ‘‘Improve IT quality” is one of the top concerns facing IT executives • But the question is: What is Software Quality? • “The extent to which software satisfies its users” • It is not only concerned with meeting user requirements but also with criteria such as ease of learning, ease of use, enjoyment to use and so on… Software Quality - An Elusive Target 12
- 13. User Developer Measurable External quality features x yes speed x x yes space x x yes network usage x x yes stability x x yes robustness x x somewhat ease-of-use x subjective determinism x x yes back-compatibility x yes security x difficult power consumption x difficult Internal quality test coverage x yes testability x hard portability x somewhat thread-safeness x hard conciseness x somewhat maintainability x hard documentation x subjective legibility x subjective scalability x somewhat Software Quality - An Elusive Target 13
- 14. Views on Software Quality* • Transcendental view: Ideal • User View: Fitness for purpose; product characteristics; product usability • Manufacturing View: “Did you build the thing right”; believes that process improvement will lead to improved product quality; may lead to standards • Product View: focus on Internal and inherent characteristics of a product eg. quality of code • Value based view: quality is what the customer is willing to pay for; value for money * D. Garvin, “What Does “Product Quality” Really Mean?” Sloan Management Review, Fall 1984, pp. 25-45 Software Quality - An Elusive Target 14
- 15. Measurement of Software Quality* The way we choose to measure quality depends on the view point we take… •User View: Quality concept can be broken down into directly measurable attributes •Manufacturing View: measuring defect counts and rework costs (cost of defect correction) •Business View: How the product enables a business to expand and gain more market share * Kitchenham, B.; Pfleeger, S.L.; , "Software quality: the elusive target [special issues section]," Software, IEEE , vol.13, no.1, pp.12-21, Jan 1996; doi: 10.1109/52.476281 Software Quality - An Elusive Target 15
- 16. Some Measures of Software Quality • Number of pre- and post-release defects • Defect Density • Mean Time Between Failures (MTFB) • Customer Perceived Quality Software Quality - An Elusive Target 16
- 17. Software Quality: Greek Chimera* • Why? – the perceived scale of the problem, – the diversity of quality defects in software, and – the difficulty of factoring high level quality attributes down to tangible properties – Market Structure – Spread of Internet – ?? *R. Geoff Dromey, "Cornering the Chimera," IEEE Software, An Elusive Targetpp. 33-43, Jan. 1996, doi:10.1109/52.476284 Software Quality - vol. 13, no. 1, 17
- 18. Addressing Software Quality Software Quality - An Elusive Target 18
- 19. The Software Development Process Waterfall Software Quality - An Elusive Target 19
- 20. The Software Development Process Agile Software Quality - An Elusive Target 20
- 21. The Software Development Process • Iterative approach should not be reactive and must be carefully planned • Frequent and unplanned iterations may lead to deviations from the desired results Software Quality - An Elusive Target 21
- 22. Addressing Product Quality* • Quality Model Framework: – Links product properties that influence Quality with a set of high-level quality attributes • Quality carrying properties (tangible): – Internal Correctness: Component’s normal form should not be violated eg. Every loop must have a termination point – Contextual Correctness: Avoiding redundancy and repetition – Descriptive: Easy to understand code, variable names etc. • Software Quality Attributes (intangible): – high-level attributes like functionality, reliability, efficiency and maintainability – Process maturity (assuming that it impacts product quality) • Define linkages: – Example: redundancy affects efficiency and maintainability • Implement the model for the key products of software development: Implementation, Design and Requirements *R. Geoff Dromey, "Cornering the Chimera," IEEE Software, An Elusive Targetpp. 33-43, Jan. 1996, doi:10.1109/52.476284 Software Quality - vol. 13, no. 1, 22
- 23. Addressing Product Quality* • Implementation Quality Model: – Role of programming languages; language structure and statements; data (variables, constants etc) • Requirements Quality Model: – Must be complete, precise, consistent, explicit, understandable, implementable, verifiable, traceable and non-redundant – Prototyping may be used to achieve this • Design Quality Model: – The design must show how each of these requirements is to be realized in the context of the overall system: focus on overall architecture – Reduced variety of software structure leads to reduced complexity *R. Geoff Dromey, "Cornering the Chimera," IEEE Software, An Elusive Targetpp. 33-43, Jan. 1996, doi:10.1109/52.476284 Software Quality - vol. 13, no. 1, 23
- 24. Addressing Product Quality • Controlling defects controls cost • But is it possible to prevent defects from passing on to the customer? Maybe. • Is testing sufficient? • What about rapidly changing requirements? • Maybe a process oriented view is better? Software Quality - An Elusive Target 24
- 25. Software Quality - An Elusive Target 25
- 26. Improving Process for Improved Software Quality • Process Improvements: – Total Quality Management – Six Sigma – Failure Mode and Effect Analysis (FMEA) – Design reviews – Voice of the Customers – Continuous Improvement (Including Process Improvements: CMMI, OPM3) – Software Quality Function Deployment (SQFD) Software Quality - An Elusive Target 26
- 27. Software Quality Function Deployment (SQFD) • Focuses on requirements planning phase • Steps: – Customer requirements are solicited – Product specifications are generated using these requirements – Customers are asked to correlate between their requirements and the product specs – Customers are asked to prioritize their requirements – Product specification priorities are developed Software Quality - An Elusive Target 27
- 28. SQFD - Benefits • Improved User and Management Involvement • Shorter development cycle • Better communication between users and developers • Improved quality overall Software Quality - An Elusive Target 28
- 29. Total Quality Management* • Instead of post-process inspection and testing focus should be on continuous Improvement in processes and response to user demands • Improvement in development quality leads to improved product quality • An increase in process standardization and tool capability can significantly increase the level of development quality • Does less reporting of defects by customers necessarily mean higher quality? *Marcus A. Rothenberger , “Total quality in Software Quality - An ElusiveAn empirical study of quality drivers and benefits software development: Target 29 in Indian software projects” Information & Management (December 2010), 47 (7-8), pg. 372-379
- 30. Process Improvement •In assembled goods industry, strong operational excellence in internal product development processes positively impacts market outcomes •In the context of software development in some industries, the relationship between quality and profitability may not be uniform across all customers Software Quality - An Elusive Target 30
- 31. Process Maturity • Increased CMM levels could: • Improved software product quality (less defects) • lead to less operating cost • better product reliability • Adoption of standards by clients resulted in better requirements (reduced requirements volatility) • Result in higher development quality • but… • Reduction in variance in quality resulting in uniformity in quality • doesn’t mean you start producing good software Software Quality - An Elusive Target 31
- 32. Software Validation* • Did you build the right thing? • Why? – Usually required for regulated industries – To ensure design is correct – To reduce defects after changes are made • Done in all phases of software development and everybody participates • Validation is considered costly, brittle and more about documentation • In practice, everybody looks for ways to avoid it * Most of the material in section is from lecture slides of Dr. Darrell Raymond from his course on Quality Systems (Winter Quality at UW). Used with permission Software 2011 - An Elusive Target 32
- 33. Managing Quality on Projects • Quality Management is a key area of project management • Some recent observations: – Release the product, fixes will come later – Minimize the cost at the expense of customer inconvenience Software Quality - An Elusive Target 33
- 34. Role of Quality Standards • There are so many of them • ISO, CMM, CMMI, IEEE, ANSI, AIAA • Used for the certification function • Can help ensure consistent process quality but their impact on product quality is unclear • Can standards be imposed on program behaviour? • Maybe be mandatory for some industries and may also provide legal cover Software Quality - An Elusive Target 34
- 35. Will COTS lead to better Organizational Quality? • Enterprise Software may be of good quality but its implementation might not be • Post-Implementation Processes are more important than the Software itself • But Software quality issues may aggravate the overall implementation • Overall IT Architecture become more important and plays an important role in determining overall quality Software Quality - An Elusive Target 35
- 36. Quality in Outsourcing* • Contracts as a means of managing risk and quality on project • Outsourcing (contracts) helps in improving overall quality • Clients can focus more on quality by outsourcing some of their activities • Fixed Price contracts provide better quality then Time and Material * Gopal, A. and Koka, B. R. (2010), The Role of Contracts on Quality and Returns to Quality in Offshore Software Development Outsourcing. Decision Sciences, 41: 491–516 Software Quality - An Elusive Target 36
- 37. Is Open Source Software (OSS) the Answer* • Does OSS have more Reliability, performance, scalability, security? • Equivalent OSS/FS applications are more reliable • MySQL database (a leading OSS/FS database) had fewer defects than a set of 200 proprietary programs used for comparison • Sites using Microsoft’s IIS web serving software have over double the average time offline than sites using the Apache software • Surveys report that GNU/Linux systems experience fewer viruses and successful cracks. * http://www.dwheeler.com/oss_fs_why.html Software Quality - An Elusive Target 37
- 38. Open Source Software Development Process* * http://cio-nii.defense.gov/sites/oss/Open_Source_Software_(OSS)_FAQ.htm Software Quality - An Elusive Target 38
- 39. Quality In OSS • Open Source Software can be of higher quality* • Users track bugs to their root cause and fix them • Peer Reviews: Developers review each other’s code • Meritocracy • No resource or time pressure * http://www.dwheeler.com/oss_fs_why.html Software Quality - An Elusive Target 39
- 40. Final Discussion Software Quality - An Elusive Target 40
- 41. Quality Issues • A software is not just a piece of code • It requires: • hardware • People • Processes • Policies and • Other software (Operating systems, databases, languages, internet and so on) • Example: Enterprise Software * http://www.dwheeler.com/oss_fs_why.html Software Quality - An Elusive Target 41
- 42. Quality Issues • So Software Quality is not an absolute concept – It is difficult to define… – But easy to blame • Propagation of internet is changing the concept of quality – More and continuous opportunity for testing – Easy to release upgrades – But requirements are also continuously evolving * http://www.dwheeler.com/oss_fs_why.html Software Quality - An Elusive Target 42
- 43. Role of Organizational Politics • Quality Manager is usually – not a position of power within the organization – Someone without a strong sponsor • Is usually viewed as someone – who creates hurdles (and so increases cost) – Who is “not on our side” • Quality Manager has to be more like a PMBOK Project Manager Software Quality - An Elusive Target 43
- 44. Role of Quality Manager • A principled friend • Advisor • Relationship builder • Wise • Empathetic • Positive • And most importantly: political Software Quality - An Elusive Target 44
- 45. Theme for the Term Paper • How has the Concept of Software Quality evolved during the last three decades? • What models and methods for managing quality have evolved? • Is there an improvement in Software Quality? • And is there any empirical evidence for this? Software Quality - An Elusive Target 45
- 46. BACK TO SLIDE 1 Thank you Software Quality - An Elusive Target 46
- 47. BACK Source: R. Geoff Dromey, "Cornering the Chimera," Quality - An Elusive Target no. 1, pp. 33-43, Jan. 1996, Software IEEE Software, vol. 13, 47 doi:10.1109/52.476284
Editor's Notes
- #5: What is quality? Timely, Attractive, desirable, robust, factual, secure, reliable, auditable, predictable, maintainable, environmental, accountable, ethical, precise? Quality is the degree to which a set of inherent characteristics fulfill requirements – PM Book Grade is a category assigned to products or services having the same functional use but different technical characteristics Not meeting quality requirements is a problem; low grade is not High quality vs low grade or low quality vs high grade: Trade off
- #6: What is quality? Timely, Attractive, desirable, robust, factual, secure, reliable, auditable, predictable, maintainable, environmental, accountable, ethical, precise? (Make notes using quality lecture)
- #10: Definition: “Project Quality Management includes the processes and activities of the performing organisation that determine quality policies, objectives and responsibilities so that the project will satisfy the needs for which it was undertaken.”
- #12: Examples of volatility and change is the change in technologies etc So software is more about process and less about product
- #14: Thread Safeness:
- #16: User view: Thus, each quality-requirement specification includes a measurement concept, unit, and tool, as well as the planned level (the target for good quality), the currently available level, the best possible level (state of- the-art), and worst level. Manufacturing View: Defect counts may lead to process improvements and test efficiency but the impact of defects on operational failures is not clear; Rework is defined as any additional effort required to find and fix problems after documents and code are formally signed-off as part of configuration management. Thus, end-phase verification and validation are usually excluded, but debugging effort during integration and system testing is included. Rework can also be pre-release and post release but only post release rework can be counted towards the measurement of quality (you didn’t find it but the customers did) because pre-release rework contributes towards process improvements.
- #17: Defect Density: MTFB:
- #18: The Chimera or Chimaera was, according to Greek mythology, a monstrous fire-breathing female creature of Lycia in Asia Minor, composed of the parts of multiple animals: upon the body of a lioness with a tail that ended in a snake's head, the head of a goat arose on her back at the center of her spine. The Chimera was one of the offspring of Typhon and Echidna and a sibling of such monsters as Cerberus and the Lernaean Hydra. The term chimera has also come to describe any mythical animal with parts taken from various animals and, more generally, an impossible or foolish fantasy.
- #24: To identify high-level quality attributes of a requirements specification, ask “What do I want to do with this specification?” Principally, you want to use it to describe a problem’s requirements; use it as the basis for design; use it as. the instrument of contract and common understanding among the client, the users, and the developer;
- #27: OPM3: Organizational Project Management Maturity Model Also mention Malcolm Baldridge awards
- #30: Development quality is indicated by the defects reported before the product release Product quality is indicated by the defects reported by the customers
- #31: In other words, control over costs and defects internally enable the firm to cut costs in the manufacturing process while still charging premiums for the high-quality products in the market
- #35: Standards may be in computer science, quality assurance, project management, systems engineering, reliability, safety, product standards, process standards and there may be company guidelines too. Standards may be used for repeatable processes, may provide consensus wisdom, cross-specialization, professional discipline, customer protection Standards reduce variance in the process by driving them towards commonality AIAA: American Institute of Aeronautics and Astronautics (e.g. AIAA R-013-1992 Recommended Practice for Software Reliability). Standards can help ensure consistent quality, but primarily for process, not product
- #36: Standards may be in computer science, quality assurance, project management, systems engineering, reliability, safety, product standards, process standards and there may be company guidelines too. Standards may be used for repeatable processes, may provide consensus wisdom, cross-specialization, professional discipline, customer protection Standards reduce variance in the process by driving them towards commonality AIAA: American Institute of Aeronautics and Astronautics (e.g. AIAA R-013-1992 Recommended Practice for Software Reliability). Standards can help ensure consistent quality, but primarily for process, not product
- #37: When a firm accepts fixed price contracts this shows that they may be more efficient in doing something T&M are usually used when the work being undertaken is new
- #40: CEO Scott Trappe explained this result by noting that the open source model encourages several behaviors that are uncommon in the development of commercial code. First, many users don’t just report bugs, as they would do with [proprietary] software, but actually track them down to their root causes and fix them. Second, many developers are reviewing each other’s code, if only because it is important to understand code before it can be changed or extended. It has long been known that peer review is the most effective way to find defects. Third, the open source model seems to encourage a meritocracy, in which programmers organize themselves around a project based on their contributions. The most effective programmers write the most crucial code, review the contributions of others, and decide which of these contributions make it into the next release. Fourth, open source projects don’t face the same type of resource and time pressures that [proprietary] projects do. Open source projects are rarely developed against a fixed timeline, affording more opportunity for peer review and extensive beta testing before release.