ROI at the bug factory - Goldratt & throughput (2004)
Download as PPSX, PDF2 likes806 views
The document discusses using concepts from manufacturing to analyze software development processes. It introduces ROI and Goldratt's Theory of Constraints, which views inventory as a constraint on throughput and profitability. The document argues that specifications and unfinished software can be viewed as inventory in software development. Seeing software development and testing processes through this "factory" lens may provide insights on effectiveness, efficiency, and improvement.
ROI at the bug factory - Goldratt & throughput (2004)
- 1. ROI at the Bug Factory: Goldratt and Throughput 1st European Conference on Software Projects, Process & People 01 - 03 December 2004 – Köln, Germany Software Projects Track session WP1 Neil Thompson Thompson information Systems Consulting Limited www.TiSCL.com © Thompson information Systems Consulting Limited 1 www.qualtechconferences.com
- 2. 1. Introduction • Primary objective of presentation – to explore some analogies between processes for software development and implementation and those for manufacture (of physical things): – to see whether recent (and not-so-recent) revolutionary changes in manufacturing have relevance for software – to relate such analogies to existing work in agile methods – to relate this thinking to the software development lifecycle whether agile or not, and to verification, validation & testing (VV&T) – to help you in planning and managing projects & programmes • Secondary objectives: – to use the concept of Return On Investment (ROI) as an introduction to thinking about process effectiveness, efficiency and improvement – to use Goldratt’s Theory Of Constraints (TOC) as the basis of considering manufacturing processes, and for its general thinking tools © Thompson information Systems Consulting Limited 2 www.qualtechconferences.com
- 3. What is ROI, why measure it? • ROI: return / investment (usually expressed as a percentage): – “return” could be profit, (value – cost), (net benefit)… – “investment” may be expressed as net cost • I’m a “crossover” from EuroSTAR (but wider background) • So I’ll start with the example of ROI for VV&T: – to decide how many test environments to buy, and how life-like – to decide how many testers & QA people to hire – to estimate duration of testing – to justify budgets – to decide how to divide effort between testing & reviews/inspections – to focus training – to justify automation – to motivate staff? – to make managers look good? © Thompson information Systems Consulting Limited 3 www.qualtechconferences.com
- 4. Why ROI difficult for IT systems • Although costs relatively easy to measure, not trivial • Benefits no longer about replacing manual systems; often are enhancing market position, or opening new market, and those can be very difficult to forecast • Misleading / unbalanced precision between costs & benefits • Misguided cost-cutting (eg through inadequate understanding of benefits) • Excuses for not calculating ROI “properly”: – “we’ve simply got to have it” – project / system too small to bother – it’s too difficult – wouldn’t be believed anyway • Problem of “construct validity”: you may not really be measuring what you think you’re measuring Cem Kaner © Thompson information Systems Consulting Limited 4 www.qualtechconferences.com
- 5. ROI even more difficult for VV&T • Partitioning difficulties: – how to distinguish as activities, when much is part of develop’t work – whether / how / why to distinguish by job title • Biggest problem is quantifying benefits in terms of costs saved by preventing failures later: – Boehm’s exponential-type cost rise questioned for modern methods – isn’t as simple as “early=good”, because some bugs not (easily) visible until later (though continuous integration helps) – how can we say fault A would have caused failure B at date X if it had not been found by test level C? – even where live failure costs calculated, validity is questionable – how distinguish poor VV&T from “good VV&T where there are very few bugs there to find”? © Thompson information Systems Consulting Limited 5 www.qualtechconferences.com
- 6. Also other complications, but let’s persist still • Additional complications to using ROI for VV&T: – “Tyranny of numbers” two different books – whether testing actually reduces risk or is merely (a misinterpretation of?) “Project Intelligence” Paul Gerrard – is effectiveness / efficiency distinction feasible / useful? • But ROI thinking should still be helpful: – still need to answer those questions, don’t we? • which VV&T activities to do, which not, which partially? • (coarse-tune) which people to use? • (fine-tune) how to maximise effectiveness & efficiency? – so Agenda… © Thompson information Systems Consulting Limited 6 www.qualtechconferences.com
- 7. Agenda • 2. Goldratt’s Theory Of Constraints • 3. The code factory and the bug factory • 4. Beyond the factory: the supply chain, and live systems • 5. The Systems Development LifeCycle as an inventory network • 6. Conclusions & way forward © Thompson information Systems Consulting Limited 7 www.qualtechconferences.com
- 8. 2. Goldratt’s Theory Of Constraints: what it is • Now, “multifaceted management philosophy”, but originally emerged in early 1980s from new approach to materials scheduling in factories • I already applied Goldratt’s thinking tools to process definition in software testing STAREast 2003 • Here looking wider, at whole SDLC, going back to Goldratt’s original arguments in manufacturing The Goal, The Race • Evidence of TOC success in reducing inventory, lead times & cycle times; improving due-date performance & revenue • Associated with Japanese improvements (eg Just In Time, Lean), but JIT came first, Goldratt was a response and a claimed improvement; Lean followed a few years later © Thompson information Systems Consulting Limited 8 www.qualtechconferences.com
- 9. Parts of TOC using here • Goal definition, eg make money: – measures cash flow (survival), profit (absolute) & ROI (relative) • Throughput better for these measures than cost management • Adverse effect of inventory on competitive edge: – it damages not only ROI & cash flow but also profit • Relationship of throughput to inventory & operating expense: – throughput (T) = money in (should maximise) – inventory = money tied up (should minimise, but not damage T) – operating expense = money out (should minimise, but not damage T) • Drum-Buffer-Rope as a technique to optimise inventory • Relationship to Synchronous Flow Manufacturing, Japanese methods and other quality techniques © Thompson information Systems Consulting Limited 9 www.qualtechconferences.com
- 10. Applicability of TOC beyond manufacturing • Military logistics • Marketing, sales & distribution • Project management • Measurements, human relationships, medicine etc • Using technology, eg assessing benefits of functionality • IT systems development: – focussing of Goldratt’s Critical Chain on hi-tech projects Robert C Newbold – methodology design Alistair Cockburn – “Lean software development” Mary & Tom Poppendieck – Agile management using Feature Driven Development David J Anderson © Thompson information Systems Consulting Limited 10 www.qualtechconferences.com
- 11. But software development isn’t like manufacturing? • Software isn’t like hardware • Intellect adds value less predictably than machines • The manufacturing part of software development is disk duplication: “development” is really a design activity • People are more important than processes • Software development doesn’t repeat exactly, people are always tinkering with the processes • Development involves discovery, production involves reducing variation • But I say: does that make all the analogies worthless, or do they just need interpreting? I suggest the latter… © Thompson information Systems Consulting Limited 11 www.qualtechconferences.com
- 12. 3. The code factory and the bug factory • No-waste factory a b c Stated Demonstrations & requirements acceptance tests Programming • Now here’s some waste: meeting/escalation (and loaded personal memories), or inventory? a b c a b d Implicit ’ requirements Documented ? Acceptance tests I I requirements ? Meeting / escalation to agree Programming © Thompson information Systems Consulting Limited 12 www.qualtechconferences.com
- 13. Specs & unfinished software are inventory • Specifications are generally not needed after go-live (I will come later to exceptions) so they are not end-product, they are work-in-progress (especially intermediate levels like functional & non-functional specs) • Untested software, and even finished software not paid for, is also inventory • Iterative lifecycles help if “adaptive” (product-based) rather than “transformational” (where specifications multiply!) a b Revised & new requirements a b c ’ ’ I May include Programming Programming redesign © Thompson information Systems Consulting Limited 13 www.qualtechconferences.com
- 14. Full W-model bulges with Business objectives inventory Test against Post-implement- Make into Verify ation review Requirements Verify & Validate Acceptance Acc. retest, Statement RS, + spec. AT test fix & reg. test Validate (incl. “QA”) Functional Verify & Validate System Sys. retest, Spec. FS, + spec. AT test fix & reg. test Retest lower Technical V&V TD, Integrat- Int. retest, levels + spec. IT fix & reg. test where Design ion test necessary Module V&V MS, + Unit Unit retest, Specs. spec. UT test fix & reg. test Code © Thompson Static-check information Systems Consulting Limited 14 www.qualtechconferences.com
- 15. In a factory, small batches reduce inventory Stages & units / hour (i) 1.3 (ii) 10.0 Multi-batch 1000 (iii) 1.0 200 200 200 200 200 (ie “iterative”) Single-batch (ie “waterfall”) (iv) 10.0 (v) 2.0 0 1 2 3 4 0 1 2 3 4 months months Inventory Inventory © Thompson Based on Goldratt, The race (North River Press 1986) information Systems Consulting Limited 15 www.qualtechconferences.com
- 16. Why inventory damages competitiveness (i) Goldratt’s objectives for My IT analogies competitiveness Product: 1. quality Quality & scope: 1. few defects now, reliable in future 2. engineering 2. amount & sophistication of funct- ionality, non-functional attributes Price: 3. higher margins Low-cost: 3. economical development & testing so flexible pricing 4. lower investment per 4. no environments panic, lower break- unit even point & flexibility to compete Responsiveness: Timely delivery: 5. rarely / never late 5. due-date performance 6. shorter quoted lead- 6. rapid application development! time © Thompson information Systems Consulting Limited 16 www.qualtechconferences.com
- 17. Why inventory damages competitiveness (ii) Objectives for Advantages of low inventory competitiveness (and/or iterative lifecycles!) Product: 1. quality Defect detection & fixing sooner after their introduction, fewer recurrences 2. engineering Quicker realisation of improvements Price: 3. higher margins Lower probability of needing overtime 4. lower investment per Lower probability of needing additional unit machines (test environments) Responsiveness: More reliable materials forecasting 5. due-date performance (smoother staff recruitment) 6. shorter quoted lead- Shorter overall time (already shown on time slide 15) © Thompson information Systems Consulting Limited 17 www.qualtechconferences.com
- 18. Throughput relationships, why good Net profit ROI Cash flow Operating Throughput Inventory expense High quality Good scope Low people costs Low machine costs Rarely late Rapid Competitive edge Extended from Goldratt, The race (North River Press 1986) © Thompson information Systems Consulting Limited 18 www.qualtechconferences.com
- 19. Drum-buffer-rope • Optimise throughput by: – (1) drumbeat based on constraining stage (a) & orders (b) – (2) buffer to protect constraining stage from upstream disruptions – (3) rope to prevent leader extending gap on constraining stage • For subassemblies feeding in, have additional buffers “troops marching” materials flow buffer raw materials (1) in constraining ”leader” stage (a) assembly (2) (3) buffer subassembly orders (b) © Thompson information Systems Consulting Limited 19 www.qualtechconferences.com
- 20. Buffer management too cautious premature interruption release of inventory healthy of inventory in in y hours x days too lean © Thompson information Systems Consulting Limited 20 www.qualtechconferences.com
- 21. Tracing buffer holes back to process steps to fix 3 5 2 1 4 © Thompson information Systems Consulting Limited 21 www.qualtechconferences.com
- 22. Relationship to other quality approaches • To recap: Goldratt argues why DBR is better than JIT • In addition to TOC, use lean techniques to minimise waste • TOC project management (critical chain, see Newbold): – deprecates multi-tasking – translates buffer management to contingency placement – recommends also time management & Pareto techniques – uses risk management to calculate buffer sizes • Goldratt is more process-oriented than product- (as was Deming) © Thompson information Systems Consulting Limited 22 www.qualtechconferences.com
- 23. Is VV&T a factory? • VV&T produces (depending on your viewpoint): – “project intelligence”; – risk reduction; – a mountain of test plans, scripts, logs, reports etc etc; – bug reports • Customer does not want to pay for bugs • Bugs are imperfections on inventory rather than inventory themselves; may not be many in the first place • “Removal of bugs” is not an easy end product to grasp, and the developers do it anyway • So the code factory & the bug factory are two aspects of the same thing; VV&T can help identify constraints © Thompson information Systems Consulting Limited 23 www.qualtechconferences.com
- 24. 4. Beyond the factory: supply chain & live systems • TOC/JIT says that functionality should be pulled by customers rather than pushed by analysts / vendors (but why does this often not happen?) • Software passing testing but not yet paid for is still WIP • Software development has its value chain • Poppendieck principles include “decide as late as possible”: contradiction with Newbold? • Biggest point here is that most systems go on for years, and need some documentation or staff continuity (this point receives insufficient attention often) © Thompson information Systems Consulting Limited 24 www.qualtechconferences.com
- 25. 5. SDLC as an inventory network • TOC suggests choosing SDLC with a view to what else is wanted other than working software, eg training material, maintenance documentation, regulatory audit • This affects value of inventory • Cockburn uses TOC principles, of which one consequence is effect on method • Another contradiction? Advice on handling slack • TOC process improvement can monitor buffers, and is not just for agile methods • Can still think of VV&T ROI as tangibles & intangibles, even if we can’t (yet?) calculate • But via the throughput route to ROI we can still seek to optimise © Thompson information Systems Consulting Limited 25 www.qualtechconferences.com
- 26. Inventory moving through SDLC Amount of functionality Requirements Inventory in this stage Lead time for Specification of process this stage Design of process Inventory in process Programming & Within each stage overall unit testing of testing, can Integration subdivide by testing pass/fail, bug states etc System testing Acceptance testing Live and paid-for Date If lines not approx parallel, inventory is growing © Thompson Based on Anderson, Agile management for software engineering (Prentice Hall 2004) information Systems Consulting Limited 26 www.qualtechconferences.com
- 27. SDLC as an inventory network (cont’d) raw materials in Where is/are the Requirements constraining stage(s)? Where should buffers Specification be / not be? Acceptance testing order(s) System testing Design assembly Integration testing sub-assemblies Programming Unit testing © Thompson information Systems Consulting Limited 27 www.qualtechconferences.com
- 28. 6. Conclusions and way forward • Although original Goldratt ideas don’t translate directly to software development, after so much success and so many analogies, can’t ignore (and thinking tools useful for “anything”) • Several of the later interpretations have plausible messages • Although a few authors have already applied TOC to agile methods, I hope that by going back to the original principles in some detail I have widened consideration to potentially any SDLC, and provided a platform for me and others to develop this thinking further… © Thompson information Systems Consulting Limited 28 www.qualtechconferences.com
- 29. Way forward project-informing Published Constraint-led methodologies thinking process definition (“Tailorable”?) tools & improvement Context → “Methodology Maintenance regime per project” Circumstance-driven © Thompson information Systems Consulting Limited 29 www.qualtechconferences.com
- 30. Summary Heavy industry, waste, burnout Light industry, lean, cool fumes “How clean is your factory?” © Thompson information Systems Consulting Limited 30 www.qualtechconferences.com
- 31. References & acknowledgements • Main references: – Goldratt, Eli: The race & Necessary but not sufficient – Newbold, Robert: Project management in the fast lane – Mellis, Werner: Process & product orientation – Cockburn, Alistair: Agile software development – Poppendieck, Mary & Tom: Lean software development – DeMarco & Lister: Waltzing with bears – Anderson, David: Agile management for s’ware engineering – Boehm & Turner: Balancing agility & discipline • Acknowledgements: – Jens Pas EuroSTAR 1998, metrics – Greg Daich STAREast 2002, documentation superstitions © Thompson information Systems Consulting Limited 31 www.qualtechconferences.com
- 32. Contact details Neil Thompson NeilT@TiSCL.com Questions? www.TiSCL.com 23 Oast House Crescent Farnham, Surrey, England GU9 0NP, United Kingdom phone +44 (0)7000 NeilTh (634584) or +44 (0)7710 305907 © Thompson information Systems Consulting Limited 32 www.qualtechconferences.com