Measure It, Manage It, Ignore It - Software Practitioners and Technical Debt
Download as PPTX, PDF4 likes1,182 views
The document discusses a study on technical debt in software engineering, examining definitions, sources, and management practices through a survey of software professionals. Key findings identify architecture as a significant source of technical debt and suggest that current practices for managing it rely heavily on issue trackers. The study highlights the need for standardized tools and practices to effectively address technical debt, which is often overlooked in project management.
![16
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
RQ3: Management of TD
“(B1) regarding static analysis we have the source code static
analysis tools, but this is to assure proper quality of source
code. But how architectural changes are impacting I don’t
know. And, in fact, this is something we don’t do.”
“(C1) it showed up on Jenkins - the CI server - there’s a
billion little warnings. And so it seems a little bit
overwhelming.”
“[we track] occasionally by explicit tech debt items, usually by
pain, or not at all...”](https://image.slidesharecdn.com/fse15-td-150903130651-lva1-app6892/85/Measure-It-Manage-It-Ignore-It-Software-Practitioners-and-Technical-Debt-16-320.jpg)
Measure It, Manage It, Ignore It - Software Practitioners and Technical Debt
- 1. © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Measure it? Manage it? Ignore it? Software Practitioners and Technical Debt Neil A. Ernst with Stephany Bellomo, Ipek Ozkaya, Robert Nord Software Engineering Institute Carnegie Mellon University Pittsburgh, PA 15213 and Ian Gorton College of Computer and Information Science Northeastern University
- 2. 2 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Copyright 2015 Carnegie Mellon University and IEEE This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. References herein to any specific commercial product, process, or service by trade name, trade mark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by Carnegie Mellon University or its Software Engineering Institute. NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. This material has been approved for public release and unlimited distribution. This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu. DM-0002708
- 3. 3 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Distribution Statement A: Approved for Public Release; Distribution is Unlimited Background • Cunningham, 1992: “Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite... The danger occurs when the debt is not repaid” • Our definition: “the obligation that a software organization incurs when it chooses a design or construction approach that's expedient in the short term but that increases complexity and is more costly in the long term.” (McConnell)
- 4. 4 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Distribution Statement A: Approved for Public Release; Distribution is Unlimited Research Questions • RQ1. Is there a commonly shared definition of technical debt among professional software engineers? • RQ2. Are issues with architectural elements (such as module dependencies, external dependencies, external team dependencies, architecture decisions) among the most significant sources of technical debt? • RQ3. Are there practices and tools for managing technical debt?
- 5. 5 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Distribution Statement A: Approved for Public Release; Distribution is Unlimited Methodology • Pilot Survey • Full survey • Triangulate with: • Multiple choice background • Open-ended questions • Likert agreement • Follow-up interviews (opportunistic)
- 6. 6 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Distribution Statement A: Approved for Public Release; Distribution is Unlimited Demographics • 1831 surveys were started (across all three collaborators) and 536 surveys fully completed (all questions answered), an overall response rate of 29% • Respondents over 6 years experience • Roles selected included developers (42%), and project leads/managers (32%) • Most projects were web systems (24%) or embedded (31%). • Projects generally consisted of 10-20 people • The systems averaged 3-5 years old, but a significant number (29%) were over 10 years old. • The systems between 100KLOC and 1MLOC in size.
- 7. 7 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Distribution Statement A: Approved for Public Release; Distribution is Unlimited Findings 1. Architecture is a key source of technical debt 2. Technical debt is broadly temporally distributed (source and symptom not synched) 3. Issue trackers are currently the best tool approach to managing technical debt. Triangulate answers to these questions by looking at quantitative (closed-ended) questions, and coding of open-ended and interview data.
- 8. 8 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Distribution Statement A: Approved for Public Release; Distribution is Unlimited RQ1: Defining Technical Debt • “(A2) I think the vocabulary of technical debt is useful for getting the interests aligned.” • ‘convincing product managers and stakeholders on the value proposition of managing the debt.’
- 9. 9 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Distribution Statement A: Approved for Public Release; Distribution is Unlimited RQ2: Architecture choices key 9
- 10. 10 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Distribution Statement A: Approved for Public Release; Distribution is Unlimited Coding of open-ended questions
- 11. 11 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Distribution Statement A: Approved for Public Release; Distribution is Unlimited RQ2: Architecture as source
- 12. 12 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Distribution Statement A: Approved for Public Release; Distribution is Unlimited RQ2 (2) “(B2) the work that we’re doing now to introduce a service layer and also building some clients using other technology is an example of, you know, decisions that could have been done at an earlier stage if we had had more time and had the funding and the resources to do them at the time instead of doing it now.” “‘platform’ was not designed with scalability in mind” “In retrospect we put messaging/communication ... in the wrong place in the model view controller architecture”
- 13. 13 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Distribution Statement A: Approved for Public Release; Distribution is Unlimited RQ2: Drift Lehman’s concept of entropy present in our data: • “over the years, other sites would begin using the system and would require changes to how the workflow operated” Weak association between system age and the perceived importance of architectural issues, using Yule’s Q. 89% of those with longer-lived systems (>=6 years old) agreed or strongly agreed with the notion that architectural issues were a significant source of debt, compared to 80% of those with newer systems (<3 years old).
- 14. 14 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Distribution Statement A: Approved for Public Release; Distribution is Unlimited RQ3: Management of TD How tracked … Where tracked …
- 15. 15 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Distribution Statement A: Approved for Public Release; Distribution is Unlimited RQ3: Tool Use 15 None/Unknown: 58%
- 16. 16 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Distribution Statement A: Approved for Public Release; Distribution is Unlimited RQ3: Management of TD “(B1) regarding static analysis we have the source code static analysis tools, but this is to assure proper quality of source code. But how architectural changes are impacting I don’t know. And, in fact, this is something we don’t do.” “(C1) it showed up on Jenkins - the CI server - there’s a billion little warnings. And so it seems a little bit overwhelming.” “[we track] occasionally by explicit tech debt items, usually by pain, or not at all...”
- 17. 17 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited • Timeline for TD management • Track TD items like we do defects • Connect Architecture Technical Debt to architectural analysis (e.g., quality attribute prioritization) Future Work and Open Questions
- 18. 18 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited 18 • Software practitioners agree on the usefulness of the metaphor • Different interpretations of what makes up technical debt in particular contexts. • Leading sources of technical debt are architectural choices • take many years to evolve • Managing this drift is vital in managing technical debt. • Developers perceive management as unaware of technical debt issues • Desire for standard practices and tools to manage technical debt that do not currently exist http://bit.ly/sei-td – @neilernst – nernst@sei.cmu.edu
- 19. 19 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Call for Papers: Negative Results in Software Engineering Special Issue of J. Empirical Software Engineering. We welcome your well-conducted yet ‘negative’ empirical studies. We also are looking for suitable reviewers and reviewer experience reports. Deadline for submission: October 7, 2015 http://bit.ly/emse-negative If you have questions/comments or would like to volunteer to be a reviewer of the papers, please contact the guest editors. Richard Paige richard.paige@york.ac.uk Jordi Cabot jcabot@uoc.edu Neil Ernst nernst@sei.cmu.edu Plug
- 20. 20 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Backup
- 21. 21 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Distribution Statement A: Approved for Public Release; Distribution is Unlimited Coding Process Id Statement Coder 1- A Coder 1-B … Coder 2-C 1 Technical debt is built over years & multiple versions by the combination of factors like monolithic code design, mix of obsolete and new technologies, cost over quality etc. Code Code … Code Cohen’s K: 0.45
- 22. 22 TD Survey September 2, 2015 © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Distribution Statement A: Approved for Public Release; Distribution is Unlimited Code definitions Final Coding Term Definition Subsumed Codes Interest The pain caused by technical debt, but not the debt itself. Rework Additional work needed to remove technical debt ‘principal’ Architecture choice While arch choice is pain caused perhaps by context shifts vs design shortcut that is deliberative we combine the two as “architecture choice” because participants may not even know whether it was deliberate or not. Must be a specific case/instance. Design Shortcut, Bad Architecture Choice Legacy modernization Changes and evolution in operating environment e.g. new tech or requirements changes. Obsolete Technology, Legacy, evolution, changing requirements, External Dependencies, Prototype become Product Limited Knowledge Respondent had no good understanding of TD No clue Awareness People in respondent’s org had no understanding of problems TD caused Management, avoidance strategy, culture, deferred integration design, metaphor Defects Responses referring to problems in code externally visible. Interest code is more suitable when text refers to linking those bugs to original decision. Bugs, Maintenance, Software Quality Time Pressure Must release to make deadline Schedule Cost Pressure Lack of financial resources or motivation to fix problem New code Code Problems Issues relating to code-level problems such as detected by FindBugs Overly complex code, inter-module dependencies, Code with Debt, Code Duplication Process Some problems arising from poor development processes. Inadequate testing, Inefficient CM, Lack of Documentation None A comment that is not possible to code Measurement codes about need for measuring, things we might need to measure such as complexity and accuracy Tools this category is about the things that are technical in nature including tool support for making intelligent decisions. Platform dependence, ways to find TD Requirements Shortfall Requirements not changing significantly, but system does not meet them. Could be due to ambiguity, lack of clarity, etc. Lack of documentation Issues due to incomplete understanding of the architecture which was often attributed to lack of documentation or some way to better understand the impact of making changes on the quality or maintainability of the system Inadequate Testing This category covers inadequate testing due to issues such as inadequate test coverage, limited test resources, lack of test automation or not enough time to complete tests
Editor's Notes
- #2: 9/3/2015
- #4: Approaches to TD identification Community perception of “TD is important” Quote Cunningham is ok but is this our preconceived definition or just motivation? (McConnell) Maybe start with show of hands/ questions
- #5: Motivation for Arch unclear. Cunningham focused on code, but since then it has broadened (landscape picture). Part of approach is “what’s in/what’s out” Not so defensive. We *think* it might be architecture to motivate second question.
- #6: We asked them about defns and examples before likert to hear their opinions Triangualted e.g arch questions #6 - s/w involved, but not software only
- #7: Make point about not homogenous group (even though only 3 orgs) – lots of different projects Average versus clusters of responses
- #8: Point about long lag time is good. Add to RQ2 discussion.
- #9: 11 Ward’s problem is you need a lot of discipline to return and repay
- #11: Define “coding. (And process) What this means is that when people give an example, it tended to involve architecture
- #13: #11 as Ward notes, the idea is really that you cannot know upfront, so you MUST fix it later - and soon Protoytpe becomes product
- #15: bigger
- #18: 1 – debt incurred (decision made) 2 – recognized 3 – ideal pay back time (should do it here) 4 – actual repayment based on pain 5 – decision to strategically manage debt (instead of forgetting, lack of arch docs and rationale).
- #20: Must cut 6 minutes or so. Faster at beginning. Less digressions.
- #22: Two of us rated each one after we came to agreement.