Bottom-up Adoption of Continuous Delivery in a Stage-gate Managed Software Organization
1 like357 views
This document discusses a case study of a software organization that was attempting to adopt continuous delivery (CD) practices while using a stage-gate process for managing releases. The study identified several direct signs of dysfunctional CD, including failing builds, flaky tests, low test coverage, and slow feedback. Through workshops, the researchers analyzed the root causes of these issues, many of which related to lack of time due to the stage-gate process and tight schedules. The stage-gate process affected CD adoption by creating branches, time pressure, and limiting the ability to improve practices like testing. Guidelines are proposed based on providing more time and flexibility to support better CD.
![Case Organization
• Nokia Networks software development unit
• Managed by product management unit
• Telecommunications product for managing self-organizing
network infrastructure
• Customers: mobile network operators around the world
• Org. structure: distributed to several sites and countries
• Time difference to one site, other sites on same timezone
"Internally, the target would be continuous deployment,
deploying to target hardware continuously [...] For continuous
deployment, the management is not needed or changes in the
processes."
4](https://image.slidesharecdn.com/presentation-160909134518/85/Bottom-up-Adoption-of-Continuous-Delivery-in-a-Stage-gate-Managed-Software-Organization-5-320.jpg)
Bottom-up Adoption of Continuous Delivery in a Stage-gate Managed Software Organization
- 1. Bottom-up Adoption of Continuous Delivery in a Stage-Gate Managed Software Organization Eero Laukkanen, Timo O.A. Lehtinen, Juha Itkonen, Maria Paasivaara, Casper Lassenius ACM/IEEE International Symposium on Empirical Software Engineer- ing and Measurement, September 2016 Department of Computer Science, Aalto University
- 2. Continuous Delivery (CD) Continuous Delivery Continuous Integration Version Control Commit Stage Acceptance Test Stage Other Stages Deploy to Production "Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time." (Fowler, 2013) 1
- 3. CD Adoption Organization, has not adopted CD Organization, has adopted CD Continuous delivery adoption: a set of adoption actions Problems CD solves Problems of CD Benefits of CD has CD adoption problems has 2
- 4. Previous Studies of CD Adoption Problems Debbiche etal. Claps etal. Leppänen etal. Chen Gm eineretal. Laukkanen etal. Hukkanen Identified Problems Build Design X System Design X X X X X Integration X X X X Testing X X X X X Release X X Human&Organizational X X X X X X X Process X Resources X X X X X X 3
- 5. Case Organization • Nokia Networks software development unit • Managed by product management unit • Telecommunications product for managing self-organizing network infrastructure • Customers: mobile network operators around the world • Org. structure: distributed to several sites and countries • Time difference to one site, other sites on same timezone "Internally, the target would be continuous deployment, deploying to target hardware continuously [...] For continuous deployment, the management is not needed or changes in the processes." 4
- 6. Stage-gate Process of the Case Organization Customer Trial Planning Development System Verification General Availability Code Freeze G3 G4G2G1 • Releases go through the stages, gates are milestones • Planning: feature specification, resource allocation • Development: implementation, testing (Scrum) • System Verification: end-to-end system testing • Customer Trial: trials at customer environments • General Availability: release to all customers 5
- 7. Research Goal and Questions Goal: Understand how the stage-gate process affected CD adoption in the case organization. RQ1. What direct signs of a dysfunctional CD practice did the case organization show? RQ2. What caused the direct signs of a dysfunctional CD practice in the case organization? RQ3. How did the stage-gate process used by the case organization explain the direct signs of a dysfunctional CD practice? 6
- 8. Research Process • ARCA root cause analysis method (Lehtinen et al., 2011) • Designed for high-level causal analysis • Two two-hour workshops on one site of the organization 1. Problem identification: What CD adoption problems the organization is facing? Which are the most significant? 2. Causal analysis: Why are the identified issues problematic? • Altogether 15 participants: managers, developers, testers • Audio recordings transcribed and thematically analyzed • Limitations • Perceived causes • Only one site heard • Only development unit heard 7
- 9. Results
- 10. RQ1. Direct Signs of Dysfunctional CD Practice Sign Description Failing builds CD pipeline builds were often failing and not fixed immediately afterwards. Flaky tests Some tests were flaky, meaning that they might fail randomly even when there was no issue in a code change. Low test coverage Test coverage was not considered high enough for having confidence to release the product after running the tests. Slow feedback Feedback about the changes made to the product came slow to the developers. 8
- 11. RQ2. Mechanisms and Root Causes Legend Stage-gate Process Failing Builds Flaky Tests Slow Feedback Low Test Coverage Multiple Branches Delayed Integration Time Pressure Process Overhead Tight Schedule Lack of Time To Improve Branch Overhead Distributed Organization Insufficient Communication Lack of Testing Strategy Duplicate Testing Unsuitable Architecture Unstable Architecture Limited Hardware Resources Sharing Test Environments Slow Build Breaking Changes Direct Sign Other Mechanism Process Mechanism Root Cause 9
- 12. RQ3. Lack of Time Mechanisms Stage-gate Process Failing Builds Flaky Tests Low Test Coverage Multiple Branches Time Pressure Process Overhead Tight Schedule Lack of Time To Improve Branch Overhead 10
- 13. RQ3. Tight Schedule Mechanism Stage-gate Process Tight Schedule Early Plans Do not Hold Same Deadlines for Dependencies and Dependents Uneven and Unpredictable Workload Plans not Readjusted after Delays Problems Hidden to Pass Quality Gates Code Freeze not Respected 11
- 14. RQ3. Branch Mechanisms Stage-gate Process Failing Builds Flaky Tests Slow Feedback Low Test Coverage Multiple Branches Delayed Integration Lack of Time To Improve Branch Overhead Limited Hardware Resources Sharing Test Environments 12
- 15. Guidelines Based on the Case Study • No time to improve test coverage, fix builds and flaky tests → Provide time with slack (Lawson, 2001) • Branches decided by management caused overhead for development → Avoid branches with feature toggles (Rahman et al., 2016) → Give decision-making power to developers • Stage-gate process substantially affected the CD adoption → CD and stage-gate process might be incompatible → Get support from other parts of the organization for CD adoption 13
- 16. References L. Chen. Continuous Delivery: Huge Benefits, But Challenges Too. Software, IEEE, 32(2):50--54, 2015. G. G. Claps, R. B. Svensson, and A. Aurum. On the journey to continuous deployment: Technical and social challenges along the way. Information and Software Technology, 57(0):21 -- 31, 2015. A. Debbiche, M. Dienér, and R. Berntsson Svensson. Challenges When Adopting Continuous Integration: A Case Study. In Product-Focused Software Process Improvement, volume 8892 of Lecture Notes in Computer Science, pages 17--32. Springer International Publishing, 2014. M. Fowler. ContinuousDelivery, 2013. URL http://martinfowler.com/bliki/ContinuousDelivery.html. J. Gmeiner, R. Ramler, and J. Haslinger. Automated testing in the continuous delivery pipeline: A case study of an online company. In 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pages 1--6, 2015. L. Hukkanen. Adopting Continuous Integration – A Case Study. M.Sc. thesis, Aalto University, 2015. E. Laukkanen, M. Paasivaara, and T. Arvonen. Stakeholder Perceptions of the Adoption of Continuous Integration – A Case Study. In 2015 Agile Conference, pages 11--20, Washington, DC, USA, 2015. M. B. Lawson. In praise of slack: Time is of the essence. The Academy of Management Executive, 15(3):125--135, 2001. T. O. A. Lehtinen, M. V. Mäntylä, and J. Vanhanen. Development and evaluation of a lightweight root cause analysis method (ARCA method)–field studies at four software companies. Information and Software Technology, 53(10): 1045--1061, 2011. M. Leppänen, S. Mäkinen, M. Pagels, V.-P. Eloranta, J. Itkonen, M. V. Mäntylä, and T. Männistö. The Highways and Country Roads to Continuous Deployment. Software, IEEE, 32(2):64--72, 2015. M. T. Rahman, L.-P. Querel, P. C. Rigby, and B. Adams. Feature toggles: practitioner practices and a case study. In Proceedings of the 13th International Conference on Mining Software Repositories, pages 201--211. ACM Press, 2016. 14