Comparison of release engineering practices in a large mature company and a startup
0 likes96 views
This study compares release engineering practices between a large mature company and a startup, highlighting the impact of organizational context on their implementation and outcomes. Findings indicate that large companies face challenges such as slow continuous integration, low feedback from production, and extensive organizational changes, while startups experience difficulties in automation due to resource constraints but benefit from frequent customer feedback. The study underscores the importance of adapting release engineering practices to specific contexts to enhance customer satisfaction.
Comparison of release engineering practices in a large mature company and a startup
- 1. Comparison of release engineering practices in a large mature company and a startup Eero Laukkanen, Maria Paasivaara, Juha Itkonen, Casper Lassenius eero.laukkanen@solita.fi +
- 2. Why is this study important? Context in software engineering (Dybå et al. 2012) - Software engineering phenomena are essentially context-specific - Attempts to produce universal results often lead to hasty generalization - It is important to link research findings to the context they are found in Adoption of modern release engineering and context (Laukkanen et al. 2017) - Few studies on the effect of the context on release engineering practices - Product context: Few reported successes outside web application domain - Organization context: Few reported successes in large companies and startups
- 3. What is modern release engineering? Release engineering: everything from source code change to being used Modern release engineering (Adams and McIntosh 2016) - Fast and frequent release cycles - Extensive version control - Build, test and deployment automation - End-to-end organizational collaboration (DevOps) - Defect detection during and after production deployments, automated rollbacks
- 4. What did we study? Goal: Understand how the organizational context affects release engineering practices in a startup compared to a large mature company RQ1. What release engineering practices had the case organizations implemented? RQ2. What outcomes did the implemented release engineering practices have for the case organizations? RQ3. What were the reasons for differences in release engineering practices in the case organizations?
- 5. Who were studied? Theme Case BigCorp Case SmallOrg Organizational context Product development unit in a large mature company Startup company Company size > 20,000 50 Case organization size 180 50 Distribution of the organization Two sites in Europe, one site in Asia One site in North America Organizations developed competing products BigCorp acquired SmallOrg after losing new customers to it
- 6. What research methods were used?
- 7. What research methods were used? 18 qualitative interviews with a flexible interview guide, inductive coding Continuous integration data (2600+ builds during 24 weeks) Preliminary research in BigCorp (1 background interview, 2 group interviews) Triangulation by - multiple researchers - multiple data collection methods - interviewees from different roles - case representative agreement
- 8. Results
- 9. What kind of product was developed? Early market, ambiguous requirements, substantial growth expected Installed to customer premises, integrated with customer-specific production systems Customer production environments vary significantly Sales process starts with proof of concept in the customer environment During proof of concepts, customer-specific changes are often requested After full production deployment, most customers do not want very frequent releases (6-12 months is OK) Some of defects can be only discovered in the customer production environments
- 10. What kind of organizations were developing the products? Theme Case BigCorp Case SmallOrg Customers Tens One, Friendly Organizational structure Large, Distributed, Functional Small, Co-located, Collaborative Resources Sufficient workforce and Test equipment Limited workforce and Test equipment
- 11. What release engineering practices had the case organizations implemented?
- 12. What release engineering practices had the case organizations implemented?
- 13. What release engineering practices had the case organizations implemented? Metric (mean) Case BigCorp Case SmallOrg Green builds / week 1.6 (min–max: 0–10) 24.3 (min–max: 8–44) Build success rate 1.9% (weekly min–max: 0–14%) 76.7% (weekly min–max: 38–97%) CI execution time 2.7 h (weekly min–max: 0.23–5.0 h) 22 min (weekly min–max: 16–41 min)
- 14. What release engineering practices had the case organizations implemented? Case BigCorp Case SmallOrg No systematic code review All code is reviewed through pull requests Slow CI, broken most of the time Fast CI, unbroken most of the time Strict definition of done, dedicated testers, integration and non-functional testing in test labs No definition of done, no dedicated testers, low unit test coverage, lack of integration and non-functional testing Pilots with existing customers performed by other organization than development, low feedback from production use Extensive testing in customer labs and production environment by domain experts and service team members, fast feedback to development
- 15. What outcomes did the release engineering practices have? Case BigCorp Case SmallOrg Low feedback from the production use Frequent feedback from the production use Low number of defect reports High number of defect reports Elementary programming mistakes revealed with new customers Scalability and other issues with new customers
- 16. What were the reasons for differences in release engineering practices?
- 17. What can be learned from the study? Adopting modern release engineering practices requires different things in different contexts In large companies, the adoption can require extensive changes in the organization In startups, extensive automation can be difficult to achieve due to low resources Internal verification and frequent customer validation seem to have positive effects on the customer satisfaction
- 18. References Adams, Bram, and Shane McIntosh. "Modern release engineering in a nutshell--why researchers should care." Software Analysis, Evolution, and Reengineering (SANER), 2016 IEEE 23rd International Conference on. Vol. 5. IEEE, 2016. Dybå, Tore, Dag IK Sjøberg, and Daniela S. Cruzes. "What works for whom, where, when, and why?: on the role of context in empirical software engineering." Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement. ACM, 2012. Laukkanen, Eero, Juha Itkonen, and Casper Lassenius. "Problems, causes and solutions when adopting continuous delivery—A systematic literature review." Information and Software Technology 82 (2017): 55-79.