Innovation Experiment Systems Practices (ICSOB 2015)
1 like240 views
The document discusses practices for moving software development companies towards continuous delivery and innovation experimentation systems. It presents findings from a case study of 5 software companies in Finland. All companies had adopted agile practices like Scrum or Kanban. 4 had continuous integration but challenges remained in continuous deployment. Benefits included faster feedback and ensuring the right features were developed. Challenges included complexity, quality risks, and changing customer expectations. The document proposes an extended Stairway to Heaven model with 20 practices across business, architecture, process and organization to help companies transition towards innovation experimentation systems.
Innovation Experiment Systems Practices (ICSOB 2015)
- 1. Teemu Karvonen, Lucy Ellen Lwakatare, Tanja Sauvola, Pasi Kuvaja. Markku Oivo MALMÖ UNIVERSITY Helena Holmström Olsson CHALMERS UNIVERSITY OF TECHNOLOGY Jan Bosch Hitting the Target: Practices for Moving towards Innovation Experiment Systems
- 2. AGENDA BACKGROUND AND MOTIVATION RESEARCH GOALS (RESEARCH QUESTIONS) RESEARCH DESIGN (METHODS AND STUFF) RESULTS • CASE STUDY FINDINGS • ”EXTENDED STAIRWAY TO HEAVEN”
- 3. Software business- & development challenges SPEED & EFFICIENCY BUSINESS ACCURACY ”Hitting the Target” SW company’s capability for continuous delivery SW company’s capability for customer experimentations
- 4. Agile software development (Highsmith J., 2009) • Agile Manifesto • Scrum, XP, TDD, Continuous Integration… Lean Software Development (Poppendieck M, 2007) • Lean thinking, Toyota Way • Continuous Improvement, Kaizen, Value stream mapping, Kanban… Beyond agile methods • Continuous delivery I.e build, test and deployment automation (Humble & Farley, 2010) • Continuous experimentation (Fagerholm et al. 2014) • Continuous innovation & Lean Startup-method (Ries E., 2011) • Continuous software engineering Continuous * (Star) (Fitzerald et.al, 2014) • Stairway to Heaven model, Innovation experiment systems & HYPEX model (Holmström Olsson H., Bosch J. 2013) BACKGROUND & RELATED STUDIES
- 5. Stairway to Heaven model ”EVOLUTION OF SOFTWARE DEVELOPMENT COMPANIES” (Holmström Olsson H., Bosch J, Alahyari H. 2012) Traditional Development Agile R&D Continuous Integration Continuous Deployment R&D as Innovation Experiment Systems Step A: Step C: Step E Step D R&D Validation R&D Product Management R&D Validation (Customer) Product Management R&D Validation Customer Validation R & D Customer Product Management Step B:
- 6. RESEARCH GOALS To understand software development companies current ways of working and their status in relation to Stairway to Heaven model steps To undertand software development companies opinions about benefits and barriers in transition towards Continuous Deployment and Innnovation Experiment Systems Research questions: RQ1: What are the benefits and barriers that software development companies experience when moving towards Innovation Experiment Systems RQ2: What are the key practices that software companies need to adopt in order to evolve their software development practices according to the Stairway to Heaven model?
- 7. Research design • Replication/rerun of multiple-case study conducted 2012 in Sweden (Holmstöm Olsson et al. 2012). Minor modifications to interview plan. • Method: Multiple-case study – Semi-structured interviews and qualitative data analysis (Data coding in Nvivo) – Interpretive data analysis approach • Interviews conducted during Oct-Dec 2014 in 5 software development companies from Finland (Company A,B,C…) • Five case companies from Finland – 4 companies mainly operating in embedded systems domain (R&D services, telco, wireless systems etc.) – 1 company mainly operating in information technology (IT) services domain – Products: Base stations, wireless device platforms, PC-client sofware, web-applications… • 24 interviews, 4-5 interviews per company • 1,5-2 hour individual semistructured interviews • Interviewees from multiple roles (product managers, project managers, architecture designers, UX designers, developers, testers, quality managers…)
- 8. RESULTS
- 9. CASE STUDY FINDINGS • All 5 companies had adopted Agile development practices (Typically Scrum or Kanban or ”some mix of them”). 1 company (interviewed team) had just started using Scrum method. • 4/5 had adopted continuous integration practices and was able to run automated builds and test cases -> Developers were able to get fast feedback of how their code changes integrates and works. • Typical challenges in Continuous Integration – Expenses and resources for CI, embedded systems hardware configurations (Special HW accelators etc.…) Improvement of test automation coverage • ”The challenge”: Moving towards continous deployment – Improvement of software development capabilities for Continuous Deployment – Increasing customer’s awereness of the benefits of Continuous Deployment? Finding lead customers. Traditional Development Continuous Integration Continuous Deployment R&D as Innovation Experiment Systems Step DStep B Agile R&D
- 10. − “You actively seek the feedback and send interim versions of SW and prototype HW, to collect that feedback. In the end, the result would be a product that the customer actually want and not a product that they specified in the beginning, if those two things are not the same anymore.” − “Mainly the customer feedback, I think that's the most valuable information that you get from deploying the early releases to customer.” − “More visibility to the, how the development is proceeding” − “It is also a way to show the customer that we are on schedule to deliver what we have promised, to do” − “We (and customer) are both saving money. We can keep our schedules better. You can have those features that you really want and need, more.. faster than earlier. “ − “I think that, we benefit, and I benefit from not doing work that’s going to be trashed, that we develop stuff that’s really required and we develop in an order that customer needs it. “ Moving towards Continuous Deployment and IES Key Benefits – Quotes
- 11. − “If we do handsets, deployment is fairly easy. We can do over the air updates, it's very simple. But then, in some cases we have very complex systems where multiple pieces of the system needs to be updated to specific versions. In those cases, it is more challenging. It might take even, days to get the system updated and back up and running again - It is almost impossible to do it continuously.“ − “In the early stages of the development it is risky. If the quality of your deliverables is R&D (quality), which it is in the early stages.” − “If you start deploying too early, to your customers, it might cause some negative effects. Because sometimes customers are not as educated as us how, the development cycle goes. They might freak out if they see the status of the deliverables too early. So I would be cautious when starting the continuous deployment.” Moving towards Continuous Deployment and IES Key Challenges – Quotes
- 12. − “I think the obstacle is the money. It is expensive but, doing it in a right manner could make the next projects cost less money.” − “Business is one limitation here. This level E (experimentation) is of course interesting, and it would be very nice to work in project with this kind of a way of working. But, I can't see, it’s not so near future for us. Maybe someday. “ − “Of course our customer they do have their own processes and way of working. It’s quite hard for us to, tell them that please change your, processes. “ − “My first idea of (R&D as Innovation Experiment System) would be a nightmare. I would not want to go there. We do this type of work when we develop, demonstration, product for some customer. “ − “Customers need to get involved and they have to be continuously willing to check things, which might be hard “ Moving towards Continuous Deployment and IES Key Challenges – Quotes
- 13. WHAT ARE THE BENEFITS OF • The innovation validation is fast • Immediate feedback is obtained • New business opportunities are identified • Development resources are focused Continuous Integration Continuous Deployment • Customers get fast and incremental delivery of relevant functionality • Customers can perform their own testing and business activities on top of deliveries a) Moving from Continuous Integration to Continuous Deployment b) Moving from Continuous Deployment to R&D as Innovation Experiment Systems Continuous Deployment R&D as Innovation Experiment Systems
- 14. WHAT ARE THE BARRIERS OF Continuous Integration Continuous Deployment a) Moving from Continuous Integration to Continuous Deployment b) Moving from Continuous Deployment to Innovation Experiment Systems Continuous Deployment R&D as Innovation Experiment Systems • The shortening of the validation and verification cycle is complex and expensive • Lack of trust in software quality • Missing functions may cause a negative impression – Traditional delivery expectations • Customer feedback integration into the short development and business planning cycle. • Difficult to conduct experiments in safety critical systems
- 15. PRACTICES FOR Continuous Integration Continuous Deployment Moving from Continuous Integration to Continuous Deployment Moving from Continuous Deployment to R&D as Innovation Experiment Systems Continuous Deployment R&D as Innovation Experiment Systems • Identify and incorporate lead customers in development process • Renew business model, contracts, marketing and sales strategies • Provide architecture where software functionalities can be deployed independently • Improve automated system testing and adopt a continuous release process • Ensure that System/UX design and business development work in short cycles and in alignment with R&D • Adopt data-driven strategic decision-making model. • Implement A/B testing with the customer • Adopt product platform (e.g. virtualization, cloud technologies) that enables flexible experimentation. • Establish a short customer feedback loop and process for data-driven decision making • Synchronize supplier and customer organization in short development cycles
- 16. Extended Stairway to Heaven model Practices for Moving towards Innovation Experiment Systems
- 17. Extended Stairway to Heaven model (Practices) 20 practices in 4 categories: – Business – Architecture – Process – Organisation 4 adoption levels Institutionalised Product Team Not adopted Business practices Architecture practices Innovation experiment systems A/B testing Product Architecture supporting run- time variation of Product Continuous Deployment Business model transformation Not adopted Componentisation for partial release & rollback Product Continuous Integration Reduce cost of bad quality Institutionalised Modularisation to improve ability to do unit & Institutionalised Agile R&D Product owner to represent the customer Institutionalised Lead architect to protect architecture from erosion Institutionalised Traditional Development Customer validation at the end of the project Not adopted Early-on requirement freeze Not adopted Process practices Organisation practices Innovation experiment systems Customers real usage data collection practices Not adopted Product management & business development Not adopted Continuous Deployment Continuous release process Team UX/system design integrated in team Product Continuous Integration Test-driven development & automated test scripts & Institutionalised V&V function integrated in agile team (continuous Institutionalised Agile R&D Sprints & daily standups Institutionalised Feature teams (cross- functional) Institutionalised Traditional Development Milestone-driven development Not adopted Large development teams divided into discipline Not adopted
- 18. Extended Stairway to Heaven model (Example) CASE COMPANY A Business practices Architecture practices Innovation experiment systems A/B testing Product Architecture supporting run- time variation of Product Continuous Deployment Business model transformation Not adopted Componentisation for partial release & rollback Product Continuous Integration Reduce cost of bad quality Institutionalised Modularisation to improve ability to do unit & Institutionalised Agile R&D Product owner to represent the customer Institutionalised Lead architect to protect architecture from erosion Institutionalised Traditional Development Customer validation at the end of the project Not adopted Early-on requirement freeze Not adopted Process practices Organisation practices Innovation experiment systems Customers real usage data collection practices Not adopted Product management & business development Not adopted Continuous Deployment Continuous release process Team UX/system design integrated in team Product Continuous Integration Test-driven development & automated test scripts & Institutionalised V&V function integrated in agile team (continuous Institutionalised Agile R&D Sprints & daily standups Institutionalised Feature teams (cross- functional) Institutionalised Traditional Development Milestone-driven development Not adopted Large development teams divided into discipline Not adopted
- 19. FUTURE STUDIES? Ideas about directions of the future studies Investigating different stakeholder’s point of view. How customers perceive benefits and challenges associated to continuous deployment / experimentation? How subcontractors and technology platform providers perceive benefits and challenges associated to continuous deployment / experimentation? Expanding view from company level to whole supply chain or ecosystem level. What are the challenges and benefits of continuous deployment/experimentation in ”system level” e.g. software ecosystem?
- 22. SUMMARY Research New ways of working - Steps towards Innovation Experiment Systems Agile & CI – ”First two steps” • Integrated and iterative Code & Test -cycles • Automated I&V, rapid feedback for developers • Sprint based planning, reviews and repetitive retrospectives (continuous improvement) Continuous deployment ”3rd Step” • Integrated and iterative Plan-Design- Code-Test-Deploy -cycles • Ability to deploy at will. Frequency not deciding factor. Could be weekly or daily deployments to production. Innovation experiment systems • Continuous customer feedback • Data driven development decision making • Plan-Design-Implement-Test-Deploy- Get Feedback • Capability to learn from customer real usage data
- 23. Hitting the Target: Practices for Moving towards Innovation Experiment Systems. Karvonen T., Lwakatare L., Sauvola T., Bosch J., Holmström Olsson H., Kuvaja P., Oivo M. RESEARH METHOD: • A multiple-case study conducted in Finland (October-December 2014). • 24 interviews, 5 software development companies, Qualitative data analysis RESEARCH QUESTIONS: • Moving towards Continuous Deployment & Innovation Experiment Systems. RQ1: What are the improvement steps and key practices? RQ2: What are the benefits and barriers? MAIN CONTRIBUTION: • Extension of the Stairway To Heaven model – Introduction of steps and practices for moving towards Innovation Experiments Systems STYDY IN A NUTSHELL: COLLABORATION PARTNERS: UNIVERSITY OF OULU, MALMÖ UNIVERSITY, CHALMERS UNIVERSITY OF TECHNOLOGY