TorontoAgile and Software 2014: Introduction to DevOps with Lego and Chocolate simulation game.

Editor's Notes

• #2: Good afternoon everyone and welcome to Chocolate, LEGO and Scrum Jambalaya!
• #3: For those of you who don’t know what Jambalaya is: It is a very popular dish in New Orleans, Louisiana, which consists of variety of different ingredients and is cooked in 3 distinct steps. Just like Jambalaya, this workshop will be using many different ingredients and in 3 sprints will give you a taste of DevOps.
• #4: And What is DevOps? According to Wikipedia, DevOps is a software development method that stresses communication, collaboration and integration between software developers and IT operations. John Willis states that “DevOps is about CAMS” ( culture, automation, measurement and sharing) There are so many different definitions out there that Gartner analysts in their “Seven Steps to Start Your DevOps Initiative” even recommends as a step 1 “Define DevOps for You” For this workshop, I would like to focus on DevOps Culture. So if you came here to learn everything about the DevOps tools, can’t wait to start looking into environments configuration with Vagrant or start examining the Chef recipes, I am going to disappoint you. The only recipe that I am referencing today is a Jambalaya recipe.
• #5: Instead, I invite you to experiment with me and use a simulation game to figure out what problem DevOps is addressing and why YOU should start building the DevOps culture in your organization. We will use game elements to make our learning effective and fun! If this sounds like your type of session, I am glad you are here!
• #6: A little bit about me. My name is Dana Pylayeva, I am an Agile Coach and a continuous learner and I like to use games to enhance the learning experience for myself and others. I’ve been in IT industry for 16 years, love wearing multiple hats, in a real live and in my professional career and I never say “NO” to an opportunity. That’s how I ended up taking on a DBA Manager position at one point, without really understanding what it entails. I recovered from that experience (not without some wounds and scars), but it really helped me develop an understanding and deep appreciation for what Operations have to go through long after developers finished celebrating a successful release of their project.
• #7: If you’d like to get some insights into the Ops World (without actually becoming one), I would highly recommend you to read “The Phoenix Project.” It is a great novel about DevOps and in fact is a #1 on the top 60 DevOps books. One of the things I found interesting while reading this book, was that it emphasized the same idea used in Gamification and The Principles of Product Development Flow. Specifically, the idea of Amplified Learning through accelerated feedback loop. Fast and early feedback is important because it allows us to learn about our incorrect assumptions, respond and adapt to the changes in the unpredictable environment of product development. The concept of the feedback isn’t new. In fact, it is one of the building blocks in Scrum! http://dev2ops.org/2010/02/what-is-devops/ Focus on amplified learning through feedback loop acceleration
• #8: Let’s look closely at what’s going on with the feedback loop in Scrum. This framework should look familiar.. Working off of a prioritized product backlog, scrum team builds potentially shippable product increments in 2- 4 week sprints and gets a feedback from the customers on the new features they released. The key here is POTENTIALLY shippable ( and potentially NOT..)
• #9: What do you think happens when the code is not getting deployed into production at the end of the Sprint? Are we getting any feedback from the customers? What if our competitors start offering the features that we are waiting to deploy? Potentially outdated features? All these factors are contributing to the increasing cost of delay in scrum. The fundamental problem with this picture is that Scrum is only used as a framework for a development process. Any process improvements that Scrum team identifies during their retrospectives are all focused on increasing development team effectiveness and velocity. They don’t really help with the overall flow of work through organization. This narrow-focused approach creates an illusion of optimization.
• #10: While Scrum team metrics are getting better, with high team velocity and optimistic burndown chart, Deployment queue piles up in front of the IT Operations Every production deployment becomes a heroic effort. It may cause outages in production and significantly reduce customer satisfaction. We may even loose a few of them..
• #11: And if it fail in productions? Not a Dev problem – It worked on my machine! We all witnessed that large deployments are painful and often result in a service disruption. If step back and look at this situation at the organizational level, we will clearly see the misalignment of goals.
• #12: What do we have here? Ever-changing market demand from our customers. Development team, excited about new technologies, interested in building cool feature and getting them to production quickly. Their slogan is “Think it, build it, ship it, tweak it” And the Operations team – focused on stability and “Keeping the lights on” When deployments are rare and manual it is close to impossible to reconcile these goals. What is the solution and how can we move forward?
• #13: Let’s run an experiment! I’d like you to self-organize around these colorful tables we need groups of 8 ( 9 and 4) I am going to distribute the packages, role cards and players instructions for each group. Please select you roles. Everyone knows their roles? Great!
• #14: What we will do now, we will run an end-to-end simulation of a ChocolateLegoScrum.com company. This company has 3 scrum teams and one operations team. Company is selling LEGO Animals with chocolate packages. Market demand is determined by Animal Exchange. We have some initial demand from the Market and the prices that customers are willing to pay for these animals. A product owner from each Scrum Team, will need to work with the Market Analyst and select the user stories for each scrum team. Before Scrum team can start the development, they all need to have their individual development environments. Adam Admin is your friend! He is the only one who knows how to do it. Any questions so far?
• #15: This is how a sample deployment package looks like: A user story, 5 small packages with Animals and chocolate. Each animal has a security label on it. Scrum team builds individual packages, but they need Robert Release to help them with a deployment package. Any questions so far?
• #16: For this first sprint, everyone will stay within his/her role. Operations team is it’s own department. Releases can only be done during release window. Once deployment package has been built, it will need to go through a security scan. We will run a 15 min Sprint: 2 min for planning, 8 min for sprint, 2 min for demo to Product owner and deployment, 3 min for retrospective. I will ask every group to share their top 2 finding from their retrospective. Any questions before we get started?
• #17: For the Sprint 2, let’s all make one big change and invite Sara Security to work with the development team! Same timing: 15 min Sprint: 2 min for planning, 8 min for sprint, 2 min for demo to Product owner and deployment, 3 min for retrospective. We will share our top 2 findings at the end of the Sprint.
• #18: Now that we’ve experienced the effects of large cycle time and the changes in market demand, we can very much relate to this quote:
• #19: Anyone knows what this number means? 11.6 sec – this a frequency of production deployments at Amazon. Back in 2009, 10 deploys per day at Flickr was a revolution! that many found hard to believe. Now it is 11.6 sec at Amazon! Think about those companies that are still deploying after multiple sprints. Or even after every 2 weeks sprint! New features are being deployed into production every two weeks. Or every 11.6 sec. Which company would you rather be a customer of? What that means to majority of companies out there, that they need to change!
• #20: They need to change to be able to compete in todays world. Unless of cause they don’t care about survival. It’s not mandatory.
• #21: We must change twice to survive. Changing reality is a gradual process ( you can think about planting a tree – it grows really slow, until one day it becomes large enough for a kid to hide behind it. That is when a change of perception happens: from a small tree to a hiding place!) With DevOps we need to change our reality – introduce new tools, new processes in business and IT, but until that change in perseption happens, when people truly embrace the spirit of DevOps, until you see the change in culture, you don’t really have the DevOps organization just yet.
• #22: All right, where do we start? How do you know what to change first?
• #23: Start with stepping back and taking a look at your overall organization to find your bottlenecks.
• #24: Once you found your biggest bottleneck, focus on making major changes to improve it’s capacity. Any improvement done before or after constrain will not have any impact on the overall system performance. We’ve seen it in the sprint 2. When we added Sara security to the Scrum team, the team was able to build more products. But it didn’t help to deliver more products to the customers as our major bottleneck was in the deployment process.
• #25: Quoting “The Phoenix project”: Our goal should be to “ensure the fast, predictable, and uninterrupted flow of planned work that delivers value to the business while minimizing impact and disruption of unplanned work.”
• #26: When you think back about your organization, think about the bottlenecks in YOUR flow of work. A great tool to help you visualize delays and bottleneck in your process is a value stream mapping. Beware of different type of bottlenecks
• #27: Bottleneck may be a result of inefficient tools or lack of appropriate tools, written or unwritten policies ( like a sign off process/gates). Bottleneck can also be a result of limited skills that people have, their outdated mental models, the culture! Tool: The way existing tools are used and/or lack of appropriate tools may limit the ability of the system to produce more. People: Lack of skilled people limits the system. Mental models held by people can cause behavior that becomes a constraint. Policy: A written or unwritten policy prevents the system from making more.
• #28: Don’t become a bottleneck yourself – embrace the idea of continuous learning! Ken Rubin speaks about I person – someone with a deep by narrow skills. Don’t be that person! Become a T person – develop additional skills!
• #29: Come to think of it, People always had a dream about multi-skilled team members!
• #31: Expand your Scrum team to include operations. In Sprint 2, when you included Sara Security into your Scrum team - you noticed less rework. Now bring System Administrators into your Scrum team. They’ve seen how your application performs in production! They can give you a very good feedback and help you design it for better performance. Learn together how to automate environment provisioning and deployments.
• #33: No one likes pain. When the deployment are manual and painful the tendancy is to avoid them. Don’t avoid deployments – simplify them! Explore an option of moving your infrastructure into the cloud. Work with the Sys Admins to automate environment provisioning to reduce surprises during deployment. Make your deployments small, automated and frequent. You will need to change the way your design and architect your systems, to allow for loose-coupling, feature toggling etc. The advantage you will get from this as organization is huge. You will be able to perform A/B testing, validate you ideas and deliver more features to your customers! Here is a crazy idea, what if you didn’t have to wait until the end of the Sprint to deploy? What if you could deploy to production as soon as the story is ready? Do you want to try that? Wide availability of virtualized[18] and cloud infrastructure from internal and external providers Increased usage of data center automation[19] and configuration management tools
• #34: Welcome to the sprint 3! I’d like you to take a look at your cards – notice that most if them have labels on the back. If you are a developer, pair up with someone from Operations and cross-train each other, exchange the labels! Now look at your lessons learnt from Sprint 2 and see how you can improve your process with the new skills that people have. We will also start using smaller packages this sprint, Particia, let’s teach teams how to break these large user stories into smaller ones! We will run the last Sprint as 2 min planning, 10 min sprint ( with ability to do a continuous deployment) and 3 min retrospective. Ready? Let’s start it!
• #36: Anyone feels like this by now?
• #37: DevOps is about creating fast flow of work. DevOps is about amplified feedback loop. DevOps is about experimentation, repetitions and practice. DevOps is about changing the culture.
• #38: The game you played today is a result of collaborative effort. We designed it with Bryan Beecham about a hear ago and presented for the first time at Scrum Gathering in New Orleans. The version you participated in today is a version 3.0, which was extensively re-designed based on the feedback from Scrum Gatherings in New Orleans and Berlin. Today is your opportunity to share ideas on how to improve this game! Please submit your feedback! Let’s make it better together! If you liked the game as is – it is available on LeanPub under creative common license.
• #40: With that I’d like to thank all of you for playing with me today! Feel free to take some chocolate with you! You’ve been an awesome audience and everyone has earned a player badge! Thank you very much!