The DevOps Centipede

Editor's Notes

• #2: I’ve managed technical teams for much of my 20 year career and wanted to share some things I’ve learned along the way. If you aren’t familiar with The Human Centipede, there’s no way for me to explain it and stay within our Code of Conduct. No, don’t Google it. You’ll get the general idea.
• #3: This is Conway’s Law. From my experience I believe it to be absolutely true. It states that when you design a system, it’s going to have internal boundaries along the lines of your organizational structure.
• #4: The most typical example of Conway’s Law in action is when various teams create a “Frankenapp” where parts of what should be one user-facing application look or act differently from the others.
• #5: And no one likes a Frankenapp. Techniques to mitigate Conway’s Law in the dev arena range from collective code ownership to Big Books of UI Standards.
• #6: But from the DevOps viewpoint we know there’s another organizational dimension that often divides us. This is equally harmful to the resulting product and resembles another famous movie monster… I call it the DevOps Centipede.
• #7: Organizations have tried many, many different org structures in an attempt to fulfill their mission, whatever it is (delivering working software is, strangely, not the primary supported goal in some cases). Let’s go through some that I’ve used or seen used and talk about their pros and cons.
• #8: We’ll start with the DevOps Centipede models. The most typical “IT Shop” org is divided by technical discipline. To deliver a single application, you need people from ten different teams to somehow work together to deliver it, which goes wrong more often than not.
• #9: One attempt to mitigate this is to create intermediary teams (some call them “DevOps!”) to add some focus and ownership. We did this at NI as a first attempt at fixing the initial problem. However, this creates brittle bottlenecks.
• #10: Another mitigation is to pick specific people from those groups and give them an affinity or “ownership area” for a given scope (app, team, etc.). This works until a single person is supposed to own a whole batch of things, and then it collapses under its weight.
• #11: All these are “zone” instead of “man to man” strategies. They are better when there’s uncertainty from lack of direction or when the ratio of staff to desired solutions is way less than one-to-one. They are designed more to mitigate large scale failure, not to facilitate large scale success. Let’s talk about man-to-man strategies now.
• #12: Here we have teams oriented by product, or if you will, service. This is basic Agile doctrine, as well as being recommended by ITIL v3 and other IT service management methodologies. It creates great focus and ownership but team members can suffer from lack of coaching and career advancement, and a lack of standardization can have later negative effects.
• #13: In this case we have people from these disciplines still reporting back to a per-discipline manager but residing semi-permanently in a product team. We did both this and the previous model at Bazaarvoice – the drawback of this model is that it can cause a reluctance to fully embed on the product teams as a primary concern and a related confusion or conflict of interest, but it helped correct the personnel management issues from the previous approach.
• #14: Halfway between these two is a rapid innovation model (popularized by Spotify) where “chapters” and/or “guilds” are used as discipline-focused “user groups” across the organization. We intended to try this, since it seems like it might reduce the ‘two boss factor’ while retaining some esprit de corps for the different types of techies, but the “above and beyond” effort required to initiate maintain it always faltered.
• #15: And what about standards and common elements? “Platform” teams can provide elements the products are built on – but must be careful to deliver software either as a tool or a service, and not to take on work for those teams – otherwise this reduces to the earlier zone model with bottlenecks.
• #16: But what if we just have a sea of people and move them where they are needed, when they are needed? This has many downsides but the worst, and the dealkiller in my mind, is that it creates a lot of engineer dissatisfaction. Whenever we didn’t let teams get through Tuckman’s stages of development and perform together and moved individual people around frequently, engineer satisfaction plummeted. However, for some use cases like a consulting firm, the flexibility this gives might (?) be worth it.
• #17: All these models have pros and cons can work for certain missions and workloads, but here’s some ways to screw any of them up.
• #18: First of all, orphans. If you develop apps, or worse, services, and then just let them become “unowned”, you are deliberately introducing chaos into your org, because they always end up “wanting some more.” Have a service catalog that lists every item developed and enforce a chain of custody of ownership.
• #19: So let’s have a sustaining team! This is often a very bad idea. It creates different classes of engineer, and also removes a lot of the benefits of owning your own solution, being responsible for its quality and learning from its operation. If you really are churning out a large volume of stuff you don’t care about further developing – you should stop and look at how you’re composing your application portfolio and ask yourself ‘why?’
• #20: Also a chronic problem in each of these models is the manager who believes he has suddenly discovered the wisdom of Solomon and can allocate “half a person” to a team or product. Sometimes teams can share a single skilled resource (like a security engineer that works with 2-3 product teams because there’s not enough demand for him full time on just one) but by and large you should figure out how to make that work serial and not parallel – besides the usual “keep WIP low” benefits, it’s very difficult to do this and not make other mistakes in the package (overallocation, etc.)
• #21: There’s not one magic org structure for all circumstances, but you should try to avoid the wrath of Conway’s Law by experimenting to get the best flow – and employee happiness – in your organization. If you’re interested in this kind of topic, I’m writing a series on the topic on the agile admin. Thanks and good afternoon!