![Copyright © 2011 Constant Contact, Inc.11This is appropriate if you are building an…“Originally planned to enter service in May 2008, the [Dreamliner] project has suffered from repeated delays and is now more than three years behind schedule.” -- Wikipedia](https://image.slidesharecdn.com/2011beyondascrumofscrums-110616100342-phpapp02/85/Beyond-Scrum-of-Scrums-26-320.jpg)
Beyond Scrum of Scrums
- 1. Copyright © 2011 Constant Contact Inc.Beyond a Scrum of Scrums: Scaling Up Agile with KanbanGil IrizarryProgram ManagerConstant ContactJune 2011
- 2. Copyright © 2011 Constant Contact, Inc.2Agenda A bit about me and Constant Contact
- 3. Scrum and Kanban at the team-level
- 4. Scaling up to an engineering orgCopyright © 2011 Constant Contact, Inc.3My background Over 20 years software development and management experience
- 5. Over 5 years in an agile software development environment
- 6. Most recently working with Kanban, prior to that Scrum and RUP
- 7. CSM and PMP certifications, Kanban coaching training with David Anderson
- 8. BS from Cornell, ALM from Harvard, certificate in Management from MIT SloanCopyright © 2011 Constant Contact, Inc.4My contact info girizarry@constantcontact.com
- 10. Contact me if you would like updated slidesCopyright © 2011 Constant Contact, Inc.5Background on Constant ContactSaaS company offering on-line e-mail marketing, event marketing and surveys. Recent enhancements extend the services to the social media space
- 11. >$200MM gross revenue per year
- 12. >800 employees
- 13. >450K paying customers
- 14. Engineering and Operations total about 150 people
- 15. First Scrum team formed in 2006Copyright © 2011 Constant Contact, Inc.6The Goal We want to release more features more frequently
- 16. Why?
- 17. React quicker to changing market conditions
- 18. Get new features to users more quickly
- 19. Frequent releases are smaller releases: our customers often cannot absorb a large set of changesCopyright © 2011 Constant Contact, Inc.7The Challenge 13 software development teams
- 20. 5 development and testing server clusters
- 21. 2 production server clusters in 2 data centers (soon to be 3)
- 22. A code base that was first created around 2004 and built up over time
- 23. And all utilizing one operations teamCopyright © 2011 Constant Contact, Inc.8What do we have? A mess of inter-dependencies
- 24. Image from http://viewyonder.com/2009/07/12/data-center-consolidation-road-to-unified-computing-service-analysis/ - Data Center Spaghetti. This problem exists in many other parts of the enterprise!Copyright © 2011 Constant Contact, Inc.9So, how do we address this? Scrum of Scrums!
- 25. Um, wait…Copyright © 2011 Constant Contact, Inc.10A little review: how things used to be
- 26. Copyright © 2011 Constant Contact, Inc.11This is appropriate if you are building an…“Originally planned to enter service in May 2008, the [Dreamliner] project has suffered from repeated delays and is now more than three years behind schedule.” -- Wikipedia
- 27. Copyright © 2011 Constant Contact, Inc.12Quick Review of ScrumFixed iterations
- 28. Daily stand-ups
- 29. What did you do yesterday, what did you do today, any impediments
- 30. Retrospectives
- 31. Burn-down chart
- 32. Board with To Do, In Progress and Done statesCopyright © 2011 Constant Contact, Inc.13Scrum in a slide
- 33. Copyright © 2011 Constant Contact, Inc.14Scrum of Scrums (of Scrums (of Scrums…)) How to integrate teams in Scrum?
- 34. Hold a Scrum of Scrums
- 35. What happens if you have too many teams?
- 36. Hold a Scrum of Scrums of Scrums, and so on…
- 37. Add a MetaScrum or twoCopyright © 2011 Constant Contact, Inc.15Foundational Principles of Kanban Start with what you do now
- 38. Agree to pursue incremental, evolutionary change
- 39. Respect the current process, roles, responsibilities & titlesFrom: http://agilemanagement.net/index.php/Blog/the_principles_of_the_kanban_method (David Anderson)
- 40. Copyright © 2011 Constant Contact, Inc.165 Core Properties of Kanban Visualize the workflow
- 41. Team board states are a reflection of the value stream
- 42. Limit WIP
- 43. Manage Flow
- 44. Implied that flow should be continuous
- 45. Make Process Policies Explicit
- 46. Improve Collaboratively (using models & the scientific method)Copyright © 2011 Constant Contact, Inc.17Sample Kanban Board
- 47. Copyright © 2011 Constant Contact, Inc.18Sample CFD
- 48. Copyright © 2011 Constant Contact, Inc.19Compare with Burndown chart
- 49. Copyright © 2011 Constant Contact, Inc.20Scrum and Kanban: Let’s combine them!(But how? And why?)http://en.wikipedia.org/wiki/Reese's_Peanut_Butter_Cups#Marketing_and_advertising
- 50. Copyright © 2011 Constant Contact, Inc.21Transistioning from Scrum to Kanban We did not change how teams were currently working
- 51. We modeled existing hand-offs within the team, i.e. each team’s kanban board reflected that teams style of workCopyright © 2011 Constant Contact, Inc.22The DilemmaRemember that teams want (and often need) to work continuously, but releases are discrete.Multiple teams working together create interdependenciesHow to resolve this?
- 52. Copyright © 2011 Constant Contact, Inc.23Transforming Scrum (and Scrum of Scrums) At the enterprise-level, team(s) management becomes a problem of dependency management
- 53. Planning, coordination and stand-ups at the departmental and organizational level need to be executed with this mindCopyright © 2011 Constant Contact, Inc.24What we have done at CTCT At the organization-level:
- 55. All At Once Planning
- 57. (Near) Continuous integration and (mostly) automated regression testing
- 58. Dependency and deliverable review
- 59. Release trainCopyright © 2011 Constant Contact, Inc.25What we have done at CTCT At the team-level:
- 61. Better metrics and metrics-driven estimationCopyright © 2011 Constant Contact, Inc.26Prioritized Project List Product Management, Engineering and Operations leads meet every two weeks to prioritize all development projects
- 62. This becomes the organizational backlog, which drives each team’s backlogCopyright © 2011 Constant Contact, Inc.27All At Once Planning
- 63. Copyright © 2011 Constant Contact, Inc.28All At Once Planning Prior to the start of an iteration, teams use the prioritized project list to plan their upcoming work.
- 64. Planning involves the identification of deliverables and dependencies.
- 65. Dependencies are discussed with dependent teams.
- 66. A meeting is held (all at once planning mtg) in which all development teams present their dependencies to each other and to the operations team.Copyright © 2011 Constant Contact, Inc.29All At Once Planning, cont’d At the end of the meeting, each team has their planned deliverables and incoming dependencies.
- 67. If they haven’t already, they determine their capacity and, based on the priorities, commit to a set of work.
- 68. This means that a team may have capacity to do work, but may not get to it in a release if that work pushes the operation team beyond its capacity.
- 69. Once the iteration starts, we will have each team’s set of commitments.Copyright © 2011 Constant Contact, Inc.30Classes of Service Inevitably, the operations team can often become a bottleneck for development.
- 70. We attempt to manage this through classes of service. We borrow a concept from Kanban that says similar projects are grouped into classes and each class is assigned an allocation.
- 71. For example, we may decide that 20% of ops time should be spent on infrastructure improvements, and 80% spent on servicing developmentCopyright © 2011 Constant Contact, Inc.31(Near) Continuous integration and (mostly)automated regression testing It is important that builds, deployments and testing are as automated and continuous as possible. Tools we use:
- 72. Hudson for continuous integration
- 73. Puppet for automated deployments
- 74. Maven for managed builds
- 75. Selenium for automated testing
- 76. svn for source control and versioning
- 77. We still have a lot of work to do in this areaCopyright © 2011 Constant Contact, Inc.32Dependency and deliverable review We have a daily meeting, about 20 minutes, long in which we review each project, not each team.
- 78. Project leads review the deliverables and dependencies to which they have committed and say if they are on track or not.Copyright © 2011 Constant Contact, Inc.33Dependency and deliverable review, cont’d We found the teams could still miss a deliverable even if they had no impediments. Deliverable tracking provides a better view of the state of the iteration
- 79. However, we still call the review meeting “Scrum of Scrums” because the name stuck!Copyright © 2011 Constant Contact, Inc.34Release Train Releases are iterative but development is continuousCopyright © 2011 Constant Contact, Inc.35Release Train, cont’d Scrum aims to make development iterative but this causes problems:
- 80. How do you handle all the testing at the end of the sprint? What if defects are found a day before the sprint ends?
- 81. If a deliverable misses a release (the train), it simply waits to capture the next one.
- 82. To be reiterate (and be explicit): we don’t penalize a team if a deliverable is not done at the end of a release and misses the release trainCopyright © 2011 Constant Contact, Inc.36Release Train, cont’d To make this work:
- 83. Releases must be short
- 84. Must have those tools for automated deployments
- 85. Marketing and support must be flexible enough to react to last-minute product changesCopyright © 2011 Constant Contact, Inc.37Kanban at the team-level Teams plan continuously
- 87. It’s OK if a team finds a defect on the last day of the release.
- 88. It’s OK if a team starts work for the next release in the current release
- 89. Development and testing can flow more smoothly, because we do not want the end of an iteration to feel like this:Copyright © 2011 Constant Contact, Inc.38Kanban at the team-level
- 90. Copyright © 2011 Constant Contact, Inc.39Metrics We gather for each team:
- 91. Cycle time on items after grouping them by size:
- 92. Completion time for small, medium and large
- 93. Spread of cycle times
- 95. Open defects in production, to give a gross measure of technical debtCopyright © 2011 Constant Contact, Inc.40Metrics guide planning and estimation Over time, we expect that the spread of cycle times for a given item size goes down.
- 96. So, over time, an estimate of completion time for items of a given size should become more accurate.
- 97. We have eliminated planning poker. Work items are just sized as smalls or mediums and the average cycle times for those sizes from the last release become the estimate for the upcoming release.
- 98. Large items are broken down in smaller itemsCopyright © 2011 Constant Contact, Inc.41
- 99. Copyright © 2011 Constant Contact, Inc.42How did we do? Releases used to take 12 weeks. Now they take 5 weeks.
- 100. More importantly, testing used to take 6 weeks. Now it takes 1 week. Testing used to be 50% of the release cycle, but now is just 20%.
- 101. We have a better picture of our release at any given moment
- 102. A year ago, CTCT was not in the social media space. Now our service has ties to Facebook, Twitter and LinkedIn. We are becoming a leader in the social media marketing space.Copyright © 2011 Constant Contact, Inc.43How did we do?
- 103. Copyright © 2011 Constant Contact, Inc.44Time for Questions