![Agile Metrics (by Venkatraman L) 10
Risk & Quality Metrics
There is a misconception in the community that Agile does not focus on risks. Agile inherently focuses on
the risks of the project and the portfolio by enabling the teams to call it out almost daily if not immediately
as they occur. Mike Cohn also talks about two methods by which Risks can be factored into determining
the schedules using the velocity of the team. They are called the Risky and Rigorous [2] process of risk
management and leads to a risk-adjusted backlog, which is nothing but the product backlog that is
interspersed with, risk responses.
Similarly, the ones that we can capture are the escaped defects that got out of a release and the
customer caught them. There are enough quality metrics that can be captured in terms of priority,
severity of the defects, defects that got moved from the previous release of the product (deprioritized
releases), the defects that got ingested into the system due to fixing few other releases.
Managing a Program / Portfolio of Releases
The Dynamics of a portfolio
Portfolio as the name suggests is used in organizations when their systems and capacity scales across
the board. Portfolios have themes that in turn can map to one or more initiatives or programs. In a fast
growing nature of the business, the themes in the portfolio might have varied number of programs and
initiatives within them. The key is to focus on what is priority for the business and for the customer.
The Prioritization Phase
During the portfolio prioritization phase, the metrics to look at are the ROI (return on investment), NPV
(Net present value) of each of the projects that are within the portfolio. This can be using the “value
points” [1] suggested by Jim Highsmith. The Cost vs Value analysis would provide the details around
whether a particular project or feature needs to be implemented. Eventually, the initiative that the feature
is part of benefits or loses based on the choices made during the prioritization phase.](https://image.slidesharecdn.com/venkatramanl-131008015800-phpapp01/85/Venkatraman-l-10-320.jpg)
![Agile Metrics (by Venkatraman L) 11
The Planning & Execution Phase
During the planning and execution phase, the key is to pick the set of stories at sprint level (user stories
as termed in Agile) that are of high value point [1]. The value flows all the way from the top at the
portfolio level and the costs flow bottom up leading to a consensus. Pie-charts can detail what the
contribution margin that each of the portfolio themes are providing to the organization as a whole.
The other metrics that are important is the cycle time of the entire program (when one team begins and
the final team ends the release) and how the cycle times can be cut down. The other metrics used are the
“Process Cycle Efficiency” that projects how much waste has been cut down in the system.
Scrum-Ban and Scrum-fall
In order to focus better on the cycle time, large programs do the release plans but execute with the
methodology of Kanban. As already called out in the introduction, a combination of waterfall + sprints +
Kanban during execution of the sprints can be really helpful in very large projects where each phase can
be considered a program by itself with clear goals and metrics to achieve. Using Kanban, the following
metrics are captured – lead-time, cycle time, work time and wait time (idle time between various
states). This has been the key to deliver value, faster.](https://image.slidesharecdn.com/venkatramanl-131008015800-phpapp01/85/Venkatraman-l-11-320.jpg)
![Agile Metrics (by Venkatraman L) 13
In short, there are enough and more metrics in Agile development that can enable the teams to make
wise decisions using them. The key is to know “what to measure and why”
References
[1] Highsmith, Jim: Agile Project Management: Creating Innovative Products
[2] Cohn, Mike. Agile Estimating and Planning](https://image.slidesharecdn.com/venkatramanl-131008015800-phpapp01/85/Venkatraman-l-13-320.jpg)
Venkatraman l
- 1. Agile Metrics (by Venkatraman L) 1
- 2. Agile Metrics (by Venkatraman L) 2 Agile Metrics (by Venkatraman L) In Projects & Portfolio Venkatraman L Head of Project Management InMobi 20-Jun-13
- 3. Agile Metrics (by Venkatraman L) 3 Table of Contents Introduction.................................................................................................................................... 4 Abstract...................................................................................................................................... 4 Keywords................................................................................................................................... 4 Introduction................................................................................................................................ 4 Context of the project............................................................................................................... 5 Context of the program............................................................................................................ 5 Context about the portfolio of releases................................................................................. 6 Overview of Agile ......................................................................................................................... 6 What is Agile ............................................................................................................................. 6 How is it different from the Traditional (waterfall) model.................................................... 7 So, how does Agile solve for it?............................................................................................. 7 How does the triple constraint stack up in Agile.................................................................. 7 Is Agile applicable on all type of projects? ........................................................................... 8 Managing Projects through Agile............................................................................................... 8 Metrics during the Plan Phase ............................................................................................... 8 The Sprint – What is measured during execution............................................................... 9 Risk & Quality Metrics ........................................................................................................... 10 Managing a Program / Portfolio of Releases......................................................................... 10 The Dynamics of a portfolio.................................................................................................. 10 The Prioritization Phase ........................................................................................................ 10 The Planning & Execution Phase........................................................................................ 11 Scrum-Ban and Scrum-fall.................................................................................................... 11 Conclusion................................................................................................................................... 12 Snapshot.................................................................................................................................. 12 References .......................................................................................................................................................13
- 4. Agile Metrics (by Venkatraman L) 4 Introduction Abstract As the cliché goes “You cannot manage what you don’t measure”, it has been always a debate about how much to measure and what to manage. Agile Projects have been of late topic of discussion for all reasons. One of them in particular has been the impact of capturing less metrics for managing and control. To define what is “less” is the tough part. Tougher is to understand what to measure and why it’s being measured. Agile does not prescribe that set of things has to be followed. If you do so, you are agile. If you don’t, you can still be agile. The key is in understanding the problem that the teams are expecting to solve. The paper discusses the various metrics that can be used in Agile to plan, execute and monitor the entire project over the lifecycle. There is also a fundamental need to measure some of the metrics that were traditionally in practice, in order to bring out a change, even in Agile teams. Also, these metrics and the changes that they facilitate are different at the project, program and portfolio levels. The paper provides a view of the metrics at project and portfolio levels broken down as Technology & Engineering, Operations and Business metrics. The paper ends with a detailed matrix of the metrics across the three. Keywords Story Points, Velocity, Escaped Defects, Burn-down, Burn-up, Ideal Time, Agile EVM, Cycle Time Introduction The paper describes a project scenario, a program and a portfolio of releases and how the agile metrics are used across the varied type of delivery needs. The key is also to understand how some of the metrics are critical for the success for a program and the portfolio (as the portfolio is a set of programs, releases and projects eventually).
- 5. Agile Metrics (by Venkatraman L) 5 Context of the project The project that is referred here is both a NPD (new product development) and enhancement of the existing features of the product. The product is a portal that enables users to watch videos of their choice. The NPD mentioned above talks about revamping the entire watching experience (the new product of the same capability was built from scratch and replaced the existing product completely). The team in this case was a self-sufficient team (front-end, middleware, backend, QA and Operations) and the team size was more or less intact through he duration of the project. However, the teams were a bit distributed (the product owner and the operations team were in the US and the rest of the team were in India). Context of the program The program here, at a different organization refers to versions of releases that define a program. It is part of the portfolio as mentioned below but its primary objective is to meet the program goals.
- 6. Agile Metrics (by Venkatraman L) 6 Context about the portfolio of releases The portfolio that is being referred is a set of releases (>40) cutting across multiple themes in the organization. There are around 8 themes and the 40+ releases map to a theme. The key to note is that each release is “contributed to” by more than one team in the organization. The teams are formed by the nature of the architecture and they are not cross-functional unified team. The colored lines below shows how multiple teams (M) contribute to multiple releases (N). The organization terms this as the “Release Dependency Matrix” and is the key behind the success of the project. Overview of Agile What is Agile Agile is a set of values and guiding principles that guide the teams and organizations to the path of successful delivery. The focus is on value driven delivery than just the triple constraints of Cost, Time and Scope (Jim Highsmith).
- 7. Agile Metrics (by Venkatraman L) 7 How is it different from the Traditional (waterfall) model So, how does Agile solve for it? Agile is based on Deming’s PDCA (Plan, Do, Check, Act) and focuses on a concept of iterative and incremental development and delivers the highest value features sprint over sprint (a time-boxed duration of work is called a sprint). The customer (or the product owner) focuses on always providing the list of high priority requirements to the team that the team commits to, sprint over sprint. The customer (in the role of a Product Owner) is expected to be available all through the project lifecycle so that the teams can collaborate well and get faster feedback from the customer. The team plans every 10 days on a set of requirements, (that have been stack ranked by the product owner) discuss and thrash the details, commit to delivering it within the 10-day cycle (sprint). Ad the end of the sprint, the customer is shown a demo of how the product or feature is progressing. This helps to provide faster feedback and WYSIWYG (what you see is what you get). How does the triple constraint stack up in Agile In case of the traditional waterfall mode, the scope is provided and the project manager develops a detailed plan to articulate the cost and the time it takes to develop the scope. When scope changes, the Waterfall model (Traditional model as it is called) focuses on staged development. Requirements, high level design, low level designs, development, unit testing, integrated testing, user acceptance testing and release. The key is that the customer is not always (some purists say never) that involved in all the phases of the project. The customer eventually takes a demo during the final phase of the project and end up realizing that what he / she wanted is a bit far from what has been implemented. This has been one of the shortcomings of the traditional waterfall model, which has led to Agile being one the most opted methodologies of delivery.
- 8. Agile Metrics (by Venkatraman L) 8 entire process is re-run or at least the additional time and cost is being worked out and re-baselined. In case of Agile, the triangle is flipped around totally. The available constants are the cost (of the team + operational), the time (2-4 weeks of sprints) and the variable in this case is the scope. One should not mis-read the variable beyond the need of delivering “high value” scope and not take it as though scope is completely open for changes all the time. Is Agile applicable on all type of projects? Yes, the reason being that Agile is not a process but a set of values, principles and practices, that, put together can generate wonderful value for the end customer. In large projects that follow waterfall model, the various phases within the waterfall can be run through agile sprints. For eg: the requirements and analysis phase that delivers the high level architecture and design can be run as multiple sprints and each has a specific goal as the deliverable. Managing Projects through Agile Metrics during the Plan Phase When planning a release for a product having multiple new or enhancements, the entire backlog is broken down into a set of requirements called the product backlog. The teams work off the product backlog in determining the set of agreeable high priority requirements and commit to the “sprint backlog”. The various metrics that are used in this case is the concept of ideal days (the duration that is required ideally if nothing else was being done by the developer / tester), the story points for each user story (which is nothing but a relative size estimation of the complexity of the work), the velocity of the team (the number of story points that the team can comfortably deliver). The number of sprints are
- 9. Agile Metrics (by Venkatraman L) 9 typically calculated by the formula total sprints = total story points for the release / velocity of the team. This is also called as the Agile EVM – Earned value management where the projection of timeline happens through the available metrics. Please note that Agile EVM is an elaborate topic in itself. The Sprint – What is measured during execution During the sprint execution (after the sprint starts), the teams use the concept of burn-down (a graph that plots how much of effort in hours) that is still pending to be completed by the team. This plots the available capacity vs the pending to be completed and compares it with the available work in hand. The team and in specific the product owner also starts looking at the burn-up (at sprint level) eventually cumulating at the release burn-up chart level, which talks about how many story points were delivered by the team. Jim Highsmith also talks about the value points that get associated with the stories that can also be calculated to understand how much of business value was delivered against the planned. One of the other key metric to capture in a team is to understand how much time the team is able to focus on new features vs fixing existing bugs. This is a key in the utilization of the team’s efforts in delivering true value to the customer. The teams can also capture the stretch factor (how much was the additional effort put by the team to meet the sprint commitments). These could be due to the improper estimations, unexpected events of effort loss etc.
- 10. Agile Metrics (by Venkatraman L) 10 Risk & Quality Metrics There is a misconception in the community that Agile does not focus on risks. Agile inherently focuses on the risks of the project and the portfolio by enabling the teams to call it out almost daily if not immediately as they occur. Mike Cohn also talks about two methods by which Risks can be factored into determining the schedules using the velocity of the team. They are called the Risky and Rigorous [2] process of risk management and leads to a risk-adjusted backlog, which is nothing but the product backlog that is interspersed with, risk responses. Similarly, the ones that we can capture are the escaped defects that got out of a release and the customer caught them. There are enough quality metrics that can be captured in terms of priority, severity of the defects, defects that got moved from the previous release of the product (deprioritized releases), the defects that got ingested into the system due to fixing few other releases. Managing a Program / Portfolio of Releases The Dynamics of a portfolio Portfolio as the name suggests is used in organizations when their systems and capacity scales across the board. Portfolios have themes that in turn can map to one or more initiatives or programs. In a fast growing nature of the business, the themes in the portfolio might have varied number of programs and initiatives within them. The key is to focus on what is priority for the business and for the customer. The Prioritization Phase During the portfolio prioritization phase, the metrics to look at are the ROI (return on investment), NPV (Net present value) of each of the projects that are within the portfolio. This can be using the “value points” [1] suggested by Jim Highsmith. The Cost vs Value analysis would provide the details around whether a particular project or feature needs to be implemented. Eventually, the initiative that the feature is part of benefits or loses based on the choices made during the prioritization phase.
- 11. Agile Metrics (by Venkatraman L) 11 The Planning & Execution Phase During the planning and execution phase, the key is to pick the set of stories at sprint level (user stories as termed in Agile) that are of high value point [1]. The value flows all the way from the top at the portfolio level and the costs flow bottom up leading to a consensus. Pie-charts can detail what the contribution margin that each of the portfolio themes are providing to the organization as a whole. The other metrics that are important is the cycle time of the entire program (when one team begins and the final team ends the release) and how the cycle times can be cut down. The other metrics used are the “Process Cycle Efficiency” that projects how much waste has been cut down in the system. Scrum-Ban and Scrum-fall In order to focus better on the cycle time, large programs do the release plans but execute with the methodology of Kanban. As already called out in the introduction, a combination of waterfall + sprints + Kanban during execution of the sprints can be really helpful in very large projects where each phase can be considered a program by itself with clear goals and metrics to achieve. Using Kanban, the following metrics are captured – lead-time, cycle time, work time and wait time (idle time between various states). This has been the key to deliver value, faster.
- 12. Agile Metrics (by Venkatraman L) 12 Conclusion Snapshot Here is a short summary of the various metrics that were captured in the above projects. Marked with (G) is also for the Program and marked with (F) is for the portfolio also along with program Tech Metrics Operational Business Project Test Coverage Escaped Defects (G) In Sprint Defects Cost of rework Code Quality (G) Performance (G) Defect ingestion Story point velocity per sprint Available capacity Capacity utilization Cost of the sprint ($) Story points accepted Story points not accepted New scope added in sprint Technical Debt added Agile EVM (G) Risk Register updates (G) Stretch factor Time spent on Bugs vs feature Value delivered (P) Customer Satisfaction (P) Program Cycle Time (P) Process cycle efficiency (%) Release Burn up Scope creep Contractual metrics ($) Portfolio Pie-chart of releases Health of the portfolio Prioritization changes ROI in the portfolio Contribution Margin As called out earlier, this is just a suggestive minimal list of metrics. Based on the governance and the type of the organization, there might be more metrics such as time spent vs billed, actual billing cards, agile contracts etc that might need to be added to this list.
- 13. Agile Metrics (by Venkatraman L) 13 In short, there are enough and more metrics in Agile development that can enable the teams to make wise decisions using them. The key is to know “what to measure and why” References [1] Highsmith, Jim: Agile Project Management: Creating Innovative Products [2] Cohn, Mike. Agile Estimating and Planning