Deeper dive into dashboards video slides

Editor's Notes

• #6: The main objective of the testing dashboard is to provide simple, meaningful and reliable information in one place to help guide the testing effort and convey that information to our clients.
• #9: In many ways a testing project is like driving a car. You need to know where you’re going, where you are, and the direction you’re headed. As the test continues, you need to know your progress, how fast you are going, if you have the time and people needed to sustain the effort, and how your process (engine) is working.
• #10: Our goal is to reach our desired destination safely. In the context of testing, that means we achieve our objectives without harming anything (for example, not destroying live data, crashing the live system, etc.)‏ To do this, we need to maintain focus and stay on the road. This means that we keep our goals and objectives in mind so we don’t get distracted. Also, we don’t want to get lost. This means we follow our map to the intended destination. In the context of testing, this means we follow the test plan. However, things change and sometimes our plans must change as well. So, we must also be flexible. We don’t want to run out of fuel. For our project, we need to make sure we have the resources to continue. This typically amounts to having enough time and people. Finally, the testing effort needs a leader. Ideally, just one leader at a time.
• #11: In this discussion, there are three effective techniques we will discuss: An effective testing strategy – this defines the test objectives, scope and approach early in the project. The great thing about a testing strategy is that it doesn’t take very long to create and it adds a lot of value to the project. The test strategy is a great vehicle for getting the entire team involved in testing. The strategy describes what the test is intended to accomplish and what makes it unique. A workable test plan – this defines the scope, resources, schedules, risk, contingencies, anything else that is needed to describe how the test should be performed. Basically a test plan is a project plan for the test. A dashboard – this monitors key information from the test in one place. We will examine these in much more detail in this session.
• #13: The concept of the dashboard is nothing new. At the end of this presentation, I will list some additional resources about dashboards. Even with all this information, testers and test team leaders still struggle with effective ways to convey information to management and clients. At the risk of covering something you may have already seen, let’s look at some examples of dashboards. Perhaps you may discover a new way to apply dashboards in your situation. Then we will look at some of the issues about test measurement and reporting.
• #18: A testing dashboard, just like a car’s dashboard, is a set of indicators that show the current status of the test. You can see dashboards from various perspectives such as project dashboards, testing dashboards and dashboards that are created to monitor ongoing system maintenance. There is no standard definition for what should be in a dashboard or even how it should look. That’s why I’m going to show several examples and approaches in the following slides.
• #19: This slide shows some of the major reasons for having a testing dashboard.
• #20: In order to have a dashboard, you must have a way to obtain accurate and meaningful measurements and metrics. You also need a culture of trust and openness or else the measurements and metrics will be at risk of manipulation. As the old saying goes, “you get what you measure.” That’s not always a good thing. Also, the measurements should be non-intrusive. That means that people should not have to take special effort to measure and record them. Ideally, that they can come from activities already being tracked. An example would be defect information derived from defect tracking systems.
• #21: In order to have a dashboard, you must have a way to obtain accurate and meaningful measurements and metrics. You also need a culture of trust and openness or else the measurements and metrics will be at risk of manipulation. As the old saying goes, “you get what you measure.” That’s not always a good thing. Also, the measurements should be non-intrusive. That means that people should not have to take special effort to measure and record them. Ideally, that they can come from activities already being tracked. An example would be defect information derived from defect tracking systems.
• #22: Once again, there is no standard or consensus as to what should be on a testing dashboard. This slide shows some of the typical items found on testing dashboards. You may think of others you would like to add.
• #35: Dashboards are also used at the project level. They have the same characteristics as a testing dashboard, but include different measurements and metrics.
• #36: This is a sample project dashboard from the U.S. Navy. It covers areas such as progress, change, quality, and risk. Not all dashboards are this elaborate!
• #40: It’s best to keep the dashboard simple. Too many items can be distracting and confusing – unless of course you’re flying a plane! The analogy here is that a more complex situation may require a more complex dashboard. As we have mentioned, metrics can be abused. This especially happens when the metrics start to shape people’s behavior. And finally, don’t forget that the unexpected happens. Things that are not shown on your dashboard can derail your test. This includes things such as reorganizations, scope increases, and changes dictated by management.
• #42: Dashboards also contain warning indicators to let you know when things are working right. In this context, the engine is the process.
• #44: It’s really great when testing can be seen as a value-added part of the project, instead of a “problem finding” part of the project.