Rolling Dice and Measuring Outcomes

Rolling Dice and Measuring Outcomes – D M Goldstein, August 2025

In the Software Support world we love our metrics, many of which are represented by three-letter acronyms (TLAs) like Net Promoter Score (NPS), Average Handle Time (AHT), First Response Time (FRT), and the king (at four letters), Customer Satisfaction (CSAT). There is a growing realization that we may have become too obsessed with chasing specific numbers and should be focusing instead on whether we are achieving the desired outcomes.

Rolling the Dice

This got me thinking about rolling two six-sided standard dice. If they are uniquely identifiable, such as having one red and one blue, then mathematically there are 36 possible combinations when you roll them. For example, “Red-1 Blue-6” has a 1:36 (2.8%) chance of coming up. If they are both generic white dice, there are 21 possible combinations (1-1 through 1-6, 2-2 through 2-6, etc.), with “doubles” each having that 1:36 chance, and the other 30 combinations each having a 2:36 (5.6%) chance, since 3-4 is the same as 4-3. But there are only 11 possible outcomes, totaling 2 through 12. Only one combination nets you either 2, 3, 11, or 12, two combinations net you 4, 5, 9, or 10, and three combinations for 6, 7, or 8. 6 and 8 each can be done with a pair of doubles, so they each have a 5:36 (13.9%) chance, while 7 has a 6:36 (16.7%) chance. Why is this relevant? Well, you need to ask yourself whether you are trying for a 1-6 combination at 5.6% or is any combination which totals 7 just as good, with a 16.7% chance? It is about paying attention to the desired outcome rather than the specific means of achieving it.

Choosing an Outcome that matters

People’s behavior will adapt to what they are being goaled on. If the goal is, “Get everything out of our warehouse and into customers’ hands”, then don’t be surprised if somebody puts it all out on the street and gives it away for free. Clarity of the objective and the reasoning behind it will drive better behaviors than an over-simplified mandate will.

Most, if not all, companies rely on increasing revenue to stay in business. So, let’s look at Net Recurring Revenue (NRR), which is a financial measurement for customer growth and retention. For the following discussion, we will make increasing NRR the measure of success - the desired outcome. All other goals and metrics can be viewed by whether and how they affect that outcome. With the dice analogy, there may be 36 possible rolls, but only 21 combinations and only 11 outcomes, with some (rolling a 7) up to six times more likely than others (rolling a 2). Understanding that may help you make wiser choices.

RIMC

I’ve come up with an acronym to measure the positive or negative aspects of various goals: RIMC - Risk, Impact, Mitigation, and Cost. Tying it to the roll of the dice, if you need a 2 or 12 then it is high risk and long odds. If you need a specific number combination, that is also high risk. But if you “just need a seven”, that is much lower risk, with multiple paths and better odds to get the same outcome. Let’s look at some examples.

Goal: “10x new logos this fiscal year.” One way to increase the NRR is to add new customers. If you are a startup who just shipped your first release, you might be starting with 1-3 customers. If so, a “10x” increase to 30 is a good one-year goal. But if you are a large established company in a stable industry, you might not get that kind of growth organically, outside of acquiring a competitor. The Risk of pushing organic growth may be investing too much in Sales or Marketing, and ignoring Product, Engineering, and the customer-facing Support and Success teams. The Impact may be signing up “bad” customers - those for whom your product is not a good fit, at which point you will either experience churn or put extra burden on other teams to try to retain these customers. The Mitigation may be strong oversight and review for new prospects, or “claw-backs” of commissions from Sales teams for customer churn. And the Cost may be high customer and staff turnover, lack of progress on the product roadmap, and a poor reputation. You may be spending more chasing new logos than the higher payoff from servicing and protecting current accounts. Of course there is upside potential as well; if done well the company has higher revenue to drive and grow the business. Measuring the outcome of increased customer count, low customer churn, and better financial results, is what matters; it’s not just a new logos game. Recognize “good” sales and successful retention efforts, many of which may be subtle in their execution. Rolling a 6-1, 5-2, or 4-3 (and vice versa) all get you to 7; celebrate and encourage each of them.

Goal: “95% CSAT.” Customer feedback is a very useful tool, but CSAT has nuances that are often overlooked. There are Risks with CSAT. Among them are surveying the wrong people (end-users instead of decision-makers), the fact that it is an after-the-fact activity (not capturing upset customers at the point where you can fix the problem), misattribution of the ratings (staff vs product vs pricing vs relationship), and game-playing (“If I threaten to give a bad survey I have more leverage during renewal negotiations”). CSAT is an abstraction that does not truly tell you where you are. The Impact is potentially making bad decisions based on a misunderstanding of the data. Blaming Support for unstable product, blaming Engineering for a “wrong” customer, and blaming any single department when CSAT is a company-wide report-card are all going to drive the wrong behaviors. Mitigation efforts may include using detailed semi-annual broad surveys instead of only transactional ones, as I have written about in an earlier article (Ref 1), or personal follow-up on negative or ambiguous feedback. There are also AI tools that can determine “customer sentiment” during interactions to get more real-time data. The Cost is what happens when people start playing games based on a bad objective. Teams may start pointing fingers at each other, or customer-facing teams may be trading favors with the customers for positive reviews. CSAT is a “lagging indicator” at best; getting feedback from customers throughout the relationship is a better way to read their sentiment. High CSAT tends to correlate with increased retention and revenue, which is the desired outcome.

Goal: “Under 5 minutes Average Handle Time.” That sure sounds like a great goal for Support issues. Or does it? Do “Average Handle Time” or “First Response Time” matter to a business’ ability to succeed? Indirectly, yes, but only in that they might affect CSAT. When I go to see a doctor, I check in and wait until an RN or PA takes me to an exam room and checks my vitals. That is somewhat like “First Response Time” for Support issues. Then the doctor comes in. Do I want my doctor to worry about how much time they are spending with me? Or would I prefer that they fully understand my issue and that I fully understand their guidance? The Risk of being goaled on AHT is that staff could be more concerned about closing the case than with helping the customer. The Impact, contrary to the presumed result of increased customer satisfaction, is that you may end up with decreased customer satisfaction. Mitigation involves understanding where time is being spent on various issues and then optimizing processes without obsessing over an explicit AHT. The Cost is that some customers will have to open multiple cases for the same issue, and other customers will silently leave for a competitive product. If we assume that CSAT leads to retention, then too strong a focus on AHT can risk missing the NRR outcome. With the dice analogy, why are you hoping for a 1-6 combination when there are multiple ways to achieve the desired outcome of 7?

The big picture

Let me clarify that the goals I discussed above are still good and valid goals. My point is that they cannot be looked at in isolation. You can’t just say, “Bring me three rocks,” without explaining what kind of rocks you want and what you need them for. Let’s drill down more on Average Handle Time (AHT).

As pointed out, a mandate to reduce AHT might improve CSAT, or it might not. And CSAT might correlate to NRR, or it might not. At its most basic level, reducing AHT implies that staff can therefore handle more issues, leading to increased throughput without raising costs. OK, that goes straight to the company’s bottom line. But that’s “rolling a 1-6”; are there other ways to get to 7?

What if, instead of reducing the time spent on each case, we found a way to reduce case volume? Having a customer self-service portal can deflect cases. Using an AI assistant or a good knowledge base makes that self-service portal more effective. The interesting side effect of this is that the cases which still go to Support tend to be much more complex and take longer to resolve. Your AHT goes up, but your work volume does not. Your customers are getting quicker resolutions on the easy issues and your Support team can focus on the harder issues where they add more value, which can improve CSAT and keep a lid on expenses. Now we’re “rolling a 2-5”, which still gets us to 7 and increases our likelihood of success.

But wait, there’s more. What if, instead of deflecting cases, we could eliminate some? Most Support cases fall into one of three categories: (1) General Information, like information about account status or general company information; (2) How-To help with using the product; and (3) Break-Fix for unexpected product behavior. General Information inquiries should be addressed through a customer portal for account information and through a good company website for the rest. How-To questions can arguably be handled with better documentation, context-sensitive in-app help, online help videos, onboarding and training classes, and more intuitive user interfaces. Break-Fix problems should be assessed with an “escape analysis” to see how and why a user encountered the issue, which is the most expensive way to find it. How are your design review, code review, unit testing, integration testing, and prerelease load testing processes? If you can prevent an issue from getting out into the wild, you have saved a serious amount of post-release effort, improved CSAT, and controlled expenses. That’s at least a “3-4” to get us to that target of 7.

You are what you measure

“You are what you measure.” Or “You can’t improve what you don’t measure” (Peter Drucker, Ref 2). It is true that a good dashboard will provide useful information but be careful that what you measure gets you where you want to be. It is also true that, "If you don't know where you're going, any road will take you there" (Lewis Carroll, Ref 3). It is important to understand what your desired outcome is. The various goals, metrics, and objectives are means to get you there. They must be clear in their intent, consistent with each other and across teams, and monitored for unintended consequences. When somebody says, “I just read an article that says our XYZ ratio should be 45%,” challenge them. What does that ratio mean? What outcome is affected by it? Assess it using RIMC analysis. “We need to roll a 1-6!” Do you explicitly need 1-6? Or will any 7 achieve it?

This whole line of thought goes beyond Software Support. Say you are hosting a fund-raising event, whether it is a show, presentation, gala, or dinner. You have various sources of funds: ticket sales, a raffle, an auction, concessions and merchandise, and non-attending donors. If you assigned each group a quota (“to incentivize them”) it is possible that they will stop pushing once their quota is met, or siphon potential donors from the other areas to meet their quota. Instead, having an overall organizational goal and having all teams pulling for it is a better way to meet your desired outcome.

While writing this article I heard about TSIA’s “KORE Score” (Ref 4). KORE stands for “Keeping Organizational Outcomes, Retention, Renewal, and References Excellent”, pointing out how it differs specifically from NPS. While it is still survey-based, like NPS and CSAT, they emphasize a different set of questions to measure the outcomes as I have discussed in this article. Is the customer achieving the goals they expected with/from your product? Will the customer be a reference for your product? Is the customer likely to renew your product? How “loyal” and satisfied is the customer (imho, this is like the renewal question)? These are intended to provide guidance about customer retention or churn, which go straight to my primary objective of increasing NRR. While I am not advocating for or against this method or the questions used, the point of trying to measure whether your company will achieve their desired outcome is what drew my attention.

The TLDR

A company must identify its key objective outcome(s). Department goals must then be set to incorporate those outcomes. Importantly, those goals must be clearly communicated and integrate with goals for all other departments. Otherwise, you may end up in a situation like this. Sales and Marketing are goaled on “New Logos”, resulting in “bad fit” accounts, causing excessive customer and staff churn and burning up financial resources trying to retain them. Support is goaled on “Support 101” stats of FRT and AHT, resulting in insufficient attention to the actual customer and hurting the customer experience. Product and Engineering are goaled on new features, distracting them from addressing “technical debt” and bugs, again hurting the customer experience. Finance is goaled on improving the bottom line, leading to insufficient investment in staffing levels to get stuff done. Individually, they all sound like legitimate goals. But when viewed in the context of doing the right thing to increase NRR, they can fall apart due to poor execution. Take a holistic approach and use RIMC to evaluate whether you are too focused on the numbers and not enough on the outcome. Go roll me a seven!

References

1.      https://www.linkedin.com/pulse/measuring-customer-satisfaction-csat-miles-goldstein

2.      https://www.forbes.com/councils/theyec/2018/12/04/you-are-what-you-measure/

3.      https://eric.ed.gov/?id=EJ997652

4.      https://www.tsia.com/blog/nps-vs-kore-score