FP&A analytics and Simpson’s Paradox
Download as DOCX, PDF2 likes305 views
1) Simpson's paradox is a statistical phenomenon where a trend appears in different groups of data but disappears or reverses when the groups are combined. 2) KPIs and other analytics used in FP&A can unintentionally induce "false" understandings if they overly aggregate data in a way that causes Simpson's paradox. 3) FP&A should be aware of Simpson's paradox and ensure their metrics and processes do not fall victim to it, such as through proper customization and testing of indicators.
FP&A analytics and Simpson’s Paradox
- 1. FP&A analytics and Simpson’s Paradox Are your KPIs, Scoreboards and other metrics safe from the Simpson's paradox? “Simpson’s paradox or Yule-Simpson effect is a paradox in probability and statistics, in which a trend appears in different groups of data but disappears or reverses when these groups are combined. It is sometimes given the descriptive title reversal paradox or amalgamation paradox”. (Extract from Wikipedia) KPI, dashboard, other types of analytics,… that finance and FP&A use for communication, may include or generate such paradox polluting the decision making process. Typically, those tools aggregate i.e. combine data in the perspective to give simpler and quicker ways to manage and communicate performance and take decisions. General trend in management practices is to highly recommend limitation of indicators’ number, and as such to increase the aggregation level. It more or less focuses the attention of top and middle management depending on company culture and on rewards and recognition systems. It is then highly possible for such aggregation to induce directly or indirectly “false” understanding or certitudes just like Simpson’s paradox would do. It should then be FP&A role to find and to fight such occurrence. Still, we hardly hear or see mention of this concept during professional studies and life. Practically, teaching and promotion of such tools includes few recommendations that would potentially exclude dropping in that trap but without formally researching or verifying whether it exists or not. Example: A sales director is looking at theratio of commercial contact turned into sales in order to follow theperformance of two teams and (among few other things) reward them on this basis. Ratio contact to sales Period 1 Period 2 Period 3 Period 4 Period 5 Period 6 Team 1 60% 10% 64% 47% 43% 20% Team 2 90% 30% 75% 50% 49% 50% Team 2 had better rewards periods after periods. Frustrated Team 1 leader ask their FP&A to analyze in more details and feed-back is “Sorry, Simpson paradox!! Here is a better view of the relative performance! In fact, the two teams had the same performance over last 6 periods“ Number of sales contact transformed in sales Period 1 Period 2 Period 3 Period 4 Period 5 Period 6 Total Nber of visits Team 1 100 10 50 150 70 20 400 Team 2 10 100 60 120 80 30 400 nber visits Turnsales Team 1 60 1 32 71 30 4 198 Team 2 9 30 45 60 39 15 198 Ratio contact to sales Team 1 60% 10% 64% 47% 43% 20% 50% Team 2 90% 30% 75% 50% 49% 50% 50% Of course, here I forced the figures to a ridiculously simplistic example to illustrate my point and make it easy to understand. In real life, it is generally far more complex to identify such issue. My example is a reversal case (or nearly) which is more easy to spot. Still, in many cases it will not be so extreme, it will only hide (make disappear) the real evolution or performance. The paradox appears thanks to few main causes:
- 2. 1. Dependent variables are hidden through the aggregation of data, 2. Difference of distribution in time (the example is it), 3. Difference of size between the data groups entering in the calculation. Similar examples can be found with return on investments, any given typeof expenses (say marketing) versus revenues, product lines revenue distribution or growth, etc.… (In fact, many of thewidely known ratios or indicators may very well includes or generate such effects if not properly customized and tested). This brings few questions: 1. Up to what point thisparadox is known (as such or in another form)? 2. How it is reflected into the definition and implementation of the company tools and processes? 3. How the verification is done? In my example, FP&A should have make sure the KPI was not reduced (aggregated)to such a level and demonstrate that a more comprehensive set of figures needs to be considered,this should have been made during the set-up of the company systems and FP&A should have “tell the story” behind those figures all along the process in such a way that recognition should have been more in line with performance. A period to date indicator (or a weighted indicator) would have been more representative e.g. Period to date 60% 55% 58% 53% 51% 50% 90% 35% 49% 50% 49% 50% In this example, it is highly intuitive, i.e. even without knowing about the paradox, a reasonably able person shall have made it happen. Still, reality can be far more complex and intuitive reasoning might not be sufficient to identify and adopt the right approach in each and every case, in particular, when the context is strongly oriented towards limiting the number of KPI, Indicators,… used by management. Who has not heard a manager saying “This is too much detail/figures, I just need to know (i.e. my compensation is only based on) these x indicators”. Obviously, this brings into picture a different subject i.e. the reward and recognition based on KPI, scoreboard and other metrics. Still, is it really different, if the metrics can reveal to be paradoxical?! Knowing (be conscious of) this paradox and systematically making sure to avoid its trap is then a key role for FP&A (i) when elaborating the systems and process that will be used (ii) when analyzing and commenting the ongoing results. We may try to draw a list of potentialcommon case but it remains a high level approach that cannot exempt from doing effectively the exercise. Volume vs price evolution i.e. where your Revenue Indicators may hide more or less important shift in the price and/or volumes. Similar for external expenditures. Regional differences, RI covering wide geographic reach may hide major evolutions in the respectiveareas, Customer profiles differences, RI covering wide customers profiles may hide customer profile shift, Product or market maturity phase, RI covering wide range may hide product or market maturity shift, Currency effect, in particular when pricing currency is different from the entities currency and from the consolidation currency. Indicators expressed in consolidation currency will aggregate and then hide any currency effects and shift. This also applies to most costs. Risk assessment or indicators, Quality indicators, Productivity indicators, … Thoseapplicable to revenues (but not only) may very well combine altogether. An (hopefully over-simplistic) example would be a global company management concentrating on an overall revenue indicator mixing revenues from different geographic areas with product and markets in different maturity phasewhere customer profiles are evolving rapidly and where business is done in few pricing currencies but over a large number of legal entities (with different currencies then) and consolidating their results in its home currency. Such a global revenue indicator would mix so many effects and potentials evolutions that it may reveal paradoxical. Even more dangerous, as it may not have been a problem for few years i.e. whereas the potentialof an issue was there but the reality of the markets was not challenging (activating) it.
- 3. We could bet in such a case that the information systems will“lose” the pricing currency within theconsolidation process, potentially shortcut theentity currency to consolidation currency effect too; will aggregate the product into a few product lines historically driving the company development, will probably lose track of the customers profiles among few other “simplifications” justified by the necessity to keep things more effective. Furthermore, the FP&A peoplemay be located in a limited number of shared service centers loosening the contact with local reality. Theend result would be that thewhole organization IS and processes, (probably culture) are cut from reality thanks to a basic misconception (lack of understanding/verification) during the process of defining and implementing tools. FP&A peoples do not need to know all the mathematics about this paradox. They still need to understand the concept, recognize that it may “slip in” when defining their tools and processes, KPI, Scoreboard and other metrics and then make sure it is not the case.