Does the Dichotomy Between Data-Driven Decisions and Managerial Experience Exist?

In today's business world, we often encounter a recurring dilemma: should strategic decisions be based on data analysis or on the experience and judgment of managers? In recent years I have returned to studying in depth the decision-making processes and I find this debate particularly fascinating because it touches the core of how organizations can maximize efficiency and competitiveness.

Managers with years of experience possess invaluable knowledge. They have witnessed economic cycles, faced crises, and seized opportunities, developing an intuition that allows them to make swift decisions. Throughout my professional life, I have implemented many data science and machine learning models in different types of companies. And most of the time, the feedback from managers was: "yes, the data is fine because I made the same decision", or "why do we need that model if I can make the same decision?"

However, behavioral economics has taught us that even experts are not immune to cognitive biases that can distort their perception and judgment (Kahneman & Tversky, 1979). This is why one of the critical steps for the viability of data science projects is to understand how biases work in companies, not just in data.

For instance, the confirmation bias leads us to favor information that confirms our preexisting beliefs while disregarding data that contradict them (Nickerson, 1998). The availability heuristic may cause us to overestimate the likelihood of events simply because they are easier to recall (Tversky & Kahneman, 1973).

Anchoring bias can cause us to rely too heavily on the first piece of information encountered, influencing subsequent judgments and decisions (Tversky & Kahneman, 1974). Loss aversion suggests that we prefer avoiding losses over acquiring equivalent gains, which can lead to overly cautious strategies (Kahneman & Tversky, 1984). Present bias leads us to prioritize immediate rewards over future benefits, potentially resulting in underinvestment in long-term initiatives (O'Donoghue & Rabin, 1999).

These biases can result in suboptimal decisions that negatively impact a company's profitability and sustainability.

On the other hand, data science (DS) and artificial intelligence (AI) have emerged as powerful tools for processing and analyzing vast amounts of information. These technologies can identify patterns and trends that elude human observation, offering a more objective basis for decision-making (Brynjolfsson & McAfee, 2017). They can also mitigate the influence of cognitive biases by grounding decisions in solid empirical evidence (Silver, 2012).

However, we must not fall into the trap of thinking that technology is a panacea that will completely replace human experience. Implementing advanced data analysis systems entails significant costs and requires organizational adaptation (McAfee & Brynjolfsson, 2012). Moreover, data-driven models are only as good as the quality of the information they receive and may lose flexibility when facing abrupt changes in the environment.

So, how do we reconcile this dichotomy? I believe the answer lies in integration. Augmented intelligence suggests that the best strategy is to combine the analytical capabilities of AI with human experience and creativity (Raisch & Krakowski, 2021). Managers can use insights derived from data to inform their intuition, while their experience can guide the focus and application of these analyses.

From a microeconomic perspective, this synergy allows for a more efficient allocation of resources and better adaptation to changing market conditions (Varian, 2014). By recognizing and mitigating biases, while also leveraging managerial experience, companies can make more informed decisions, reducing risks and capitalizing on opportunities more effectively.

So, it's not about choosing between data or experience but about how to combine them strategically. In a world where information is abundant, but time is limited, the ability to integrate data analysis with expert judgment will be a key differentiator for business success.

In the coming years, I plan to investigate deeper into how AI, ML, and data science models help executives make better decisions by enhancing their experience and knowledge. I look forward to sharing my findings through research and teaching. Stay tuned.

References

• Brynjolfsson, E., & McAfee, A. (2017). Machine, Platform, Crowd: Harnessing Our Digital Future. W. W. Norton & Company.
• Kahneman, D., & Tversky, A. (1979). Prospect Theory: An Analysis of Decision under Risk. Econometrica, 47(2), 263-291.
• Kahneman, D., & Tversky, A. (1984). Choices, Values, and Frames. American Psychologist, 39(4), 341-350.
• McAfee, A., & Brynjolfsson, E. (2012). Big Data: The Management Revolution. Harvard Business Review, 90(10), 60-68.
• Nickerson, R. S. (1998). Confirmation Bias: A Ubiquitous Phenomenon in Many Guises. Review of General Psychology, 2(2), 175-220.
• O'Donoghue, T., & Rabin, M. (1999). Doing It Now or Later. American Economic Review, 89(1), 103-124.
• Raisch, S., & Krakowski, S. (2021). Artificial Intelligence and Management: The Automation–Augmentation Paradox. Academy of Management Review, 46(1), 192-210.
• Silver, N. (2012). The Signal and the Noise: Why So Many Predictions Fail—but Some Don't. Penguin Press.
• Tversky, A., & Kahneman, D. (1973). Availability: A Heuristic for Judging Frequency and Probability. Cognitive Psychology, 5(2), 207-232.
• Tversky, A., & Kahneman, D. (1974). Judgment under Uncertainty: Heuristics and Biases. Science, 185(4157), 1124-1131.
• Varian, H. R. (2014). Intermediate Microeconomics: A Modern Approach. W. W. Norton & Company.