Reimagining Theories of Change – A shift from Deterministic to Probabilistic Models

In the development sector we know that some of the interventions work some of the times, in some of the contexts, with some of the people. But it’s rare that we know in advance, with certainty, which one will create what impact. That is why we develop Theories of Change – our best guess about what a project can achieve. These deterministic theories of change are not only inadequate, they are also invariably wrong. And here is why.

 Let us assume you are running a health or education program that has 5 interventions being applied across 5 geographic/cultural contexts. Can you try to calculate the possible outcomes of these interventions? Here is the answer:

If each intervention has two possible outcomes (success or failure) for each context, the total number of combinations is (25)5 or a staggering 33,554,432 outcomes. This complexity further escalates and the number of possibilities start to look insane when considering multiple outcomes for each intervention and their interdependencies across different contexts. Open Syllabus has mapped over 20 million learning outcomes of educational programs and these can be used to create a range of permutations, mapping interventions to outcomes.

 Due to limitation of data and compute, the landscape of development planning and intervention has long been navigated through the lens of traditional deterministic theories of change. These models, grounded in a linear perspective, assume a direct cause-effect relationship between interventions and outcomes. We make a leap of faith that input A will lead to Output B and that would in turn lead to outcome C. These models were borrowed from pure sciences where experiments conducted in controlled environments, following the laws of physics, would lead to 'theoretically determinable' results.

 Unfortunately, the real world is far more complicated where a myriad of confounding factors (human attitudes and behaviour, politics, religious beliefs, financial priorities, speed of implementation, quality of implementation, access to media, mobiles and internet, social and gender norms, regional variation etc) make prediction of impact far more complex. So we make a few assumptions and several leaps of faith to develop a ToC that we ‘hope’ will work. We lean on Randomized Control Trials (again borrowed from the world of pure sciences) to validate our hypothesis. RCTs by definition are lab environments where we try to eliminate as many variables as possible.

 So the question I have been pondering about is: Can we shift from deterministic models of change to an AI-based probabilistic models of change where millions of possible outcomes can be predicted? This can help us get a broader, holistic view of both positive and negative outcomes of our interventions, the relative impact of different interventions and the dependency and dose effect of different interventions.

 Probabilistic theories of change, powered by AI, embrace the complexity and uncertainty of development interventions. AI, with its advanced data processing and machine learning capabilities, can analyze vast datasets, uncover patterns, and predict a range of potential outcomes with associated probabilities. This approach acknowledges that interventions in complex social systems can lead to multiple possible outcomes, influenced by a confluence of factors.

 For example, in a literacy project, a probabilistic model would consider variables such as socio-economic backgrounds of students, local cultural attitudes towards education, teacher quality, and even political stability. AI algorithms can assess how these factors might interact in complex ways, offering a spectrum of potential outcomes rather than a single predicted result. This provides planners with a more nuanced understanding of what might happen, allowing for more flexible and adaptive planning.

 The integration of AI-based predictive analytics in development planning offers a transformative approach. By harnessing probabilistic models like Bayesian Networks, AI is equipped to predict a spectrum of possible outcomes, each assigned a probability, while navigating through layers of uncertainty. For example, In the context of urban development, these models can assess the probable impact of new infrastructure projects, taking into account variables such as urban density, traffic flow, environmental impact, and local socio-economic factors. Surgo Health conducted a study called, "Discovery of interconnected causal drivers of COVID-19 vaccination intentions in the US using a causal Bayesian network," to map out the complex, intertwined drivers of COVID-19 vaccine hesitancy across the United States.  Customizing AI to analyse regional data ensures that its predictions and recommendations are acutely relevant to specific contexts. The deployment of natural language processing to analyze regional news reports and social media can further refine these strategies, ensuring they resonate with local needs and aspirations.  According to Vincent Huang, Director of AI at Surgo Health, “AI-based mapping provides a system-level understanding of barriers and behavioral drivers to help practitioners, leaders, and governments design more effective, holistic intervention strategies.”

 Even in cases where detailed data for each variable is not available, techniques like Monte Carlo simulations allow for the exploration of multiple scenarios, providing a sandbox for development planners to understand the impact of various variables under different conditions. Such simulations are used by analysists like ARK Invest for predicting long term value of specific stocks, taking into account a range of variables – industry trends, company’s execution, product adoption rates, health of economy etc. – and deliver bull, base and bear case scenarios. In the field of environmental conservation, such simulations can project the outcomes of conservation strategies, factoring in variables like species migration patterns, climate change effects, and habitat alteration, to present scenarios that guide conservation efforts. A great use case for scenario planning in areas like disaster risk reduction and pandemic management, where understanding the range of potential impacts of natural disasters on communities, health systems and economy is crucial for developing resilient response strategies.

 The value of probabilistic theories of change, while quite obvious, is often met with scepticism. When discussing these ideas, I am bombarded with questions: How useful would it be to have thousands of predicted outcomes. How will that help in decision making? Will it not increase complexity and paralyze decision making if nothing is certain?

The answer is that Nothing is certain in the real world. Probabilistic models help us understand likelihood of impact far better than deterministic models. Clustering of probable results can help us understand the ratio of positive and negative outcomes, a quantified assessment of the probability of expected results and thereby help us make more informed choices about the relative focus and interplay of different interventions.

 Integrating AI into development planning may seem daunting and could disrupt traditional models. However, the need to move from deterministic to probabilistic models of change is imperative.

 Up Next: How Gaming can help in transitioning from deterministic to probabilistic model of behaviour change Interventions