Modelling Pitfalls - introduction and some cases
Download as PPTX, PDF0 likes102 views
This document discusses several case studies of agent-based models (ABMs) and some of the potential pitfalls of ABMs. It begins by outlining Bruce Edmonds' approach to discussing pitfalls, which is to start with the purpose of the simulation and think about how it could go wrong. Several examples of ABMs are then summarized, including Schelling's model of racial segregation, opinion dynamics models, a water distribution model of Bali, models for predicting US elections, and models of cooperation evolution. The document concludes by providing a summary of different ABM purposes and some of the particular risks associated with each purpose.
Modelling Pitfalls - introduction and some cases
- 1. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 1 the Pitfalls of ABM – intro and some cases Bruce Edmonds Centre for Policy Modelling Manchester Metropolitan University
- 2. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 2 The Approach • Start with the purpose of the simulation – which derives from how the simulation can be justified • E.g. Predict something, support an explanation, describe, explore theoretical consequences of some assumptions, as an illustration of process, a way of thinking about some phenomena or a way of mediating between people • Then think of the ways in which using an ABM for this purpose might go wrong – e.g. how we might fool ourselves trying to do this • Then do activities that mitigate against these risks
- 3. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 3 The Schelling 2D Model of Racial Segregation • Sakoda/Schelling’s 2D Model of segregation based on a checkerboard space, with agents moving from space to space. • This showed that a relatively low level of racial intolerance could result in spatial segregation • It was a counter-example to the natural assumption that the observed segregation was due to high racial intollerance • It has been extended and interpreted in many different ways since
- 4. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 4 An Opinion Dynamics Model • Previous models showed how opinions converge but not how they may polarise • No evidence or data is applied, this is just an exploration of the general outcomes of some abstract mechanisms • So the space of behaviours is thoroughly explored with many thousands of runs • The results are then characterised in general terms • Deffuant, G., et al. (2002) How can extremism prevail? jasss.soc.surrey.ac.uk/5/4/1.html
- 5. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 5 Water Distribution in Bali • Lansing & Kramer’s (1993) model of water distribution in Bali, explained how the system of water temples act to help enforce social norms and facilitate a complicated series of negotiations • The implemented the system of rivers and how norms about negotiation about water allocation are medicated by a system of water temples • It explained how this social system was effective in facilitating an equitable and sufficient allocation of water
- 6. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 6 Predicting US Presidential Elections • Nate Silver’s team tries to predict the outcome of elections using computational models. • The structure of the electoral college is built in • Opinion polls from each area are fed into the simulation plus some noise • The simulation is then run lots of times to give a distribution of outcomes • From this distribution one can extract probabilities of the different outcomes • The predicted a 1/3 chance of Trump being elected and got ALL of the electoral colleges right in Obama’s election • (http://fivethirtyeight.com and Silver 2013)
- 7. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 7 Evolution of Cooperation • Axelrod’s (1984 etc.) facilitated simulations and competitions where different interaction strategies competed against each other in tournaments • This was also done within an evolutionary framework where winners tend to be reproduced into the next generation, so more successful strategies proliferate • It was found that “tit for tat” did better than other strategies in both cases • This was interpreted as showing how cooperation could evolve even when individuals behave selfishly
- 8. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 8 A Socio-Ecological Test Bed • In this a ecology of ~100 interacting species is evolved ‘from scratch’ in a complex food web within a 2D grid of locations • This starts from plants, then herbivores appear, then predators etc. Then the state saved as a consistent starting point for experiments • The balance between species is constantly changing as new species emerge and interact • This is used to do an uncertainty/risk analysis – what might happen if humans do X • When something unexpected happens, one can ‘drill down’ into the details to understand how it occurred • This does not say what will happen, but shows a range of possibilities
- 9. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 9 A Model of Domestic Water Demand • This combined – input of weather data (rainfall, sun, wind etc.) – a social network of households – each household has a different statistic of using different appliances (frequency, amount of water etc.) – which were differently selfish, social influenced or influenced by official advice – new water-spending/saving devices are introduced – during droughts the authority advises reduce water use • The idea was to help inform policy for water companies and government as to future water supply and reaction to droughts
- 10. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 10 Summary of Purposes, features and risks Modelling Purpose Essential features Particular risks (apart from that of lacking the essential features) Prediction Anticipates unknown data Conditions of application unclear Explanation Uses plausible mechanisms to match outcome data in a well-defined manner Model is brittle, so minor changes in the set-up result in bad fit to explained data; bugs in the code Description Relates directly to evidence for a set of cases Unclear provenance; over generalisation from cases described Theoretical exposition Systematically maps out or establishes the consequences of some mechanisms Bugs in the code; inadequate coverage of possibilities Illustration Shows an idea clearly as a particular example Over interpretation to make theoretical or empirical claims; vagueness Analogy Provides a wayof thinking about something; gives insights Taking it seriouslyfor anyother purpose Social learning Facilitates communication or agreement Lack of engagement; confusion with objective modelling
- 11. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 11 There are a whole lot of other things that can go wrong when dealing with the policy world! For example: • Promising what one can not deliver (e.g. prediction) • Model spread – when models are used outside the context they were originally designed for • People believe the model too much based on surface plausibility and animations • Narrowing the evidence used, by focusing attention on what can be modelled • Takes focus away from a discussion of values (see extra slides in “Pitfalls – resources.pptx” in folder)
- 12. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 12 The End Bruce Edmonds: http://bruce.edmonds.name Centre for Policy Modelling: http://cfpm.org A version of these slides will be in the shared dropbox folder and at: http://slideshare.com/BruceEdmonds