Risk-aware policy evaluation using agent-based simulation
1 like238 views
The document discusses risk-aware policy evaluation through agent-based simulation, emphasizing the complexity of systems and necessary changes in modeling approaches. It critiques traditional forecasting and quantification methods that do not adequately capture the dynamics of complex systems, urging a shift in modeling techniques and iterative processes. The presentation includes examples of models concerning domestic water demand and housing rental markets, highlighting the need for more nuanced understanding in policy development.
Risk-aware policy evaluation using agent-based simulation
- 1. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 1 Risk-aware policy evaluation using agent-based simulation Bruce Edmonds Centre for Policy Modelling Manchester Metropolitan University
- 2. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 2 Simple systems… … may be complicated but behave in predictable ways, allowing them to be represented by models... • where one can use them to numerically forecast • where uncertainty can be analytically estimated • where one can get rough estimates cheaply, and better estimates with increasing investment • which one can sensibly plan and execute systematically • where there is a basically one right way of doing it • so that one can fully understand the model
- 3. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 3 However… Even with only two bits of wood the result can be complex See video at: http://www.youtube.com/watch?v=czLIj-4suOk
- 4. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 4 The Main Point of the Talk… …is that complex systems need to be dealt with in a different way to that of simple systems... ...not only using different techniques but also how models about complex systems are used in policy development process needs to change including moving away from prediction. • Simulation modelling will be increasingly important as we try to develop better policies and deal with complex and fast moving situations • But it can not be ‘business as usual’ – just doing better modelling with the same modeller–policy actor relationship will not work well
- 5. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 5 Structure of the (rest of the) Talk 1. A bit about modelling context, purposes and tensions 2. Some of the underlying assumptions and habits that need to change 3. An eample model – A model of Domestic Water Demand 4. An example model – Stefano Picascia’s Modelling of the Housing Rental Market 5. Some suggestions as to ways forward
- 6. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 6 Tensions and difficulties for the modeller Part 1
- 7. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 7 The Complexity facing Modellers • Many of the situations or issues we need to understand are mixtures of: technical, social, behavioural and ecological factors • They are not only complicated, but also unexpected outcomes can ‘emerge’ from the interaction of the actors and internal processes • We do not have good general models for how people behave (regardless of what economists claim) • How to approach using models to understand complex phenomena is not fully developed
- 8. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 8 Different modelling purposes Models can be used for a wide variety of different purposes, and these impact upon the kind of techniques needed and its difficulties, e.g. • Forecasting – predicting unknown (e.g. future) situations and outcomes • Explanation – understanding how known outcomes might have come about • Theoretical Exploration – understanding a complex model by exploring some of its properties and behaviours • Analogy – using a model as a way of thinking about something else
- 9. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 9 Model Scope • The scope of a model is the conditions under which it is useful for its planned purpose • Whilst this is implicit and stable for many simple systems, this is not the case for many complex ones • Thus trying to make scope explicit is important, and these relate to model assumptions • A process not included in the model (and hence outside its scope) can overwhelm the results… • ..but in complex systems internal processes of change can also emerge, and some of these can be usefully modelled (but only in more complex ways)
- 10. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 10 Possible modelling trade-offs • Some desiderata for models: validity, formality, simplicity and generality • these are difficult to obtain simultaneously (for complex systems) • there is some sort of complicated trade-off between them (for each modelling exercise) simplicity generality validity formality Analogy Solvable Mathematical Model Data What Policy Actors Want
- 11. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 11 A picture of modelling whatisobservedor measured themodel themodellers themodelusers
- 12. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 12 Assumptions and expectations from Policy Actors Part 2
- 13. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 13 Expectations of Scientists • What works well with simple systems does not necessarily work well with complex ones • Many of the expectations of complexity scientists by policy makers and the public come from: – What economists have claimed to be able to do – Or how physical scientists have been able to do • As I hope will be clear, complex simulation modelling can usefully inform policy making • But these expectations can get in the way • So we will look next at some of these expectations
- 14. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 14 The Cost-Benefit Approach • Basically weighing the benefits – the costs • As if an economist had written a manual for policy actors in how to think (i.e. as their theory states) This assumes that one can: 1. list the main alternative options 2. forecast the results of these 3. put meaningful numerical values on these 4. decide on the best one, adopt that option • Allows policy optimisation… • ...if it were possible
- 15. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 15 Quantification • Makes life much easier for policy actors – choose the one with the biggest (or smallest) number! • Especially when asked to justify an approach • But can be more misleading than helpful because it gives a false impression of accuracy • And implicitly leads to a focus on the measurable and that things will ‘average out’ etc. • Was a limitation of purely mathematical approaches, but computer simulation does not have to be focused on these aspects • 1D quantification is often an inadequate representation of what we need to understand
- 16. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 16 Planning and Managing Modelling • In a simple case one can apply an approach where one carefully plans, manages and evaluates models • As if this was like building a bridge! • But in complex cases complications about what needs to be included or not requires a more iterative approach… • ...where models are repeatedly built for a purpose and the lessons learnt as you go along... • Becuase the difficulties can not be predicted in complex cases!
- 17. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 17 No gradual approximation, but scope-limited usefulness It is often assumed that as time and effort increase the accuracy of the results improve, but this is not the case with complex systems and models Rather in order for the outcomes to be within scope enough iterative development has to occur Before this the results are worse than nothing Time and cost Error
- 18. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 18 Compartmentalism • That some problems can be separated into smaller sub-problems which can be modelled more simply • Not true in many complex cases, where the scope of modelling is dependent on having enough of the key processes represented • Sometimes several different modelling approaches with different (but overlapping) assumptions can be more helpful • Just fiddling, incrementally expanding an existing (and failing) model will probably not help here
- 19. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 19 An Example: A model of Domestic Water Demand Part 3
- 20. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 20 Context of model • As part of a broader model which sought to understand the impact of climate change on the domestic demand for water in the UK • For the UK government and water companies • Looked at the impact of some present and extrapolated weather patterns under four different future economic/cultural scenarios • Included sophisticated statistical models for prediction of demand • Plus our agent-based model as a contrasting approach
- 21. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 21 Monthly Water Consumption REL_CHNG .88 .75 .63 .50 .38 .25 .13 0.00 -.13 -.25 -.38 -.50 20 10 0 Std. Dev = .17 Mean = .01 N = 81.00
- 22. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 22 Relative Change in Monthly Consumption in a small village Date FEB 2001 SEP 2000 APR 2000 N O V 1999 JU N 1999 JAN 1999 AU G 1998 M AR 1998 O C T 1997 M AY 1997 D EC 1996 JU L 1996 FEB 1996 SEP 1995 APR 1995 N O V 1994 JU N 1994 REL_CHNG 1.0 .8 .6 .4 .2 -.0 -.2 -.4 -.6
- 23. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 23 Purpose of the Model • Not long-term prediction • But to begin to understand the relationship of socially-influenced consumer behaviour to patterns of water demand • By producing a representational agent model amenable to fine-grained criticism • And hence to suggest possible interactions and outcomes
- 24. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 24 Model Structure - Overall Structure • Activity • Frequency • Volume Households Policy Agent • Temperature • Rainfall • Sunshine Ground Aggregate Demand
- 25. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 25 Model Structure - Microcomponents • Each household has a variable number of micro- components (power showers etc.): bath other_garden_watering shower hand_dishwashing washing_machine sprinkler clothes_hand_washing hand_dishwashing toilets sprinkler power_shower • Actions are expressed by the frequency and volume of use of each microcomponent • Actions-Volume-Frequency distribution in model calibrated by data from the Three Valleys
- 26. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 26 Model Structure - Household Distribution • Households distributed randomly on a grid • Each household can copy from a set of neighbours (those within a certain distance ) • Households have different mixtures of motivations: self, social, global • They decide which is the neighbour most similar to themselves – this is the one they are most likely to copy – but all neighbours have some influence • Depending on their evaluation of actions they might adopt that neighbour’s actions • Or do the action they are used to (habit) • Or that suggested by the policy agent
- 27. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 27 An Example Social Structure (main influence only) - Global Biased - Locally Biased - Self Biased
- 28. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 28 Household Behaviour - Endorsements • Action Endorsements: recentAction neighbourhoodSourced selfSourced globallySourced newAppliance bestEndorsedNeighbourSourced • 3 Weights moderate effective strengths of neighbourhoodSourced selfSourced globallySourced endorsements and hence the bias of households • Can be summarised as 3 types of households influenced in different ways: global-; neighbourhood-; and self-sourced depending on the dominant weight (though this is a simplification, all three weights and factors can play a part)
- 29. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 29 History of a particular action from one agent’s point of view Month 1: action 1330 used, endorsed as self sourced Month 2: action 1330 endorsed as recent (from personal use) and neighbour sourced (used by agent 27) and self sourced (remembered) Month 3: action 1330 endorsed as recent (from personal use) and neighbour sourced (agent 27 in month 2). Month 4: action 1330 endorsed as neighbour sourced twice, used by agents 26 and 27 in month 3, also recent Month 5: action 1330 endorsed as neighbour sourced (agent 26 in month 4), also recent Month 6: action 1330 endorsed as neighbour sourced (agent 26 in month 5) Month 7: replaced by action 8472 (appeared in month 5 as neighbour sourced, now endorsed 4 times, including by the most alike neighbour – agent 50)
- 30. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 30 Policy Agent - Behaviour • After the first month of dry conditions, suggests AFV actions to all households (reducing water usage) • These actions are then included in the list of those considered by the households • If the household’s weights predispose it, it may decide to adopt these actions • Some other neighbours might imitate these actions etc. • Others, more self-sourced may not be influenced
- 31. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 31 Number of consecutive dry months in historical scenario 0 1 2 3 4 5 6 7 8 9 J-73 J-74 J-75 J-76 J-77 J-78 J-79 J-80 J-81 J-82 J-83 J-84 J-85 J-86 J-87 J-88 J-89 J-90 J-91 J-92 J-93 J-94 J-95 J-96 J-97 Simulation Date Numberofconsequativedrymonths
- 32. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 32 Simulated Monthly Water Consumption REL_CHNG .075 .063 .050 .037 .025 .012 -.000 -.013 -.025 -.038 -.050 120 100 80 60 40 20 0 Std. Dev = .01 Mean = -.000 N = 325.00
- 33. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 33 Monthly Water Consumption (again) REL_CHNG .88 .75 .63 .50 .38 .25 .13 0.00 -.13 -.25 -.38 -.50 20 10 0 Std. Dev = .17 Mean = .01 N = 81.00
- 34. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 34 Simulated Change in Monthly Consumption Date SEP 1997 APR 1996 N O V 1994 JU N 1993 JAN 1992 AU G 1990 M AR 1989 O C T 1987 M AY 1986 D EC 1984 JU L 1983 FEB 1982 SEP 1980 APR 1979 N O V 1977 JU N 1976 JAN 1975 AU G 1973 M AR 1972 O C T 1970 REL_CHNG .10 .08 .06 .04 .02 0.00 -.02 -.04 -.06
- 35. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 35 Relative Change in Monthly Consumption (again) Date FEB 2001 SEP 2000 APR 2000 N O V 1999 JU N 1999 JAN 1999 AU G 1998 M AR 1998 O C T 1997 M AY 1997 D EC 1996 JU L 1996 FEB 1996 SEP 1995 APR 1995 N O V 1994 JU N 1994 REL_CHNG 1.0 .8 .6 .4 .2 -.0 -.2 -.4 -.6
- 36. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 36 30% Neigh. biased, historical scenario, historical innov. datesAggregate demand series scaled so 1973=100 0 20 40 60 80 100 120 140 160 180 200 J- 73 J- 74 J- 75 J- 76 J- 77 J- 78 J- 79 J- 80 J- 81 J- 82 J- 83 J- 84 J- 85 J- 86 J- 87 J- 88 J- 89 J- 90 J- 91 J- 92 J- 93 J- 94 J- 95 J- 96 J- 97 Simulation Date RelativeDemand
- 37. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 37 80% Neigh. biased, historical scenario, historical innov. datesAggregate demand series scaled so 1973=100 0 20 40 60 80 100 120 140 160 180 200 J- 73 J- 74 J- 75 J- 76 J- 77 J- 78 J- 79 J- 80 J- 81 J- 82 J- 83 J- 84 J- 85 J- 86 J- 87 J- 88 J- 89 J- 90 J- 91 J- 92 J- 93 J- 94 J- 95 J- 96 J- 97 Simulation Date RelativeDemand
- 38. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 38 80% Neigh. biased, medium-high scenario, historical innov. datesAggregate demand series scaled so 1973=100 0 20 40 60 80 100 120 140 160 180 200 Jan- 73 Jan- 74 Jan- 75 Jan- 76 Jan- 77 Jan- 78 Jan- 79 Jan- 80 Jan- 81 Jan- 82 Jan- 83 Jan- 84 Jan- 85 Jan- 86 Jan- 87 Jan- 88 Jan- 89 Jan- 90 Jan- 91 Jan- 92 Jan- 93 Jan- 94 Jan- 95 Jan- 96 Jan- 97 Simulation Date RelativeDemand
- 39. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 39 What did the model tell us? • That it is possible that social processes within communities: – can cause a high and unpredictable variety in patterns of demand – can ‘lock-in’ behavioural patterns and partially ‘insulate’ them from outside influence (droughts only occasionally had a permanent affect on patterns of consumption) • Thus identifying and taking measures at high- usage areas at an early stage might be sensible • Also that the availability of new products could dominate effects from changing consumptions habits
- 40. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 40 An Example: A Model of the Rental Housing Market Part 4
- 41. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 41 The model • By Stefano Picascia, an PhD student of mine, now at Sienna University, Italy • Is an agent-based simulation that represents both tenants and developers co-adapting • Is geographically based with tenants making decisions as where to move to based on location as well as quality of housing and price • Developers put in captial to build/rennovate housing for tenants • Rents are determined by the quality and prices of surrounding housing
- 42. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 42 The Manchester Case Waves of price changes can spread Can have different outcomes each time it is run Has also been applied to London and Beirut
- 43. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 43 Average prices in a run
- 44. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 44 Different Sectors of the City in a run
- 45. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 45 What it does and does not tell us In the model (which is the private rental sector only): • That change is fundamentally internally driven as well as due to outside events • Price oscillations are endemic to the system • That some regions of cities will be stuck as low quality housing for long periods of time depending on the state of neighbouring areas • The very high price regions stay that way • That under certain conditions sudden ‘gentrification’ may occur to some degree raising standards but maybe also displacing existing functional communities • For poorer districts decline is gradual and continual between any such periods
- 46. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 46 Concluding discussion and some ways forward Part 5
- 47. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 47 From Probabilistic to Possibilistic • When outcomes can not be sensibly forecast… • And especially numerically forecast… • …where even probability zones or 90% bounds are misleading • Then moving to an approach that models and understand (more of) underlying processes... • ...in terms of the different kinds of outcome might be much more informative • Each outcome tagged with its own assumptions and scopes (if they differ)
- 48. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 48 From Forecasting to Risk Analysis • However much one might like forecasting, often it is simply not possible… • ...let alone in a way such that the outcomes from different options can be compared! • Predicting outcomes can be more misleading than helpful • Rather it may be more approapriate to use models for risk analysis – finding all the ways a policy might go wrong (or right!) • Techniques are available to help discover and understand how endogenous processes might result in different future possibilities • Which can then inform the design of ‘early warning’ monitors giving the most immediate feedback to policy makers
- 49. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 49 Informing the adaptive ‘driving’ of policy • Complex models are no good for policy makers! • Because they have to make decisions on grounds they understand and know the reliability of • They can not (and should not) delegate this to ‘experts’ and their inscrutable models • Rather modellers should use their modelling to understand the key emergent kinds of outcome • To inform: – the consideration of these kinds of outcome – the design of appropriate data visalisations – the design of ‘earl warning indicators • …So that policy can adapt to changing trends and events as quickly and fluidly as possible
- 50. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 50 Conclusions • Modelling of complex phenomena is not cheap or quick and requires iterative development • It will not forecast the impact of potential policies or events, but can anticipate possible future outcomes in a way intuition can not • There will always be a ‘scope’ – a set of conditions/assumptions a model depends upon • But a good model can repay its investment in terms of cost and improving people’s lives many, many times over
- 51. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 51 Summary It is no good wishing that the world or modelling is simple and trying to ‘force’ it to be so, one has to adapt to suit reality… …this includes how models and modelling are used by the policy process
- 52. Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 52 The End The Centre for Policy Modelling: http://cfpm.org These slides will be available at: http://slideshare.net/BruceEdmonds Stefano’s model of housing was developed under this project, funded by the EPSRC, grant number EP/H02171X Social Science Aspects of Fisheries for the 21st Century – with two Icelandic partners: MATIS and the University of Iceland