DSD-INT 2022 The choice - A workshop for modelers
0 likes89 views
The document outlines a workshop focused on modeling decisions, exploring the various roles and experiences of participants in the modeling process. It discusses the impact of different choices on modeling outcomes, emphasizing quality assurance and collaboration among modellers and stakeholders. Key themes include the perceived influence of modellers, the importance of data quality, and the regulatory context of modeling practices.
DSD-INT 2022 The choice - A workshop for modelers
- 1. The Choice: A workshop for modelers (and others) Lieke Melsen, assistant professor, Wageningen University Carine Wesselius, consultant/researcher, Deltares Janneke Remmers, PhD-candidate, Wageningen University
- 2. This Workshop • Who is joining today • “The Choice” • Theory on choices in modeling • Break • Roles and responsibilities in Modeling • Discussion • Closure
- 4. Different people, different backgrounds, different experiences, different choices! But….what is the effect on the end-result?
- 9. Interview study 14 interviews ~ 1400 minutes ~ 350 pages
- 10. My study: example questions How do you generally make this decision? Why option X? When would you choose a different option?
- 11. My study: example questions How do you generally make this decision? Why option X? When would you choose a different option? How much confidence do you have in a model? How do you estimate your influence as a modeller?
- 13. Two modellers: introduction Governmental organisation Senior Rainfall-runoff modelling
- 14. Two modellers: model structure
- 15. Two modellers: model structure “It is of course dependent on the goal.”
- 16. Two modellers: model structure “It is of course dependent on the goal.” “Chosen because municipality already worked with it.”
- 17. Two modellers: model structure “It is of course dependent on the goal.” “Chosen because municipality already worked with it.” “Generally used within the governmental organisations.”
- 18. Two modellers: model structure “It is of course dependent on the goal.” “Chosen because municipality already worked with it.” “Generally used within the governmental organisations.” “We trust the model. Why, I don’t know.”
- 19. Two modellers: sensitivity analysis
- 20. Two modellers: sensitivity analysis We always ask a consultancy company to do this.
- 21. Two modellers: sensitivity analysis “We think it is important.” We always ask a consultancy company to do this.
- 22. Two modellers: sensitivity analysis “We think it is important.” We always ask a consultancy company to do this. “Well... It is maybe not very explicitly done.”
- 23. Two modellers: sensitivity analysis “We think it is important.” We always ask a consultancy company to do this. “Well... It is maybe not very explicitly done.” “I would find it difficult anyway. How do you do such a thing?”
- 25. Two modellers: calibration “Actually, it depends on what the consultancy company suggests.”
- 26. Two modellers: calibration “The quality of the data also plays a role.” “Actually, it depends on what the consultancy company suggests.”
- 27. Two modellers: calibration “The quality of the data also plays a role.” “The data should be available for calibration.” “Actually, it depends on what the consultancy company suggests.”
- 28. Two modellers: calibration “The quality of the data also plays a role.” “The data should be available for calibration.” “Actually, it depends on what the consultancy company suggests.” “You look at what is there, if it is trustworthy and if I can execute my calibration based on this.”
- 29. Perceived influence of modellers
- 30. Perceived influence of modellers “I think you have a lot of influence as a modeller.”
- 31. Perceived influence of modellers “I think you have a lot of influence as a modeller.” “Too much” “Quite some”
- 32. Perceived influence of modellers “I think you have a lot of influence as a modeller.” “Too much” “Quite some” “I don’t think it matters that much.”
- 33. Perceived influence of modellers “I think you have a lot of influence as a modeller.” “Too much” “Quite some” “I don’t think it matters that much.” “Same result” “Different method”
- 34. Investigating the modeller... 14 in-depth interviews 3 different institutes (France, Belgium, Germany) February – June 2020 835 minutes of material (13.9 hours) 110.767 words (about 220 pages) Inductive thematic content analysis 3 Group leaders 4 Senior researchers 4 Post-docs 3 PhD-candidates 107 codes, 87 motivations
- 35. Melsen, 2022. It takes a village to run a model – the social aspects of hydrological modelling, WRR
- 37. Epistemic presumed to promote the truth-like character of science Contextual
- 38. Epistemic presumed to promote the truth-like character of science Contextual
- 39. Epistemic presumed to promote the truth-like character of science Contextual
- 40. This is a self-sustaining system
- 41. • Model etiquette to move decisions from contextual to epistemic Contextual Epistemic • Determine which decisions are most sensitive, and invest in underpinning those decisions Data availability Time constraints projects Computer resources
- 42. BREAK
- 43. Quality assurance in modelling Quality Assurance
- 44. Quality assurance in modelling – a shared responsibility CLIENT CONTRACTOR
- 45. Quality assurance in modelling – a shared responsibility Problem owner CONTRACTOR Contract lead Requires high quality to justify decisions to stakeholders and constituency Actions: Non-biased, open scope. Requires high quality to justify results to stakeholders and problem owner Actions: Provide resources for quality assurance. Explicit requirements in contract on model quality (assurance).
- 46. Quality assurance in modelling – a shared responsibility Problem owner Employer Contract lead Requires high quality to justify decisions to stakeholders and constituency Actions: Open scope. Requires high quality to justify results to stakeholders and problem owner Actions: Provide resources for documentation. Assurance of documentation. Explicit requirements in contract on model quality (assurance). Modelling team Modeller A good market position, legal responsibility? A good market position, internal position Work performance, appreciation Actions: Facilitation of peer-review, internal model guidelines, intervision, professional development employees Actions: Peer-review, development model guidelines, intervision, training of new employees Actions: Make choices with best intentions, provide substantiation, document choices, sensitivity analysis where necessary
- 47. Quality assurance in modelling Quality assurance as part of Model etiquette Social norms and practices that are considered accepted and good behaviour, in this case related to modelling and documentation practices
- 48. Quality assurance in modelling – Regulatory contexts Certification • Requirements and standards are determined per project • No implications if standards are not met Voluntarily Legal/juridical • Can be organized at different levels (national, internally) • Cost of Compliance involved • Easy for client to require certain standards, specified in the certification • Model set-up / choices are prescribed by law • All contractors have to obey the same requirements • Slow uptake of new insights “statutory or legislative tools and frameworks that are used to enforce action toward specific goals”
- 50. Thank you! Lieke Melsen Assistant Professor Wageningen University lieke.melsen@wur.nl Carine Wesselius Consultant/Researcher Deltares carine.wesselius@deltares.nl Janneke Remmers PhD-candidate Wageningen University janneke.remmers@wur.nl