LAK19 - Towards Value-Sensitive Learning Analytics Design
1 like1,005 views
This document discusses the implementation of value-sensitive design (VSD) within learning analytics, emphasizing the importance of ethical considerations and stakeholder values in the design process. It presents two studies: one addressing a social learning analytics tool and another focused on a recommendation system for Wikipedia projects, detailing a five-stage approach for stakeholder engagement and iterative development. The findings highlight the necessity of balancing design trade-offs and value tensions to enhance learning analytics systems.
LAK19 - Towards Value-Sensitive Learning Analytics Design
- 1. Towards Value-Sensitive Learning Analytics Design Bodong Chen, Curriculum & Instruction Haiyi Zhu, Computer Science & Engineering LAK Conference | 2019-03-08 1 / 37
- 2. Ethical considerations in learning analytics (Slade & Prinsloo, 2013; Ferguson et al., 2016) 2 / 37
- 3. Ethical frameworks Codes of ethical practice Ethical consideration checklists Empirical work of stakeholders (Drachsler & Greller, 2016; Ifenthaler & Tracey, 2016; Sclater, 2016; Willis, Slade, & Prinsloo, 2016) 3 / 37
- 4. 2017 - The ACM Statement on Algorithmic Transparency and Accountability 2018ā - The ACM FAT* Conference 2018 - #LAK18 & JLA Special Section on Human-Centered Learning Analytics 2019 - #LAK19 FairLAK workshop; papers on evaluating fairness (Photo Credit) Transparency, Accountability, Fairness 4 / 37
- 5. Learning Analytics System Integrity (Buckingham Shum, 2017, Black box learning analytics? Beyond algorithmic transparency) 5 / 37
- 6. Ethical Design Critique (EDC) (Buckingham Shum, 2018, Ethical Design Critique for Learning Analytics Dashboards) 6 / 37
- 7. Value-Sensitve Design Another toolkit for a very hard problem 7 / 37
- 8. "Something important, worthwhile, or useful;" "one's judgment of what is important in life" (Oxford English Dictionaries, 2018). "What a person or group of people consider important in life" (Borning & Muller 2012). "What is important to people in their lives, with a focus on ethics and morality" (Friedman, Hendry, & Borning, 2017) (Source: "It's about power", ACM Communications) Values (vs. Ethics) 8 / 37
- 9. Value Sensitive Design (VSD) is a methodology and an established set of methods for addressing values a pioneering endeavor to proactively consider human values throughout the process of technology design (Davis & Nathan. 2013) offers a systemic approach with speci c strategies and methods to explicitly incorporate the consideration of human values into design (Friedman et al., 2017) Value-Sensitve Design 9 / 37
- 10. Tripartite Methodology (Boston, 2016; Friedman et al., 2017) 10 / 37
- 11. Direct and Indirect Stakeholder Analysis Value Source Analysis Co-evolution of Technology and Social Structure Value Scenario Value Sketch Value-oriented Semi-structured Interview Value-oriented Coding Manual Value-oriented Mock-up, Prototype or Field Deployment Value "Dams" and "Flows" (Friedman et al., 2017, A Survey of Value Sensitive Design Methods) VSD Methods 11 / 37
- 12. Study 1: A conceptual investigation of a social learning analytics tool Study 2: A holistic application to the design of a recommendation system for WikiProjects 12 / 37
- 13. Social Media for Learning (Greenhow & Lewin, 2016; Haythornthwaite, Priya, et al., 2018) 13 / 37
- 15. (Photo Credit) Provide another way of engaging with the class discussion, Enhance student learning and social interaction, and Assist the instructor to grasp discussion content in order to make informed instructional decisions Yellowdig Visualization Tool 15 / 37
- 16. Tool Features (demo) Multi-mode, interactive Temporal Lexical (instrutor only) 16 / 37
- 17. Methods 1. Stakeholder analysis 2. Value analysis 17 / 37
- 18. Direct Instructors Students Active In-active Indirect Future cohorts of the class Academic advisors 1. The Stakeholders 18 / 37
- 19. 2. Values DesignĀ Choices Instructors ActiveĀ Students InactiveĀ Students WhatĀ toĀ show RicherĀ data Utility:Ā navigating,Ā senseĀmaking Utility:Ā assessing FreedomĀ fromĀ bias Autonomy Privacy Usability:Ā cognitiveĀ load (Bereiter, 2014, Principled Practical Knowledge) 19 / 37
- 20. 2. Values DesignĀ Choices Instructors ActiveĀ Students InactiveĀ Students HowĀ toĀ show NodeĀ highlightingĀ (studentĀ orĀ post) Utility:Ā navigating,Ā senseĀmaking Utility:Ā intervening SocialĀ interaction SelfĀimage SelfĀimage Privacy ForceĀdirectedĀ layout Utility:Ā intervening SenseĀ ofĀ community SelfĀimage FreedomĀ fromĀ bias ContentĀ filterĀ (instructorĀ only) Utility:Ā senseĀmaking Utility:Ā senseĀmaking FairĀ access 20 / 37
- 21. Design trade-offs ValueĀ tensions Analysis Representation Action utilityĀ <ĀĀ>Ā autonomyĶ¾ privacyĶ¾Ā selfĀimageĀ Ā Ā activeĀ <ĀĀ>Ā inactive students makeĀ studentĀ nodesĀ not highlightable provideĀ theĀ optionĀ ofĀ not revealingĀ one'sĀ name freedomĀ fromĀ biasĀ Ā <ĀĀ>Ā selfĀimage considerĀ otherĀ graph layoutĀ algorithms studyĀ student perceptionsĀ ofĀ layouts fairĀ accessĀ Ā <ĀĀ>Ā utility:Ā senseĀ makingĀ Ā <ĀĀ>Ā privacy giveĀ studentĀ accessĀ to textĀ miningĀ features reserveĀ theĀ student filterĀ toĀ theĀ instructor (Siemens, 2013) 21 / 37
- 22. Study 1: A conceptual investigation of a social learning analytics tool Study 2: A holistic application to the design of a recommendation system for WikiProjects 22 / 37
- 23. WikiProjects Self-organized groups of contributors within the English Wikipedia community. 23 / 37
- 24. Problem and Goal ļ±Ā WikiProjectsĀ haveĀ beenĀ declining Only 12% projects have 10+ active members ļ Ā ToĀ buildĀ aĀ recommendationĀ system to help WikiProjects identify and recruit suitable new members 24 / 37
- 25. A Five-Stage Approach (Zhu et al., 2018, CSCW) 25 / 37
- 26. Stage 1. Understand Stakeholders 26 / 37
- 27. Stage 1. Understand Stakeholders A literature review + A survey study --> Stakeholders' values DesignĀ Choices NewĀ project comers CurrentĀ project members Wikipedia community WhoĀ to recruit Experienced editors Collaboration Productivity MemberĀ retention BrandĀnew editors Mentorship Productivity NewĀ member retention HowĀ to defineĀ fit InterestĀbased MutualĀ interest MutualĀ interest RelationĀbased Personal connection Productivity HowĀ to decide Automation Control HumanĀinĀtheĀ loop Control 27 / 37
- 28. Stage 2. Prototyping - "Who to Recruit" DesignĀ Choices NewĀ project comers CurrentĀ project members Wikipedia community WhoĀ to recruit Experienced editors Collaboration Productivity MemberĀ retention BrandĀnew editors Mentorship Productivity NewĀ member retention Value Tension -> Design Choice Evaluateing both brand-new and experienced editors Ranking the two types of editors separately 28 / 37
- 29. Interest-based Rule-based Category-based Relationship-based Bond-based Co-edit-based Stage 2. Prototyping - "How to de ne t" DesignĀ Choices NewĀ project comers CurrentĀ project members Wikipedia community HowĀ toĀ define fit InterestĀ based MutualĀ interest MutualĀ interest RelationĀ based PersonalĀ connection Productivity Four algorithms (see Zhu et al., 2018, CSCW) 29 / 37
- 30. Stage 2. Prototyping - "How to decide" DesignĀ Choices NewĀ project comers CurrentĀ project members Wikipedia community HowĀ to decide Automation Control HumanĀinĀtheĀ loop Control Pitfall -> Design choice Design a dashboard to communicate recommendations to project members 30 / 37
- 31. Stage 2. Prototyping - "How to decide" Analytic results communicated to current WikiProjects members: 31 / 37
- 32. Stage 3-4. Deploy, Iterate and Re ne Over a six-month period: The analytic system analyzed information from 16,000 editors delivered 4 distinct batches of 385 recommendations to 18 Wikiprojects Re ned the system prototype based on stakeholder feedback 32 / 37
- 33. Stage 5: Evaluate Messages from current WikiProjects members "This puts some science behind recommendations, and will be a great supplement to the current processes. " Messages from invited new members "Thank you for reaching out to me and thank you for informing me about the WikiProject Africa talk page... I appreciate it. " (see Zhu et al., 2018, CSCW) 33 / 37
- 34. Stage 5: Evaluate Response from the Wikipedia community: a Signpost (news story) about the design 34 / 37
- 35. Summary Study 1: A conceptual investigation of a social learning analytics tool Identi ed stakeholders and their values Elicited value tensions Suggested design tradeoffs to address value tensions Study 2: A holistic application to the design of a recomendation system Developed a ve-stage approach that integrate multiple methods Designed with end-users Involved the user community in the design process Iterated over an extended period of time 35 / 37
- 36. Implications and Future Directions Value Sensitive Design provides a toolkit to: 1. guide the evaluation and re nement of existing learning analytics applications 2. help holistically addressing values in the design of new learning analytics Currently applying VSD to the design of the Personal Learning Compass tool ... 36 / 37
- 37. Thank You! Full Paper: https://doi.org/10.1145/3303772.3303798 Preprint: https://arxiv.org/abs/1812.08335 ļ chenbd@umn.edu ļ @bod0ng ļ¬ Personal website: http://meefen.github.io/ ļ¬ Research group: https://colig.github.io/ 37 / 37