Prioritizing Ethical Use of Learning Data.pdf
0 likes68 views
The document discusses the ethical use of learning data in adult workforce development, emphasizing the importance of empathy and ethical practices when using analytics like xAPI. It highlights various types of analytics—descriptive, diagnostic, predictive, and prescriptive—and their implications for learner success, bias, and decision-making. Continuous inquiry and reflection are encouraged to improve ethical practices within analytics and use data responsibly to benefit real people.
Prioritizing Ethical Use of Learning Data.pdf
- 1. PRIORITIZING ETHICAL USE OF LEARNING DATA (REGARDING ADULTS IN WORKFORCE DEVELOPMENT CONTEXTS) xAPI Cohort - The Learning Guild 20 October 2022 Aaron E. Silvers, Elsevier - a.silvers@elsevier.com
- 2. MACHINE LEARNING ANALYTICS METRICS CONSIDER HOW XAPI AMPLIFIES YOUR IMPACT IN THE LONG TERM 2 Descriptive What happened? Diagnostic Why did it happen? Predictive What will happen? HINDSIGHT INSIGHT FORESIGHT Di ffi culty Value Prescriptive What makes it happen? “I DID THIS” + CONTEXT, RESULTS, ATTACHMENTS, MORE… A TINY BAD DECISION CAN HAVE EXPONENTIALLY DEADLY RESULTS GARTNER ANALYTIC ASCENDANCY MODEL BOYER AND BONNIE, 2016
- 3. WHAT’S COVERED IN THIS PRESENTATION 3 EYEBROW / CAPLINE HEADER Given the context of adult learners in workforce development, this presentation will… ▪︎ Model a lifecycle of an xAPI statement to understand the potential long-term impacts of this data on real people. ▪︎ Curate relevant ethical challenges related to using learning data from a corpus of research literature speci fi cally about the ethics of learning analytics. ▪︎ Impart the need to continuously train the capacity for empathy to strengthen your inquiry skills and prioritize ethical use.
- 4. PRIORITIZING ETHICAL USE OF LEARNING DATA STARTS WITH YOU. 4 PART I A professional working with xAPI MUST be able to describe and model what happens with the data.
- 5. MACHINE LEARNING ANALYTICS METRICS GARTNER ANALYTIC ASCENDANCY MODEL 5 Descriptive What happened? Diagnostic Why did it happen? Predictive What will happen? HINDSIGHT INSIGHT FORESIGHT Di ffi culty Value Prescriptive What makes it happen? BOYER AND BONNIE, 2016
- 6. SCENARIO: US DEPARTMENT OF DEFENSE’S ENTERPRISE DIGITAL LEARNING MODERNIZATION STRATEGY 6 ADL, 2022 https://adlnet.gov/projects/edlm/
- 7. “ 7 WILLIAMS, 2022 The only 'good news' I have here is for fans of mythology: You get to Cassandra *and* Sisyphus, at the same time. A tiny bad decision can have exponentially deadly results, but a tiny kind decision can have exponentially bene fi cial results.” - Dr. Damien P Williams, Department of Science, Technology and Society, Virginia Tech (2022). Dr. Damien P. Williams, https://twitter.com/Wolven
- 8. THE CORPUS 8 PART II A professional working with xAPI MUST be able to demonstrate evidence for design and architecture choices, including referencing peer-reviewed research that supports and strengthens the effort.
- 9. DESCRIPTIVE ANALYTICS 9 ▪︎ Analytics focused on description, detection and reporting. ▪︎ Mechanisms to query from multiple sources, fi lter and combine data. ▪︎ Outputs include visualizations; pie charts, tables, bar charts or line graphs. ▪︎ Uses include de fi ning key metrics, identify data needs, de fi ne data management practices, prepare data for analysis, and present data to a viewer. ▪︎ Tracking user interaction data is required. The purpose of tracking is to measure systems performance and institutional compliance. ▪︎ xAPI is used to collect and store activity data that supports dashboards that helps learners analyze learning logs and provide evidence of learning. DOWNES, 2020 Stephen Downes, https://twitter.com/downes
- 10. DIAGNOSTIC ANALYTICS 10 ▪︎ Diagnostic analytics look more deeply into data in order to detect patterns and trends ▪︎ Security is a common application: automated checks for plagiarism, as an example, looks at the recurrence of words, trains machine learning applications to suss out keywords and phrases from a sample, and automated checks against databases in a complicated but consistent and veri fi able series of processes. ▪︎ Assessing competencies from actual performance data, sourced outside the learning context, uses technologies like analytics-based assessment of personal portfolios (in design profession contexts) aor using data-driven skills assessment in the workplace. DOWNES, 2020
- 11. PREDICTIVE ANALYTICS 11 ▪︎ Predictive Analytics identify factors statistically correlated. ▪︎ Example: Supporting employee retention, Elsevier’s Transition to Practice product leverages data from multiple activity sources to suggest newly graduated nurses in the program who may be at- risk of feeling unsupported and/or lack con fi dence in their professional skills, which statistically results in 35% of all newly graduated registered nurses leaving the profession within their fi rst two years. ▪︎ Sentiment analysis of journal entries ▪︎ Sharp swings in shift survey results ▪︎ Professional Skills Curriculum activity ▪︎ Like “precision medicine” such “precision learning” systems require more variables than data from a single activity: learner pro fi ling, learner preferences, learning environments, the work environment, learning strategies contribute to the data set. DOWNES, 2020
- 12. PRESCRIPTIVE ANALYTICS 12 ▪︎ A popular application for prescriptive analytics are recommendation engines. ▪︎ Adaptive Learning systems are powered by prescriptive analytics, of which there are various forms ▪︎ More on Adaptive Learning approaches and how they compare/contrast can be found here: https:// www.slideshare.net/aaronesilvers/adaptive- learning-43673699 BOWE AND SILVERS, 2014
- 13. POST-PREDICTIVE 13 DOWNES, 2020 ▪︎ Using analytics to generate new analytics ▪︎ Example: using data from assessment items to generate new assessment items for the learner ▪︎ Looks at competencies linked ▪︎ Looks at dif fi culty level of the assessment items ▪︎ Looks at previous attempts against sampled assessment items ▪︎ Determines without a human what is correct/incorrect and delivers that directly to the learner GENERATIVE ▪︎ “What ought to happen?” using analytics to mediate the experience. ▪︎ Example: Recommenders, Bots that enforce explicit media policies, anti-bias detection, Simulators DEONTIC
- 14. ETHICAL CHALLENGES FOR LEARNING ANALYTICS 14 AI is brittle. Error raises ethical concerns. Veri fi ability and replicability of data, algorithms and processes. Bias. DOWNES, 2020
- 15. AI / MACHINE LEARNING IS BRITTLE 15 When the data are limited or unrepresentative, it can fail to respond to contextual factors our outlier events. It can contain and replicate errors, be unreliable, be misrepresented, or even defrauded. In the case of learning analytics, the results can range from poor performance, bad pedagogy, untrustworthy recommendations, or (perhaps worst of all) nothing at all. DOWNES, 2020
- 16. ERRORS 16 Errors invite concerns. “Analytics results are always based on the data available and the outputs and predictions obtained may be imperfect or incorrect. Questions arise about who is responsible for the consequences of an error, which may include ineffective or misdirected educational interventions” Misinterpretations invite questions of validity. Because analytical engines don’t actually know what they are watching, they may see one thing and interpret it as something else. For example, looking someone in the eyes is taken as a sign that they are paying attention. And so that’s how an AI interprets someone looking straight at it. But it might just be the result of a student fooling the system. For example, students being interviewed by AI are told to “raise their laptop to be eye level with the camera so it appears they’re maintaining eye contact, even though there isn’t a human on the other side of the lens” (Metz, 2020). The result is that the AI misinterprets laptop placement as “paying attention”. DOWNES, 2020
- 17. VERIFIABILITY AND REPLICABILITY 17 Reliable data requires foresight and investment, “as distinguished from suspicion, rumor, gossip, or other unreliable evidence” A “reliable” system of analytics must be consistent. “an AI experiment ought to ‘exhibit the same behavior when repeated under the same conditions’ and provide suf fi cient detail about its operations that it may be validated.” The reliability of models and algorithms used in analytics “concerns the capacity of the models to avoid failures or malfunction, either because of edge cases or because of malicious intentions. The main vulnerabilities of AI models have to be identi fi ed, and technical solutions have to be implemented to make sure that autonomous systems will not fail or be manipulated. DOWNES, 2020
- 18. BIAS 18 In one sense, bias is one speci fi c way analytics can be in error or unreliable. The outcome of bias is re fl ected in misrepresentation and prejudice. The problem of bias pervades analytics: it may be ▪︎ in the data, ▪︎ in the collection of the data, ▪︎ in the management of the data, ▪︎ in the analysis, ▪︎ in the application of the analysis. DOWNES, 2020
- 19. PRIORITIZING ETHICAL USE OF LEARNING DATA STARTS WITH YOU. 19 PART III A professional working with xAPI furthers their inquiry skills and ethical practice by continuously building their muscles for empathy.
- 20. “ 20 BROWN, 2012 When it comes to our sense of love, belonging, and worthiness, we are most radically shaped by our families of origin – what we hear, what we are told, and perhaps most importantly, how we observe our parents engaging with the world…” - Brené Brown, Daring Greatly: How the Courage To Be Vulnerable Transforms the Way We Live Brené Brown, https://twitter.com/BreneBrown
- 21. DISSOLVING THE BARRIERS OF OUR BIAS 21 Your capacity for empathy stretches when you awaken loving-kindness for… ▪︎ Yourself “May I enjoy happiness and know the root of happiness” ▪︎ Someone you feel unequivocal goodwill and tenderness “May ___ enjoy happiness and know the root of happiness” ▪︎ Someone slightly more distant, like a friend or neighbor Use the same words. ▪︎ Someone about whom you feel indifferent Again, use the same words. ▪︎ Someone you fi nd dif fi cult or offensive Only when you can mean it, use the same words. ▪︎ All of the above, all together, at the same time — dissolving the barriers we put up for ourselves “May I, my beloved, my friend, the neutral person and the dif fi cult person all together enjoy happiness and know the root of happiness.” ▪︎ Toward all beings CHÖDRÖN, 2002 Pema Chödrön, https://twitter.com/AniPemaChodron
- 22. “FROM A PLACE ON NON- JUDGMENTAL CURIOSITY AND UNDERSTANDING” 22 ▪︎ Intended to facilitate collaboration and emotional support for both patients and health center staff through the social needs screening process ▪︎ Evoke patient priorities relating to social determinants of health needs ▪︎ Integrate patient feedback into subsequent care planning and delivery processes OPCA, 2022 Empathic Inquiry was created by Oregon Primary Care Association through motivational interviewing and trauma-informed care approaches, along with input from patients and other stakeholders.
- 23. QUESTION- STORMING 23 ▪︎ Does “an activity” mean “learning activity” or, like, a survey? ▪︎ What does it mean to “get through” an activity, speci fi cally? ▪︎ All the way to the last frame of linear content? ▪︎ Do you need to have seen all the content? ▪︎ Must you have passed in order to be considered complete? ▪︎ Do multiple conditions need to be met to “get through?” SILVERS, 2021 “How long does it take to get through an activity?” Seems like a really simple question, right? Give it a few seconds to get your own doubts going. Turn those into follow-up questions that demonstrate considering other perspectives or needs. As a mental challenge, and especially as a group activity, one could prioritize by coming up with as many questions as you can, given a single question as a prompt. This is how I’d approach producing and vetting enterprise analytics requirements. People don’t know what they want until they get perspective on what’s possible. This is why inquiry is important.
- 24. ETHICAL CHALLENGES FOR LEARNING ANALYTICS 24 Use data and analytics whenever they can contribute to learner success, ensuring that the analytics take into account all that is known about learning and teaching Equip learners and educators with data literacy skills, so they are suf fi ciently informed to give or withhold consent to the use of data and analytics Take a proactive approach to safeguarding in an increasingly data-driven society, identifying potential risks, and taking action to limit them. Work towards increased equality and justice, expanding awareness of ways in which analytics have the potential to increase or decrease these. Increase understanding of the value, ownership, and control of data. Increase the agency of learners and educators in relation to the use and understanding of educational data. FERGUSON, 2019 Prof Rebecca Ferguson, https://twitter.com/R3beccaF
- 25. RECAP 25 IF NOTHING ELSE, REMEMBER THIS Ethical Use Starts With You A tiny bad decision can have exponentially deadly results, but a tiny kind decision can have exponentially beneficial results. Learning Analytics Is More Than xAPI A professional working with xAPI MUST be able to describe and model what happens with the data. Ethical Practice Is Smart Learning Business A professional working with xAPI MUST be able to demonstrate evidence for design and architecture choices, including referencing peer-reviewed research that supports and strengthens the effort. Empathy Drives our Future Practice A professional working with xAPI furthers their inquiry skills and ethical practice by continuously building their muscles for empathy. TL;DR:
- 26. AARON E. SILVERS 26 Work: a.silvers@elsevier.com Need a hand? aaron@makingbetter.us Twitter: @aaronesilvers GO BIRDS! GO PHILS! GO SIXERS! WOOOO! Manager, Analytics / Learning Scientist & Architect Clinical Solutions, Elsevier Inc / MakingBetter
- 27. PRIMARY SOURCES 27 SOURCES Association for the Advancement of Computing in Education. 2019. “Global Guidelines: Ethics in Learning Analytics” https://www.jisc.ac.uk/ sites/default/ fi les/ jd0040_code_of_practice_for_learning_analytics_190515_v1.pdf Bowe, Megan & Aaron Silvers. 2015. The Big Picture of Adaptive Learning. https://www.slideshare.net/aaronesilvers/adaptive- learning-43673699 Boyer and Bonnie. 2016. Gartner Analytic Ascendancy Model. Downes, Stephen. 2020. Ethical Codes and Learning Analytics. Human and Arti fi cial Intelligence for the Society of the Future European Distance and E-Learning Network (EDEN) Proceedings 2020 Annual Conference | Timisoara, 22-24 June, 2020. ISSN 2707-2819. doi: 10.38069/ edenconf-2020-ac0003 Ferguson, Rebecca. 2019. “Ethical Challenges for Learning Analytics” https:// fi les.eric.ed.gov/fulltext/EJ1237573.pdf JISC. 2015. Codes of Practice for Learning Analytics https://www.jisc.ac.uk/sites/default/ fi les/ jd0040_code_of_practice_for_learning_analytics_190515_v1.pdf Oregon Primary Care Association. 2022. Empathic Inquiry. https:// www.orpca.org/initiatives/empathic-inquiry Silvers, Aaron. 2021. Question-storming. https://makingbetter.us/news/ learning-analytics/question-storming/