Towards Educational Recommender Systems for Self-Directed Learning
0 likes32 views
This document discusses the development of educational recommender systems (ERS) aimed at enhancing self-directed learning by examining decision-making processes among students through qualitative interviews. It highlights the importance of understanding student goal orientations and identity formation, and proposes evaluation metrics for ERS that cater to individual student needs such as accuracy, novelty, and utility. Ultimately, it advocates for future research into ERS that supports both exploratory and convergent thinking based on students' learning goals.
![Recommendation Systems in Education:
A Systematic Mapping Study [17]
4
→ Existing
approaches do
not consider
learner
characteristics
C1. Areas in education; C2. RS approach; C3. RS development](https://image.slidesharecdn.com/paperpresen-songer-210723123442/85/Towards-Educational-Recommender-Systems-for-Self-Directed-Learning-4-320.jpg)
![RecSys Evaluation Metrics
5
[5], [7], [11]
Accuracy-
The most common evaluation
metric in ERS, it effectively
calculates sameness or likeness
based on similar users and
previous choices.
Novelty
Diversity
Utility
Serendipity](https://image.slidesharecdn.com/paperpresen-songer-210723123442/85/Towards-Educational-Recommender-Systems-for-Self-Directed-Learning-5-320.jpg)
![Exploration in STEM
An exploratory attitude is linked to adaptability, motivation,
self-confidence, self-efficacy and student retention in STEM
Exploration is represented in two psychology theories:
■ In Achievement GoalTheory, the student's goals are
oriented towards discovery, growth and learning
■ In Identity StatusTheory, the student seeks new information
to form a personal or professional identity
8
[6], [16], [18]](https://image.slidesharecdn.com/paperpresen-songer-210723123442/85/Towards-Educational-Recommender-Systems-for-Self-Directed-Learning-8-320.jpg)
![Achievement GoalTheory
9
[3]
Intrinsic motivation
Desire to learn or grow
Aim to prove self-
worth
:
demonstrate
knowledge or skill
: escape
being assessed for lack
of skill
Resist spending time
or effort
■ Students value achievement relative to their personal goals
within socio-cultural and situational contexts.](https://image.slidesharecdn.com/paperpresen-songer-210723123442/85/Towards-Educational-Recommender-Systems-for-Self-Directed-Learning-9-320.jpg)
![Identity StatusTheory
• Open to different
identities
• Collect information and
make decisions rationally
• Committed to protecting or
validating an identity
• Make decisions based on
social clues
10
[14]
■ Decision-making is influenced by the development of a
personal or professional identity.](https://image.slidesharecdn.com/paperpresen-songer-210723123442/85/Towards-Educational-Recommender-Systems-for-Self-Directed-Learning-10-320.jpg)
![Thank you for your attention
[3] Carol S. Dweck. 1986. Motivational processes affecting learning. American Psychologist 41, 10, 1040–1048. DOI:https://doi.org/10.1037/0003-
066X.41.10.1040
[5] Mojisola Erdt, Alejandro Fernández, and Christoph Rensing. 2015. Evaluating Recommender Systems for Technology Enhanced Learning: A Quantitative
Survey. IEEE Transactions on Learning Technologies 8, 4, Article 1 (Oct.-Dec. 2015), 326-344, DOI:https://dx.doi.org/10.1109/TLT.2015.2438867
[6] Hanoch Flum and Avi Kaplan. 2006. Exploratory Orientation as an Educational Goal. Educational Psychologist 41, 2, 99-110,
DOI:10.1207/s15326985ep4102_3
[7] Mouzhi Ge, Carla Delgado-Battenfeld, and Dietmar Jannach. 2010. Beyond accuracy: evaluating recommender systems by coverage and serendipity. In
Proceedings of the fourth ACM conference on Recommender systems (RecSys '10). Association for Computing Machinery, New York, NY, USA, 257–260.
DOI:https://doi.org/10.1145/1864708.1864761
[11] Komal Kapoor, Vikas Kumar, Loren Terveen, Joseph A. Konstan, and Paul Schrater. 2015. "I like to explore sometimes": Adapting to Dynamic User Novelty
Preferences. In Proceedings of the 9th ACM Conference on Recommender Systems (RecSys '15). Association for Computing Machinery, New York, NY,
USA, 19–26. DOI:https://doi.org/10.1145/2792838.2800172
[14] James E. Marcia. 1993. The Status of the Statuses: Research Review. Ego Identity. Springer, New York, NY. DOI:https://doi.org/10.1007/978-1-4613-8330-
7_2
[16] Tony Perez, Jennifer G. Cromley, Avi Kaplan. 2014. The role of identity development, values, and costs in college STEM retention. Journal of Educational
Psychology 106, 1 (Feb. 2014), 315-329.
[17] Abdon Carrera Rivera, Mariela Tapia-Leon, and Sergio Lujan-Mora. 2018. Recommendation Systems in Education: A Systematic Mapping Study. In
Proceedings of the International Conference on Information Technology & Systems (ICITS 2018). Advances in Intelligent Systems and Computing, Vol. 721.
Springer, Cham. DOI:https://doi.org/10.1007/978-3-319-73450-7_89
[18] Montrischa Williams and Casey George. 2014. Using and doing science: gender, self-efficacy, and science identity of undergraduate students in STEM.
30](https://image.slidesharecdn.com/paperpresen-songer-210723123442/85/Towards-Educational-Recommender-Systems-for-Self-Directed-Learning-30-320.jpg)
Towards Educational Recommender Systems for Self-Directed Learning
- 1. TOWARDS EDUCATIONAL RECOMMENDER SYSTEMS FOR SELF-DIRECTED LEARNING RobertW. Songer International College ofTechnology, Kanazawa TomohitoYamamoto Kanazawa Institute ofTechnology ICEMT 2021,Virtual Conference, July 23 – 25
- 2. Towards Educational Recommender Systems for Self- Directed Learning ■ A qualitative study into the context of self-directed learning ■ From interviews, we analyzed dispositional, contextual, and situational aspects of decision- making in projects ■ Analyzed responses from the perspectives of Achievement Goal Theory, Identity StatusTheory, and evaluation metrics for Educational Recommendation Systems (ERS). Profiling Student Achievement Goals and Identity Formation to Inform Recommendation Evaluation Metrics 2
- 3. INTRODUCTION Educational Recommender Systems (ERS) 3
- 4. Recommendation Systems in Education: A Systematic Mapping Study [17] 4 → Existing approaches do not consider learner characteristics C1. Areas in education; C2. RS approach; C3. RS development
- 5. RecSys Evaluation Metrics 5 [5], [7], [11] Accuracy- The most common evaluation metric in ERS, it effectively calculates sameness or likeness based on similar users and previous choices. Novelty Diversity Utility Serendipity
- 6. Our Cross-DisplinaryApproach From Educational Psychology: ■ Decisions are made based on perceived value of choices ■ Achievement GoalTheory and Identity StatusTheory describe how choices are valued by individuals ■ ERS recommendations provide choices, so we should understand how students perceive their value 6
- 7. THEORETICAL BACKGROUND Achievement GoalTheory & Identity StatusTheory 7
- 8. Exploration in STEM An exploratory attitude is linked to adaptability, motivation, self-confidence, self-efficacy and student retention in STEM Exploration is represented in two psychology theories: ■ In Achievement GoalTheory, the student's goals are oriented towards discovery, growth and learning ■ In Identity StatusTheory, the student seeks new information to form a personal or professional identity 8 [6], [16], [18]
- 9. Achievement GoalTheory 9 [3] Intrinsic motivation Desire to learn or grow Aim to prove self- worth : demonstrate knowledge or skill : escape being assessed for lack of skill Resist spending time or effort ■ Students value achievement relative to their personal goals within socio-cultural and situational contexts.
- 10. Identity StatusTheory • Open to different identities • Collect information and make decisions rationally • Committed to protecting or validating an identity • Make decisions based on social clues 10 [14] ■ Decision-making is influenced by the development of a personal or professional identity.
- 11. INTERVIEWS Student Stories, Questions & Responses 11
- 12. Setting: International College of Technology, Kanazawa ■ Students of ages 16–17 in 2nd year Computer Skills course within the Department of Science andTechnology ■ CS topics include animation, video editing, programming, etc. ■ Self-directed "focus area" project at the end of term – Students chose the topic, skills, software, & criteria – Teacher approves, evaluates challenge level 12
- 13. Interview Objectives Q1. What decision-making struggles did the students have? – Student disposition – Contextual factors – Situational aspects Q2. What were the students' goals and orientations for learning? – Mastery vs. Performance Goals – Exploratory vs. Normative Identity Status 13
- 14. Interview Method When: End of term, after the "focus area" project ■ 7 students (5 male, 2 female) agreed to the interview ■ Semi-structured format for 30-40 minutes ■ 1 teacher (interviewer) with 1-2 students ■ Each session recorded and transcribed w/ pseudonyms Guiding questions based on Achievement GoalTheory and Identity Formation StyleTheory 14
- 15. Interview Question Analytical Clues What were your personal goals for the project? Mastery/Performance Goal Orientation What was your motivation for choosing your topic? Exploratory/Normative Identity Formation Style What did you consider to be your own success on the project? Disposition (ACH) How difficult was it for you to choose a topic? How difficult was your chosen topic for you? Disposition (ACH) What did you think about doing a project like this for the Computer Skills course? Disposition (ID) (ACH = Achievement goal orientation) (ID = Identity formation style) 15
- 16. Interview Question Analytical Clues How did you handle the responsibility of writing the rubric items? Disposition (ID) What impression do you think other students and the teacher had about your results? Context (ACH) How do you think you changed in the eyes of other students and the teacher? Context (ID) After hearing the project introduction, what did you think was most important for you to do? Situation (ACH) What questions or doubts did you have for yourself when choosing your topic and rubric items? Situation (ID) (ACH = Achievement goal orientation) (ID = Identity formation style) 16
- 17. Focus Area Projects (1) • 3D Model in Fusion 360 • Highly motivated with lofty goals Kenta • Name logo in Photoshop • A comfortable task to avoid "teacher slave labor" Takeo • Music video in Premier with After Effects • Maximize assessment, minimize effort Aya 17
- 18. Focus Area Projects (2) • Video in Premiere with After Effects • Wanted to learn and express himself Kazu • Reinforced Learning in Python • Focused on tasks, not opinions or evaluation Kei • AppearanceTools in Illustrator • Wanted to learn new tools; not committed to field Shin • Image Editing in Photoshop • Not confident to explore topics; chose a tutorial Sakura 18
- 19. DISCUSSION Student Goal Orientations, Identity Statuses, Recommendation Needs, and Evaluation Metrics for an ERS 19
- 20. Analysis ■ We identified achievement goal orientations and identity statuses for each student ■ Additionally, we propose RecSys evaluation metrics based on each student's decision-making needs: – Accuracy of predicting the user's rating of the item – Utility based on a measurable benefit to the user – Novelty as different from user's past experiences – Diversity as difference between items within a selection – Serendipity as the union of accuracy and novelty 20
- 21. Goals Identity Recommendation Needs Metrics Kazu Mastery Exploratory Novel and fun, easy enough to be enjoyed. Novelty Kei Mastery Exploratory Convergent on specific problem definitions. Accuracy Shin Mastery Exploratory Descriptive of specific software functions. Accuracy Goals, Statuses & Needs (1) 21 ■ During the project, all 3 were relaxed with no feelings of stress ■ Shin and Kei had a topic from the beginning and only needed to narrow down their tasks
- 22. Goals Identity Recommendation Needs ERS Metrics Kenta Mastery Normative Novel but closely related to existing interests. Serendipity Goals, Statuses & Needs (2) 22 ■ Kenta had a strong desire to complete his chosen task ■ Pressure to finish did not come from worry over his grade, but something personal instead
- 23. Goals Identity Recommendation Needs ERS Metrics Sakura Avoidance (work) Normative Varying according to challenge level. Diversity/Utility Goals, Statuses & Needs (3) 23 ■ Sakura had limited ideas about what to do with Photoshop ■ She also had low confidence for choosing tasks and worried about her grading criteria
- 24. Goals Identity Recommendation Needs ERS Metrics Takeo Avoidance (work) Normative Applicable to project and ideas of a future self. Serendipity Goals, Statuses & Needs (2) 24 ■ Takeo was happy in his role helping others with Photoshop ■ He did not explore topics related to his future goals as he didn't want to do a lot of work
- 25. Goals Identity Recommendation Needs ERS Metrics Aya Performance Exploratory Time-efficient and highly rated topics. Utility Goals, Statuses & Needs (2) 25 ■ Aya had exploratory goals at the educational level, but performance goals at the course level ■ Pressure to complete work from other courses afforded her little room to explore in the focus area project
- 26. An UnexpectedObservation Students expressed patterns of divergent thinking and convergent thinking while choosing topics for their project 26 DivergentThinking • Expand choices; Discover topics • ERS Metrics: Diversity & Novelty Convergent Thinking • Eliminate Choices; Specify tasks • ERS Metrics: Accuracy & Utility
- 28. Profiling ERS Users 1. Students who already explore may need help converging on a task 2. Other students need diverse options to support divergent thinking 3. Metrics such as difficulty ratings and time estimates may support utility-based convergence 28
- 29. Future Research A proposal for self-directed learning ERS based on goal orientations & convergent/divergent thinking model: ■ Use diversity metric to present topics to student users in a way that promotes exploration/divergence ■ Provide utility-based metrics (difficulty, time/effort) for specific tasks to assist with convergence 29
- 30. Thank you for your attention [3] Carol S. Dweck. 1986. Motivational processes affecting learning. American Psychologist 41, 10, 1040–1048. DOI:https://doi.org/10.1037/0003- 066X.41.10.1040 [5] Mojisola Erdt, Alejandro Fernández, and Christoph Rensing. 2015. Evaluating Recommender Systems for Technology Enhanced Learning: A Quantitative Survey. IEEE Transactions on Learning Technologies 8, 4, Article 1 (Oct.-Dec. 2015), 326-344, DOI:https://dx.doi.org/10.1109/TLT.2015.2438867 [6] Hanoch Flum and Avi Kaplan. 2006. Exploratory Orientation as an Educational Goal. Educational Psychologist 41, 2, 99-110, DOI:10.1207/s15326985ep4102_3 [7] Mouzhi Ge, Carla Delgado-Battenfeld, and Dietmar Jannach. 2010. Beyond accuracy: evaluating recommender systems by coverage and serendipity. In Proceedings of the fourth ACM conference on Recommender systems (RecSys '10). Association for Computing Machinery, New York, NY, USA, 257–260. DOI:https://doi.org/10.1145/1864708.1864761 [11] Komal Kapoor, Vikas Kumar, Loren Terveen, Joseph A. Konstan, and Paul Schrater. 2015. "I like to explore sometimes": Adapting to Dynamic User Novelty Preferences. In Proceedings of the 9th ACM Conference on Recommender Systems (RecSys '15). Association for Computing Machinery, New York, NY, USA, 19–26. DOI:https://doi.org/10.1145/2792838.2800172 [14] James E. Marcia. 1993. The Status of the Statuses: Research Review. Ego Identity. Springer, New York, NY. DOI:https://doi.org/10.1007/978-1-4613-8330- 7_2 [16] Tony Perez, Jennifer G. Cromley, Avi Kaplan. 2014. The role of identity development, values, and costs in college STEM retention. Journal of Educational Psychology 106, 1 (Feb. 2014), 315-329. [17] Abdon Carrera Rivera, Mariela Tapia-Leon, and Sergio Lujan-Mora. 2018. Recommendation Systems in Education: A Systematic Mapping Study. In Proceedings of the International Conference on Information Technology & Systems (ICITS 2018). Advances in Intelligent Systems and Computing, Vol. 721. Springer, Cham. DOI:https://doi.org/10.1007/978-3-319-73450-7_89 [18] Montrischa Williams and Casey George. 2014. Using and doing science: gender, self-efficacy, and science identity of undergraduate students in STEM. 30