Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing

Editor's Notes

• #2: Hi I’m Hatim Lahza, a PhD candidate at the University of Queensland. Today, I’m going to present our work that has been done collaboratively with my great supervisory team, Professor Hassan Khosravi and Professor Gianluca Demartini from the University of Queenslad, and Professor Dragan Gasevic from Monash University. Our paper title is “Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing”.
• #3: I’m going to start with a brief introduction and abit of a background, then methods, then results, and then Discussion and conclusion.
• #4: As we all know, a part of instructors' responsibilities is supporting students in need. But, when class size increases, it's challenging for them to meet each individual student needs. This issue is mitigated with modern computer-supported learning environments as they can scale instructional solutions or scaffolds in large classes. However, there is no clear guidance on how self-regulation strategies can be integrated with them -------------------------------------------------------------------------------- Here I'm going to share the motivation of this study. As we all know, a part of instructors' responsibilities is supporting students in need. But, when the class size increases, it's challenging for them to assist students. Fortunately, computer-supported learning environments can scale instructional solutions very well in large classes. But, there is no clear guidance on how they can be integrated with self-regulation strategies
• #5: In our study we have two main objectives: Designing an online educational environment interfaces based on self-regulated learning strategies Conducting an in-the-field experiment to evaluate our design ==================================================================================================== We choose self-regulated learning as it’s an essential lifelong learning skill that can help students to take hold of their learning tasks and reduce their dependency on teachers. As a background, there are many different models of self-regulated learning from different psychological and educational perspectives. But, in this study we choose to look at their commonalities. In Panadero’s work, we could see that most of the common SRL models are cyclic processes and have preparatory, performance and evaluation phases. Moreover, a self-regulated learner should practice agency by choosing, adapting and developing strategies and instructions to reach their goals. On the other hand, a poor self-regulated learner can be supported by scaffolds within educational technologies. However, evidence of self-regulated learning effectiveness in online learning environments are still non-conclusive and insufficient to rigorously guide the practice. In our study, we use a learnersourcing environment: Which refers to the activity of engaging students in creating learning resources. This type of activity can benefit learners in many areas such as: Promoting active learning Supporting evaluative judgment skill Reducing Instructor load And more Past studies suggest that learner has the ability to create high-quality resources. However, not all contents created by students are high-quality. We assume that the creation interface might be part of the issue as they are usually designed for the instructor. ----------------------------------------------------------------------------- So, why self-regulation strategies and what do they consist of? Self-regulated learning is considered an essential competence and lifelong learning skill. The most common SRL models are cyclic and have three common phases: task analysis in which the plans and goals are set; followed by Performance in which tactics and strategies are implemented with monitoring And the third phase is “evaluation”, which involves reflection, self-assessment, regulation and adaptation In addition, a self-regulated learner should practice agency by choosing, adapting and developing strategies and instructions to reach their goals. On the other hand, a poor self-regulated learner can be supported by scaffolds within educational technologies. However, evidence of SRL effectiveness in online learning environments are still non-conclusive and insufficient to rigorously guide the practice. In our study, we use a learnersourcing environment: Which refers to the activity of engaging students in creating learning resources. This type of activity can benefit learners in many areas such as: Promoting active learning Supporting evaluative judgment skill Reducing Instructor load And more Past studies suggest that learner has the ability to create high-quality resources. However, not all contents created by students are high-quality. We assume that the creation interface might be part of the issue as they are usually designed for the instructor.
• #6: Why self-regulated learning? We choose self-regulated learning as it’s an essential lifelong learning skill that can help students to take hold of their learning tasks and reduce their dependency on teachers. There are many different models of self-regulated learning from different psychological and educational perspectives. But, in this study we choose to look at their commonalities. In the literature, we could see that most of the common SRL models are cyclic processes and have preparatory, performance and evaluation phases. Moreover, a self-regulated learner should practice agency by choosing, adapting and developing strategies and instructions to reach their goals. On the other hand, a poor self-regulated learner can be supported by scaffolds within educational technologies. However, evidence of self-regulated learning effectiveness in online learning environments are still non-conclusive and insufficient to rigorously guide the practice.
• #7: In our study, we use a learnersourcing environment: Which refers to the activity of engaging students in creating learning resources. This type of activity can benefit learners in many areas such as: Promoting active learning Supporting evaluative judgment skill Reducing Instructor load And more Past studies suggest that learner has the ability to create high-quality resources. However, not all contents created by students are high-quality. We assume that the creation interface might be part of the issue as they are usually designed for the instructor.
• #9: In our study we used RiPPLE which is an adaptive learning environments that enables learnersourcing of different types of learning resources such as MCQs and worked examples. It also supports learner models so that students can understand and monitor their performance in the platform. The process graph here is a high level overview of the process showing the three main types of activities, creating learning resources, moderating learning resources and practicing resources tailored to students abilities. And here is the original interface for creating a learning resource which is in this case a multiple choice question. So, the students first write the stem of the question, then choose the associated topic, then create the correct response and the distractors, and lastly write an explanation for the question.
• #10: To reach or objectives, we first developed interventions for SRL strategies In our study, we considered three main interventions, namely planning then creating externally-facilitated monitoring And, self-assessing after creating Additionally, we considered a fourth intervention that combine all of the strategies Then we created an interface corresponding to each of the interventions as I'm going to show in the next slides
• #11: In our study, the first self-reglation strategy is the planning then creating which includes six main elements. We hypothesis that the planner can benefit both students who have a clear idea of what to create and those looking for new ideas Here in the figure, the first step is: Inspiration source of the resource ides The target concept or skill The learning level according to Bloom’s taxonomy. The students were also provided with a short presentation that explains how to choose the appropriate level which we will see in the next slide Level of difficulty of the item Mistakes or misconception related to the question Type of media to be used It’s important to mention that here we focus only on multiple choice questions And also these components are optional
• #12: This slide shows the presentation provided to students on learning levels. The presentation includes the definitions of the learning levels and the associated verbs. Also, exemplars and general tips were provided
• #13: The second intervention is the externally-facilitated monitoring that should support the SRL “performance” phase. We aim to provide monitoring elements while creating a resource These elements support: Best practices in writing the body of the question Best practices in writing plausible distractors And best practices in writing explanation of the correct answer We presented these elements as tick boxes
• #14: The self-assessing after creation is the third intervention and it aims to support the SRL evaluation phase Our interface consists of self-assessment and reflection. The self-assessment was built according to the rubric with a set of criteria used in the moderation activity. In the reflection component, the students were provided with an open-ended question to write their reflection on the resource they created.
• #15: We used the following research questions to guide our study To reach our second objective, we conducted an in-the-field randomised control experiment that considers a control group A and four other treatment groups: planning then creating (B) monitoring while creating (C) self-assessing after creating (D) All (E)
• #16: The study was conducted at the university of Queensland. We used data from two courses, Neuroscience & Information systems. The participants were 797 undergraduate students who provided their consent on the platform. RiPPLE was used for assessment over multiple rounds and contributed 10% to the final grade. But engagement with the intervention was optional. So, the creation task along with other tasks such as the moderation or practices were required to gain the 10%, but not the intervention. The resources created by the students for both courses were about 2-3 per student. We considered a student to be engaged if they at least completed all the components of an intervention. So, an engaged student is the student how at least completed all the intervention components at one session
• #17: Table 1 provides an overview of the experimental groups. In total, the students created 1907 resources. The students in each group created from 300 to 400 resources. The engaged students were more than 60% in both courses.
• #18: Here are the RQs again. In our analysis, we used the combined data. First, we measured the impact of the interventions on the creation process and effort. To do so, we modelled the creation processes using the trace data and First Order Markuv Model. We also conducted a qualitative analysis using the time spent by students in min. Second, we measured the impact on student perception. Students rated t he effectiveness of the interfaces using a scale of 5 after each creation. Also, we used 137 responses to an open-ended question provided by 103 students. Third, we used descriptive and inferential statistics on the effectiveness rating, and we used a general inductive approach and sentiment analysis on the written feedback. We measured the impact on performance by using qualitative analysis on the resource quality rating provided by the system.
• #20: Here we present the result of RQ1 in which we investigate the impact of the interventions on the process and effort The visualisation of the process consists of connected oval shapes representing the type of activity and arrows showing the direction of the connection. The strength of the connection is indicated by the thickness of the arrows and the transition probability next to it. Inside the oval shapes, we added the fractional time the students spent on the activities. Key observations In general the interventions changed the process in which the students create learning resources, but with additional complexity. Condition B: The students used the planner and there was a strong interaction between the planning and creation components However, the students sometimes skipped the planning component (28%). This might mean many different things, such as the students knew what to create. We can see that the panner required extra time to complete the creation task and this was statistically different than the control group. Condition C: Similar to the planner, the students used the monitoring components. There was large probability of ending the process after writing the resource (40%). I expected this to be lower. The students spent only 4% of the total time on the scaffolding. This is similar to previous findings [1]. Condition D: strong interactions between the self-assessment and creation (Cr –> Sa: 87% and Sa –> Cr: 32%). Condition E: Combination of the processes observed in the other groups
• #21: In RQ2, we investigate the impact of the interventions on Student Perceptions Table 2 summarises the results And here, we order the conditions based on the average of the student rating It seems that the students liked the control interface more than the treatment interfaces Next, the qualitative analysis will reveal more information
• #22: Here we present the qualitative analysis results. We identified three main themes, technology, pedagogy and generic In table 3 we summarise the results with some examples of students’ responses In summary,
• #23: Here we investigate the impact on the performance. It’s important to recall that the performance here is measured by quality rating that the resource receive from the moderation process. Figure three summarises the results of RQ3. We can see that for both “all students” and “engaged students”, condition “B” and “C” had the higher median followed by condition D. The control group and condition E had the least median. The difference between condition C and A was statistically significant for the engaged students with small effect size.
• #25: Our results indicate that: Scaffolds increased task complexity and time with no significant improvements Similar results observed in previous research [1] Many factors can help explain why the SRL strategies might have a small or no impact on content creation. The students could have different regulation levels Some students could low prior knowledge Technical difficulties that we found in the qualitative analysis Also, our results indicate that the Interventions were not perceived as effective Which means that the metacognitive prompts were not assisted by the learners positively: This can be explained by Winne [2] four conditions According to Winne [35], for learners to use a learning tool that introduces metacognitive scaffolds, four conditions need to be met: - learners are aware that the scaffold is useful to them, learners know that the scaffold can be useful for the task at hand, learners have enough skills to use the scaffold and learners are motivated to use it.
• #26: Our work contribute to the growing interest in creating analytics-based environments that enhance learning and using learning analytics to evaluate such environments We highlight the need for combining the newly introduced scaffolds with rationale for their value along with training to facilitate student acceptance and adoption. The training can be delivered with LA personalised feedbacks We emphasize the importance of empirical research in measuring the effectiveness of theory-based technological interventions and the involvement of stockholders in assessing interventions
• #27: We acknowledge some limitation of our work that we plan to overcome in our future work. The duration was relatively short which may not have allowed the benefits of the interventions to be fully achieved. We only used the rating provided by the system which might not be as accurate as instructors rating We also did not consider the impact of the interventions on students learning Finally, we did not use data from multiple courses