Srl research lecture
3 likes2,628 views
This document outlines different methods that can be used to measure self-regulated learning. It discusses measuring SRL as both an aptitude and an event, and some of the challenges in measurement. Some key methods mentioned include self-report questionnaires, observations of student behavior, think-aloud protocols, learning diaries, interviews, and analyzing digital traces of student interactions in online learning environments. The document advocates for using mixed methods to address limitations of individual approaches and gain a more comprehensive understanding of students' self-regulated learning.
Srl research lecture
- 1. MEASURING SELF-REGULATED LEARNING Piia Näykki Learning and Educational Technology Research Unit 1
- 2. OUTLINE OF THE LECTURE 1. Key points of SRL and measurement 2. Research protocols that are currently used to measure SRL 3. Constructive critique for the measures 4. Primary vs. secondary research 5. Forecast what measurements of SRL might be in the future 27.11.2015 2
- 3. KEY POINTS IN SRL The general agreement in the research field: • The term SRL is associated with forms of learning that are metacognitively guided, intrinsically motivated, and strategic • There are a number of SRL models • SRL models emphasizes different phases • SRL is a cyclical process 3 This is what we already know! hmmmmm.. But how to study SRL?
- 4. 4 BRAINSTORM in pairs (5 min): 1. Think, what do you allready know about Self-Regulated Learning and 2. What aspects of SRL you would be interested to study? 3. Formulate a list of interesting questions related to SRL. 4. Prepare yourself to share your ideas to the class.
- 5. 5 SHARE your ideas to the class, As we continue the lecture, think following question: - How could this topic of my interest be studied?
- 6. KEY POINTS OF MEASUREMENTS OF SRL 1. Measurement of SRL should reflect a model of SRL. 2. From the description of SRL it is obvious that many facets of SRL are not readily observable. 3. One challenge of studying SRL is to find ways to document its components. 4. How to study SRL as an aptitude and/ or as an event? 27.11.2015 6
- 7. SRL AS AN APTITUDE… AS AN EVENT 1. When SRL is measured as an aptitude, a single measurement aggregates over or abstracts some quality of SRL based on multiple SRL events. Example: - student studying for a test. A researcher may be interested in mc monitoring concerning a rehearsal tactic. - The student’s description of SRL might be recorded in several forms: a rating on a questionnaire item, an interview’s classification or the proportion of the particular kinds of notes student has written to the textbook. - To predict whether a student will/ will not, can/ can not act on an SRL-related cognition. 27.11.2015 7
- 8. SRL AS AN APTITUDE… AS AN EVENT 1. When SRL is measured as an event has 3 levels: occurency, contingency and patterned contingency. Example: - Student is solving a geometry problem and thinking aloud. - The student says: ”wow, this is hard”. - The researcher may presume that s/he have monitored the current state of the task (task difficulty). - What are the standards against this monitoring action has been done? (we don’t know). - Student’s report is interpreted as indirect evidence. 27.11.2015 8
- 9. WHY IT IS SO DIFFICULT TO MEASURE SRL • SRL is an internal process • Can we ask individuals to tell their internal processes? • What problems are related to that? • SRL has an external signs, • What kind of signs? • Are we sure that our interpretations of the signs are valid? • What problems are related to that? 9
- 10. TENSIONS IN MEASURING SRL • General Reliability & validity • Big sample size vs. sample with a meaning • Self-report critique: Self-report is too static and ”Individuals tend to tell what they expect researchers want to hear”. • Qualitative approaches critique: video observation methods are subjective and based on researchers’ interpretation, not objective. • In sum: Some of the features of the classical QUANTITATIVE VS. QUALITATIVE divide 10
- 11. OUTLINE OF THE LECTURE 1. Key points of measurements in general 2. SRL: components researchers and practioners seek to measure 3. Research protocols that are currently used to measure SRL 4. Constructive critique for the measures 5. Primary vs. secondary research 6. Forecast what measurements of SRL might be in the future 27.11.2015 11
- 12. PROTOCOLS FOR MEASURING SRL Winne & Perry; Boekaerts & Corno (2005) + Handbook2011 AS AN APTITUDE • Self-report questionnaire • Structured interviews • Teacher judgements AS AN EVENT • Observations of overt behavior (video/ real time) • Thinking aloud protocols • Traces of mental events and processes (computer) • Learning diaries • Interviews (possible with a stimulus) 12
- 13. 13 BRAINSTORM in pairs (5 min): 1. Take your list of interesting questions related to SRL. 2. Now think and discuss in pairs: how could your topic of interest be studied? 3. Prepare yourself to share your ideas to the class.
- 14. OBSERVATIONS OF OVERT BEHAVIOR • The famous Marshmallow experiment http://www.youtube.com/watch?v=QX_oy9614HQ • 1960’s Standford University • 4 year old kids, 15 minutes to waite • 2/3 could not waite.. • 1/3 – delayed gratification • 14-15 years later follow-up studies with the same kids. • Around 100% of those who waited were successful in school • Around 80% of those who could not waite were in troubles. 27.11.2015 14
- 15. OBSERVATIONS OF OVERT BEHAVIOR • Capture what the learner is doing • Based on verbal (talk), actions (gestures) and interactions • Use a coding categories (pre-defined or based on the data) • Recording for inter-judges reliability Problems: • Are we really accessing the students SRL? • Do we make inferences as researchers? • How does a SRL ”look like”? How it can be recognized? How to operationalize the phenomenon? 27.11.2015 15
- 16. THINKING-ALOUD PROTOCOLS Example of thinking-aloud (02:45-03:40) http://www.youtube.com/watch?v=px0A4jV02RA&feature=relate d • Students say out loud what they are doing, thinking, etc. • Ericsson, K. A., & Simon, H. A. (1980) • Coding and inter-judges • Researcher presence should not affect but activate • Direct measure on-task • Problems: requieres training, reliability with younger children, task difficulty, high cognitive load, frequency affected by personal factors. 27.11.2015 16
- 17. DIARIES • Students report in a scheduled basis their strategy use and problems • Bernhard Schmitz & Julia Klug (Handbook, 2011) • Qualitative but also quantitative (structured questions) • Time series analysis • Type of Self-report • Problems: Reactivity, high level of students implication 27.11.2015 17
- 18. INTERVIEW • Asking questions about individuals’ self-regulated learning • Interview can be unstructured, semi-structured or structured • Stimulated recall interview • Participants explain what was happening in a certain situations (i.e. in the video). • Coding needed: Inter-judges • Problems: off-task, students’ accuracy, laborous: transcriptions needed. 27.11.2015 18
- 19. TRACES OF MENTAL EVENTS & PROCESSES Example: Eye-tracking experiment, online dating http://www.youtube.com/watch?v=WqD2pXqT0Z0 • Follow the students actions called traces. • Students behavior (Computer technology based) • Examples: gStudy & nStudy (Winne et al.) • Objective collection 27.11.2015 19
- 20. LOGFILE TRACES • Time stamped records of learners activity in computer based learning environment. • Provide data ”on the fly” of learners actions. • Possible to capture in details each action lerner produces (Perry & Winne, 2006). • Questionnaires and self-reports do not reliably answer what the students actually do (Hadwin et al., 2002). • Depending on the age group, often children cannot destinquish they actions from actual behavior (Paris & Paris 1990).
- 21. WHEN THERE IS A NEED TO USE ON-LINE METHODS • Usability problems • Learner – hypermedia interaction, such as navigation • Modeling learners actions • Investigation of cognitive processes during the learning • To better capture intra-individual differences on learners actions (Rouet & Passerault, 1999; Winne 2001; Hadwin et al., 2007)
- 22. DURATION, FREQUENCY AND SEQUENCY OF EVENTS • Duration = how long time something is happening • i.e. duration of reading (eg. Time spent in reading) • duration of model events (eg. Time spent in processing or manipulating the information). • Frequency = how often a certain event occurs? • i.e. amount of checking the instructions. • Sequency = how the events are sequenced. • i.e. sequency of the specific events occuring. • Data parsing to identify the meaningful events from the extagenous events that occur in the data (Nesbit et al., 2008). XWSAJNSHBHCLSOD XWSAJNSHBHCLSOD -> ABC
- 23. • gStudy is designed in order to supports students to became and practise self-regulated learning skills (Winne et al., 2005). • gStudy provides a number of cognitive tools, for example highlighting with different labels, creating notes that prompts to elaborate or summarize or making searches from the contents of the current kit (Kumar, Groeneboer, Chu, Jamieson-Noel. & Xin, 2006; Winne & al., 2006). • Students cognitive tool use in gStudy is considered to reflect students use of study tactics (Hadwin et al., 2008) GSTUDY
- 24. LEARNING PATTERNS IN CHALLENGING LEARNING SITUATIONS •In the challenging learning situations, 88 learning patterns emerged between the low achieving students. •These patterns varied from lenght of three to eight. •Each pattern was used eight to twelve times between challenging gStudy sessions. 2 1 1 2 1 2 2 2 2 2 1 1 2 3 4 5 6 7 8 Label: Interesting information Label: Important information 1 11 3 3 1 2 3 4 Label: Interestimg detail Make note in C-map
- 25. nStudy Winne & Hadwin, 2009
- 26. TRACES PROBLEMS • Problems: students can use actions for different purposes (researcher inferences), Massive data: analysis problematic 26 27.11.2015
- 27. SELF-REPORT • A self-report inventory is a type of psychological test in which a person fills out a survey or questionnaire with or without the help of an investigator • Likert-scales (5 or 7) • Quantitative data • Open questions could be qualitative • Large sample size • Easy interpretation • High reliability if well-constructed • MSLQ, LASSI, OSLQ, ILS… 27.11.2015 27
- 28. SELF-REPORT PROBLEMS • Validity • Honesty & accuracy (Introspective ability) • Decontextualized • Too general if not tailored • In sum: Like other methods have problems 27.11.2015 28
- 29. MIXED METHODS TO STUDY SRL • Triangulation • Validity • Different information: bigger picture • Example of combinations: • Self-report with other • TAP + stimulated recall • Traces + stimulated recall 27.11.2015 29
- 30. RESEARCH EXAMPLE Mixed methods approach: video data analysis of group interaction and video-stimulated recall interview. Näykki, P. Järvelä, S. Kirschner, P., & Järvenoja, H. (2014). Socio-emotional conflict in collaborative learning – A process-oriented case study in a higher education context. International Journal of Educational Research, 68, 1-14.
- 31. How did the students interpret the conflict? Emma and Anna reacted the most strongly to the conflict EMMA: ‘It was a huge shock to me; I was so surprised that strange people can talk to each other like that. So I shut down, and I thought, “Oh my god, can I say what I think at all?” ANNA: ‘I think we were overruled, and I didn’t enjoy the group work after the conflict; it was just to get the course done. At first, I tried to negotiate the task with Erik, but in the end, I took a yes-man role and tried not to care and to agree on every solution’.
- 32. Erik and Maria explained the conflict more neutrally ERIK: ‘The personal chemistry didn’t work between two of the girls in our group, Tiina and Maria. Actually, there were three central characters, me, Maria, and Tiina, and then the two that were a bit quieter, Anna and Emma. Very clearly, Maria was bothered of Tiina’s occupation; I think she was jealous of her and was making fun of her’. Maria andTiina had this fight, and after that, everything was kind of hushed up and—I don’t know—I sometimes felt that I was some kind of a leader; others wanted me to comment on everything’. My goal was to learn as much as possible because this whole area is so interesting and very valuable to me’. But after the conflict, it was more like, let’s just try to do the task, and we just tried to get group work done as soon as possible’.
- 33. KEY POINTS OF MEASUREMENTS IN GENERAL • Reliability: • Overall consistency of a measure: Does it measure what we are aiming for? • Inter-judges reliability. • Validity: • Are these results meaningful: How to interpret the findings? • Internal validity: extent to which a causal conclusion based on a study is warranted. Is my study valid? Did I interpret the results correctly? • External validity: extent to which the results of a study can be generalized to other situations and to other people. Does this happen in the real world? 33
- 34. OUTLINE OF THE LECTURE 1. Key points of measurements in general 2. SRL: components researchers and practioners seek to measure 3. Research protocols that are currently used to measure SRL 4. Constructive critique for the measures 5. Primary vs. secondary research 6. Forecast what measurements of SRL might be in the future 34
- 35. PRIMARY VS. SECONDARY RESEARCH • Primary research consists of the collection of original primary data by the researcher. It is often undertaken after the researcher has gained some insight into the issue by reviewing secondary research or by analyzing previously collected primary data. • Secondary research (also known as desk research) involves the summary, collation and/or synthesis of existing research rather than primary research, where data is collected from, for example, research subjects or experiments. 35
- 36. TYPES OF SECONDARY RESEARCH • Different types • Systematic Review (Meta-Analysis) • Best-Evidence Synthesis • Narrative Review • Different Objectives • Integrative Research Review • Theoretical Review • Methodological Review • Thematic Review • State-Of-The-Art Review • Historical Review • Comparison of two perspectives review 36
- 37. META-ANALYSIS • A meta-analysis refers to: • Methods focused on contrasting and combining results from different studies… • …in the hope of identifying patterns among study results… • …giving statistics to evaluate the effect of the relationships. • It uses the different studies’ effect sizes comparing among them taking into account other variables (e.g. sample size) to weight the importance of a particular study for the general research aim. 37
- 38. CRITIQUES TO META-ANALYSIS • A meta-analysis of several small studies does not predict the results of a single large study (medicine). • Golden standard Randomized Controlled Trial vs. meta- analysis –RCT is just one study. • Publication bias: the file drawer problem. • Different study goals, methods, etc.: Mixing apples and oranges • Inclusion criteria is CRUCIAL: Garbage in, garbage out • In sum: there are bad meta-analysis as much as there are excellent narrative reviews. MA per se does not guarantee quality. 38
- 39. META-ANALYSIS AND SRL SRL relatively a new field • Four MAs • Hattie, Biggs, & Purdie, (1996). • Dignath, & Büttner, (2008). • Sitzmann, & Ely, (2011). • Dignath, Büttner, & Langfeldt, (2008). 39
- 40. DIGNAGTH’S MA FEATURES • Goal: reviewed 48 studies derived from 30 articles on the effectiveness of self-regulatory training with primary school students. • Detailed eligibility criteria. • SRL training programmes have a positive effect on learning, strategy use & motivation, even for primary school children • Highest benefits can be gained in mathematics, motivational outcomes, and cognitive & metacognitive strategies (in line with Hattie et al.,1996). • Primary students benefit even more than older (in line with Hattie et al.,1996). 40
- 41. DIGNAGTH’S MA FEATURES • Surprisingly, effect sizes were significantly higher for interventions that did not train students by means of group work but… • Several meta-analyses investigated this method and revealed positive effects: • In the studies included in this meta-analysis, we found only very little information about the implementation of group work in the learning setting. • Hence, a possible reason for the negative effect of group work on training effects at primary school level might be that students were not used to work in groups and did not receive enough instruction about collaboration. • Too many self-report data (2006) 41
- 42. DIGNAGTH’S MA FEATURES • Summarizing the most effective characteristics of interventions yields that a training programme should be based on social- cognitive theories, should train cognitive (especially elaboration and problem solving strategies), metacognitive (especially planning strategies), and motivational strategies (especially feedback), and provide knowledge about strategy use and about its benefit. • Future research: learning environment & how teachers can be trained. • PREP EXAMPLE (Preparing Teacher Education Students for 21st Practice Learning) 42
- 43. 43 BRAINSTORM in pairs (5 min): 1. Take your list of how to study (a certain topic of) SRL 2. Now think and discuss in pairs: what do you think about measuring SRL now? 3. Prepare yourself to present 3 key points to the class.
- 45. FORECASTING THE FUTURE 11/27/2015 45 • Too little has been achieved yet in measuring SRL as an event. • Challenges in this arena are significant! • Protocols are needed for collecting longitudinal measurements that span multiple brief episodes as well as expended periods (i.e. grade levels). • Methods are needed that characterize temporally unfolding patterns of engagement with tasks • in terms of the tactics and strategies that constitute SRL. • In terms of comparing patterns over time • New ideas from psychophysiological measurements • Triangulation needed!
- 46. CONCLUSIONS Different ways to measure • Choose the one that fits your research goals. • But don’t avoid because lack of knowledge. • Try to triangulate data • Don’tbe afraid to be innovative but remember: • Reliability and validity 27.11.2015 46
- 47. CONCLUSIONS 1. Measuring intervenes in a student’s environment. • We design measurement with an intention to cause the student to recall or to generate a particular kind of response. WHEN WE MEASURE, WE CHANGE THE ENVIRONMENT – ALLWAYS. 27.11.2015 47
- 48. REFERENCES • Boekaerts, M., & Corno, L. (2005). Self-regulation in the classroom: A perspective on assessment and intervention. Applied Psychology-an International Review-psychologie Appliquee-revue Internationale, 54(2), 199-231. • Butler, D. L. (2011). Investigating self-regulated learning using in-depth case studies. In B. J. Zimmerman & D. H. Schunk (Eds.), Handbook of self-regulation of learning and performance(pp. 346-360). New York: Routledge. • Efklides, A. (2011). Interactions of metacognition with motivation and affect in self-regulated learning: The MASRL model. Educational Psychologist, 46(1), 6 -25. doi: 10.1080/00461520.2011.538645 • Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87(3), 215-251. doi: 10.1037/0033-295X.87.3.215 • Samuelstuen, M. S., & Bråten, I. (2007). Examining the validity of self-reports on scales measuring students' strategic processing. British Journal of Educational Psychology, 77(2), 351-378. doi: 10.1348/000709906x106147 • Schmitz, B., Klug, J., & Schmidt, M. (2011). Assessing self-regulated learning using diary measures with university students. In B. J. Zimmerman & D. H. Schunk (Eds.), Handbook of self-regulation of learning and performance(pp. 251-266). New York: Routledge. • Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated engagement in learning. In D. Hacker, J. Dunlosky & A. Graesser (Eds.), Metacognition in educational theory and practice(pp. 277-304). Hillsdale, NJ: Erlbaum. • Zimmerman, B. J., & Moylan, A. R. (2009). Self-regulation: Where metacognition and motivation intersect. In D. J. Hacker, J. Dunlosky & A. C. Graesser (Eds.), Handbook of Metacognition in Education(pp. 299-315). New York: Routledge. • Zimmerman, B. J., & Schunk, D. H. (2011). Handbook of self-regulation of learning and performance. New York: Routledge. 27.11.2015 48
- 49. REFERENCES • Cooper, H. (2010). Research synthesis and meta-analysis. Thousand Oaks, California: SAGE. • Dignath, C., & Büttner, G. (2008). Components of fostering self-regulated learning among students. A meta-analysis on intervention studies at primary and secondary school level. Metacognition and Learning, 3, 231-264. doi: 10.1007/s11409-008-9029-x • Dignath, C., Büttner, G., & Langfeldt, H. (2008). How can primary school students learn self-regulated learning strategies most effectively? A meta-analysis on self-regulation training programmes. Educational Research Review, 3(2), 101-129. doi: 10.1016/j.edurev.2008.02.003 • Dochy, F. (2006). A guide for writing scholarly articles or reviews for the Educational Research Review. Educational Research Review. http://www.journals.elsevier.com/educational-research-review/ • Hattie, J., Biggs, J., & Purdie, N. (1996). Effects of learning skills interventions on student learning: A meta-analysis. Review of Educational Research, 66(2), 99-136. doi: 10.3102/00346543066002099 • Sitzmann, T., & Ely, K. (2011). A meta-analysis of self-regulated learning in work-related training and educational attainment: What we know and where we need to go. Psychological Bulletin, 137(3), 421-442. doi: 10.1037/a0022777 27.11.2015 49
- 50. THANK YOU! QUESTIONS AND COMMENTS? piia.naykki@oulu.fi 50