Mixed Methods Research: A Critical Reading
- 1. Mixed Methods Research An overview and critical reading Nicole Crozier EDCI 515 Summer 2019
- 2. Get ready! 1. Researcher background 2. Intro to mixed methods 3. My paper of choice 4. My paper mixed methods 5. Questions & insights 2
- 3. 1. Researcher Background Who is Alicia O’Cathain?
- 4. The Researcher: Alicia O’Cathain ● Professor of Health Services Research and Director of Medical Care Research Unit, University of Sheffield ● Well-educated, older white woman ● Researches evaluation of new health services, patient views of health care, urgent care ● Writes, teaches and uses mixed methods 4
- 5. 2. Intro to Mixed Methods The basics of a research methodology
- 6. Quantitative Closed-ended info that results in a numerical representation . Qualitative Subjective and open-ended info that allows the participant voice to be heard 6 Mixed Methods +
- 7. Provides a more thorough understanding of the results 7
- 8. Examples Data Interview Data is collected (i.e. using a survey); participants are interviewed to learn more about some/all responses. Interview Survey Interview participants to learn perspectives on or feelings about a topic; use that info to develop a survey. 8 Interview Survey Conduct interviews to determine whether participants are appropriate for a quantitative study; often used in health research/RCTs. Focus Group Survey Host one or more focus groups about a topic; validate responses with a larger group via a survey.
- 9. ● Fills in the gaps when it comes to weaknesses ○ Quantitative: Limited in understanding of context ○ Qualitative: Can include bias and difficult to generalize ● Provides a more comprehensive answer to the research question ● Results can be validated within the study Advantages ● May require more resources (time, people, money) ● Research procedure is more complicated (to create and carry out) 9 Challengesvs.
- 10. Quantitative + Qualitative 10 With the goal of providing a more thorough understanding of the results
- 11. 3. My Chosen Paper What is the rest of this presentation based on?
- 12. Accessing online learning material: . Quantitative behavior patterns and their effects on motivation and learning performance 12 Authors: Liang-Yi Li and Chin-Chung Tsai Journal: Computers & Education Date: July 2017
- 13. Quick Background ● Learning management systems (LMS) support teachers in conducting a variety of activities. ● Over the years, researchers have analyzed LMS system logs answer a number of questions and improve online learning. ● Learning on an LMS is a self-regulated process. 13
- 14. Research Questions 14 What online learning materials did students spend more time on? • Lecture slides, video lectures, shared assignments or posted messages 1 What kinds of behavioural patterns exist when students viewed online learning materials? 2 How did the different behavioural patterns relate to students’ learning performance? 3 How did the different behaviour patterns relate to students’ learning motivation? 4
- 15. Method Participants 59 third-year computer science undergraduate students enrolled in a blended course called Mobile Phone Programming. LMS usage All clicking events were recorded in the LMS system log; what they are clicking on, when, from where, for how long. 15 Learning performance Evaluated via multiple homework assignments each week (completing a program) and a final examination (paper based and programming based) Motivation Measured by the Motivated Strategies for Learning Questionnaire, which students filled out via LMS.
- 16. Results 16 What online learning materials did students spend more time on? • Lecture slides, video lectures, shared assignments or posted messages 1 What kinds of behavioural patterns exist when students viewed online learning materials? 2 “Students accessed the lecture slides more often and, on average, spent more time on each access than they did for other learning materials.” Researchers classified the students with similar viewing behaviours into three groups: “consistent use” students, “slide intensive use students” and “less use students”.
- 17. Results 17 How did the different behavioural patterns relate to students’ learning performance? 3 How did the different behaviour patterns relate to students’ learning motivation? 4 a) “Consistent use” and “slide intensive use” students had higher homework and examination scores than “less use” students. b) “Consistent use” students had higher homework scores than “slide intensive use” students, but the same exam scores. The “consistent use” students were higher in intrinsic goal orientation, self-efficacy and task value.
- 18. 4. My Paper as Mixed Methods What would happen if they did mixed methods instead of quantitative?
- 19. Quantitative Mixed Methods The Research 19 The Reader The Researcher The Researched
- 20. The Research 20 Learn the WHY behind online learning material choices1 Learn the why behind behavioural patterns2 a) Why did they make those choices re: online learning materials? b) Why did students not highlight or comment on the online materials? Why did “less use” students use the content less? Are there outside influences? Are there ways we could influence/entice use?
- 21. The Research 21 Validate the motivation data3 Explore how students are actually using the material4 Does motivation actually influence behavioural patterns, or are there also other conflicting factors? “It cannot be determined if a student is actively and effectively learning from a material once they have accessed it.”
- 22. The Researcher Currently Liang-Yi Li: Professor, Department of Computer Science and Engineering, National Central University, Taiwan Chin-Chuang Tsai: Professor and Chair, Program of Learning Sciences, National Taiwan Normal University, Taiwan Future More time and resources Would likely require the project to take on additional collaborators: ○ To provide expertise on the method ○ To provide help in distributing the labour 22
- 23. The Researched Who is being researched would not change. Still studying 59 third-year computer science undergraduate students enrolled in Mobile Phone Programming. . The impact on the research participants would change. A subset of students would have to spend time actively engaging in the research project. 23
- 24. The Reader ● Have a better understanding of the value of the research and be able to apply it. ● What materials should I use/create? ● How should my course be set-up? ● What digital literacy skills might my students be missing? ● How can I motivate my students? 24
- 25. 5. Questions & Insights Thoughts that arose during this examination of mixed methods research
- 26. Questions & Insights ● Mixed methods can really help you to understand why something is happening. ● Eliminates the need for (some) future research ● How do you know when to stop going deeper? ● Can you switch to mixed methods part way through a study? 26
- 27. References 27 Li, L.-Y., & Tsai, C.-C. (2017). Accessing online learning material: Quantitative behavior patterns and their effects on motivation and learning performance. Computers & Education, 114, 286– 297. O’Cathain, A. (2010). Assessing the Quality of Mixed Methods Research: Toward a Comprehensive Framework. In A. Tashakkori & C. Teddlie, SAGE Handbook of Mixed Methods in Social & Behavioral Research (pp. 531–556). Overview of Mixed Methods. (n.d.). Retrieved July 10, 2019, from Centre for Innovation in Research and Teaching website:
Editor's Notes
- #5: Professor of Health Services Research and Director of Medical Care Research Unit, University of Sheffield Medical Care Research Unit focuses on the development and evaluation of complex interventions for chronic conditions (CF, MS, etc.) Researcher, teacher and administrator Based in England, so likely has a Western perspective Well-educated, older white woman (as far as I can tell) BSc, MSc, MA, PhD Published her first paper in 1988 This could inform her perspective (is she clinging to a past? Moving forward with the times) Researches evaluation of new health services, patient views of health care, urgent care Does LOTS of research (so has lots of experience!). 11 current research projects listed on website, 13 publications listed in the last 5 years. Much of her research uses mixed methods, and/or randomized control trials. Writes, teaches and uses mixed methods Helps explain why she is considered to be an expert on mixed methods for this textbook chapter. 4 different textbooks chapters, numerous papers focused on the use of applicability of mixed methods for certain situations Mixed methods in in biomedical and health research The role of group dynamics in mixed methods health research teams Three techniques for integrating data in mixed method studies Runs mixed methods workshops and clinics for PhD candidates Based in England (western perspective) Teacher, researcher, administrator Unit focuses on the development and evaluation of complex interventions for chronic conditions. 11 current research projects, 13 publications in the last 5 years.
- #8: Puts the emphasis on the research question, and how to really get the answer Uses all approaches available to come to a better understanding Often allows us to better understand both the what and the why It’s not really a simple math equation- both the quantitative and qualitative parts can influence and affect one another.
- #9: Different types of mixed methods design, which differ on timing, order of pieces, and how they influence each other Example in mixed methods paper could happen concurrently, because they were pretty separate, but allowed for a much better answer to the question. (and findings that weren’t just misleading!!) Data Interview: could be a survey, could be data pulled from an LMS, could be data pulled from social media Interview survey: “why didn’t you participate” question on orientation survey Interview survey (health): many randomized control trials, to ensure you meet criteria Focus group survey: market research; could have done this with the women’s scholar online harassment study
- #14: Variety of activities Online or blended course (or just extra) Reading materials, asynchronous discussion (and synchronous discussion), quizzes, self/peer assessment understand engagement levels discover behaviour patterns explore the relationship between behaviour and learning performance Self-regulated process. It’s up to you to decide what course material to access, when, for how long; you control your learning
- #17: Consistent use = accessed more often for shorter periods, all types od materials Slide intensive use = accessed less often but for longer; spent same time overall, but mostly on the slides. Less use = just didn’t use as often
- #20: Obviously, need to add a qualitative component in order to achieve the mixed methods methodology.
- #21: According to the conclusion, “understanding how students view different learning materials and how their viewing behaviours affect their learning has been an important issue. “ But this study doesn’t actually determine student VIEWS in any way… just use. They may not be related . We’re assuming views based on behaviour A) why did they make these choices? Did they spend more time on content they found more useful? Did some content just TAKE less time (i.e. posted messages. Shared assignments if you’ve solved the assignment)? Was the system set-up to encourage use of one over the other? LMS provided a notification page where students could view how many posts they had not viewed- students then didn’t have to access?? Method of access B) Why did they not highlight? YOU COULD LITERALLY JUST ASK THEM. Not used to doing it Didn’t like digital annotation tools Copied and executed codes in the IDE; made comments there. Also: didn’t know the tool existed?? 2. The behavioural patterns aren’t all based on choice… they are also based on total time spent. Why did “less use” students not engage with the content? Is it all based on motivation? Did the know the system existed? Did outside factors (work, mental health) prevent them from engaging as much. Are their outside influences on behavioural patterns (what’s happening outside the LMS? Backchannels? How are students perceiving their learning? Do they just think they already know it all?) Are there any strategies that could have made a difference?
- #23: Liang Yi: probably only familiar with quantitative research, coming from computer science (and based on quick skim of other published articles) Chin Chuang: educated in both science and education; likely not overly familiar with qualitative either (views, self-efficacy) Require expertise, but also more resources Interviews and analysing them takes a lot of time!!
- #24: Who wouldn’t change. They’re not recruiting participants, they have a set group. Partly wouldn’t change because the qualitative part is being added AFTER the quantitative.