PROCEDE 2014- Asking the Right Questions
- 1. Asking the right questions: Collecting and validating data – refining, rephrasing, and refocusing Dr. Corinne Haigh, The School of Education, Bishop’s University, chaigh@ubishops.ca PROCEDE 2014: People, Passion, Proof – The Data Difference PROCEDE 2014 1
- 2. Session Outline What makes a good research question? Where do I begin? What is the danger of a single story? How can I be sure that the measurements I take match the question I am trying to answer? How do we define what we wish to measure? What are reliability and validity? What are some ethical issues to consider? PROCEDE 2014 2
- 3. Pollev.com/chaighbu From any browser (code or keyword) <your response> From a text message Participating with Poll Everywhere: How to vote via the web or text messaging 22333
- 4. Participating with Poll Everywhere How to vote via the web
- 5. Participating with Poll Everywhere How to vote via texting
- 10. Reflection What types of questions was I able to ask using the polling software? What type of information was I able to gather? How could I analyze this information to answer a specific question? How might I present it to others? Take a moment to reflect on these questions individually. PROCEDE 2014 10
- 13. Reflection What types of questions was I able to ask using the polling software? What type of information was I able to gather? How could I analyze this information to answer a specific question? How might I present it to others? Take a moment to reflect on these questions individually. PROCEDE 2014 13
- 14. Now, think of a recent “problem” you have encountered in your professional life… Briefly describe it. How did/might you approach it? What did/do you hope to learn as a result? Take a few minutes to reflect on these questions individually, and then compare notes with a colleague PROCEDE 2014 14
- 15. The danger of a single story Asking the right questions involves considering multiple perspectives We will now watch one video that illustrates the danger of being exposed to only a single story And I’ve included a link to another for you to watch at another time… PROCEDE 2014 15
- 16. Chimamanda Adichie Novelist Chimamanda Adichie tells the story of how she found her authentic cultural voice -- and warns that if we hear only a single story about another person or country, we risk a critical misunderstanding. The Danger of a Single Story PROCEDE 2014 16
- 17. Ben Goldacre Doctor and epidemiologist Ben Goldacre shows us, at high speed, the ways evidence can be distorted, from the blindingly obvious nutrition claims to the very subtle tricks of the pharmaceutical industry. Battling Bad Science PROCEDE 2014 17
- 18. Checkpoint Take 2-3 minutes and write about an educational example of the danger of a single story Try to write for the full time You will then discuss your writing with a partner (or in a trio) and then with the whole group PROCEDE 2014 18
- 19. Asking a question often starts from necessity How do I know when…? Think up some endings to my sentence starter. PROCEDE 2014 19
- 20. The Assessment for Learning (AL) Cycle 1. What is known 2. What might be happening and why 3. What you need to find out 4. What you think is happening 5. What you say 6. What you need to do 7. Did what you chose to do work? Gathering referral information Generating hypotheses Checking your hypotheses by collecting information Arriving at tentative judgement Reporting findings Planning instruction Evaluating success Further information needed? 20 PROCEDE 2014
- 21. Refining Questions Initial questions may be vague, but vague questions can never be answered Refine questions by: Consulting past research in the area Speaking with colleagues Applying theory Replicating someone else’s study Clarifying contradictory findings PROCEDE 2014 21
- 22. What is a variable? A variable is something that can be changed, such as a characteristic or value We are generally looking at whether change in one variable results in change in another variable Can you generate some examples of variables from your practice? What kind of relationships between these variables might be interesting to investigate? PROCEDE 2014 22
- 23. Learning environment Academic Physical Emotional Social Initial Concern Gaps in learning; opportunity to learn; history of successes/failures; teachers; past and present programs Vision; hearing; health history Reading profiles; academic profile; experiences; cognitive development; aptitude (IQ); Language profile Family; self concept; motivation; security Culture; friends; maturity 23 PROCEDE 2014
- 24. Operationally defining a variable Specifies exactly how to measure and/or manipulate the variables in a study Go back to one of the examples you generated for the previous question – how would you define it? Remember, it must be measureable (think about setting SMART goals for IEPs) PROCEDE 2014 24
- 25. Types of variables Independent vs. dependent Independent: precedes, influences, or predicts the dependent variable (e.g., teaching method) Dependent: affected or predicted by the independent variable (e.g., student engagement) Extraneous and confounding Extraneous: source of error affecting the results (e.g., time of day) Confounding: varies systematically with the independent variable (e.g., the teacher) PROCEDE 2014 25
- 26. 26 How can we make sense of a complex process? We make inferences… 1. 2. 3. PROCEDE 2014
- 27. 27 BALLOON - Recognize the picture as balloons - Select the appropriate word - Produce the appropriate articulation What mental activity needs to occur to identify the picture and say what it is? How is making an inference about learning different than blowing up a balloon? - we cant see the “mental activity” -Therefore, we must make inferences about the mental activity from observable behaviours. PROCEDE 2014
- 28. Qualitative research A means for exploring and understanding the meaning that individuals or groups ascribe to a social or human problem Emphasizes natural settings, understanding, verbal narratives, and flexible designs 28 PROCEDE 2014
- 29. Qualitative techniques and data Data: Verbal descriptions, field notes, observations, documents, photographs, people’s own words, narrative Techniques: observation, participant observation, open-ended interviewing, review of documents and artifacts PROCEDE 2014 29
- 30. Quantitative research A means for testing objective theories by examining the relationship among variables Emphasizes numbers, measurements, deductive logic, control, and experiments 30 PROCEDE 2014
- 31. Quantitative techniques and data Data: quantiles, counts, numbers, statistics, measures/instruments Techniques: experiments, quasi-experiments, structured observations, structured interviews, surveys PROCEDE 2014 31
- 32. How do I know which to use? It is critical to match the reason for the research with the appropriate method Determined by the purpose and the research question Using each method will be a challenge Either can be used well, or poorly Depth of understanding is important, regardless of the design Leads to greater impact PROCEDE 2014 32
- 33. Checkpoint Think about some of the potential research questions you generated earlier – what sort of approach would be best suited to answering them? Consider the data you’d want to collect (or existing data you’d like to access) and the techniques you could use to do so PROCEDE 2014 33
- 34. Reliability Test-retest reliability: Agreement between scores on the same test, given to the same people, at different times Inter-rater reliability Agreement among different raters or observers Inter-item reliability Are all of the items on the test equally good at assessing the same underlying ability? 34 PROCEDE 2014
- 35. Validity Content validity The range and relevance of the items Can be very difficult to measure for some more complex aspects of learning Construct validity Concerns the underlying theoretical construct: is the test measuring the target skill or behaviour Test this by comparing scores in two different measures – should be highly related if they’re measuring the same construct 35 PROCEDE 2014
- 36. Reliability and validity 36 PROCEDE 2014
- 37. Some broad ethical questions to consider: Does answering this question benefit the individuals being studied? Will the answer be meaningful for others? Are participants being marginalized or disempowered? Are participants being deceived about the intent of the study? Are participants at more than minimal risk? Minimal risk: “Research in which the probability and magnitude of possible harms implied by participation in the research is no greater than those encountered by participants in those aspects of their everyday life that relate to the research.” (TCPS 2, Chapter 2) PROCEDE 2014 37
- 38. Some broad ethical questions to consider: Did all participants benefit from the “treatment”? Is there a possibility that participants will disclose harmful and/or intimate information? What will be done with this information? How will you protect the anonymity of participants? PROCEDE 2014 38
- 39. Checkpoint Describe an ethical question or concern you’ve had when trying to collect data to answer a specific question (or speculate about a potential ethical issue that may arise in future). How did/could you address it? First reflect on this individually and then have a discussion at your table. PROCEDE 2014 39
- 40. Some final points for reflection… Sometimes coming up with a good research problem or question is the hardest part The approach you take to answering your question will depend on: The problem you’re trying to address Your worldview or assumptions about research The specific inquiry strategies you wish to use Your personal experience There are multiple ways to approach data collection and analysis and no one is better than the other as long as you aware of their strengths and limitations PROCEDE 2014 40
- 41. Thank you! chaigh@ubishops.ca PROCEDE 2014 41