Itr validation 1
- 1. Introduction to the use of validated questionnaires Dr Tan Chai Eng Introduction to Research Dr. Fam. Med. 2014
- 2. Objectives • To explain about basic concepts of reliability and validity • To explain about the need for cross-cultural adaptation of questionnaires
- 3. Questionnaires • One of the commonest study tools used by Family Medicine postgraduates for their research project. • To collect information / responses from study subjects for analysis
- 4. Validity and Reliability Validity Reliability • How well the tool measures what it is supposed to measure • Will the tool give consistent measurements?
- 5. Reliability • Repeatability • Reproducibility • Consistency Weighing Scale A Reading 1 Reading 2 Reading 3 Reading 4 60kg 61kg 59.5kg 60.5kg Adapted from slides by Dr Jamaiyah Haniff, CRC Weighing Scale B Reading 1 Reading 2 Reading 3 Reading 4 60kg 57kg 56kg 55kg Which one is reliable?
- 6. Validity • Correctness • Accuracy Adapted from slides by Dr Jamaiyah Haniff, CRC
- 7. Is it enough to have a reliable tool? Weighing Scale A with standard weight 50kg Conclusion ? Reading 1 Reading 2 Reading 3 Reading 4 Reliable but not valid 60kg 60kg 60kg 60kg Weighing Scale B with standard weight 50kg Conclusion ? Reading 1 Reading 2 Reading 3 Reading 4 Not valid and not reliable 59kg 50kg 52kg 49kg Weighing Scale C with standard weight 50kg Conclusion ? Reading 1 Reading 2 Reading 3 Reading 4 Valid and 50kg 50.5kg 50 kg 50kg reliable
- 8. How does this apply to questionnaires? • Questionnaires measure psychological constructs that cannot be seen or felt… QoL Same scores in same person after a certain period of time? Test-retest reliability Same scores in same person rated by different observers? Inter-rater reliability Same person, same test and same time Internal consistency (Cronbach alpha) Scores in same person at the same time but split into 2 tests Split-half reliability
- 9. How to interpret? Kappa Reliability 0- <0.2 None to slight 0.2- <0.4 Fair 0.4 - <0.6 Moderate 0.6 - <0.8 Good 0.8 – 1.0 Excellent Cronbach alpha Reliability 0 - <0.4 Poor 0.4- < 0.7 Moderate 0.7 - <0.90 Good >0.90 Good reliability but may have redundant items in the questionnaire Poor alpha can be due to: • low number of items • poor inter-relatedness of items (heterogeneous constructs) (Tavakol, 2011) Cohen’s kappa – 2 observers, Fleiss – 3 or more observers
- 10. A test cannot be valid if it is not reliable!
- 11. Back to validity… • Types of validity – Face validity – Content validity – Construct validity • Convergent • Discriminant – Criterion validity • Concurrent • Predictive
- 12. How do I know whether the questionnaire is valid? • Face validity – it "looks like" it is going to measure what it is supposed to measure.
- 13. Example: PHQ-9 What does the Patient Health Questionnaire-9 measure? Kroenke K, Spitzer RL, Williams JB (2001). The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 16(9):606
- 14. Example: PHQ-9 PHQ-9 does not measure general health. It is actually used for diagnosing depression and for monitoring symptom severity.
- 15. How do I know whether the questionnaire is valid? • Content validity – Content experts have seen the questionnaire items and agree that it measures what it is supposed to measure
- 17. How do I know whether the questionnaire is valid? • Criterion validity – how this questionnaire predicts the results of another tool – Concurrent validity – the questionnaire agrees with similar tool. • E.g. Epworth Sleepiness Scale with Multiple Sleep Latency Test (Chervin et al., 1997) – Predictive validity – questionnaire is able to predict an outcome • E.g. PHQ-9 predicts diagnosis of depression (Kroenke, 2001)
- 18. How do I know whether the questionnaire is valid? • Construct validity Does the questionnaire measure the latent constructs it is supposed to measure? Verbal Mathematical Visuospatial Inductive reasoning Mechanical reasoning
- 19. Do school examinations have the construct validity to measure intelligence? Verbal Mathematical Visuospatial Inductive reasoning Mechanical reasoning Oral test Mathematics, Add Math Comprehension, Essays How well does it measure each construct?
- 20. Construct Validity Convergent & Discriminant Validity
- 21. Convergent Validity • Are the items that are supposed to belong together really belong to each other? Next example from McCrae, 1987: Validation of the five factor model of personality across instruments and observers
- 23. Extraversion subscale Are all the items in this subscale theoretically relate to extraversion? What about those that load into two different factors? What about those that don’t belong here at all?
- 24. Discriminant validity • Are the items that are NOT supposed to belong together really DO NOT belong to each other?
- 25. Discriminant validity Does this statement belong to this factor/domain? Or should it be in another factor?
- 26. DASS21 Name: Date: Please read each statement and circle a number 0, 1, 2 or 3 which indicates how much the statement applied to you over the past week. There are no right or wrong answers. Do not spend too much time on any statement. The rating scale is as follows: 0 Did not apply to me at all 1 Applied to me to some degree, or some of the time 2 Applied to me to a considerable degree, or a good part of time 3 Applied to me very much, or most of the time 1 I found it hard to wind down 0 1 2 3 2 I was aware of dryness of my mouth 0 1 2 3 3 I couldn't seem to experience any positive feeling at all 0 1 2 3 4 I experienced breathing difficulty (eg, excessively rapid breathing, breathlessness in the absence of physical exertion) 0 1 2 3 5 I found it difficult to work up the initiative to do things 0 1 2 3 6 I tended to over-react to situations 0 1 2 3 7 I experienced trembling (eg, in the hands) 0 1 2 3 8 I felt that I was using a lot of nervous energy 0 1 2 3 9 I was worried about situations in which I might panic and make a fool of myself 0 1 2 3 10 I felt that I had nothing to look forward to 0 1 2 3 11 I found myself getting agitated 0 1 2 3 12 I found it difficult to relax 0 1 2 3 13 I felt down-hearted and blue 0 1 2 3 14 I was intolerant of anything that kept me from getting on with what I was doing 0 1 2 3 15 I felt I was close to panic 0 1 2 3 16 I was unable to become enthusiastic about anything 0 1 2 3 17 I felt I wasn't worth much as a person 0 1 2 3 18 I felt that I was rather touchy 0 1 2 3 19 I was aware of the action of my heart in the absence of physical exertion (eg, sense of heart rate increase, heart missing a beat) 0 1 2 3 20 I felt scared without any good reason 0 1 2 3 21 I felt that life was meaningless 0 1 2 3 Depression Anxiety Stress
- 27. Can I report the results item by item? DASS21 Name: Date: Please read each statement and circle a number 0, 1, 2 or 3 which indicates how much the statement applied to you over the past week. There are no right or wrong answers. Do not spend too much time on any statement. The rating scale is as follows: 0 Did not apply to me at all 1 Applied to me to some degree, or some of the time 2 Applied to me to a considerable degree, or a good part of time 3 Applied to me very much, or most of the time 1 I found it hard to wind down 0 1 2 3 2 I was aware of dryness of my mouth 0 1 2 3 3 I couldn't seem to experience any positive feeling at all 0 1 2 3 4 I experienced breathing difficulty (eg, excessively rapid breathing, breathlessness in the absence of physical exertion) 0 1 2 3 5 I found it difficult to work up the initiative to do things 0 1 2 3 6 I tended to over-react to situations 0 1 2 3 7 I experienced trembling (eg, in the hands) 0 1 2 3 8 I felt that I was using a lot of nervous energy 0 1 2 3 9 I was worried about situations in which I might panic and make a fool of myself 0 1 2 3 10 I felt that I had nothing to look forward to 0 1 2 3 11 I found myself getting agitated 0 1 2 3 12 I found it difficult to relax 0 1 2 3 13 I felt down-hearted and blue 0 1 2 3 14 I was intolerant of anything that kept me from getting on with what I was doing 0 1 2 3 15 I felt I was close to panic 0 1 2 3 16 I was unable to become enthusiastic about anything 0 1 2 3 17 I felt I wasn't worth much as a person 0 1 2 3 18 I felt that I was rather touchy 0 1 2 3 19 I was aware of the action of my heart in the absence of physical exertion (eg, sense of heart rate increase, heart missing a beat) 0 1 2 3 20 I felt scared without any good reason 0 1 2 3 21 I felt that life was meaningless 0 1 2 3
- 28. The Chipsmore version of construct validity
- 29. Can I use English questionnaires in my study? • Can your respondents understand English? • Are the questionnaire items worded in a way that is suitable to the local culture/language? – E.g. toilet vs loo vs restroom vs washroom • Pre-test among local population and check their understanding of the wording in the questionnaire.
- 30. Can I translate the questionnaires to Malay myself? • Need translate in a way that you retain the semantic and content equivalence • Back-to-back translation • Pilot-test, check reliability of the scales within the questionnaire • Best practice: validate the questionnaire locally
- 31. Can I verbally translate the questionnaires to Malay myself? • Difficulty in achieving standardized questions to respondents – may get different responses depending on what you say at that point of time • Affects the validity of your study results • Not suitable for self-administered questionnaires, especially where sensitive information is requested
- 32. Permissions • Development of a good questionnaire for research takes a lot of work from a team of researchers. • Ethically, need permission from the corresponding author who developed the questionnaire. • Public domain questionnaires do not require permission. Eg DASS21
- 33. Permissions • Validated Malay-language versions, need permission from the researcher who validated it locally. Eg Malay version DASS21 – Prof Ramli Musa • Copyright holder may be a company, e.g. QualityMetrics – may need to pay for licence to use the questionnaire. Include into your proposal budget for funding.
Editor's Notes
- #6: Consistency – different method, different condition, repeatability – same method, same condition; reproducibility – same method, different condition
- #7: A measuring tape cannot be used to measure weight. It can be used to measure length. Similarly, questionnaires should be proven to measure what it was developed to measure.
- #13: For example, PHQ-9 does not measure health but measures depressive symptoms
- #29: Chipsmore packet is your questionnaire. The cookies are the constructs. The chocolate chips are your items. You cannot analyse results based on chocolate chips only.