Research 101: Scale Validity & Reliability
0 likes193 views
This document discusses the importance of validity and reliability in scale development and measurement. It defines reliability as the consistency of measurement and identifies several ways to calculate reliability, including test-retest, split-half, composite, and internal consistency methods. Validity is defined as the degree to which a scale measures the intended construct. Theoretical evaluations of validity include expert reviews of apparent and content validity. Empirical evaluations include assessing convergent, discriminant, predictive, and concurrent validity through statistical analyses and exploring factor structures. Both theoretical and empirical evaluations are needed for a fully validated scale.
Research 101: Scale Validity & Reliability
- 1. Research 101: Scale Validity & Reliability Harold Gamero
- 2. Validity and reliability of a scale • Sometimes, the constructs to be measured represent imaginary concepts and are multidimensional. • When choosing a scale we must ensure that it is valid and reliable:
- 3. Reliability Definition Degree of consistency in the results of an instrument. • Consistency does not mean accuracy (validity). • Some of the main factors affecting reliability are: a) The subjectivity of the researcher (especially with qualitative methods). b) Ask ambiguous or vague questions. c) Questions on topics that the respondent is not familiar with. d) Questions about private or sensitive topics.
- 4. Reliability How can the reliability of a scale be improved? a) Use methods that are less sensitive to the subjectivity of the researcher. b) Ask questions whose answers are known to the respondent. c) Avoid ambiguous or very sensitive indicators. d) Simplify the way indicators are written.
- 5. Reliability Ways to calculate the reliability of a scale: Total amount of true score variance relative to the total scale score variance. Composite reliability Consistency between two measurements of the same construct, in the same sample but at two different points in time. Test-retest Consistency between two halves of the scale. The more extensive the instrument, the more likely it is to be highly consistent. Split-half Consistency between different items of the same scale (Cronbach's alpha coefficient). Internal consistency
- 6. Validity Definition Degree to which an instrument measures the construct it is construct it is supposed to measure. • The validity of an instrument can (should) be obtained through two approaches: vs. Empirical evaluation (Validity of the criterion) Verify that the test results are similar to other external criteria. Four subtypes: convergent, discriminant, concurrent and predictive. Theoretical evaluation (Translational or representational validity) Verify that the construct image has been adequately represented in the empirical test. Two subtypes: apparent and content
- 7. Validity Theories / ideas Conceptualization Valid instruments Statistical analysis Remarks Theoretical level Empirical level
- 8. Validity Theoretical evaluations Apparent validity • Degree to which the instrument appears to measure the underlying construct. • Evaluated by a panel of experts. Content validity • Degree to which the instrument includes items relevant to the domain of the construct. • Evaluated by a panel of experts.
- 9. Validity Empirical evaluations Convergent validity Relationship of the test items to the construct being measured. Discriminant validity Separation of test items from other constructs that are not measured. Predictive validity Degree to which the measurement satisfactorily predicts results that should theoretically occur Concurrent validity Relationship of the measurement to other criteria that should occur simultaneously.
- 11. Measurement theory Classical test theory or true score theory: • In the data set observed in the sample: • The objective of the psychometric analysis is to estimate and minimize var(E) X = T + E Observed value Actual value Error var(X) = var(T) + var(E)
- 12. Measurement theory • Measurement errors can be of two types: vs. Systematic error • They systematically affect the entire sample. • They create positive or negative biases as they are consistent across observations. • Systematic errors must be identified and corrected. Random error • It is attributed to unknown and uncontrollable external factors. • They affect the measurement randomly. • They are considered "noise" in the measurement and are generally ignored. X = T + Er + Es Observed value Actual value Random error Systematic error
- 13. Measurement theory Standard error Adds variability to the distribution Reduces reliability Systematic error Moves the central tendency of the distribution Reduces validity (more severe) Distribution with error Distribution without error Distribution without error Distribution with error
- 14. An integrated vision for validation Conceptualization of constructs 2. Creation / selection of indicators 3. Refinement or discard analysis 4. Assessment of face and content validity 5. Conduct pilot test 6. Assessment of convergent and discriminant validity 7. Reliability assessment by dimensions 8. Evaluate predictive validity Instrument validated Theoretical evaluation Empirical evaluation
- 15. An integrated vision for validation Because inferences made from measurements made with compromised scales are worthless, scale validation is one of the most important steps in empirical research. !