Presentation Review on the validity and reliability of self-reported work-related illness
Download as PPT, PDF1 like134 views
The document reviews the validity and reliability of self-reported work-related illness, highlighting common issues such as social desirability and recall problems. It analyzes various self-report measures used in occupational health studies, noting that while some questionnaires show high reliability and validity, self-reported work-relatedness has limited evidence. Recommendations include further research on self-assessment of work-relatedness and the development of specific measurement tools.
![Aim of review
Extend the knowledge on the reliability and
validity of self-reported work-related illness.
by
1] exploring the use of various self-report
measures of work-related illness in
occupational health studies, and
2] evaluating the evidence on the reliability
and validity of the self-report measures of
work-related illness.](https://image.slidesharecdn.com/presentatiereviewself-reporticohen-180328143457/85/Presentation-Review-on-the-validity-and-reliability-of-self-reported-work-related-illness-4-320.jpg)
Presentation Review on the validity and reliability of self-reported work-related illness
- 1. Coronel Instituut, Academisch Medisch Centrum Amsterdam Review on the validity and reliability of self-reported work-related illness Annet Lenderink and Ilona Zoer Supervised by Prof.Dr. Frank J H van Dijk Prof. Dr. Monique H.W. Frings-Dresen Dr. Henk F. van der Molen Dr. Dick Spreeuwers
- 2. Self-report in occupational health research • Very common • Relatively cheap • Sometimes the only way • Measure of perception of health • Vulnerable to distortion due to – Social desirability – Dissimulation – recall problems – Language/interpretation barriers – Etc.
- 3. Self-reported work-related illness Requires two steps: • Self-report of illness • Attribution of illness to work or work situation; causal attribution or causal inference
- 4. Aim of review Extend the knowledge on the reliability and validity of self-reported work-related illness. by 1] exploring the use of various self-report measures of work-related illness in occupational health studies, and 2] evaluating the evidence on the reliability and validity of the self-report measures of work-related illness.
- 6. First research question Which methods or measures are used for the assessment of work- related illness based on self- reported illness and self attribution to work-related factors?
- 7. Measures and methods for self-reporting work-related illness • 768 combinations of self-report measure and health status • 75 interview measures vs. 693 self-administered questionnaires • 20% of measures were on health as well as exposure
- 8. Self-report questionnaires 0 50 100 150 200 health status number untitled 131 135 37 57 49 5 5 52 titled 42 28 134 3 2 5 0 8 GH MSD PSY RESP SKIN EAR EYE MISC
- 10. Second research question What is the evidence on the reliability and validity of these methods or measures to assess self-reported illness and its attribution to work- related factors?
- 11. Two categories of studies 1. Questionnaire Testing: 38 studies studies that test the reliability and/or validity of a self-report questionnaire in a work-related situation 2. Clinical Examination: 33 studies studies that compare self-reported health with an expert opinion as “golden standard” (i.e. physician diagnosis based on clinical examination and/or clinical testing)
- 12. Questionnaire testing (1) Extracted from each study: • Questionnaire studied • Health status measured • Aimed at work or work situation • Population (type, number) • Type(s) of reliability/validity tested: reliability: test-retest, internal consistency, responsiveness, reproducibility/ repeatability, validity: content, criterion, construct, concurrent, predictive (sensitivity/specificity) • Outcome measures • Assessment of outcome measures against predefined criteria H = high, M = moderate, L = low, A = acceptable
- 13. QT Results (1) • 38 studies on 37 questionnaires • Work aimed: yes 14; no 19; unclear 4; • Most tested types of reliability: – test-retest reliability (23x, moderate to high) – internal consistency (17x, moderate to mainly high). • Most tested types of tested validity – concurrent validity (10x, moderate to high) – construct validity, including convergent and divergent validity and CFA (9x, low to high)
- 14. QT Results (2)
- 15. QT Results (3)
- 16. QT Results (4) • Most studied: Nordic Musculoskeletal Questionnaire (NMQ): – reliability/validity mainly high • Reliable and valid alternatives: – Curtin Back Screening Questionnaire (CBSQ) – Upper Extremities Questionnaire • QQ aimed at psychosocial complaints and disorders: – Reliability/validity variable, mainly moderate to high • Several QQ aimed at psychosocial complaints designed for use in work situations
- 17. Clinical examination (1) Extracted from each study: • Self-report measure studied • Health status measured • Aimed at work or work situation • Population (type, number) • Outcome measures: % agreement, sensitivity, specificity, PPV, NPV, ratio self- diagnosis/physician diagnosis • Assessment of study quality against predefined criteria VH = very high, H = high, M = moderate, L = low
- 19. CE Results (1) • Self-report QQ that focus on symptoms*: high sensitivity, low to moderate specificity • Self-report QQ that combine a number of relevant symptoms** to one health condition score: lower sensitivity, higher specificity • Self-report QQ that ask workers to self- diagnose a health problem in one single question***: lower sensitivity, higher specificity • Validity of self-report depends on its purpose, health status, population, exposure etc.
- 20. CE Results (2) • On an individual level: big differences between self-reports and clinical examination (physician’s diagnosis) • On population level: prevalence based on aggregated scores of self-reports might be close to prevalence measured by physician’s diagnosis • Substantial numbers of false positives and false negatives when using self-reported illness or self-reported work-relatedness measures • Self-reported workrelatedness hardly studied
- 21. Main findings (1) • Frequent use of self-report to assess self- reported work-related illness • Many self-report measures are not aimed at work or at the work situation • For epidemiological research: extensive choice in valid and reliable self-report measures on musculoskeletal disorders and mental health problems • Very few information on validity of self- assessed work-relatedness of a health condition.
- 22. Main findings (2) • No final judgement on the validity of self-reported work-related illness in general • Validity of measuring work-related illness by self- reports depends on the purpose, the context and the health condition measured • Low to moderate agreement on an individual level between medical examination and self-report • Better agreement on group or population level; especially when using self-diagnose questionnaires and questionnaires that use a specific combination score of health symptoms.
- 23. Recommendations • More research needed on workers’ self-assessment of work-relatedness of health problems • Development and testing of a specific measure of self-reported work- relatedness of ill health • Develop a professional practice guideline to assess the work- relatedness of a health complaint or disease
- 24. Reliable? Valid?
- 25. Questions?
Editor's Notes
- #2: Thank you for the opportunity to tell you about the review I did with my colleague Ilona Zoer on the validity and reliability of self-reported work-related illness. It was an assignment of the Health and Safety Executive in the UK and we were supervised by the people named here, all working at the Coronel Institute.
- #3: Self-report is often used in health care research. Major reasons are that it is relatively cheap and that sometimes the only way to collect specific information. Nevertheless, self-report also vulnerable because it can be strongly influenced by social desirability of responses, unreliability of memory, language and interpretation problems, etc. For the surveillance of work-related diseases self-report is important, for instance, because it is used in Labor Force Surveys that generate figures and statistics on work-related health. The first critical point is the moment a worker realizes his or her signs and symptoms may be an illness and could be cause by work or work situation.
- #4: Uiteindelijk vraagt zelfrapportage van werkgerelateerde aandoeningen dus om zelfrapportage van een aandoening, ziekte, symptomen en vervolgens een al dan niet toeschrijven van die aandoening aan werk.
- #5: HSE has asked us to examine what forms of self-reporting of work-related illnesses are used in research in occupational health care and what is known in the literature about the reliability and validity of these self-reporting. I will only briefly talk about the use of self-report measures and then come to the more interesting point of validity and reliability of self-report.
- #6: We built a search strategy and searched PubMed, Embase, Psychinfo and OSH Update for studies published since 1990 in English, German, French, Dutch and Spanish. This is a quick overview of the proceeds of our search strategy. To answer the first question we checked the abstracts and gathered the self-report measures and methods applied. To answer the second question we finally included 71 articles
- #8: We scanned all abstracts and found 768 different combinations of a self-report measure and health status. Questionnaires were much more often used than interview methods. Approximately 20% of the questionnaires was also used to collect information on exposure.
- #9: Here you can see how the self-report questionnaires were distributed according to health status. Most self-report was on general health, musculoskeletal disorders and psychological health problems. We distinguished between questionnaires of which the title or name was mentioned in the abstract and the questionnaires that were only referred to as “a questionnaire”. Eventually we found 99 different questionnaires with a name or title. Of these 99 questionnairs, 41 also inquired about work.
- #10: We looked for information on all these questionnaires and in the final report we described the questionnaires shortly focusing on aim, short description of use, developer and link to further information.
- #11: This is the research question that we formulated to investigate the reliability and validity of the self-reported work-related illness. We included 71 studies and extracted the data. The next step was finding a way to summarize the results in a logical way. After experimenting with the extracted data and discussion in the counseling group we chose to divide the studies in two groups.
- #12: We divided the studies into two categories: a group of 38 studies investigating the validity and / or reliability of questionnaires. These are more or less classic studies on the diagnostic quality of questionnaires to all their criteria. Usually you need a reference method, a gold standard to answer this question. But there is no such thing for self-report. So we decided to use a physician’s opinion as reference standard. A second group of 33 studies in which self-reports of health / illness was compared with the opinion of an expert as the gold standard: in most cases, a diagnosis made by a physician based on clinical and / or testing. At first I’ll tell you something about the first group, which we labelled Questionnaire testing.
- #13: In the data extraction of these 38 studies, the following aspects were extracted:
- #14: In 38 studies, 37 questionnaires were tested, of which 14 were aimed also at work. In particular, test-retest reliability and internal consistency were often tested, as concurrent validity and construct validity.In the report all findings are summarized for each study. Here, I’ll try to summarize the results.
- #15: Here the results of the questionnaires on musculoskeletal disorders and mental health assessment: green means high reliability or validity, yellow means moderate, red means low and blue means acceptable, usually based on the authors conclusions on content, face or cross cultural validity not assessed with figures.
- #16: Results for general health, respiratory health and a small category of miscellaneous studies
- #17: summarizing
- #18: The second group containsf 33 research studies that examine self-report compared with the opinion of an expert.From these studies, the following data collected: self-reporting method, the health / disease in which the self-report focused, work directed or not, the participants, results in size and number and the assessment of study quality compared with pre-formulated criteria.
- #19: The assessment of quality produced the following results, where it is clear that the investigation of musculoskeletal and skin meet the highest quality scores.
- #20: Summarizing *(pain, ache and discomfort, itch, papules or fissures, cough, phlegm, dyspnoea) ** (for instance based on factor analysis) *** (e.g. “Did you suffer from hand eczema in the last 12 months?”).
- #22: Belangrijkste bevindingen: Vragenlijsten en interviews worden vaak toegepast voor zelfrapportage van werkgerelateerde aandoeningen Veel van deze instrumenten zijn echter niet gericht op werk of werksituatie Voor epidemiologisch onderzoek naar klachten van het HBA en psychische klachten is er een uitgebreide keus uit valide en betrouwbare zelfrapportage methoden Betrouwbaarheid en validiteit van zelfrapportage wordt gewoonlijk getest tov van het oordeel van een expert Zeer weinig studies gaan expliciet in op zelfgerapporteerde werkgerelateerdheid
- #24: with the aim to develop a questionnaire that can be combined with valid and reliable self-report instruments to assess health status