Evaluation methods in heathcare systems

3
Introduction:
 From 1995s in the hope of :
Medical information systems involve:
Computer-
stored databases
• patient
information to
support medical
order entry
• Results
reporting
• Decision support
systems
• clinical
reminders
Comprehensive
systems
• coordinates
patient care
activities by
linking
computer
terminals in
patient care
areas to all
departments
Smaller separate
systems
•Link patient care
areas to only one
department
•Laboratory system
•Radiology system
•Pharmacy system
•Expert systems
•Computerized
databases
CPOE
• Concerns
about patient
safety and
medical error
• Need to Evaluation
 Unfortunately reports of system failures have continued
1.Anderson, J.G. and C. Aydin, Evaluating the Organizational Impact of Healthcare Information
Systems. 2005: Springer.pp 5-8.
Hope
• Increasing efficiency, reducing costs, and improving patient care
1995 • Healthcare applications appeared

4
Evaluation Definition:
 Evaluation can be defined as “the act of measuring or exploring properties
of a health information system (in planning, development , implementation,
or operation),the result of which informs a decision to be made concerning
that system in a specific context.”
 Evaluating should perform not only in technology assessment but also in
the social and behavioral processes
1.Ammenwerth, E., et al., Evaluation of health information systems—problems and challenges.
International Journal of Medical Informatics, 2003. 71(2-3): p. 125-135.
2. Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
During IT Evaluation
Not only the technology itself
But also the interaction between IT
and human players in their information
processing
Must be
Considered

5
Evaluation is the means to assess:
5
Quality
Value
Impacts of IT in the health
care environment
Effects

6
Aim of evaluation
:
To provide the basis for a decision about the IT system investigated
Decision-making context is also the context of the evaluation .[2]
1- Yusof, M. M., et al. (2008). "Investigating evaluation frameworks for health information systems." Int J Med Inform
77(6): 377-385.
2-Brender, J., Handbook of Evaluation Methods for Health Informatics ,2006: Academic Press.p 9-11.
Improve performance
Outcomes
Safety
Effectiveness

7
Questions should be answered:
 Evaluation seeks to answer the :
Why (objective of evaluation),
Who (which stakeholders’ perspective is going to be evaluated)
When (which phase in the system development life cycle),
What (aspects or focus of evaluation)(What is it going to be used for ?)
How (methods of evaluation) questions [1],[2]
1- Maryati Mohd. Yusof, R.J.P., Lampros K. Stergioulas. Towards a Framework for Health Information Systems
Evaluation. in Proceedings of the 39th Hawaii International Conference on System Sciences. 2006. Hawaii IEEE.
2-SYMONS, V.J., A review of information systems evaluation-content ,context and process. Eur J Inf Syst, 1991. 1(3): p.
205-212.

8
1.Friedman C., W.J., Evaluation methods in biomedical informatics. 2 006: Spr
inger .p 6.

9
SDLC
Maintenance
Implementation
Testing
Design
Analysis
Planning

10
Pre-implementation evaluation
Inform difficult decisions
Prior to starting any program
As well as post-implementation
evaluation
1.Nykanen, P., et al., Guideline for good evaluation practice in health informatics
(GEP-HI). Int J Med Inform, 2011. 80(12): p. 815-27.

11
Assessment:
There are two basic types of assessment:
I. Summative
II. Formative [1]
 Measured :
Qualitative and Quantitative:
Putting the results into a metric context .
 Qualitatively and subjectively
 Quantitatively and objectively :With the aid of a questionnaire
study..[2]
1-Brender, J., Handbook of Evaluation Methods for Health Informatics ,2006: Academic Pr
ess. pp 9-14.
2-Brender, J., Handbook of Evaluation Methods for Health Informatics ,2006: Academic Pre
ss. pp 20-29.

12
Formative and Summativ
e evaluation
Formative evaluation
Throughout the systems
lifecycle
It provides information for
improving the system under
development
Summative evaluation
Focused on assessing the effect or
outcome of the evaluation object
At a certain point of time after
implementation
1.Nykanen, P., et al., Guideline for good evaluation practice in health informatics
(GEP-HI). Int J Med Inform, 2011. 80(12): p. 815-27.

13
Quantitative & qualitative methods:
• Task analysis
• Interface design
• Time motion analysis
• Software log
• Questionnaire
Quantitative methods
• Think aloud protocol
• Unstructured interviews
Qualitative methods
To assess
• user satisfaction
• user-perceived
• usefulness
• usability of this
framework.
Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.

14
Categories of most evaluatio
n studies:
I. CPR evaluation studies,
II. Telemedicine evaluation studies,
III. DSS evaluation studies.
 Evaluation can be conveniently classified into:
Objectivist and Subjectivist approaches . [2]
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
2- C.P. Friedman, J. Wyatt, Evaluation methods in biomedical informatics, 2nd ed., Springer Science
& Business,Media, New York, US, 2006.

15
Methodology and Methods:
A method is based on a well-defined theory and includes a
consistent set of techniques, tools and principles to
organize it.
Methodology
Method
metric
metric
measure
measure
Method
metric
Method
Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press. p 13.

16
Methodology
• Methodology is supposed to:
(1) provide the answer to what to do next, when to do what, and how to do it
(2) to describe the ideas behind such choices and the suppositions (for instance
, the philosophical background) behind them.
 A methodology must comprise:
a) The basic philosophies and theories, so that a user of the methodology can
judge the validity of its use
b) Perspective
c) Assumptions
d) Areas of use
e) Applicable methods, tools, and techniques
Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press. PP14-15.

17
Complexity of evaluation
in biomedical informatics
1.Friedman C., W.J., Evaluation methods in biomedical informatics. 2006:
Springer .p 6.
Evaluation
methodology
Computer-based
information
systems
Medicine and
healthcare
delivery
Evaluation in
medical
informatics

19
User Requirements Specificati
on
• Analysis of Work Procedures
• Assessment of Bids
• Balanced Scorecard
• BIKVA
• Delphi: Qualitative assessment
• FieM Study
• Focus Group Interview
• Future Workshop
• Grounded Theory
• Heuristic Evaluation
• Interview (no standardized)
• KUBI
• Logical Framework Approach
• Organizational Readiness
• Pardizipp
• Questionnaire (no standardized)
• Requirements Assessment
• Risk Assessment
• Social Network Analysis
• Stakeholder Analysis
• SWOT
• Usability
• Video recording
• WHO: Framework for Assessment of
Strategies

20
Technical Development Phase
Used to provide feed-back for the technical development.
 Balanced Score card
 Clinical/Diagnostic Performance
 Cognitive Assessment
 Cognitive Walkthrough
 Heuristic Evaluation
 Risk Assessment
 SWOT
 Technical Verification
 Think Aloud
 Usability

21
Assessment Methods : Adapta
tion Phase
• Analysis of Work Procedures
• BIKVA
• Clinical/Diagnostic Performance
• Cognitive Assessment
• Cognitive Walkthrough
• Equity Implementation Model
• Field Study
• Focus Group Interview
• Functionality Assessment
• Grounded Theory
• Interview (nonstandardized)
• Prospective Time Series
• Questionnaire (nonstandardized)
• RCT, Randomized Controlled Tria
l
• Risk Assessment
• Root Causes Analysis
• Social Network Analysis
• SWOT
• Technical Verification
• Think Aloud
• Usability
• User Acceptance and Satisfaction
• Videorecording
Real operational assessment can take place.

22
Assessment Methods: Evaluat
ion Phase
1. Analysis of Work Procedures
2. Balanced Score card
3. BIKVA
4. Clinical/Diagnostic Performance
5. Cognitive Assessment
6. Cognitive Walkthrough
7. Delphi
8. Equity Implementation Model
9. FieMStudy
10. Focus Group Interview
11. Functionality Assessment
12. Grounded Theory
13. Heuristic Evaluation
14. Impact Assessment
15. Interview (nonstandardized)
16. KUBI
17. Prospective Time Series
18. Questionnaire (nonstandardized)
19. RCT, Randomized Controlled Trial
20. Risk Assessment
21. Root Causes Analysis
22. Social Network Analysis
23. Stakeholder Analysis
24. SWOT
25. Technical Verification
26. Think Aloud
27. Usability
28. User Acceptance and Satisfaction
29. Videorecording
30. WHO: Framework for Assessment
of Strategies

23
• Usability methods, such as heuristic evaluation, cognitive
walk-throughs , RCT and user testing, are increasingly used
to evaluate and improve the design of clinical software
applications.
• Evaluation studies do not focus solely on the structure and
function of information resources; they also address their
impact on persons who are customarily users of these
resources and on the outcomes of users’ interactions with
them to understand users’ actions.
1.Friedman C., W.J., Evaluation methods in biomedical informatics. 2006: Springer .p 6.
2- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic
Press.

24
1- Analysis of Work Procedures
• Assessing what actually happens compared to the expectations.
They may clearly have a role as part of an evaluation study.
• Some well-known options :
1. The Learning Organization
2. Enterprise modeling
3. Business Process Reengineering
4. Use Cases and scenarios
5. Total Quality Management
6. Health Technology Assessment (HTA)
7. Computer-Supported Cooperative Work (CSCW)
8. Cognitive Task Analysis
1- Brender, J., Handbook of Evaluation Methods for Health Informatics
2006: Academic Press.

25
Analysis of Work Procedures
*Note: The use of diagramming techniques and
other forms of graphical modeling requires
experience and understanding the principles
and procedures involved.
Applied in
a health
care setting
Provide
clear
structure:
Framework
• Potential
Views
• Level of
systems
Systems analyses in
a health care
environment
We can use
Diagramming
techniques

26
Enterprise Modeling
Enterprise Architecture (EA) is a strategic activity and
planning tool for an enterprise, which facilitates decision-
making by enabling a conceptual view of the enterprise.
The main objective of an EA approach is to define the
layout of organizational components and relationships
among them.

29
Health Technology Assessme
nt (HTA):
• HTA is concerned with the systematic evaluation of the
consequences of the adoption and use of new health
technologies and improving the evidence on existing
technologies. [1,2]
• One of the basic lessons learned in the area of HCI is
that usability evaluation should start early in the design
process
• Goodman and Ahn (1999) provide a basic overview of
HTA principles and methods. [3]
1. O'Reilly, D., K. Campbell, and R. Goeree, Basics of health technology assessment.
Methods Mol Biol, 2009. 473: p. 263-83.
2. Stevens, A., R. Milne, and A. Burls, Health technology assessment: history and de
mand. J Public Health Med, 2003. 25(2): p. 98-101.
3. Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Acade
mic Press.

30
2- Balanced Scorecard
• Created in 1992 by Drs. Robert S. Kaplan and David P.
Norton, the Balanced Scorecard (BSC) is a revolutionary
way to handle strategy management.
• Balancing focus areas by means of a set of indicators for
a set of strategic objectives.
• The Balanced Scorecard is a tool used to measure an
organization’s activities and initiatives against its Vision,
Mission and Values as outlined in its Strategic Plan.
1. http://www.hrh.ca/balancedscorecard, Humber River Hospital, 10-20-2016
2. https://2gc.eu/- Balance Scorecards reports-10-20-2016-
2GC is a strategic execution consultancy with particular experience in
implementing the latest generation Balanced Scorecards. With a global client
list,

31
Balanced Scorecard is a Manage
ment System
Not just a
measurement system
You do not just measure
your heart beat
You use the
heart rate monitor
to manage your exercise

33
3-BIKVA :User Involvement in
Quality
Development A tool for making critical, subjective decisions about an
existing practice.
1. Identifying user satisfaction or dissatisfaction.
2. Summarizing the information in group interview.
3. User encouraged to identify the reasons behind the incidents and
interactions referred to.
4.Interview management in iterative process with the objective of
clarifying all the disparities that relate to the issues of quality
identified by the users.
5.conclusion will then be presented to the decision makers for an
assessment
Press.

34
BIKVA :User Involvement in Q
uality
Development
 Developed in Denmark and it is an evaluation and quality
enhancement method .
 The evaluation method is interviewing. The evaluation
process starts from the clients, then moves to the front-line
staff (employees in direct contact with the clients) and
finally ends to managers and politicians. Clients are asked
to express and justify "why they are satisfied or dissatisfied
" with the services offered.
(2016). "Linkki-hanke - yhteys työelämään." Retrieved 11/9/2016, 2016, from :
http://www.palmenia.helsinki.fi/linkki/.

35
3- Questionnaires
 Advantage:
• most people can manage it
 Different Types :
A. Open questions
B. Checklist questions
C. The Likert scale consisting of a bar with fields to tick on a scale
D. Multipoint scale
E. Semantic differential scale, which in tabular form uses columns wit
h a value scale (for instance, "extremely", "very“,….)
F. Categorical scale
 Need verification : need thorough scientific validation of the questi
onnaire in order to provide results leading to optimal action
Press. p 164

36
Questionnaires:
• Questionnaires contain items that are pre-determined by the in
vestigators and consequently are of limited value in identifyi
ng new or emergent issues in the use of a system that the inve
stigators have not previously thought of. [1]
1.Kushniruk, A.W. and V.L. Patel, Cognitive and usability engineering methods
for the evaluation of clinical information systems. J Biomed Inform, 2004. 37(1
): p. 56-76.

37
4- Clinical/Diagnostic Perform
ance Measurement of the diagnostic performance in measures of accuracy
and precision for IT-based expert systems and decision-support
systems before the system is implemented.
 Measurement is doing in Technical Development phase but it can
continue during the operational phase (the Adaptation Phase and
Evolution Phase )
 The clinical performance of the systems (for diagnostic, prognostic ,
screening tasks, etc.) is typically measured with measures from
medicine, such as accuracy, precision, sensitivity, specificity, and
predictive values.
1.Kaplan, B., Evaluating informatics applications—clinical decision systems. Internationa
l Journal of Medical Informatics, 2001. 64: p. 15-37.
2. Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic
Press.

39
*Point*
• Kaplan reviews the literature (only for the years
1997- 1998) on clinical performance of
decision-support systems for the period and
includes other review articles on this subject
and he concluded that most studies use an
experimental or randomized controlled clinical
trials design (RCT) to assess system
performance.
1.Kaplan, B., Evaluating informatics applications—clinical decision systems. International Journal of
Medical Informatics, 2001. 64: p. 15-37.

40
RCT is a trial in which subjects are randomly assigned to two groups:
one (the experimental group) receiving the intervention that is being
tested, and the other (the comparison group or controls) receiving an
alternative treatment.
The two groups are then followed up to see if any differences between
the result. This helps in assessing the effectiveness of the intervention.
40

41
Hierarchy of Study Types
Descriptive
•Case report
•Case series
•Survey
Analytic
Observational
•Cross sectional
•Case-control
•Cohort studies
Experimental
•Randomized
controlled trials
Strength of evidence for causality between a risk factor and outcome
41

42
‫مطالعات‬ ‫انواع‬
‫توصيفي‬ ‫تحليلي‬
‫مداخله‬‫اي‬‫اي‬ ‫مشاهده‬
‫باليني‬ ‫مايي‬‫ز‬‫آ‬‫ر‬‫كا‬
‫اجتماعي‬ ‫مايي‬‫ز‬‫آ‬‫ر‬‫كا‬
‫ميداني‬ ‫مايي‬‫ز‬‫آ‬‫ر‬‫كا‬
‫مقطعي‬
‫شاهدي‬ ‫مورد‬
‫كوهورت‬
‫اكولوژيك‬
‫مورد‬‫ش‬‫ر‬‫ا‬‫ز‬‫گ‬
‫د‬‫ر‬‫موا‬‫ش‬‫ر‬‫ا‬‫ز‬‫گ‬
Goal:
To test
hypotheses
about the
determinants
of disease
In contrast, the goal of intervention
studies is to test the efficacy of
specific treatments or preventive
measures by assigning individual
subjects to one of two or more
treatment or prevention options
42

43
5-RCT, Randomized Controlle
d Trial
 Randomized controlled trials are the ideal study design to
evaluate the effectiveness of health-care interventions.
 RCT is used to identify marginal differences between two or more
types of treatment.
 Useful in assessment studies of IT-based solutions for decision-
support systems and expert systems, only to a limited degree for
other types of systems.
1.Navaneethan, S.D., et al., How to design a randomized controlled trial. Nephrology (Carlton), 2010. 15(8): p.
732-9.
2. Bothwell, L.E. and S.H. Podolsky, The Emergence of the Randomized, Controlled Trial. N Engl J Med, 2016.
375(6): p. 501-4.
P 172
3- 1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.

44
All RCTs are controlled clinical trial
BUT
Not all controlled trials are RCTs
Randomization
simple cluster stratified

45
5-RCT, Randomized Controlle
d Trial
 An RCT is the most rigorous scientific method for
evaluating the effectiveness of health care
interventions.
 However, bias could arise when there are flaws in the
design and management of a trial.
1.Akobeng, A., Understanding randomized controlled trials. Archives of Disease in C
hildhood, 2005. 90(8): p. 840-844.

46
• ‫اثربخشی‬
• ‫عوارض‬RCT
‫روش‬ ‫بهترين‬ ‫انتخاب‬
‫پيشگيري‬
‫يا‬‫درمان‬
‫به‬ ‫رسيدن‬ ‫براي‬‫هدف‬

47
‫به‬ ‫ساده‬ ‫تصادفی‬ ‫روش‬ ‫به‬ ‫ها‬ ‫نمونه‬
‫و‬ ‫مورد‬ ‫گروههای‬‫ميشوند‬ ‫تقسيم‬ ‫شاهد‬
‫روشهای‬ ‫قويترين‬ ‫از‬ ‫يکی‬ ‫روش‬ ‫اين‬
‫ميباشد‬ ‫پزشکی‬ ‫درعلوم‬ ‫تحقيقاتی‬
‫از‬ ‫عبارتند‬ ‫که‬ ‫دارد‬ ‫مهم‬ ‫ويژگی‬ ‫سه‬:
‫به‬ ‫ها‬ ‫نمونه‬ ‫انتساب‬ ‫بودن‬ ‫تصادفی‬
‫بررسی‬ ‫مورد‬ ‫گروههای‬
(Random Allocation)
‫کنترل‬ ‫گروه‬ ‫وجود‬.
‫ها‬ ‫نمونه‬ ‫ومواجهه‬ ‫محقق‬ ‫وکنترل‬ ‫دخالت‬
‫فرضی‬ ‫علت‬ ‫با‬.
Randomization

48
National Cancer Institute at the National Institutes of Health;
https://www.cancer.gov/about-cancer/treatment/clinical-trials/what-are-
trials/randomization; 11/2/2016.

49
RCT usage:
I. By some patients being treated under the old s
ystem and others under the new system, or equ
ally with regard to staff;
II. By some departments keeping the old system a
nd other department getting the new system;
III. Comparing similar departments in different ho
spitals.

50
‫نیست؟‬ ‫مناسب‬ ‫مواردی‬ ‫چه‬ ‫برای‬RCT
• Testing the etiology of disease
• rare outcome
• Low participation
• Exclusion
• Low cost

51
RCT IS NOT suitable for:
* ETIOLOGY AND CLINICAL COURSE
smoking and cancer
* RARE & PROLONGED OUTCOME

52
CASP
RCTs - a checklist
• Good randomisation procedures
• Patients blind to treatment
• Clinicians blind to treatment
• All participants followed up
• All participants analysed in the groups to
which they were randomised (intention to
treat)

53
6- Delphi
• Delphi may be characterized as a method for structuring
a group communication process so that the process is
effective in allowing a group of individuals, as a whole,
to deal with a complex problem. [1,3]
• When judgment is necessary with controlled opinion
feedback: we use Delphi
• Qualitative assessment of an effect
• Method for the prediction of the future
• Key advantage : avoids direct confrontation of the
experts.
• Tool for expert problem solving [1,2]
2.Okoli, C. and S.D. Pawlowski, The Delphi method as a research tool: an example, design considerations
and applications. Information & Management, 2004. 42(1): p. 15-29.
3-‫تكنيك‬‫دلفي‬:‫علوم‬ ‫در‬ ‫آموزش‬ ‫ايراني‬ ‫مجله‬ ،‫اباذري‬ ‫پروانه‬ ،‫نصيرياني‬ ‫خديجه‬ ،‫احمدي‬ ‫اله‬ ‫فضل‬ ،‫تحقيق‬ ‫در‬ ‫ابزاري‬‫بهار‬ ،‫پزشكي‬‫ت‬ ‫و‬‫ابستان‬-138
7185(:1)8‫؛‬

54
Delphi:
• Delphi survey, including guidelines for data collection, d
ata analysis and reporting of results .
• Three ways of Delphi methods:
1- Classic Delphi, 2- Policy Delphi, 3- Decision Delphi
• Preparation of a questionnaire, including the perils and
pitfalls associated with a Questionnaire approach
1.Okoli, C. and S.D. Pawlowski, The Delphi method as a research tool: an example, design considerations a
nd applications. Information & Management, 2004. 42(1): p. 15-29.
2.Kushniruk, A.W. and V.L. Patel, Cognitive and usability engineering methods for the evaluation of clinical informati
on systems. J Biomed Inform, 2004. 37(1): p. 56-76.
3.‫دلفي‬ ‫تكنيك‬:‫تا‬ ‫و‬ ‫بهار‬ ،‫پزشكي‬ ‫علوم‬ ‫در‬ ‫آموزش‬ ‫ايراني‬ ‫مجله‬ ،‫اباذري‬ ‫پروانه‬ ،‫نصيرياني‬ ‫خديجه‬ ،‫احمدي‬ ‫اله‬ ‫فضل‬ ،‫تحقيق‬ ‫در‬ ‫ابزاري‬‫بستان‬-1387
185(:1)8‫؛‬

57
‫دلفی‬ ‫روش‬ ‫اجرايی‬ ‫مراحل‬:
‫دلفی‬ ‫روش‬ ‫اصول‬‫گرايی‬ ‫کثرت‬ ‫و‬ ‫جمعی‬ ‫خرد‬ ‫از‬ ‫استفاده‬‫شرکت‬ ‫نظرات‬
‫ميباشد‬ ‫کنندگان‬.
‫قابل‬ ‫منابع‬ ‫و‬ ‫موضوع‬ ‫گستره‬ ‫و‬ ‫دامنه‬ ‫به‬ ‫توجه‬ ‫با‬ ‫دلفی‬ ‫روش‬ ‫در‬
‫دسترس‬‫متفاوت‬ ‫کنندگان‬ ‫شرکت‬ ‫تعداد‬‫بود‬ ‫خواهد‬.
‫هم‬ ‫برای‬ ‫کنندگان‬ ‫شرکت‬ ‫که‬ ‫است‬ ‫اين‬ ‫بر‬ ‫تالش‬ ‫روش‬ ‫اين‬ ‫در‬‫ناشناس‬
‫بمانند‬ ‫باقی‬.
‫مراحل‬ ‫اين‬ ‫طی‬ ‫در‬ ‫که‬ ‫ميباشد‬ ‫تکرار‬ ‫دوره‬ ‫چندين‬ ‫شامل‬ ‫روش‬ ‫اين‬
‫که‬ ‫است‬ ‫آن‬ ‫بر‬ ‫سعی‬‫گروهی‬ ‫نظرات‬ ‫از‬ ‫استفاده‬ ‫حداکثر‬‫عمل‬ ‫به‬ ‫را‬
،‫آورده‬‫و‬ ‫مخالفت‬‫برسد‬ ‫حداقل‬ ‫به‬ ‫ناسازگاری‬.‫در‬ ‫معموال‬2‫يا‬3‫راند‬
‫يابد‬ ‫می‬ ‫پايان‬.
‫صورت‬ ‫به‬ ‫اول‬ ‫مرحله‬ ‫ليست‬ ‫چک‬‫مند‬ ‫ساختار‬ ‫نيمه‬ ‫يا‬ ‫و‬ ‫باز‬ ‫فيلد‬
(‫جديد‬ ‫های‬ ‫آيتم‬ ‫پيشنهاد‬ ‫جهت‬ ‫باز‬ ‫فيلد‬ ‫همراه‬ ‫به‬)‫می‬ ‫طراحی‬
‫گردد‬.
‫صورت‬ ‫به‬ ‫مراحل‬ ‫ساير‬ ‫ليست‬ ‫چک‬‫مند‬ ‫ساختار‬‫گردد‬ ‫می‬ ‫طراحی‬( .‫بدون‬
‫باز‬ ‫فيلد‬)
‫از‬ ‫معمول‬ ‫طور‬ ‫به‬‫معيار‬5‫ليکرت‬ ‫تايی‬‫گردد‬ ‫می‬ ‫استفاده‬.

58
6- Delphi
1.Friedman C., W.J., Evaluation methods in biomedical informatics. 2006: Springer .p 182.

59
7-Usability
• Usability evaluation is a method for identifying specific problems with IT p
roducts and specifically focuses on the interaction between the user and tas
k in a defined environment.[1,4]
• The usability evaluation methods are so various which makes the choice of
the appropriate one(s) a difficult task for evaluators. Usability factors are va
rious features that are used to measure how easy systems are in supporting
users’ tasks.[2,3]
1.Brown Iii, W., et al., Assessment of the Health IT Usability Evaluation Model (Health-ITUEM) for evaluating
mobile health (mHealth) technology. Journal of Biomedical Informatics, 2013. 46(6): p. 1080-1087.
2.Dhouib, A., et al., A classification and comparison of usability evaluation methods for interactive adaptive syst
ems. 2016: p. 246-251.
3.Borycki, E., et al., Usability Methods for Ensuring Health Information Technology Safety: Evidence-Based Ap
proaches Contribution of the IMIA Working Group Health Informatics
for Patient Safety. IMIAYearbook of Medical Informatics, 2013: p. 20-27.

60
USABILITY EVALUATION METHODS FOR INTER
ACTIVE ADAPTIVE SYSTEMS:
• Cognitive Walkthrough
• Focus Group
• User-as-wizard
• Task experiment based
• Simulated users
1.Dhouib, A., et al., A classification and comparison of usability evaluation meth
ods for interactive adaptive systems. 2016: p. 246-251.

61
Usability Evaluation
Methods
Advantages Disadvantages
Cognitive
walkthrough
(+) Makes predictions
about the design
without
involving users,
(+) Can be used early
in the preliminary evalu
ation phase.
(–)Certain adaptation
aspects not covered,
(–)Time consuming.
Heuristic evaluation (+) Quick and low-cost
evaluation method,
(+) Can be done early
in the IAS development
process
(–) Evaluators must
be experts,
(–) Need to choose
The appropriate
heuristics.
Focus group (+) Fast to conduct,
(+) Can provide large
amounts of data in
short time
(–) Support only
subjective opinions
(–) Depends on
experienced
moderator.
1.Dhouib, A., et al., A classification and comparison of usability evaluation methods for interactive adaptive
systems. 2016: p. 246-251.

62
8- Cognitive Assessment
• Task-oriented
• Assessment of the cognitive aspects of the interaction
between an IT system and its users
• Cognitive aspects deal with the extent to which the system
functions in accordance with the way the users
• Think and work in respect of user interaction with the IT
system in general and with the user dialogue in particular .

63
9- Cognitive Walkthrough
• Cognitive Walkthroughs an analytical method designed to evaluate
usability
• Assessment of user 'friendliness' on the basis of a system design,
from specification, to muck-ups and early prototypes of the system,
aimed at judging how well the system complies with the users' way
of thinking
• The cognitive walkthrough is a formalized way of imagining people
’s thoughts and actions when they use an interface for the first time.
1. Wharton, C., et al., The cognitive walkthrough method: a practitioner's guide, in Usability i
nspection methods, N. Jakob and L.M. Robert, Editors. 1994, John Wiley & Sons, Inc.
p. 105-140.
2. A. W. Kushniruk, A. W. Kushniruk, E. M. Borycki-Human, Social, and Organizational Aspe
cts of Health Information Systems-IGI Global, Premier Reference Source (2008)
3. Xiao, T., Broxham, W., Stitzlein, C., Croll, J., & Sanderson, P. Two human-cent
ered approaches to health informatics: Cognitive systems engineering and usa
bility. in Proceedings of the WCC IFIP-IMIA International eHealth Joint Conf
erence. 2010. Brisbane, Qld.

64
Cognitive Walkthrough
• CW is an inspection method for evaluating usability in a user
interface.
• The questions are aids to simulating the user’s cognitive process:
(1)Will the user be trying to achieve the right effect?
(2)Will the user discover that the correct action is available?
(3)Will the user associate the correct action with the desired effect?
(4)If the correct action is performed, will the user see that progress is being
made?
answered with YES or NO and reasons why the user will succeed or fail in
performing the action (“failure/success stories”). [1,2]
1.Blig, L.-O. and A.-L. Osvalder, Enhanced cognitive walkthrough: development of the cogniti
ve walkthrough method to better predict, identify, and present usability problems. Adv. in Hum.-
Comp. Int., 2013. 2013: p. 9-17.
2.Wharton, C., et al., The cognitive walkthrough method: a practitioner's guide, in Usability ins
pection methods, N. Jakob and L.M. Robert, Editors. 1994, John Wiley & Sons, Inc. p. 105
-140.

65
CW
• The method focuses on simplicity in learning, especially
through exploratory learning. CW has been employed to
evaluate medical equipment, such as clinical information
systems, patient information systems, clinical order
systems , dialysis machines, and patient surveillance
systems .[1]
1.Blig, L.-O. and A.-L. Osvalder , Enhanced cognitive walkthrough: development of the c
ognitive walkthrough method to better predict, identify, and present usability problems. A
dv. in Hum.-Comp. Int., 2013. 2013: p. 9-17.

66
10-Focus Group Interview
• A focus group is a form of qualitative research in which
a group of people are asked about their perceptions,
opinions, beliefs, and attitudes towards a product.
• The moderator speaks very little, and encourages the
group to generate the information required by stimulating
discussion through terse provocative statements.
 Challenge: analyzing qualitative data
1.Rabiee, F., Focus-group interview and data analysis. Proc Nutr Soc, 2004. 63(4): p. 655-60.
2.Chapter 5: Personal Interviews. Marketing research and information systems. (Marketing and Agribusiness
Texts 2016 [cited 2016 10/31/2016]; Available from: http://www.fao.org/docrep/w3241e/w3241e06.htm.

67
Time of focus group interview
:
During the
early analysis
phases
During
operations

68
 The focus used for eliciting descriptive data from population subgroups
 Usually, a group of eight to twelve persons are gathered together for a
group interview or discussion on a focused topic.
 Focus groups are widely used in the investigation of applied-research
problems and are recognized as a distinct research method.
 The method enables researchers to generate new hypotheses;
 To explore intermediate variables as a means of explaining certain
relationships found in survey data.
1. Bender, D.E. and D. Ewbank, The focus group as a tool for health research: issues in design and analysis. Health Transit
Rev, 1994. 4(1): p. 63-80.

69
11-Equity Implementation Mo
del (EIM)
• Examine users' reaction to the implementation of a new system
• It may be applied as a measuring instrument to provide the decision-
making basis relation to policy planning and priority setting, as well
as for examination of barriers and drivers at the implementation of
change.
• The EIM helps provide a theory-based understanding for collecting
and reviewing users' reactions to, and acceptance or rejection of, a
new technology or system.
1. Lauer, T.W., K. Joshi, and T. Browdy, Use of the equity implementation model to review clini
cal system implementation efforts: a case report. J Am Med Inform Assoc, 2000. 7(1): p. 91-10
2.

70
• The method provides a means for explanation of system
implementation events, leading to an understanding of user
resistance or acceptance of the new technology.
• It may be applied as a measuring instrument to provide the decisi
on-making basis relation to policy planning and priority setting, as
well as for examination of barriers and drivers at the implementati
on of change.

71
12- Functionality Assessment
1. Validation of objectives fulfillment
2. Impact Assessment
3. Identification of problems
• It is best suited to qualitative studies

72
13-FieMStudy
 Observation of an organization to identify its practices
and to expose mechanisms that control change.
 This method is widely used in psychology, sociology,
anthropology, and so on that is, in professions that deal
with different perspectives of human factors - to identify
what goes on and how

73
14-Grounded Theory
• Grounded Theory is a supportive analytical method for
data acquisition methods that generate textual data, some
open Questionnaire methods, and Interviews.
• This method is subjective.
• In a social context through an iterative and comparative
process in 4 phase:
 Open coding
 Axial coding
 Selective coding
 Theoretical coding

74
15-Heuristic Evaluation
• Used when no other realizable possibilities exist
• This method can be used for nearly everything but in practice it is m
ost commonly used for the assessment of user interfaces
• Heuristic evaluation, an established evaluation method in the field o
f Human-Computer Interaction, was originally proposed by Jakob N
ielsen .
 Five phase:
1. Selection of appropriate heuristics,
2. An individual and independent inspection of the design using heuristics to i
dentify features that conflict with some aspect of best practice,
3. Editing the joint material to identify and resolve duplicates and related findi
ngs,
4. Prioritization to identify severities
5. Analysis and report writing
1.Westbrook, J. and J.I. Westbrook, Information Technology in Health Care 2007: Procee
dings of the 3rd International Conference on Information Technology in Health Care: So
cio-Technical Approaches. 2007: IOS Press . pp 205-207

76
Zhang and colleagues incorporated Nielsen’s 10 heuristics ,
Shneiderman’s eight golden rules , and the results of their research to
formulate a list of 14 heuristics.
1.Allen, M., et al., Heuristic evaluation of paper-based Web pages: A simplified inspect
ion usability methodology. Journal of Biomedical Informatics, 2006. 39(4): p. 412-423.

77
16- Impact Assessment
• Measurement of the effect plus assessment
of side effect

78
17- Interview (Nonstandardized)
• For qualitative studies of subjective
as well as objective circumstances.
1- Brender, J., Handbook of Evaluation Methods for Health
Informatics 2006: Academic Press.

79
18- KUBI
• Translated: Quality development through user involvement
• It has many points in common with the Balanced Scorecard
method

80
19- SWOT
• Allows the assessment of an organization from a
neutral perspective through a detailed discussion
of the organization’s strengths, weaknesses, opp
ortunities and threats.

81
SWOT
• The method is useful as a brainstorming method to elucidate a
nd evaluate each of the aspects brought forward.

82
20- Think Aloud
• Think Aloud is a method that requires users
to speak as they interact with an IT-based
system to solve a problem or perform a
task.

83
• Thinking aloud technique dates back to the works of experimental
psychology and was first ever described by Karl Duncker (1945) wh
ile he studied productive thinking.
• TAP is popularly used by usability researchers today. But what do
researchers think they get from TAP? Is it right to assume that there
is a one-to-one mapping between verbal protocols and ‘pure data’?
• TAP adds cognitive load on users and can hinder primary tasks
(Preece, 1994). So how do the users experience it? What do users thi
nk of it?
Research Paper: Janni Nielsen, Torkil Clemmensen, and Carsten Yssing. 2002. Getting access to what goes
on in people's heads?: Reflections on the think-aloud technique. In Proceedings of the second Nordic
conference on Human-computer interaction (NordiCHI '02). ACM, New York, NY.

84
• Thinking is much more that what can be explicitly expressed in words.
•To get access to human cognitive processes, a way forward may be to
develop a practice of introspection; to expand our knowledge about the
reflective activity of the user in the expert-guided think aloud session.
•The authors argue that access to subjective experience is possible in
terms of introspection where user has to become a participant in the
analysis of his or her own cognitive processes.
• The paper suggests that use of think aloud should have, as a
prerequisite, explicit descriptions of design, test procedure and
framework for analysis.

85
21- Prospective Time Series
 Measurement of a development trend
 Time series
 Before-and-after studies
 Measurement of a number of measures over time shows how an acti
vity or an outcome changes as a function of either time alone or as a
function of different initiatives
 This method requires control all over time
 Carried out as a matched-pair design in:
 Controlled studies including RCTs
 Simple before-and-after study
 In cohort studies with the follow-up of cases over time
 In decision support system with before-after study
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press. P 160.

86
22-Risk Assessment
• Monitoring of risk factors in a
development or assessment project to make it
possible to take preemptive action.
• Risk assessment, risk management, and risk
control can be handled either retrospectively
or prospectively.

87
23-Root Causes Analysis
• Exploration of what, how, and why a given
incident occurred to identify the root causes of
undesirable events.
Press.

88
24-Social Network Analysis
• Social network analysis is carried out by means of diagra
mming techniques in relationships between elements wit
hin an organization (such as individuals, professions, de
partments or other organizations)
• Most commonly applied to help improve the effectivenes
s and efficiency of decision making processes in commer
cial organizations.
• The potential value of SNA to measure team function an
d use the information to improve working processes was
another finding of studies in different settings
2- Chambers, D., Wilson, P., Thompson, C., & Harden, M. (2012). Social Network Analysis in Health
care Settings: A Systematic Scoping Review. PLoS ONE, 7(8), e41911. http://doi.org/10.1371/journal.
pone.0041911

89
Example of social network: the case of the eye care programme in
the Brong Ahafo region, January 2010.
Karl Blanchet, and Philip James Health Policy Plan.
2011;heapol.czr055
Published by Oxford University Press in association with The London School of Hygiene and
Tropical Medicine © The Author 2011; all rights reserved.

90
25-Stakeholder Analysis
• Assessment of stakeholder features and their inner dynamics, aimin
g to identify participants for the completion of a given task, a proble
m-solving activity or a project.
Namazzi, G., et al., Stakeholder analysis for a maternal and newborn health project in Eastern Ugand
a. BMC Pregnancy Childbirth, 2013. 13: p. 58.

91
Stakeholder Analysis. Service quality 2016 [cited 2016 11/2/2016]; Available from: http://asq
.org/service/body-of-knowledge/tools-stakeholder-analysis.

92
26-Technical Verification
• Functions are present, work correctly and are in
compliance with the agreement.
A. Completeness of the functionality
B. Correctness of the functionality
C. Coherence of the functionality
D. Consistency of the functionality
E. Interconnectivity of the functionality
F. Reliability
G. Performance
Press.

93
27-User Acceptance and
Satisfaction
 Assessment of user opinions, attitudes, and perceptions of an IT syst
em during daily operation
 Most common methods :
 Interview techniques : Interviews or Focus-Group Interviews.
 Questionnaires: most commonly used when measuring user satisfact
ion.
 The Equity Implementation Model
*User-satisfaction questionnaires
 Aim: verify and correct for aspects such as reliability,
predictive value, accuracy and content validity and
internal validity.

94
28-Videorecording
Video recording is a capability for recording videos of
every new Automate test session that is requested on
BrowserStack. While your test is running, you can see a
live screencast that is recorded and saved with a proper
name that can be accessed and downloaded later.
https://www.browserstack.com/question/651

95
29-WHO: Framework for As
sessment of Strategies
 The method may be applied for assessment of different (
development) strategies, either individually or as a
comparative analysis.
 The method is particularly relevant early in the
explorative Phase for exploring the feasibility of basic
ideas and solutions.
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.p 222

96
30-WHO: Framework for Asse
ssment of
Strategies

97
Conclusion
Literature reviews

98
Literature reviews:
• Evaluation studies have been performed with different aims and obje
ctives, and are concerned with different domains and areas of focus.
1.Rahimi, B. and V. Vimarlund, Methods to Evaluate Health information Systems in
Healthcare Settings: A Literature Review. Journal of Medical Systems, 2007. 31(5): p. 397-
432.

99
Literature reviews about ICT Evaluation
 Most important study conducted by Kapalan about CDSS Evaluation[1].
• Conclusion of 27 reviews : RCT is standard method of evaluation approach for
CDSS.
• Problems of RCT: Do not answer questions such as why some systems tend to
be used while others are not
•Unrecognized ineffectiveness reason of CDSS.
 1,035 articles reviewed by Ammenwerth and Keizer’s from 1982 to 2002
• Concluded that the number of evaluation studies in the area of medical informatics is
rising significantly.
• appropriateness of patient care, efficiency of patient care, user satisfaction, and software
quality.
• The quality of care processes and patient outcomes was found to have increased.
• Interpreted this shift as a sign of the maturation of evaluation research in medical
informatics.
1. Rahimi, B. and V. Vimarlund, Methods to Evaluate Health information Systems in Healthc
are Settings: A Literature Review. Journal of Medical Systems, 2007. 31(5): p. 397-432.

10
 Delpierre et al reviewed 26 articles about CBPRS , from January
2000 to March 2003.
• Increased satisfaction of users and patients
• Could lead to significant changes in medical practice.
• Most of the studies did not include qualitative factors such as
characteristics of the disease and the tool
Rahimi, B. and V. Vimarlund, Methods to Evaluate Health information Systems in He
althcare Settings: A Literature Review. Journal of Medical Systems, 2007. 31(5): p. 3
97-432.

10
• Some of the studies in this review evaluate the effects of the
new system’s implementation on the quality of work
performance, such as user job performance , computer
knowledge, and investigation of skill among other users
• Most of the studies included in this paper have used survey
methodology as their research method
• Some of the studies used a clinical trial or cohort study to
research the systems 'outputs.
• No studies have been conducted to explore the impact of IT
on the system as a whole.

10
Free PPT Chart : ALLPPT.com
ALLPPT.com _ Free Powerpoint Templates, Diagrams and Charts
Your own sub headline
Text here
Add text Add
Text here Text here
Add text add text add text add text
add text add text add text add text
Text here Text here
Text here Text here
Add text add text add text
Text here Text here
Text here Text here

Evaluation methods in heathcare systems

More Related Content

What's hot (20)

Viewers also liked (11)

Similar to Evaluation methods in heathcare systems (20)

Recently uploaded (20)

Evaluation methods in heathcare systems

Editor's Notes