Grieco_Luz gradworks.proquest.com 10103879

Meeting the Demands of the 21st
Century Workplace: Effects of Critical Thinking
Instruction on the Application of Critical Thought
Dissertation Manuscript
Submitted to Northcentral University
Graduate Faculty of the School of Education
in Partial Fulfillment of the
Requirements for the Degree of
DOCTOR OF PHILOSOPHY
by
L. NERY GRIECO
Prescott Valley, Arizona
March 2016

Approval Page
Meeting the Demands ofthe 2151
Century Workplace: Effects of Critical Thinking
Instruction on the Application of Critical Thought
By
L. Nery Grieco
Approved by:
Barry K. Spiker April 20, 2016
Chair: Dr. Barry Spiker Date
Certified by:
Dean of School: Dr. Rebecca Wardlow Date
ii

iii
Abstract
One of the primary goals of education is to foster critical thinking. Since approximately
1980, both educators and employers have questioned whether the educational system in
the United States has adequately prepared students in the area of critical thinking
sufficient to meet the demands of 21st
century workforce. The problem addressed,
therefore, is that students consistently continue to graduate from the U.S. educational
system with inadequate critical thinking skills. This problem prompted the researcher to
question the transformational effects, if any, of teaching critical thinking within higher
education. The intent of explicit critical thinking instruction is not only to increase
students’ critical thinking skills, but equally important is to enhance their ability to
consistently apply those skills in a myriad of situations. The purpose of this quantitative,
quasi-experimental study was to examine the change, if any, in critical thinking skills and
application of critical thinking among undergraduates based on whether they receive
explicit critical thinking instruction throughout an introductory Physics course. The
study explored the effects of deliberately teaching critical thinking skills and the
application of those skills to real-world situations. Participants included 85 freshman and
sophomore cadets enrolled in an introductory physics course at the U.S. Air Force
Academy in Colorado Springs, Colorado during the fall semester of 2015. Cadets were
randomly sampled from a possible 218 cadets. Data were collected with two instruments:
the Halpern Critical Thinking Assessment and a Real World Outcomes (RWO) inventory.
Although the raw data revealed increases in participants’ critical thinking skills and their
ability to apply critical thinking, results of statistical analyses were not consistent with
existing literature on critical thinking skills. Of the intended sample size n=85, only 36

iv
participants completed all pre- and post- intervention assessments. With a low sample
size all statistical tests indicated no significant relationship between the explicit critical
thinking instruction and participants’ critical thinking skills or their ability to apply
critical thought to real-world situations. Replication of this study, with a larger sample
size, is recommended to further examine whether teaching critical thinking would have a
transformative effect on participants’ critical thinking skills and, more importantly, on
their ability to consistently apply those skills in real-world situations.

v
Acknowledgements
I would like to thank, first and foremost, my amazingly supportive husband and
daughter: Mike and Stephanie Grieco. Not only did they instill confidence in me as a
wife, mother, and as a researcher, but they also provided the continuous encouragement
that I needed along the way. To my parents, Victor and Eneida, who have always been so
proud of me; les agradezco con todo el alma los sacrificios que han tomado por mi bien,
los quiero mucho as los dos. Sandra, my loving sister, her husband Hector, and my niece
Jacklyn all motivated me to pursue my dreams; thank you! I took this journey in hopes to
inspire my extended family to believe in life-long learning and to serve as an example for
them of how tenacity and dedication can overcome all obstacles in life.
It is with great appreciation that I also acknowledge the three members who
served on my Dissertation Committee, Dr. Barry Spiker, Dr. Nicole Avena, and Dr. Cary
Gillenwater. These professionals encouraged me through immediate feedback and
support throughout the process. Without the statistical analysis support of Dr. Victoria
Brione, I would not have appropriately completed Chapter 4 of this study, many thanks to
her as well. To Dr. Diane Halpern and Dr. Heather Butler, whose work stimulated my
interest in the topic for this study, directly took time off their busy schedules to provide
insightful guidance. A very special thanks also go to Dr. Kimberly De La Harpe, Dr.
Gregor Novak, Dr. Rajani Ayacitula, and Lt Col Steve Novotny, faculty members at the
United States Air Force Academy, whose tireless efforts helped shape the study and
accelerate both the Institutional Review Board approval process and data collection.
Finally, to the participants of this study, I thank you for your participation and wish you

vi
all the best as you embark in one of the greatest, most honorable professions anyone can
hope to be part of, an officer in the United States Air Force.
Disclaimer: This work was created in the performance of a Cooperative Research and Development
Agreement with the Department of the Air Force. The Government of the United States has certain rights
to use this work. The conclusions expressed in this document are my own. They do not reflect the official
position of the US Government, Department of Defense, the United States Air Force, or the United States
Air Force Academy.

vii
Table of Contents
Chapter 1: Introduction....................................................................................................... 1
Background................................................................................................................... 2
Statement of the Problem.............................................................................................. 3
Purpose of the Study..................................................................................................... 4
Theoretical Framework................................................................................................. 5
Research Questions....................................................................................................... 7
Nature of the Study....................................................................................................... 8
Significance of the Study.............................................................................................. 9
Definition of Key Terms............................................................................................. 10
Summary..................................................................................................................... 12
Chapter 2: Literature Review............................................................................................ 14
Documentation............................................................................................................ 14
Taxonomy of Critical Thinking .................................................................................. 15
Significance of Critical Thinking................................................................................ 24
Historical Foundation of Critical Thought.................................................................. 26
Environments that Promote Critical Thinking Experiences ....................................... 28
Practical Application of Critical Thought................................................................... 33
Critical Thinking Skills and Individual Dispositions towards Critical Thinking ....... 42
Effects of Diversity on Critical Thinking ................................................................... 48
Transferability of Critical Thinking Theory to Practice ............................................. 49
Assessing the Application of Critical Thinking Skills................................................ 52
Summary..................................................................................................................... 54
Chapter 3: Research Method............................................................................................. 57
Research Method and Design ..................................................................................... 58
Population ................................................................................................................... 62
Sample......................................................................................................................... 63
Instruments.................................................................................................................. 64
Operational Definition of Variables............................................................................ 66
Data Collection, Processing, and Analysis ................................................................. 67
Assumptions................................................................................................................ 70
Limitations.................................................................................................................. 71
Delimitations............................................................................................................... 72
Ethical Assurances...................................................................................................... 72
Summary..................................................................................................................... 75
Chapter 4: Findings........................................................................................................... 77
Results......................................................................................................................... 77
Evaluation of Findings................................................................................................ 93
Summary................................................................................................................... 100
Chapter 5: Implications, Recommendations, and Conclusions ...................................... 101

viii
Implications............................................................................................................... 106
Recommendations..................................................................................................... 112
Conclusions............................................................................................................... 116
References....................................................................................................................... 118
Appendixes ..................................................................................................................... 130
Appendix A: Halpern Critical Thinking Assessment Manual ....................................... 130
Appendix B: Real-World Outcomes Inventory ............................................................. 131
Appendix C: Permission Letter...................................................................................... 133
Appendix D: United States Air Force Academy IRB Approval.................................... 134
Appendix E: Sample Lesson Introducing Concept Maps.............................................. 135

ix
List of Tables
Table 1. Comparison between Standardized Critical Thinking Tests............................... 53
Table 2. Actual Statistical Power...................................................................................... 78
Table 3. Demographic Characteristics of Participants...................................................... 80
Table 4. Frequencies and Percentages for Participants with Missing Data ..................... 82
Table 5. Frequency Table of Faculty Participants ............................................................ 86
Table 6. Descriptive Statistics for the Dependent Variables ............................................ 86
Table 7. Means and Standard Deviations for HCTA Scores ............................................ 90
Table 8. Mixed ANOVA Results for Changes in HCTA Scores as a Function of
Instruction ......................................................................................................................... 91
Table 9. Means and Standard Deviations for RWOI- Part A Scores................................ 92
Table 10. Mixed ANOVA Results for Changes in RWOI- Part A Scores as a Function of
Instruction and Faculty Rank............................................................................................ 92
Table 11. Means and Standard Deviations for RWOI- Part B Scores.............................. 93
Table 12. Mixed ANOVA Results for Changes in RWOI- Part B Scores as a Function of
Instruction and Faculty Rank ........................................................................................... 93
Table 13. Relationship between Transformative and Critical Thinking Learning Theories
........................................................................................................................................... 97

x
List of Figures
Figure 1. Concept Map for Independent and Dependent Variables.................................. 60
Figure 2. Diagram of Participants’ Progress through Study Phases ................................ 79
Figure 3. Two by Two Factorial Design.......................................................................... 81
Figure 4. Histograms of Pre and Post HCTA Scores...................................................... 87
Figure 5. Histograms of Pre and Post RWOI-Part A Scores ........................................... 88
Figure 6. Histograms of Pre and Post RWOI-Part B Scores............................................ 89
Figure 7. Comparison between Themes in Defining Critical Thinking, Transformative
Learning Factors, and Explicit Critical Thinking Instruction......................................... 109

1
Chapter 1: Introduction
Noah Webster and Thomas Jefferson, both of whom led the founding of the
American educational system, recognized the value of an educated citizenry (Behar-
Horenstein & Niu, 2011; Ravitch, 2008). However, whereas Webster believed that
education could be used to shape society Jefferson believed education should foster the
critical intelligence of the citizenry so that each person might understand and defend his
or her rights (Ravitch, 2008). Echoing Jefferson’s philosophy on education, Miller, Hall,
and Tice (2009) recognized that critical thought was essential for making decisions,
solving problems, reasoning, innovation, and effective practice in an increasingly
complex society. These skills are also the foundation for developing officers at the
United States Air Force Academy. People living in society need citizens as well as
military officers who can use critical thinking skills to resolve multifaceted problems
(Behar-Horenstein & Niu, 2011; Miller et al., 2009; Periklis, 2010).
One of the primary goals of education is to foster critical thinking skills (Behar-
Horensten & Niu, 2011; Marin & Halpern, 2011). Critical thinking skills give students
the ability not only to understand what they have read or been shown, but also to ask
independent questions about how they can build upon that knowledge (Fahim &
Masouleh, 2012). Since approximately 1980, both educators and employers have
questioned whether the educational system in the United States has adequately prepared
students in the area of critical thinking sufficient to meet the demands of 21st
century
workforce (Association of American Colleges and Universities (AAC&U), 2010; Bureau
of Labor Statistics, 2011; Butler, 2012; Marin & Halpern, 2011; Lansiquot, Blake, Liou-
Mark, and Dreyfuss, 2011). The literature on critical thinking reveals that students

2
consistently continue to graduate from the U.S. educational system with inadequate
critical thinking skills (Stedman & Adams, 2014). This problem prompted the researcher
to question the transformational effects, if any, of teaching critical thinking within higher
education.
Background
Curricula based on fostering critical thinking encourage students to think for
themselves by reflecting and questioning the assumptions they make when addressing
cause and effect relationships to justify their conclusions (Mathews & Lowe, 2011).
Critical thinkers should not only make sound decisions within an educational
environment, but should also have the skills to make better decisions about other aspects
of their lives, such as in legal, medical, or financial areas (Butler, 2012; Carmel &
Yezierski, 2013). The challenge facing educators, however, is the gap that exists
between knowing critical thinking concepts and being able to consistently apply the
critical-thinking process to real-world situations (Butler, 2012; Flores, Matkin, Burbach,
Quinn, & Harding, 2012; Miller et al., 2009).
One reason for this difficulty is that educators often do not understand the
concepts inherent to critical thinking and therefore continue to practice traditional
teaching strategies such as lectures and requiring students to memorize (Flores et al.,
2012; Stedman & Adams, 2012). Another reason that students fail to apply critical
thinking to real-world situations may be that critical thinking has typically been studied
as a set of skills pertaining to the individual, with little attention placed on measuring the
efficacy of critical thinking pedagogies (Ku & Ho, 2010) or on assessing the application
of these skills to real-world problems (Dwyer, Boswell, & Elliott, 2015). To facilitate the

3
development of critical thinkers, it is necessary first to understand the nature of critical
thought and then to examine closely the effects of critical thinking pedagogies on
individuals’ preparedness for habitually applying critical thought processes in a myriad of
situations (Thomas, 2009). The understanding and examination of critical thinking
pedagogies could establish a basis for measuring the transformational efficacy of
teaching critical thinking within disciplines. Knowledge gained from studying the
effectiveness of teaching critical thinking within disciplines may help students enhance
their propensity for applying critical-thinking processes to successfully address real-
world problems as well as provide educators with insight into improving faculty
development programs.
Statement of the Problem
This study examined the pervasive problem of students continuing to graduate
from institutions of higher education with inadequate critical thinking skills despite an
ever-increasing emphasis placed on critical thinking as a desired outcome of higher
education (AAC&U, 2011; Carmel & Yezierski, 2013; Flores et al, 2012; Khandaghi,
Pakmehr, & Amiri, 2011; Stedman & Adams, 2014). According to Flores et al. (2012),
Holley (2009), and Khandaghi et al. (2011) educators have neither been challenging
students to think critically within academic disciplines nor encouraging the development
of reasoning skills essential for addressing the complexities of modern life. The results
of a large-scale longitudinal study of 2,322 American college students from 2005 to 2009
indicated that 45% of students made no significant improvement in their reasoning skills
during their first four years of college (Davies, 2011). The authors of the same study also
found that 36% of students showed no significant improvement in critical thinking skills

4
after four years (Kiener, Ahuna, & Tinnesz, 2014). This problem prompted the
researcher to question the transformational effects, if any, of teaching critical thinking
within higher education. The intent of explicit critical thinking instruction is not only to
increase students’ critical thinking skills, but equally important is to enhance their ability
to consistently apply those skills in a myriad of situations. Further research was needed
to examine whether teaching critical thinking had a transformational effect on students’
critical thinking skills and the application of those skills to real-world situations.
Purpose of the Study
The purpose of this quantitative, quasi-experimental study was to examine the
change, if any, in the dependent variables (DV) critical thinking skills (DV1) and
application of critical thinking (DV2) among United States Air Force Academy cadets
based on whether: (a) they received explicit critical thinking instruction, which was the
first independent variable (IV1), (b) they did not receive explicit critical thinking
instruction (IV2), (c) instruction was provided by a junior faculty member (IV3), or (d)
instruction was provided by a senior faculty member (IV4) as part of the cadet’s
introductory physics course. Participants included a minimum of 85 cadets randomly
sampled from a possible 218 cadets at the United States Air Force Academy in Colorado
Springs, Colorado. The target population were freshmen and sophomore cadets enrolled
in the Physics-110 course during the fall semester of 2015. Physics-110 is a mandatory
course requirement for all cadets assigned to the United States Air Force Academy. This
course was selected because introductory physics courses, in general, emphasize the
scientific method for problem solving. There are many similarities between the scientific
method and the focus of this study, critical thinking processes. The sample size (n=85)

5
was determined using an a priori power analysis. A power analysis is defined as the
probability of rejecting a false null hypothesis (Faul, Erdfelder, Buchner, & Lang, 2009).
Knowledge gained from examining the effects of critical thinking instruction on students’
application of those skills may not only provide students with an awareness of their own
critical thinking skill levels, but may also provide insight into enhancing critical thinking
curricula and faculty development programs to address the problem of students
graduating with inadequate critical thinking skills.
Theoretical Framework
Educational leaders may be better equipped to improve students’ critical thinking
skills and application of those skills by recognizing the strengths and weaknesses of
contemporary learning theories (Abu-Dabat, 2011). A review of the literature on
contemporary learning theories indicated that the transformative learning theory may be
related to the development of critical thinkers (Carawan, Knight, Wittman, Pokorny, &
Velde, 2011). The focus of transformative learning is on the analytical and rational, as
well as on the behavioral, steps of the adult learning process (Herlo, 2010).
Transformative learning, also known as Mezirow’s theory, is learning that
stimulates more extensive changes in the learner than other kinds of learning, especially
learning experiences which form the learner and produce major impacts affecting the
learner's subsequent experiences (Herlo, 2010). According to transformative learning
theory, the learning process is when learners critically reflect on their assumptions and
beliefs about something and then change their frames of reference by consciously taking
actions that bring about new ways of defining their worlds (Herlo, 2010). The process,
although fundamentally rational and analytical, is an experience that can be described as

6
a behavioral transformation (Herlo, 2010). An important part of transformative learning,
according to Herlo, is for individuals to change their behavior or frames of reference by
“critically reflecting on their assumptions and beliefs and consciously making and
implementing plans that brings about new ways of defining their worlds” (p. 108). These
behavioral transformations can take place within the classroom if the two domains of
transformative learning theory are incorporated into existing lesson plans (Sammut,
2014).
The transformative learning theory offers two domains of learning. The first
domain is based on task-oriented problem solving accomplished with the development of
causal relationships. The focus of the second domain is on communicative learning,
which involves understanding the meaning of what others communicate (Carawan et al.,
2011). Additionally, there are four key factors influencing transformative learning: (a) a
learning experience, (b) critical reflection, (c) rational discourse, and (d) taking action
(Carawan et al., 2011; Sammut, 2014). At the foundation of a learning experience is a
learner-centered environment in which learners feel unthreatened and can engage in open
dialogue (Carawan et al., 2011). Drawing from a safe learning environment, learners
experience a common base for constructing meaning via personal reflection and group
discussion (Carawan et al., 2011). These experiences lead to both critical reflection and
taking action through rational discourses, which are the basis for transformative learning.
By exploring the United States Air Force Academy’s Physics-110 course, for indications
of the four key factors influencing transformative learning, the efficacy of the specific
teaching approach may be better understood as it relates to the habitual application of
critical thinking processes.

7
Research Questions
The research questions for this study were designed to assess the importance of
exposure to critical thinking instruction to cadets in terms of their application of critical
thinking skills to given problems and situations. Following are the research questions for
this study.
Q1. What are the effects of explicit critical thinking instruction on the critical
thinking skill levels of a control group as compared to a treatment group of freshman and
sophomore cadets enrolled in the Physics-110 course at the U.S. Air Force Academy?
Q2. What are the effects of explicit critical thinking instruction on the application
of critical thinking skills of a control group as compared to a treatment group of freshmen
and sophomore cadets enrolled in the Physics-110 course at the U.S. Air Force
Academy?
Hypotheses
H10. There is no statistical difference between the critical thinking skills scores
of cadets who completed Physics-110 with explicit critical thinking instruction as
compared to those who completed Physics-110 without explicit critical thinking
instruction.
H1a. There is a statistical difference between the critical thinking skills scores of
cadets who completed Physics-110 with explicit critical thinking instruction as compared
to those who completed Physics-110 without explicit critical thinking instruction.
H20. There is no statistical difference between the application of critical thinking
compared to those who completed Physics-110 without critical thinking instruction.

8
H2a. There is a statistical difference between the application of critical thinking
Nature of the Study
Accentuating critical thinking as an outcome of higher education has been a
common practice however, actually learning how to think critically has been less than
successful (Lansiquot et al., 2011). Through a quantitative Non-equivalent Group Design
(NEGD), the researcher addressed this problem by comparing the results of critical
thinking assessments between control and treatment groups. The NEGD is a quasi-
experimental research design that compares the observations of a treatment group with
those of a non-treatment group of participants (Shaughnessy, Zechmeister, E.,
Zechmeister, J., 2014). The goal of this study was to quantify any changes in the skills
and the ability to apply critical thinking to real-world situations, between participants
who received explicit critical thinking instruction within Physics-110 and those who did
not. The skills were measured by the Halpern Critical Thinking Assessment (Appendix
A) while the application of skills were measured by a Real-World Outcomes Inventory
(Appendix B).
Upon Northcentral University Institutional Review Board (IRB) approval, the
researcher assessed participant responses to two critical thinking assessments; one
measured skills and the other assessed the ability of participants to apply critical thinking
skills to address real-world situations on a pre- and post-intervention basis. The second
phase of this study consisted of data processing and analysis using a factorial mixed

9
ANOVA procedure. Finally findings, implications, and recommendations for practice
and future studies are also reported in the second phase of this study.
Significance of the Study
Acquiring and using the cognitive skills of interpretation, analysis, evaluation,
inference, explanation, and self-regulation is valuable to many aspects of daily life. For
example, a study of over 1,100 college students showed that scores on a college level
critical thinking skills test markedly correlated with college GPA (Wilson-Mulnix, 2012).
Not only does strong critical thinking skills have a positive impact on grades, but it has
also been demonstrated that critical thinking skills can be learned, suggesting that
explicitly learning critical thinking concepts has a direct relationship to improved grades
(Heijltjes, Van Gog, & Paas, 2014; Moore, 2011b; Wilson-Mulnix, 2012).
Grades are not the only positive outcome of learning and applying critical
thinking concepts. Critical thinking can be considered a tool of inquiry (Ku, Ho, Kau, &
Lai, 2014). The dependence on technology coupled with the pressures of competing
cultural and social influences highlight the need for individuals to analyze, synthesize,
and evaluate an overwhelming amount of data on a daily basis (Weiner, 2011). There is
nearly unanimous consensus on the value of students graduating with the ability to think
critically; however, there are still many institutions of higher education that do not offer
explicit critical thinking instruction (Marin & Halpern, 2011).
Further research was necessary to study the effects of explicit critical thinking
instruction on the ability of students to apply critical thinking skills in real-world
situations. The results of this study may provide a strategy for enhancing critical thinking
outcomes throughout the United States Air Force Academy and other institutes of higher

10
education. Additionally, an understanding of the relationship between explicit critical
thinking instruction and individuals’ ability to habitually apply critical thought processes
could establish a basis for measuring the efficacy of teaching critical thinking within
varying disciplines.
Knowledge gained from studying the effectiveness of teaching critical thinking
may also help individuals enhance their own ability to apply critical-thinking processes to
successfully address real-world problems. It may also provide a strategy for future
faculty development/curriculum enhancements at the Academy and beyond. Moreover,
information realized from this study may help narrow the gap that exists between
knowing critical thinking concepts and being able to consistently apply the critical-
thinking process to real-world situations. This study facilitates an understanding of the
concepts inherent to critical thinking and examines the change in critical thinking skills
and application of critical thinking among undergraduates based on whether they
received explicit critical thinking instruction. Finally, although the results of this study
identified a statistically insignificant relationship between deliberately teaching critical
thinking concepts and the critical thinking skills and application of those skills to real-
world situations, results of future similar studies may support attempts to reverse the
trend of students consistently continuing to graduate from the U.S. educational system
without having the skills necessary to reason well.
Definition of Key Terms
Active learning environment. An active learning environment is an
environment in which students are engaged in activities related to relevant issues to
advance the making of meaning (Zimmerman & Land, 2014).

11
Critical reflection. Critical reflection is one of four conditions required for the
transformative learning process to take place. Critical reflection allows for individuals to
recognize, analyze, and question experiences and perspectives (Carawan, Knight,
Wittman, Pokorny, & Velde, 2011, p. 395).
Experience. Experience is one of four conditions required to enact
transformative learning. Experience refers to encounters individuals live through and the
meanings they attach to these occurrences (Carawan et al., 2011).
Just-in-Time Teaching (JiTT). A teaching and learning strategy that combines
out of classroom web-based resources with in-class active activities; in-class content can
be rapidly adjusted to meet learner needs based on results of web-based activities (Novak,
Patterson, Garvin, & Christian, 1999).
Metacognition. Metacognition, also referred to as critical reflection, is one of the
four conditions required to be present for transformative learning to take place (Carawan
et al., 2011). Metacognition is the process of thinking about one’s own thinking and
monitoring one’s own learning (Jones, 2012).
Peer-Instruction (PI). A teaching and learning strategy that capitalizes on
student interaction during class and focuses on attention to underlying concepts; students
are encouraged to discuss or convince others of their understanding of the given concept
(Mazur, 1997).
Rational discourse. Rational discourse is one of four conditions required to be
present for transformative learning to take place. Rational discourse refers to questioning
what is being asserted in efforts to comprehend or validate the assertion further or to
question the credibility of the individual making the assertion (Carawan et al., 2011).

12
Transformative learning. Transformative learning is a form of learning that
explains changes in individuals understanding of knowledge then helps guide future
action (Taylor, 2007).
Summary
This study examined the problem of students consistently continuing to graduate
from institutions of higher education with inadequate critical thinking skills (AAC&U,
2011; Carmel & Yezierski, 2013; Flores et al, 2012; Khandaghi, Pakmehr, & Amiri,
2011). The purpose of this quantitative, quasi-experimental study was to examine the
change, if any, in critical thinking skills and application of critical thinking among United
States Air Force Academy cadets based on whether they received explicit critical
thinking instruction throughout eight lessons of an introductory Physics course. A
sample size of 85 participants (n=85) was administered two critical thinking assessments
on a pre/post-intervention basis to examine the relationship between explicit critical
thinking instruction and the level of critical thinking skills and ability to apply those skills
of participants given varying situations.
This quantitative research encompassed a quasi-experimental, Non-equivalent
Group Design (NEGD). The chosen design aligned with the study’s purpose and
research questions as reflected by the operational variables of explicit critical thinking
instruction (independent variable) and critical thinking skills and application of critical
thinking (dependent variables). Information realized from this study may help narrow
the gap that exists between knowing critical thinking concepts and being able to
consistently apply the critical-thinking process to real-world situations because the study
provided an understanding of the concepts inherent to critical thinking and examined the

13
change in critical thinking skills and application of critical thinking among
undergraduates based on whether they receive explicit critical thinking instruction.
Finally, the results of this study identified a statistically insignificant relationship between
deliberately teaching critical thinking concepts and the critical thinking skills and
application of those skills to real-world situations.
A brief description of the theoretical framework, nature, and significance of this
study established the foundation for how the researcher collected, analyzed, and reported
data. Prior to expanding on the research method that was used in this study, a review of
what was already known about critical thinking put into perspective the rationale and
significance of this study.

14
Chapter 2: Literature Review
This literature review focuses on the application of the phenomena of critical
thinking. The goal is to integrate existing research on critical thinking interventions in
order to examine the effectiveness of critical thinking pedagogies on the habitual practice
of critical thought. By evaluating the pros and cons of existing research on critical
thinking interventions, as well as the central theories that have been used to explain
critical thinking, this literature review presents the need for further research to examine
the effectiveness of teaching critical thinking in higher education.
Documentation
Empirical and theoretical studies were included in the synthesis of the literature.
This literature review consists of approximately 100-peer-reviewed articles; the majority
of which were published within the last five years. The articles were accessed from
databases such as ProQuest, Science Direct, EBSCOhost, and several other library
resources. The following search words were used in a multitude of combinations:
“critical thinking”, “dispositions for”, “teaching strategies”, “critical thinking
assessments”, “critical reflection”, “rational thought” and “problem solving”.
This review begins with a brief comparison of the various definitions researchers
have used to explain the phenomenon of critical thinking. The taxonomy of critical
thinking sets the foundation for the next section which explains the value of thinking
critically throughout history. The main focus of this review is on prominent teaching
environments and strategies educators have used to cultivate critical thinking skills within
the classroom. A synopsis of individual dispositions towards critical thinking is also
presented to help explain how student’s dispositions may affect their application of

15
critical thinking processes. Finally this review illustrates a comparison of the different
methods used to assess critical thinking skills and dispositions attained through the
various teaching strategies. Assessments of critical thinking can help determine the
effectiveness of teaching critical thinking in higher education.
Taxonomy of Critical Thinking
The phenomenon of critical thought has been studied from a cognitive
psychological and a philosophical perspective (Fahim & Masouleh, 2012; Kennedy,
Fisher & Ennis, 1991). The taxonomical structure of critical thinking associates each
perspective with a particular theory for critical thinking which tends to cause confusion
when discussing a normative definition for critical thinking within higher education
(Cassum et al., 2013; Lloyd & Bahr, 2010). Whereas some scholars use “critical
thinking” and “higher order thinking” interchangeably (Marin & Halpern, 2011), others
make pointed distinctions that include the extent to which critical thinking can be defined
as either a set of skills or an innate cognitive process (Facione, 2013). The relationship
between the term “critical thinking” and other terms such as “informal logic”,
“metacognition”, “problem solving”, and “critical reflection” causes further confusion.
Definitions for critical thinking have stemmed from a multitude of views but there is little
empirical basis for a consensual definition of this phenomena within the context of higher
education (Celuch, Kozlenkova, & Black, 2010; Moore, 2011a). Each of the two
perspectives for defining critical thinking is explored more fully below to provide a
foundation for a study on the effectiveness of critical thinking teaching strategies.
Critical thinking as a cognitive psychological concept. From the cognitive
psychological theoretical perspective, critical thinking can be described as the actions,

16
behaviors, or attitudes individuals incorporate to acquire knowledge (Celuch,
Kozlenkova, & Black, 2010; Dewey, 1933; Ennis, 1996). The cognitive psychological
perspective explains how we think and why we follow a sequence of tasks in the mind
(Abu-Dabat, 2011; Dewey, 1933; Ennis, 1996). It also relies on the individual’s
disposition to willingly reflect on the internal questions and answers they make about
alternative possibilities in a given situation (Celuch et al., 2010). Cognitive
psychological theorists claim critical thinking is a transformative process the human mind
incorporates to acquire knowledge, solve a problem, or make decisions (Carawan,
Knight, Wittman, Pokorny, & Velde, 2011; Dewey, 1933; Ennis, 1996). Fahim and
Masouleh (2012) defined critical thinking by the types of actions or behaviors that
individuals exhibit during problem solving. Lloyd and Bahr (2010) added that critical
thinking is an attitude that individuals adopt when logically applying acquired skills in a
problem solving context.
Although the cognitive psychological classification of critical thinking has merits
in the context of describing the skills and dispositions critical thinkers use to solve
problems or acquire knowledge, one disadvantage of defining critical thinking
cognitively is that this definition relies primarily on the individual with little
acknowledgement of other variables that may contribute to a definition of critical
thinking such as the learning environment or the teaching or learning strategies that may
further encourage the individual to apply critical thinking processes/skills. Another
contentious aspect of the cognitive psychological definition of critical thinking is the
disagreement among critical thinking scholars over whether or not critical thinking is
dependent on specific disciplines such as the sciences as opposed to the humanities and

17
whether or not the critical thinking processes can be transferable to various situations or
environments (Kennedy, Fisher & Ennis, 1991; McArthur, 2010). For these reasons the
cognitive psychological concept of critical thinking should not be exclusively used in the
context of higher education.
Philosophical theories on critical thought. Critical thinking philosophers such
as Richard Paul and Matthew Lipman (as cited in Moore, 2011b; as cited in Niu, Behar-
Horenstein, & Garvan, 2013; as cited in Kennedy, 2012; Paul, 2013) focus on the ideal
qualities or characteristics that individuals should be capable of doing under the best of
circumstances. Whereas cognitive theorists believe critical thinking is about how we
think, critical thinking philosophers believe critical thinking is about how we should
think. The philosophical perspective can be described as a systematic examination of
rules, ideas, or principles that would explain the phenomena of critical thought (Lai,
2011). Riggs and Hellyer-Riggs (2010) stated philosophical definitions for critical
thinking tend to list qualities that critical thinkers should possess, such as open-
mindedness, fair-mindedness, motivation, and reasoning skills. Moore (2011a) on the
other hand, described critical thinking as a habit of the mind where individuals feel the
need to question acquired beliefs. These philosophical classifications of critical thinking
are supported by the American Philosophical Association’s description of a critical
thinker:
Inquisitive in nature, open minded, flexible, fair-minded, well-informed,
understands diverse viewpoints, has the will to suspend judgment, and
considers others perspectives (Lai, 2011, p. 6).
The philosophical perspective of critical thinking focuses on the application of

18
formal rules of logic however this perspective does not always correspond to
reality (Sternberg, 1986). Emphasis on the attributes of an ideal critical thinker
may limit discussion to the ideal and not take into account how individuals
actually think. To define critical thinking in the context of higher education the
pros of each the cognitive psychological and philosophical descriptions of critical
thinking combined may provide a more applied definition of this phenomenon.
Critical thinking in an educational context. Education is not merely the
acquisition of knowledge under controlled environments; Periklis (2010)
suggested one of the desired outcomes of education is to foster basic human
capabilities such as the ability to think critically. Generalist and specialist
however, continue to debate over which definition captures what critical thinking
really means in the realm of education (Moore, 2011b). Generalist claim critical
thinking is a set of skills that can be learned in a systematic way and which can be
applied across all academic disciplines (Moore, 2011b; Wilson-Mulnix, 2012).
In a comparative analysis of critical thinking generalists, Wilson-Mulnix
(2012) suggested that some critical thinking consider critical thinking a
methodical evaluation of beliefs or statements using rational standards that can be
taught, for example, in an Introduction to Logic course. Generalists claim
individuals do not necessarily need to be aware of how they think because their
critical thinking skills tend to be utilized or applied across a broad range of
contexts and circumstances (Moore, 2011b). For example, certain aspects of
critical thinking such as rational discourse can be applied, generally, across
different reasoning contexts. In this sense, critical thinking skills can be taught in

19
a generic way (Lai, 2011; Moore, 2011a; Wilson-Mulnix, 2012). Specialists on
the other hand tend to be more skeptical and believe critical thinking is contextual
and therefore relegated to specific academic disciplines (Behar-Horenstien & Niu,
2011). In the discipline of history, for example, Jones (2009) explained that
students are taught to consider the validity of an argument which tends to be
concerned with political power relationships, whereas in physics logic and
accuracy of solutions are vital to enhancing critical thinking skills. Law and
medical students, according to staff members, examine evidence to make logical
decisions (Jones, 2009). The conflicting philosophies between generalist and
specialist regarding critical thinking has been a long standing debate. McPeck,
Ennis, and Bailin all argued that the most useful thinking skills are those that are
domain- specific whereas other scholars such as Halpern, Lipman, and Van
Gelder maintain that critical thinking relies on criteria. These criteria may vary
across domains, yet the fundamental meaning of critical thinking remains the
same and therefore can be generalized (Lai, 2011). These opposing views should
be taken into consideration when designing future research on critical thinking
pedagogies.
This study considered both a generalist and specialist viewpoint.
Although this study focused on one specific domain, an introductory physics
course, the intervention, explicit critical thinking instruction, may be generalized
into other domains to elicit a transformation in student learning across different
contexts and environments.
Despite the variety of critical thinking definitions, the aim of this section

20
is to identify crucial themes among the various definitions and propose an applied
definition of critical thinking in the context of higher education. Although a
normative definition for this phenomenon may remain elusive due to the diversity
of meanings individuals apply to the words describing critical thinking (Moore,
2011a), the following discussion is organized by four major themes of critical
thought. The consensus among critical thinking scholars regarding critical
thinking is that it involves reflection about the actual thinking process (Facione,
2013; Herlo, 2010; Lampert, 2011; Niu, Behar-Horenstein, & Garvan, 2013;
Riggs et al., 2010), making judgments based on using rational reasoning skills
(Boghossian, 2012; Halpern, 2014; Khandaghi, Pakmehr, & Amiri, 2011; Riggs et
al., 2010), being skeptical about an individuals’ skills and dispositions to reach a
desired outcome (Ku, Ho, Kau, & Lai, 2014), and creating new knowledge
(Dondlinger & Wilson, 2012; Myo-Kyoung, Patel, Uchizono, & Beck, 2012).
Each theme will be correlated to the factors influencing transformative learning to
illustrate the link between critical thinking and the transformative learning theory.
Reflection. The first repetitive theme found in the literature on critical thinking
is that critical thinking involves reflection (Facione, 2013; Herlo, 2010; Lampert, 2011;
Niu et al., 2013; Riggs et al., 2010). Whereas Tishman and Jay (1993) described critical
thinking as being reflectively aware of one’s own basic beliefs, Niu, Behar-Horenstein,
and Garvan (2013) stated critical thinking is a process of purposeful reflection on
information that requires logic in order to make judgments and informed decisions.
Monitoring the quality of one’s thought makes it more likely that one will engage in
high-quality thinking (Lai, 2011). Within a classroom, reflection begins when students’

21
ambiguity or uncertainty about a problem or unfamiliar experience compels them to
identify and evaluate options to problem resolution; these are functions of critical
thinking (Dewey, 1933). As Dewey pointed out, the challenge is for educators to
employ teaching strategies that allow enough uncertainty to trigger reflection which in
turn results in the application of critical thought (Bleicher, 2011). Reflection about a
problem or situation generally refers to making a judgment based on evaluation of
knowledge. This leads the discussion to the next major theme in defining critical
thinking which is judgment.
Judgment. Judgment is the rendering of a verdict or taking a stand based on
rational questioning using systematic forms of logic (Boghossian, 2012). Critical
thinkers tend to make different types of judgment based on the validity, truthfulness,
reliability, usefulness, or persuasiveness of the resulting verdict or stance (Khandaghi,
Pakmehr, & Amiri, 2011). Cognitive psychologist tend to classify judgment as an
individual’s action or behavior whereas philosophers tend to define the formal process
an ideal critical thinker should follow to judge situations in one way or another.
Transformative learning theory suggests one of the domains of learning is the task of
problem solving (Carawan et al., 2011; Herlo, 2010). This relates to the actions and to
formal processes of the mind that takes place while judging. Using systematic forms of
rational questioning to evaluate arguments or alternative solutions serve as examples of
applying judgment within the critical thinking process (Riggs et al., 2010).
Individual skepticism. A third theme noted in the literature on defining critical
thinking is the view that acquisition of knowledge should involve individual skepticism
(Ku, Ho, Kau, & Lai, 2014). This theme emphasizes Socratic questioning and negative

22
judgment in arriving at a verdict, stance, decision, or solution (Khandaghi, Pakmehr, &
Amiri, 2011).
Whereas reflection, judgment, and skepticism pertain to an individual’s cognitive
behavior, the fourth theme for defining critical thinking is dependent on both judgment
and skepticism in producing new knowledge. This theme is referred to as evaluation
(Myo-Kyoung, Patel, Uchizono, & Beck, 2012). Benjamin Bloom (as cited in Myo-
Kyoung et al., 2012) and his colleagues proposed taxonomy of educational objectives
and categorized information processing skills into hierarchical levels, placing
comprehension at the bottom and evaluation at the top level. Evaluation includes
descriptions such as being able to analyze, synthesize and create new knowledge. These
attributes are frequently said to represent critical thinking (Dondlinger & Wilson, 2012;
Myo-Kyoung, Patel, Uchizono, & Beck, 2012).
For critical thinking to be transformative it should consider the four components
of the transformative learning theory: a) reflection, b) rational discourse, c) taking
action, and d) the learning experience (Carawan et al., 2011; Herlo, 2010). The major
themes found in the myriad of critical thinking definitions correlate with three of the
four factors that influence transformative learning with one exception. Definitions for
critical thinking have not taken into consideration the learning experience. Therefore, in
order for critical thinking to be transformative and consistently applied to real-world
situations and within different domains/contexts, the experience must be addressed
through appropriate classroom environments and teaching strategies. Educators may
benefit by knowing that both the cognitive and the philosophical approaches to critical
thinking can be incorporated into a transformative definition of the phenomenon. The

23
relationship between the cognitive psychological and the philosophical definitions of
critical thinking as they relate to the transformative learning theory can be summarized
by comparing the key components of each theory. Whereas transformative learning is
influenced by four key factors, (a) critical reflection, (b) rational discourse, (c) taking
action, and (d) the learning experience (Carawan et al., 2011). Definitions for critical
thinking relate to three of the four key factors that influence transformative learning.
The definitions for critical thinking addresses (a) purposeful reflection, (b)
communication of rational thought and skepticism, and (c) actions of the mind, such as
judgment or creating new knowledge. There is a clear gap in the current literature on
critical thinking. For critical thinking to be transformative a definition in the educational
context should also address a learners’ experience.
In the context of higher education, a more appropriate definition for critical
thinking should not only take into account the skills, dispositions, and descriptions of an
ideal critical thinker, the definition should also provide a description of the
transformative influences that impart critical thought. A more appropriate definition of
critical thinking in an educational context would therefore include descriptions of
optimum learning environments for imparting a long-lasting experience. An educational
definition for critical thinking would also address the specific discipline or subject
matter that the critical thinking process is to encompass, as well as the teaching
strategies that have been proven to positively influence the cognitive as well as the
philosophical descriptions for critical thinking.
Despite the fact that critical thinking is primarily dependent on individuals, it is
essential to explore the additional factors that influence critical thinking. The next

24
sections examine optimum learning environments, teaching strategies that have
enhanced critical thought, how to transform theory into practice, possible influences of
demographics on critical thinking, and assessing critical thinking. Familiarity with all of
these factors will facilitate an understanding of the relationships between them and the
application of critical thinking to real-world problems.
Significance of Critical Thinking
Critical thinking scholars such as Robert Ennis (1996) and Richard Paul (2013)
agreed that the ability to think critically is essential for success in a world where new
knowledge is created at an ever-accelerating rate. The information age, for example,
ushered an overwhelming amount of data that demands individuals possess unique skills
to properly utilize data in problem solving and decision making (Martin, 2013; Weiner,
2011). There is far too much information in cyberspace to discern relevant and accurate
information from misinformation (Weiner, 2011).
Inherent to students’ academic journey through higher education, for example, is
the ability to organize knowledge in a logical and rational manner within specific time
constraints. To survive in the constrained environment students must possess strong
critical thinking skills (Martin, 2013). Having these skills facilitate the access and proper
use of precise information because critical thinking processes consists of rational,
structured, systematic and analytical steps which can lead to effective decision making in
the classrooms and beyond (Martin, 2013; Weiner, 2011).
The quality of life is highly dependent on the quality of thought (McArthur,
2010). The decisions individuals make about their education, finances, short, and long
term goals, as well as how they interact in society as citizens are all impacted by their

25
ability to apply critical thinking skills. Additionally, strong critical thinkers not only are
more likely to do better academically, they tend to also be more employable (Dwyer et
al., 2012). It is therefore vital to examine the efficacy of critical thinking teaching
strategies.
Students who graduate from college without being able to figure facts from
opinions further exemplify the implications of deficient critical thought. Educators
should therefore be responsive to an increasingly complex society by developing critical
thinkers (Holley, 2009). Being a critical thinker enables citizens to engage with the
broader society in a creative and transformative dialectic (McArthur, 2010).
Unfortunately, the American education system has been perceived as failing to cultivate
critical thinking skills and dispositions (Frodeman, Klien & Mitcham, 2012; Wilson-
Mulnix, 2012).
Educators, in general, tend to be fixated on teaching at lower cognitive levels in
efforts to meet federally mandated standards (Frodeman et al., 2010; Marin & Halpern,
2011). The prominent perception of the failure to cultivate critical thinking has
motivated some educators to elicit unconventional teaching strategies in hopes of
cultivating the critical thinking skills students need for future success in the workforce
(Butler, 2012; Hodge & Lear, 2011). Colleges such as the University of Colorado,
Colorado Springs, the United States Air Force Academy, and other institutions of higher
education have experimented with various strategies for teaching critical thought.
Although the effectiveness of some of these interventions has been measured with respect
to individuals’ skill levels, the effectiveness of critical thinking teaching strategies have
not been evaluated from the perspective of individuals’ consistent application of critical

26
thinking processes. A better understanding of environments that may promote critical
thinking may foster further interest in evaluating the effectiveness of teaching critical
thinking. The emphasis on thinking critically however is not exclusive to the
complexities of modern society.
Historical Foundation of Critical Thought
The intellectual foundation of critical thinking can be traced to the teaching and
practices of Socrates (470-399 BC) who believed the best way to lead to reason in the
interest of finding truths was through a process of rigorous questioning (Bareham, 2012).
His use of questions to elicit deeper and broader thinking compelled others to challenge
fallacious thinking and empty rhetoric (Bareham, 2012). The permanence of this
philosophers’ influence is evidenced by the fact Socratic questioning, and its concern
with clarity and logic, remains widely used in educational settings to this day (Bareham,
2012).
John Locke revolutionized education during the 1600’s by developing the
theoretical foundation for critical thinking (Stuart, 2010). Locke argued that reflection,
intuition, reason, and sensation produce critical thinking (Allen, 2013; Stuart, 2010). He
believed ideas came from reflections and that intuition and reasoning were considered
complex ideas (Allen, 2013; Stuart, 2010). As a result of Locke’s philosophy on critical
thinking, educators began using a building block approach to reasoning to be understood
by students in the same way that a malfunctioning mechanism could be taken apart and
reassembled with fixed components (Allen, 2013). Students were encouraged to make
thinking rational, to grapple with concepts rather than to accept conditions as taught. The
phenomenon of thought continued to expand throughout the 17th
through 19th
centuries

27
with a corresponding awareness of educational tools that, at the time, were considered to
enhance critical thinking, such as textbooks and blackboards (Allen, 2013; Stuart, 2010).
In the 20
th
century, the concept of critical thinking became more explicit primarily
through the works of William Graham Sumner who studied the origins of sociology and
anthropology (Fahim & Ghamari, 2011). Sumner described criticism as the examination
and test of propositions of any kind that are offered for acceptance, to determine whether
or not they correspond to reality. His descriptions of critique supported the need for
critical thinking in education since education at the time was evolving from simple
questioning to making judgments based on logic (Fahim & Ghamari, 2011).
Similar to Sumner, John Dewey addressed the concept of critical thinking in
education with an emphasis on the good habits of thinking (Davies, 2011). Dewey
discussed reflection as an aspect of sound thinking and he provided a pragmatic approach
to human thought as being grounded in actual human purposes, goals, and objectives
(Dewey, 1933). Both Sumner and Dewey's work made a significant impact to the
growing concept of critical thinking in the United States and provided a foundation for it
both in education and practice (Kennedy, 2012).
The 1980’s ushered in a Critical Thinking Movement with scholars such as Paul,
Ennis, Elder, Halpern, and Lipman (Niu, Behar-Horenstein, & Garvan, 2013). These
scholars emphasized the ideal characteristics and qualities of critical thinkers as opposed
to the actions or behaviors critical thinkers are capable of performing (Behar-Horenstein
& Niu, 2011; Fahim & Masouleh, 2012). Today critical thinking is the cornerstone of
higher education however the effectiveness of various teaching strategies has yet to be
definitively established. It is necessary to examine the efficacy of critical thinking

28
teaching strategies because critical thinking skills that are systematically cultivated can
build responsible citizens who are equipped to respond effectively in a myriad of
situations (Behar-Horenstein & Niu, 2011).
Environments that Promote Critical Thinking Experiences
Critical thinking skills can be learned; however individual dispositions or attitudes
need to be encouraged through learning experiences (Halpern, 1998; Marin & Halpern,
2011). These learning experiences tend to be dependent on numerous factors such as the
classroom’s physical location or how the classroom furniture is arranged (Cleveland &
Fisher, 2014). Classroom environments have often been associated with the type of
learning experience (such as active or student-centered learning) or the pedagogy the
educator incorporates within the classroom. Little attention has been placed on the
effects of the physical space associated with the type of learning experience or pedagogy
that is used (Cleveland & Fisher, 2014). In a comprehensive review of literature on the
effects of the physical space on learning outcomes, Cleveland and Fisher (2014)
illustrated the complex nature of evaluating the impacts of the physical classroom space
and configuration on learning outcomes. Although conclusive evaluations require further
research, the findings indicated that physical space should be designed to enable students
to feel a sense of identity and belonging, it should facilitate student engagement in
activities, and that the seating arrangements should correlate with the specific pedagogy
that is used (Cleveland & Fisher, 2014). These elements of the physical classroom space
tend to positively influence students’ learning experiences.
The relationships between the classroom learning environment, student’s
cognition, and learning outcomes were explored in a study conducted by Pitkaniemi and

29
Vanninen (2012) who suggested that the classroom environment produces learning
experiences and it is the learning experience that has been shown to be the most
significant factor in students’ learning and attitudes. The learning experience refers to the
cognitive and psychosocial properties linked to student learning through mediating
factors such as motivation and metacognition (Pitkaniemi & Vanninen, 2012). In other
words a positive learning experience and associated learning outcome is not only related
to the teacher’s instruction or classroom configuration, but it is also related to the
students’ motivation to learn and to the students’ metacognition. This is consistent with
Mathews and Lowe’s (2011) finding that revealed teachers should create classroom
experiences that center around learners’ sense of control. Despite the myriad of studies
that suggest student-centered environments enhance learning, Pascarella, Wang, Trolian,
and Blaich (2013) argued that it is the clear and organized classroom instruction and
student deep learning activities that provide positive impacts to critical thinking. The
Cleveland and Fishers’ (2014) study, Pitkaniemi and Vannienen (2012) study and the
Pascarella et al. (2013) study all shared a similar suggestion--learners who are actively
engaged in their classroom environments tend to think in a more logical and structured
way. This common finding of emphasizing active student engagement can be correlated
to transformative learning theory and critical thinking theory whereby task-orientation
and actions of the mind are congruent within both learning theories.
Learning experiences have been categorized into the following types: student-
centered, team-based, active-learning, and traditional lecture-based learning (Carawan et
al., 2011; Drummond, 2012; Yin Yin, Kanesan Abdullah, & Alazidiyeen, 2011; Zapatero,
Maheshwari, & Chen, 2012). A student-centered experience is one where learners feel

30
unthreatened and can engage in open dialogue (Carawan et al., 2011; Mathews and Lowe,
2011). When students can draw knowledge from a safe learning environment, they
experience a common base for constructing meaning via personal reflection and group
discussion (Carawan et al., 2011; Pitkaniemi & Vanninen, 2012). Within student-
centered environments, individual students are empowered to determine their own
learning goals using means they are most comfortable with (Mathews & Lowe, 2011).
These approaches are fundamentally different from traditional teacher-led instruction
hence student-centered learning has many critics. Opponents of student-centered
learning contend that this approach lacks compelling evidence to document effectiveness
(Pascarella et al., 2013). As technologies evolve and different teaching approaches
emerge, future research is needed to document evidence of the effectiveness of the
various approaches in supporting learning experiences that foster critical thinking.
Whereas student-centered experiences tend to focus on individual students’ self-
esteem, team-based learning experiences focus on student interaction within the class
learning setting (Drummond, 2012). Yin Yin, et al. (2011) studied college classroom
experiences and found a positive relationship between critical thinking and the amount of
student interaction, instructor support and questioning. The concept of team-based or
collaborative learning, refers to an instruction method whereby students work together in
small groups toward a common learning objective (Zapatero, Maheshwari, & Chen,
2012). One of the benefits of team-based learning approaches is that students are
responsible not only for their own learning but also for one another’s learning; hence the
success of one student benefits the success of the other students (Drummond, 2012;
Ofstad & Brunner, 2013; Parmelee & Michaelsen, 2010).

31
Advocates of team-based learning claim that the active exchange of thoughts and
ideas within small groups promotes critical thinking and increases the interest level
among participants (Macke, Taylor & Taylor, 2013; Thomas, 2009). Learning activities
that are designed for team-based learning tend to challenge students to apply the content
knowledge gained from readings. The activities, while feasible, tend to be challenging
enough to require engagement by every group member. In addition, activities do not
have clear-cut answers because the intent of team-based learning is to require group
discussion, problem solving, and critical thinking (Macke et al., 2013). For team-based
learning to be effective, Parmelee and Michaelsen (2010) suggested using twelve
guidelines which begins with a robust and solid course design and contains such tips as
including application exercises to fully engage teams in deep thinking and focused
discussion. The use of similar steps was suggested by Macke et al. (2013) in their
description of a step-by-step sequence for team-based instruction that fosters critical
thinking. While team-based learning has many proponents, its critics tend to believe that
implementation of this approach in the classroom is too complex and therefore students
do not enjoy the experience (Parmelee and Michaelsen, 2010). Additionally, opponents
point to the academic performance of the team is dependent on the lowest performing
team member which may jeopardize the motivation of higher performing students within
the team (Ofstad & Brunner, 2013). Perhaps the key would be to ensure that students,
whether in a student-centered or team-based learning environment, are engaged in active
learning versus listening to teachers lecturing subject matter.
Research demonstrated that active learning versus traditional lectures enhances
critical thinking skills (Drummond, 2012; Zapatero et al., 2012; Kim, Sharma, Land, &

32
Furlong, 2013). A 2006 National Commission on the Future of Higher Education
revealed that college students are not interested in the traditional lecture based instruction
style of yesteryear because their attention spans tend to be shorter leading to a lack of
student engagement and resulting in poor levels of learning and understanding (Zapatero
et al., 2012). Active learning on the other hand engages students in activities about
relevant issues to advance meaning making (Meltzer & Thornton, 2012; Zimmerman &
Land, 2014).
The similarity between all four of these learning experiences is the fact that they
each ensure student engagement and reflection are incorporated as the key approaches to
understanding the course content (Pascarella et al., 2013). Divergences between the
learning experiences stem primarily from the focus on individual versus a team. These
learning experiences can enhance critical thinking however many obstacles to their
implementation exist. According to Fraser, Timan, Miller, Dowd, Tucker and Mazur
(2014), limited time available posed the most crucial barrier to successfully implementing
positive learning experiences. A workaround strategy to this barrier may be for
instructors to build student trust and buy-in when adopting a new teaching strategy
(Fraser et al., 2014). Another challenge to fostering a positive learning experience is that
each of the four learning experiences described above depend on how well the teacher
organizes and presents the content (Pitkaniemi & Vanninen, 2012). Designing the
pedagogical approach to capitalize on strong learning experiences should consider how
the design may encourage the application of critical thought.

33
Practical Application of Critical Thought
Many critical thinking scholars maintain that critical thinking skills can be taught
(Behar-Horenstein & Niu, 2011; Fahim & Masouleh, 2012). Several different
approaches to teaching critical thinking skills have evolved. These instructional
interventions have been empirically studied during the past decades in efforts to examine
their effects on critical thinking skills development (Niu, Behr-Horenstein, & Garvan,
2013). The results of 61 empirical studies were analyzed by Niu et al. (2013) to reveal
mixed conclusions. While some studies have demonstrated that interventions have been
effective under certain conditions and populations, others provided non-significant results
under similar interventions. The mixed conclusions can be attributed to two key factors.
First the length of participant exposure to the intervention provided support to the notion
that the longer the exposure the greater the positive effect on critical thinking skills.
While some studies used similar interventions, the length of exposure to the intervention
varied thereby producing differing results. The second reason for differences in
conclusions between similar studies was based on the different discipline or subject
where the intervention took place. Within science courses the impact of the critical
thinking intervention was higher than within humanities. This may be attributed to the
strict processes inherent within the sciences as compared to the humanities.
The commonality among the empirical research however was the fact that the
majority of the studies utilized standardized measurements of critical thinking skills.
Critical thinking skills were measured by administering pre-post standardized
assessments such as the International Critical Thinking Basic Concepts and

34
Understandings Test, California Critical Thinking Disposition Inventory, California
Critical Thinking Skills Test, and Halpern Critical Thinking Assessment, to name a few.
Although assessment of students’ critical thinking skills is nearly common
practice at institutions of higher education, measuring the practical application of those
skills has eluded researchers. Further research is therefore required to explore whether or
not teaching interventions enhance the consistent application of critical thinking skills.
Developing critical thinking through explicit instruction. Critical thinking is
rarely explicitly taught, according to Heijltjes, Van Gog, and Paas (2014), because
research based guidelines for maximizing what to teach, when to teach, and how to teach
critical thinking are elusive. Another reason for the rarity of explicit critical thinking
instruction is the fact that a separate course on critical thinking tends to be resource
dependent (Marin & Halpern, 2011). Despite the elusive nature of standardized
guidelines for explicit critical thinking instruction, research has been conducted by
numerous scholars who reached similar findings. Marin and Halpern (2011) evaluated
the acquisition of critical thinking by conducting an experiment using an explicit form of
transferring critical thinking skills to students. The explicit form of transferring critical
thinking skills consisted of an online tutorial using topics of interest to high school
students such as video gaming, sports, dieting, and music videos. The online materials
required students to actively respond to questions. In addition to the online tutorial,
teachers provided classroom materials that corresponded to each online tutorial.
Teachers used these materials to introduce, discuss, and close each session. Marin and
Halpern’s findings supported the notion that explicit critical thinking instruction had a
greater positive impact on students’ critical thinking levels. A variation of Marin and

35
Halpern’s online tutorial and in-classroom materials strategy for transferring critical
thinking skills is noted in an interactive engagement pedagogy used across disciplines
called Just-in-Time Teaching.
Just-in-Time Teaching. The basic premise of Just-in-Time Teaching is that
instructors adjust in-class lessons based on students’ responses to pre-class, web-based
“warm-up” activities (Novak, 2011). Students complete the warm-up assignments online
a few hours before class. Teachers incorporate students’ understanding, or lack thereof,
of key concepts into the in-class lesson. In a sense this pedagogical approach fosters a
teacher-student team creating relevant learning experiences. Classroom instruction is
then more of a mix between pre-planned activities and student’s feedback. In a study to
examine the extent to which a new science teacher adopted Just-in-Time Teaching,
Osmond and Goodnough (2011) explored how pedagogical knowledge and practice
would be enhanced through the use of Just-in-Time Teaching. Osmond and Goodnough
(2011) concluded that the Just-in-Time Teaching strategy reinforced many areas of the
new teachers’ educational content knowledge. Just-in-Time Teaching activities helped
the teacher assess her students prior knowledge allowing her to address, in the classroom,
any gaps in their knowledge by challenging their thinking. In-class activities compelled
the teacher to reflect on her own instructional methodologies as she tried to instill active
participation through discussion sessions. Novak (2011) explained that timely web-based
assignments help both students and teachers prepare for enhanced in-class engagement as
well as provide students with some control over their own learning. The Osmond et al.
study supported the notion that Just-in-Time Teaching benefits student learning and
strengthens faculty development.

36
Student and teacher engagement has been examined in different capacities over
the past 30 years (Sun, Martinez, & Seli, 2014). Just-in-Time Teaching was first
developed over 16 years ago to enhance students’ learning experiences by providing
Physics instructors with insights to which concepts their students had difficulty
understanding (Novak, 2011). This approach has continued to be used in Physics and
other disciplines such as Economics and Biology. The goal of this strategy is to optimize
class time discussion among peers to ensure a common understanding of key concepts.
Just-in-Time Teaching makes attainment of knowledge an explicit outcome
primarily because of the nature of pre-class assignments. For Just-in-Time Teaching to
be successfully implemented, instructors must hold students responsible for on-time
delivery of meaningful responses (Scott, Gray, & Yates, 2013). Not only should the pre-
class assignment require students to include a written response in their own words, but
they should also indicate the process that led them to the response (Novak, 2011). By
reflecting on their thinking processes, students practice essential critical thinking skills.
Through a study comparing traditional lecture-based instruction with Just-in-Time
Teaching, Scott et al. (2013), measured the effectiveness of these two pedagogical
approaches in a short course on Newtonian mechanics. By assessing participants’ scores
on the Force Concept Inventory (FCI) on a pre-post intervention basis, Scott et al. (2013)
concluded that there were greater learning gains among participants in Just-in-Time
Teaching as compared to participants in the lecture-based program. Interestingly, the
female participants in Just-in-Time Teaching demonstrated greater gains in problem-
solving skills than the male participants in Just-in-Time Teaching whereas the male
participants in the lecture-based program showed improvement in problem-solving skills

37
as compared to the male participants in the Just-in-Time Teaching. The difference
between male and female outcomes was examined in two separate studies suggesting
similar conclusions. The study results suggest groups which are subject to negative
stereotypes, such as females in the science fields, tend to actively reflect and reaffirm
their core values which in turn counteracts the psychological effects of difficult and
stressful programs such as physics or mathematics (Scott et al., 2013).
This study capitalized on the Just-in-Time Teaching pedagogical approach by
introducing participants in the treatment groups to a four-part model for learning critical
thinking within the Physics-110 course warm-up activities as well as within classroom
discussion and in homework assignments. The four part model consist of (a) explicitly
learning the skills of critical thinking as identified by the Physics Department
chairperson, (b) developing the students dispositions for effortful thinking and learning,
(c) directing learning activities in ways that increase the probability of transformative and
trans contextual transfer (structure training), and (d) making metacognitive monitoring
explicit and overt (Halpern, 2014).
The goal of the Just-in-Time Teaching approach is to explicitly introduce
treatment groups to a common definition of what critical thinking is, in the context of
Physics, and provide them with the description of the critical thinking skills necessary to
solve real-world Physics applications. These two explicit concepts were introduced as
part of the pre-class warm-up activities, repeated during classroom discussions, and
assessed on a pre-test and post-test basis. During classroom time the teacher-student
team, emphasized the key concepts that some students struggled to understand during
pre-class warm-ups. Whereas students were expected to participate in and reflect on the

38
learning and teaching process, instructors fostered a community of mutual help through
Peer Instruction.
Peer Instruction. Peer Instruction, introduced by Mazur in 1997, is a teaching
method whereby students are grouped into small teams and rationally discuss their
individual answers to multiple choice questions (Scott et al., 2013). Mazur (1997)
contended that active involvement of students in the teaching process, facilitates
reflection among peers and allows instructors to continuously assess knowledge
attainment (learning). Peer Instruction (PI) is at the cornerstone of the Just-in-Time
Teaching strategy. Although Just-in-Time Teaching and Peer-Instruction had their
origins in the discipline of Physics, many other science-based disciplines have adopted
these teaching strategies (Scott et al., 2013).
Simon & Cutts (2012) suggested that the computing education community can
learn from the Physics community to foster deep understanding of computing mechanics.
These two scholars explained how Peer-Instruction is not merely talking about what the
“right” answer is; the right answer becomes apparent when the students use the
appropriate core concepts in their attempts to explain how they each define what the
problem is and articulate how they arrived at the solution (Simon & Cutts, 2012). They
acknowledged that the computing education community’s use of Peer-Instruction may
help educators learn what areas about computing that their students have difficulty
learning. The positive impacts of Peer-Instruction were empirically examined by Gok
(2012) in a quasi-experimental research design. Findings indicated that students
receiving traditional lecture-based instruction experienced significantly more difficulties
understanding the Physics phenomena then those who participated in Peer-Instruction and

39
problem solving.
This study was a quasi-experimental research design, similar to the Gok study;
however the intervention was explicit critical thinking instruction using Just-in-Time
Teaching, Peer Instruction, and incorporated concept mapping techniques in efforts to
optimize students’ deep understanding of critical thinking concepts so that they can
consistently apply those concepts and processes to real-world situations.
In examining explicit critical thinking instruction, several consistent themes
emerged. The themes consistent within explicit critical thinking instruction include
repetition, the use of argument-mapping or concept-mapping, Socratic questioning, and
student-centered active engagement (Bensley & Spero, 2014; Fahim & Masouleh, 2012;
Niu et al., 2013; Wilson-Mulnix, 2012). Students receiving explicit critical thinking
instruction must have a metacognitive awareness of the processes of thought. This can be
achieved through substantial repetition of thinking exercises (Wilson-Mulnix, 2012).
Through repetition of action oriented tasks students practice the critical thinking
principles. With repeated practice critical thinking becomes a habit of the mind.
The second theme emerging from the literature on explicit critical thinking
instruction was the active use of argument or concept maps in the classroom. Argument
or concept maps are diagrams depicting chains of reasoning and evidence that are
structured hierarchically, with premises reinforced by others in support of a conclusion
(Wilson-Mulnix, 2012). Argument or concept mapping engages students in rational
discourse while actively writing down their argument and supporting their argument with
evidence all in search for a solution or conclusion to a given problem or situation (Yeo,

40
2014). Argument or concept maps are designed to instill questioning in efforts to move
to the next logical and rational concept (Carr-Lopez, Galal, Vyas, Patel, & Gnesa, 2014).
Students can increase understanding by asking questions (Jones, 2012). The
philosopher, Socrates revolutionized instruction by basing it on questioning. Socratic
questioning was grounded on a series of methodical questions that helped learners gain
awareness towards their own misconceptions, erroneous assumptions, and false
conclusions. Teaching using the Socratic Method consists of questioning for which there
are no definitive answers in order to stimulate the thinking. According to Fahim and
Bagheri (2012) it is through questioning that students can be led to new discoveries, so
the function of questions is not limited to assessing the amount of knowledge obtained
but creating new levels of understanding, to uncover contradictions. Estes, Gunter, and
Mintz (2011) argued that "good questions are educative – they provide the opportunity
for deeper thought" (p.192). Based on Bloom's Revised Taxonomy and Paul's Taxonomy
of Socratic Questioning, Estes et al. (2011) introduced common types of questions
educators can ask to assess student knowledge and raise their level of understanding.
Bloom's Taxonomy contains six types of questioning for six cognitive levels:
1. Remembering questions which ask students to recall what they have learned,
2. Understanding questions which ask students to explain what they have learned,
3. Applying questions which ask students to use new learning in other familiar
situations,
4. Analyzing questions which ask students to break what they have learned into its
parts and explore the relationships among them,
5. Evaluative questions which ask students to render a judgment,

41
6. Creating questions which ask students to generate new ways of thinking about
issues and subjects. Likewise, Paul categorized Socratic questioning into six types as
well; however Paul’s taxonomy is also the foundation for the International Critical
Thinking Basic Concepts and Understandings Test (FCT, 2013). The six types of
Socratic questions Paul developed are:
1. Questions for clarification,
2. Questions that probe assumptions,
3. Questions that probe reasons and evidence,
4. Questions about viewpoints and perspectives,
5. Questions that probe implications, and
6. Questions that probe consequences.
Socratic questioning is not relegated to the teacher asking questions of the
student, it entails students posing question of their own, which according to Jones (2012),
is vital to critical thinking.
A student-centered active-learning environment, such as having students edit their
own and others assignments, is the fourth theme of explicit critical thinking instruction.
There is an extensive consensus among critical thinking scholars about the importance of
engaging students in authentic practices (Kim, Sharma, Land, & Furlong, 2013; Zapatero
et al., 2012), that provide meaningful contexts that will enhance their ability to apply
what they have learned. To explore the effect of active learning on critical thinking, Kim,
Sharma, Land, and Furlong, (2013) conducted a study on undergraduate environmental
science students who were tasked to develop arguments and write a final report
addressing a real-world problem, in this case, the response to a Hurricane emergency.

42
Findings from the study indicated that students who actively engaged in reflection of the
problem, held rational discussions, and wrote down argument maps scored high on a post
exercise critical thinking assessment than those who did not engage in a task-oriented
teaching module (Kim et al., 2013). While the Kim et al. study incorporated task
oriented instruction modules, Nelson and Crow’s (2014) intervention consisted primarily
of students’ repeated engagement in the practice of problematizing given situations.
Students would then collaborate on possible strategies and solutions to address the type
of situation, which theoretically developed new critical thinking skills that “improved
their ability to interpret, analyze, and address similar situations in the field of practice”
(p. 78). Both studies arrived at similar conclusions; active learning promotes critical
thinking.
Despite the myriad of empirical research concluding that explicit instruction
imparts critical thinking, a prevailing view seems to be that critical thinking can be
learned primarily through immersion into the real world (Halpern, 1998; Heijltjes et al.,
2014). Immersion into real world application of critical thinking provides learners with
opportunities to practice critical thinking skills.
Critical Thinking Skills and Individual Dispositions towards Critical Thinking
Traditionally, being a critical thinker was described in terms of ideal cognitive
abilities or skills such as being logical, analytical, open-minded, fair-minded, and rational
(Riggs & Hellyer-Riggs, 2010). A critical thinker certainly possesses these skills
however these particular skills are not meant to be definitive or exhaustive because
critical thinking is a much more complex phenomenon. For example, Halpern (2014)
discussed a much longer general list of thinking skills that would be applicable in almost

43
any situation or classroom. It is useful to identify those key thinking skills to help clarify
some of the underlying concepts of, what Halpern explained, are five categories of skills
that can be defined within a rubric for using an explicit approach to teaching critical
thinking skills. The five categories of critical thinking skills are: 1) verbal reasoning, 2)
argument analysis, 3) thinking as hypothesis testing, 4) likelihood and uncertainty, and 5)
decision making and problem solving.
The first category of critical thinking skills emphasizes verbal reasoning, using
existing knowledge about one statement believed to be true and comparing it to another
statement, the conclusion, to determine if it is true. The underlying skills of this category
include being pragmatic and logical, organizing thoughts linearly, using “if, then”
statement or graphic diagrams (Halpern, 2014). In general, the skills in this category are
those that are needed to comprehend and defend against the complexities of everyday
language. Language and thinking are integrated concepts therefore the skills required to
reason verbally have a reciprocal relationship where thoughts determine the language
used to express them, and the language used forms the thoughts. For example, verbal
reasoning includes the ability to discriminate between deductive and inductive reasoning,
the ability to avoid the fallacies of confirming the consequent and denying the antecedent
of a statement or argument, and understanding the difference between truth and validity
among several other skills. Analysis of arguments on the other hand, focuses on the skills
of making assumptions, qualifiers and counterarguments.
An argument, Halpern (2012) defined as an attempt to convince another that a
specific conclusion is true based on the rationale that is presented. Arguments must have
at least one premise (reason) and one conclusion and tend to have structures that can be

44
identified and diagramed. Although it takes a lot of effort to diagram an argument,
knowing the tools that facilitate analysis of arguments such as, identifying conclusions,
rating the quality of reasons, and determining the overall strength of an argument, are
vital to the success of teaching critical thinking in higher education (Halpern, 2014).
Whereas verbal reasoning and analysis of arguments tend to be cognitive skills
that take a great deal of effort to accomplish, thinking as hypothesis testing is a bit easier
because much of human thinking is like the scientific method of hypothesis testing
(Halpern, 2012). Like scientific reasoning, the skills needed for thinking as hypothesis
testing include the gathering of observations or information, formulating beliefs, and then
using the information collected to determine whether or not the beliefs are confirmed.
Similar to thinking as hypothesis testing, the fourth category of critical thinking skills that
should be explicitly taught, is using likelihood and uncertainty to make judgments.
Understanding probabilities and how they can affect the likelihood of an outcome
or, in uncertain situations, the unlikelihood of an outcome is the fourth essential
component of critical thinking skills. Probability in this context refers to the number of
ways a particular outcome or belief can occur divided by the number of possible
outcomes (Halpern, 2012). The skills in this category Halpern (2012) explained, require
formal instruction on concepts such as regression to the mean (another term for the
average) or conjunction errors (a misconception that the co-occurrence of two or more
events is more likely than the occurrence of the event alone). Despite the difficulty in
comprehending these skills, they are nonetheless necessary to the critical thinking process
because individual estimates of the likelihood of certain outcomes with unknown

45
frequencies tend to be inaccurate. By teaching how to calculate the likelihood of
uncertain outcomes, educators may reverse the counterintuitive nature of probabilities.
The fifth and last category of critical thinking skills is problem solving or making
a decision. There is a consensus among critical thinking scholars such as Paul, Elder, and
Halpern that critical thinking skills are used in the process of making decisions or solving
problems (Halpern, 2012; Paul, 2013). This category of skills however emphasizes how
to phrase problem statements in a variety of ways, how to identify objectives,
alternatives, and use of precise criteria to make judgments among the alternatives.
In addition to these five critical skill categories, attitudes such as motivation and
self-efficacy, values and habits of mind all play important roles in critical thinking
(Tishman & Jay, 1993). Critical thinking scholars use the term disposition to refer to the
characterological attributes of individuals (Ennis, 1996). Ennis (1996) defined critical
thinking disposition as reflectively exercising a tendency to act on certain conditions.
Individual dispositions have a significant impact to determining whether or not
individuals use their thinking skills consistently and when it matters most (Facione,
2013).
According to Facione (2013) individual dispositions can be classified as either
positively or negatively influencing critical thinking results or as not having formed a
strong habit of mind one way or the other. Ambivalent tendencies toward the use of
critical thinking should be discouraged in a classroom environment since one of the
primary goals of education is to produce critical thinkers (Butler, 2012; Halpern, 2014;
Marin & Halpern, 2011). Under the sponsorship of the Committee on Pre-College
Philosophy of the American Philosophical Association, a panel of 46 experts conducted a

46
strict-method Delphi research project which resulted in a comprehensive
conceptualization of critical thinking which included descriptions of both positive
attributes towards critical thought and negative habits of thought (Facione P., Facione N.,
& Giacarlo, 2000; Zhang, 2003). Positive dispositions or attributes of critical thinking
outcomes included inquisitiveness, judiciousness, truth-seeking, open-mindedness,
analytical, systematic, and confidence in reasoning. The bad habits of mind are the
antithesis of the positive attributes and include intellectual dishonesty, intolerance,
inattentiveness, haphazardness, indifference, mistrustfulness of reason, and having
simplistic reasoning (Facione et al., 2000; Zhang, 2003).
The following section highlights three positive dispositions that tend to be
considered antecedents to critical thinking: engagement, cognitive maturity, and
innovativeness (Khandaghi & Pakmehr, 2012; Mathews & Lowe, 2011; Ricketts & Rudd,
2004). The intent is to synthesize individual attributes that may be necessary in order for
any critical thinking teaching method to be effective.
Engagement. The term engagement as it relates to critical thinking disposition
refers to an individual’s inclination towards seeking opportunities to use reasoning or
anticipating situations that require reasoning (Khandaghi & Pakmehr, 2012; Ricketts &
Rudd, 2004). Having an engaged disposition also means an individual is confident in
their reasoning abilities (Khandaghi & Pakmehr, 2012; Ricketts & Rudd, 2004).
Whereas Khandaghi et al. (2012) uses the term engagement to describe a propensity for
recognizing opportunities for critical thinking, Mathews and Lowe (2011) use the term
sensitivity for critical thought to mean one’s capacity to apply critical thinking to
different situations. It is one’s sensitivity, according to Mathews et al. (2011), that can

47
initially trigger the processes for critical thinking because the individual is disposed to be
“vigilant for recognizing situations that might be enhanced by instantiating one’s skills
and strategies for critical thinking” (p. 61). A person disposed to engagement or
sensitivity for critical thinking does not necessarily mean that person will capitalize on
that disposition at every given opportunity. That individual also should have the
cognitive maturity to act on his/her sensitivity.
Cognitive Maturity. A person with cognitive maturity is one who is aware of the
complexity of problems, is open to other perspectives, and is cognizant of their own
biases (Ricketts & Rudd, 2004). Mathews and Lowe (2011) used the term inclination to
describe the engagement in mental behavior that includes critical thinking (cognitive
maturity). In a study exploring students’ critical thinking dispositions, Khandaghi and
Pakmehr (2012) noted that the implemented curricula, including the contents of
textbooks, did not enhance students’ cognitive maturity abilities. This knowledge may be
beneficial towards future research examining the effectiveness of teaching critical
thinking skills as skills alone do not ensure critical thinking is consistently applied
beyond the classroom. Teachers need to nurture the cognitive maturity of their students
by encouraging their open-mindedness.
Innovativeness. An innovative disposition refers to a predisposition to be
intellectually curious and possess a desire to know the truth (Ricketts & Rudd, 2004).
Innovativeness can also mean an ability to engage in cognitive behavior associated with
critical thinking (Matthews & Lowe, 2011). The innovativeness of participants in a
Khandaghi and Pakmehr (2012) study revealed that students should be actively engaged
in real problem-solving situations and share their experiences with their instructors. A

48
suggested method for improving innovativeness included the use of concept maps.
According to Khandaghi and Pakmehr (2012), concept mapping allows students to
visualize the relationships between differing beliefs and concepts which would require
analysis, organization, and evaluation. All of these processes relate to critical thinking.
Educators must model the positive dispositions needed for critical thinking in
order for students to become motivated to emulate their role model. To succeed in
student learning of critical thinking, educators need to teach critical thinking skills and
nurture student’s internal motivation to use those skills but setting the example. Simply
learning the skills of critical thought does not guarantee that those skills will be
consistently applied. This is why educators should include both critical thinking skills
and dispositions training into professional development as well as student curricula.
Effects of Diversity on Critical Thinking
The ability to think critically is influenced by individuals’ dispositions, the
learning experience, and the instructors’ ability to organize and deliver critical thinking
content (Deskissa, Liang, Behera, & Harkness, 2014). There is, however, a dearth of
research on the effects of student diversity on critical thinking (Loes, Pascarella, &
Umbach, 2012). Exposure to diversity experiences might foster the development of
critical thinking (Loes et al., 2012). A learner’s academic background, ethnicity, and
gender, should be taken into account when designing curriculum to foster critical
thinking (Deskissa, Liang, Behera, & Harkness, 2014).
Although Loes et al. (2012) uncovered no significant general effects of diversity
experiences on critical thinking, they did conclude that White students versus students of
color were more involved in interactional diversity activities. The rationale for this

49
difference, Loes et al. concluded, was that approximately 80% of While students in the
study sample attended secondary schools that were composed totally or mostly of White
students and that when the White participants attended their first year of college it was
the first real opportunity to interact with students of different racial and cultural
backgrounds. This is contrary to the participants of color who responded that they had
encountered diverse student bodies throughout their secondary education. Additionally,
Loes et al., 2012 uncovered that students who entered college with different levels of
academic achievement scores (as measured by ACT or ACT equivalent tests) tended to
be White and had a marginally significant positive net influence on the development of
critical thinking skills as compared to students of color. Although race/ethnicity play a
role in the ability to develop critical thinking skills (Loes et al., 2012), the effect of
gender on critical thinking has not had a significant effect on learner’s critical thinking
skills (Fahim & Barjesteh, 2012; Gok, 2014).
Transferability of Critical Thinking Theory to Practice
The transfer of critical thinking theory to practical application can be explained by
how individuals transmit their acquired knowledge of critical thinking concepts from one
context to another context with shared characteristics (Ya-Ting, Yung-Hsin, & Cowan,
2014). The entire enterprise of having critical thinking as an outcome of education would
be of little value if the thinking skills taught would only be used in the classroom. It
would be ideal for critical thinking skills to be used in multiple situations and contexts
such as being able to recognize weak arguments by analogy, faulty logic, unrealistic
promises of wealth, beauty, etc. whenever they are encountered (Halpern, 2014). The

50
best way to promote this transfer is with the conscious and deliberate use of skills that are
learned in a wide variety of contexts (Halpern, 2014).
A close relationship exists between transfer of learning and problem-solving,
since transfer of learning generally occurs when previous knowledge is applied to solve a
problem in a new situation (Ya-Ting, Yung-Hsin, & Cowan, 2014). To become a better
thinker it is important to recognize when critical thinking is needed and then be willing to
engage in the effortful process of using previous knowledge and skills to address
problems (Halpern, 2014). Transferring critical thinking skills among different contexts
tends to be difficult because there are no obvious cues in the context to trigger the recall
of the thinking skill that is required. For this reason, it is important for educators to direct
learning so that the skills of critical thinking are learned in a way that will facilitate
students’ recall in novel situations. It is the actions learners take that determines what
gets learned (Halpern, 2014). For example, Halpern listed several tasks that are designed
to help with the transfer of critical thinking skills. Some of these tasks included drawing
a diagram or other graphic display that organizes information, or stating what the learner
believes is the problem in at least two different ways. Actively engaging the learners’
thought processes in multiple ways helps in the transfer of learning. Task-orientation is
not the only aspect of learning transfer that is important for critical thinking to be long
lasting, individual dispositions, skills, and the learning environment also have an effect
on the application of critical thinking.
The success of learning transfer is impacted by the learners’ dispositions and
skills, by the features of the task at hand, and by the learning environment (Halpern,
1998; Pitkaniemi & Vannienen, 2012; Ya-Ting, Yung-Hsin, & Cowan, 2014). At the

51
root of critical thinking is an individuals’ attitude or disposition to reflect on his/her own
thinking and recognize when a skill is needed then be willing to apply that critical
thinking skill (Halpern, 1998). Halpern (1998) further defined critical thinking
dispositions as an individuals’ willingness to apply skills such as: (a) verbal reasoning,
(b) argument analysis, (c) skills in thinking as hypothesis testing, (d) analyzing the
likelihood and uncertainty of a situation, and (e) problem solving and decision making.
The transfer of knowledge on critical thinking skills is also impacted by the
features of the task at hand, of understanding how cause is determined, recognizing and
analyzing assumptions, and using analogies to solve problems. The features of the task at
hand include such factors as the learning objective, expected behavior after meeting the
objective, and opportunities to practice the task, as well as the learning environments are
all described by the design of curricula and the physical classroom environment
(Pitkaniemi & Vannienen, 2012; Ya-Ting, Yung-Hsin, & Cowan, 2014). It is clear that
for domains of learning such as critical thinking, the features of the task, learner, and
environment are all equally important. Additionally, the design of teacher professional
development programs and school placement must also be considered (organizational
features) for critical thinking transfer to take place from knowledge to application (Herlo,
2010).
Relating the learning transfer process to the theory of transformative learning
becomes increasingly important for effective integration of a new perspective (Ya-Ting,
Yung-Hsin, & Cowan, 2014). Ensuring teachers are engaged in and experienced with the
knowledge domain being transferred to students, particularly the critical thinking content
knowledge, facilitates transformative learning (Carawan, Knight, Wittman, Pokorny, &

52
Velde, 2011). Transformative learning emphasizes cognitive, affective, and behavioral
change (Carawan et al., 2011; Herlo, 2010). For transformative learning to be successful
the teachers’ responsibility to be a role model for critical thinking is crucial (Carawan et
al., 2011; Herlo, 2010). Additionally, the teachers’ personal experiences and critical
reflection play a key role to ensuring an environment that is conducive to transforming
the learners (Cassum et al., 2013; Marin & Halpern, 2011). Therefore the role of teacher
professional development in fostering both critical thinking skills and dispositions should
not be underestimated. This study examined the factors influencing the learning transfer
process by assessing students’ critical thinking skills and application of those skills.
Additionally the researcher observed the classroom environment and assessed the
student-teacher relationship as it related to the transformative learning theory.
Assessing the Application of Critical Thinking Skills
Much debate exists about the suitability of assessing critical thinking skills and
dispositions using standardized critical thinking tests (Butler, 2012; Hatcher, 2011;
Miller, Hall, & Tice, 2009). Controversy tends to stem from opponents of standardized
critical thinking tests who claim that the results of these assessments can be misleading
because the “effect size gains (defined as the mean gain expressed as a percentage of a
standard deviation) on critical thinking tests can vary significantly, even under ideal
conditions where the course, text, and teachers remain relatively the same throughout a
lengthy assessment period” (Hatcher, 2011, p. 29). Furthermore, empirical data reveals
that student gains are largely dependent on their instructor’s ability to teach to the test
(Hatcher, 2011). As a result, test outcomes may vary greatly depending on which test is
chosen as well as the teacher’s own critical thinking capabilities.

53
The following comparison of critical thinking tests may be helpful when
determining which of these four widely used tests (the California Critical Thinking Skills
Test, the Halpern Critical Thinking Assessment, the International Critical Thinking Basic
Concepts and Understanding Test, and the Watson-Glaser Critical Thinking Appraisal)
should be considered based on each tests’ reliability, validity, and whether or not the
assessment measures skills, dispositions, or both. Reliability and validity can be
influenced by its design as being either multiple-choice/ranking which rely on
recognition memory, or short-answer/essay relying on recall memory (Butler, 2012).
Table 1 below highlights the differences among the four critical thinking standardized
assessments.
Table 1.
Comparison between Four Standardized Critical Thinking Tests
Criteria CCTST HCTA ICTBCU WGCTA
Recognition  
Recall   
Reliability  
Validity    
Skills (S) or
Dispositions
(D)
S S / D S /D S
California Critical Thinking Skills Test (CCTST)
Halpern Critical Thinking Appraisal (HCTA)
International Critical Thinking Basic Concepts and Understanding (ICTBCU)
Watson-Glaser Critical Thinking Assessment (WGCTA)
From the comparison highlighted in Table 1, it is apparent that the Halpern
Critical Thinking Appraisal (HCTA) meets the most criterions and may be an optimum
instrument for future research on critical thinking assessment. The HCTA assesses
student’s critical thinking skills and dispositions by eliciting analysis and critique of 25
everyday scenarios in five subcategories (Butler, Dwyer, Hogan, Franco, Rivas, Saiz, &

54
Almeida, 2012; Butler 2012). It is a reliable and well-validated assessment primary due
to the method of scoring. In contrast, the Watson-Glasser Critical Thinking Assessment
(WGCTA) and California Critical Thinking Skills Test (CCTST) lack the reliability that
both the HCTA and the International Critical Thinking Basic Concepts and
Understanding (ICTBCU) assessment have (FCT, 2013; Miller et al., 2009).
Whereas the CCTST is a 34 item multiple-choice exam that can be completed in
50 minutes, the ICTBCU is a 100–item short answer exam that can be completed in 45
minutes. Both have been widely incorporated into empirical studies which examine their
reliability and validity. The CCTST evaluates many of the skills normally associated
with critical thinking: interpretation, argument analysis and appraisal, deduction, logical
puzzles, and induction however the developers of the ICTBCU took a more substantive
approach to critical thinking assessment by evaluating elements of thought which consist
of : 1. the analysis of thought, 2. the assessment of thought, 3. the dispositions of thought,
4. the skills and abilities of thought, and 5. the obstacles or barriers to critical thought
(FCT, 2013).
Summary
The literature reviewed included empirical studies on the phenomena of critical
thinking. Some studies concluded that critical thinking skills can be taught, learned, and
transferred (Cassum et al., 2013; Halpern, 1998; Marin & Halpern, 2011). Other studies
indicated that dispositions for critical thinking can be encouraged through pedagogical
interventions such as explicit critical thinking instruction (Drummond, 2012; Fahim &
Masouleh, 2012; Holley, 2009). While some researchers, such as Behar-Horenstein and
Niu (2011), Celuch, Kozlenkova, and Black (2010), and Marin and Halpern (2011) have

55
agreed that explicit critical thinking instruction increases students’ ability to think
critically, other scholars agreed that critical thinking skills can be evaluated through
standardized instruments (Behar-Horenstien & Niu, 2011; Khandaghi, Pakmehr, &
Amiri, 2011; Zimmerman & Land, 2014). Assessing the application of critical thinking
is problematic for several reasons. First, a consensual definition for the term critical
thinking remains vague leading to varying interpretations which exacerbate attempts to
cultivate and measure it (Behar-Horenstein & Niu, 2011; Cassum et al., 2013; Facione,
2013; Khandaghi, Pakmehr, & Amiri, 2011; Miller et al., 2009).
The lack of appropriate professional development programs available for
educators to adequately understand the concept of critical thinking further challenges
sound application of critical thought processes. Often educators do not even realize they
operate with this deficiency (Flores, et al., 2012). Compounding these problems is the
idea that critical thinking skills are the exclusive responsibility of the individual (Flores et
al., 2012). Attempts at assessing the efficacy of pedagogical approaches to critical
thinking have focused on the capabilities of the individual. Little empirical evidence
exists that assesses the critical thinking dispositions and, more importantly, the
application of critical thinking processes (Butler, 2012).
The evolution of the United States, from a manufacturing society where its
citizens were required to learn specific discipline-based knowledge, to a society where
knowledge and information dominate how work is conducted, drove the need for
innovation and critical thinking (Hodge & Lear, 2011). Although numerous instruments
are available that assess the critical thinking skills of individuals, the efficacy of critical

56
thinking pedagogies in transforming individuals’ critical thinking outcomes requires
further study.

57
Chapter 3: Research Method
This study addressed the problem of students, enrolled in higher education, who
continue to graduate with inadequate critical thinking skills (AAC&U, 2011; Carmel &
Yezierski, 2013; Flores et al, 2012; Khandaghi, Pakmehr, & Amiri, 2011). A great deal
of research exists on the extent to which critical-thinking pedagogies affect critical
thinking skill levels; however, the consistent application of critical thinking skills to real-
world situations remains an unanswered question. The purpose of this quantitative,
quasi-experimental study was to examine the change in the dependent variables (DV)
critical thinking skills (DV1) and application of critical thinking (DV2) among United
States Air Force Academy cadets based on whether: (a) they received explicit critical
thinking instruction which is the first independent variable (IV1), (b) they did not receive
explicit critical thinking instruction (IV2), (c) instruction was provided by a junior faculty
member (IV3), or (d) instruction was provided by a senior faculty member (IV4) as part of
the cadet’s introductory physics course.
The research questions for this study were designed to assess the importance of
exposing cadets to critical thinking instruction in efforts to enhance the application of
critical thinking skills to given problems and situations. Following are the research
questions for this study. The first question this study addressed was: what are the effects
of explicit critical thinking instruction on the critical thinking skill levels of a control
group as compared to a treatment group of freshman and sophomore cadets enrolled in
the Physics-110 course at the U.S. Air Force Academy? The null hypothesis for this
research question is that there is no difference between the critical thinking skills scores
of participants who complete Physics-110 with explicit critical thinking instruction as

58
compared to those who complete Physics-110 without explicit critical thinking
instruction. The second question this study addressed was: what are the effects of
explicit critical thinking instruction on the application of critical thinking skills of a
control group as compared to a treatment group of freshmen and sophomore cadets
enrolled in the Physics-110 course at the U.S. Air Force Academy? The null hypothesis
for the second research question is that there is no difference between the application of
critical thinking skills of participants who complete Physics-110 with explicit critical
thinking instruction as compared to those who complete Physics-110 without critical
thinking instruction.
This chapter describes the research method and design, elaborates on the
population and sample size of the study, and defines the instruments used to collect data.
The variables for this study are defined as well as the procedures that were used to
collect, process, and analyze the data. By highlighting the assumptions, limitations,
delimitations, and ethical assurances of this study, the researcher demonstrates the nature
of the research method and design.
Research Method and Design
This study incorporated a quantitative, quasi-experimental research method and
Non-equivalent Group Design (NEGD). Experimental designs are intended to test the
effects of a treatment (or intervention) on an outcome (Trochim & Donnelly, 2008). For
example, the purpose of this study was to examine the effect of explicit critical thinking
instruction, the intervention or treatment, on the critical thinking skills and application of
those skills in different situations (in other words, the outcome).

59
This study consisted of two treatment groups and two non-treatment groups. The
two treatment groups (20 participants in one group and 23 participants in the other group)
comprised of freshmen and sophomore cadets who received explicit critical thinking
instruction (using Just-in-Time (JiT), Peer-Instruction and concept-mapping teaching
approaches) within the Physics-110 course from either a senior or a junior faculty
member. The remaining two groups (20 participants in one group and 22 participants in
the other group) were freshman and sophomore cadets who did not receive explicit
critical thinking instruction from the same senior and junior faculty members who
provided the intervention to the treatment groups. The latter two groups are referred to as
the control groups. The dependent variables are the critical thinking skills scores (DV1)
and application of critical thinking skills scores (DV2), whereas the study consisted of
four independent variables: explicit critical thinking instruction (IV1), no critical thinking
instruction, (IV2), instruction provided by a junior faculty member (IV3) and by a senior
faculty member (IV4).
The dependent variable is described as the variable influenced by the independent
variable to produce a certain outcome. For example, the critical thinking skills and
application of critical thinking skills (DV1 and DV2 in Figure 1) depended on whether or
not participants were exposed to explicit critical thinking instruction (IV1) and (IV2)
taught by either a junior faculty member (IV3) or a senior faculty member (IV4). The
relationship between the independent and dependent variables are not exclusive as
demographic variables such as gender, academic performance, motivation,
metacognition, and ethnicity may influence the outcome of the dependent variable.
These variables can be referred to as mediating variables that may have either positive or

60
negative relationships to the dependent variables. Mediating variables however, were not
examined in this study due measurement limitations. The variables applicable to this
study are illustrated in the causal visual model in Figure 1 below.
Figure 1. Concept Map for Independent and Dependent Variables
Figure 1 is a concept map of four independent variables and two dependent variables.
Two research questions, indicated by Q1 and Q2, precede the dependent variables.
The research questions this study addressed are indicated by Q1 and Q2 on Figure
1 above. The first research question is intended to examine any differences, in terms of
critical thinking skills, between participants who received explicit critical thinking
instruction (treatment group) and those who did not (control group). The second research
question helped identify any differences, in terms of the ability to apply critical thinking
skills, between the control and treatment groups.

61
The NEGD is structured like a pretest-posttest randomized experiment, but the
researcher does not randomly assign participants to groups but rather the researcher
selects intact groups (Shaughnessy et al., 2014). For example, this researcher selected
freshmen and sophomores enrolled in the one of the two morning sessions of Physics-110
and one of the two afternoon sessions of the course to be the treatment groups. The
remaining morning and afternoon session groups were the control groups. In other
words, one of the two morning session groups was a treatment group while the other
morning session group was a control group. Likewise, the afternoon sessions were
categorized as one treatment and one control group.
The quantitative NEGD was selected because it best quantified the level of critical
thinking skills and abilities to apply critical thinking of each treatment and control
groups. The comparison of assessment results among the treatment and control groups
may provide a better understanding of whether the explicit form of critical thinking
instruction within the Physics-110 course produces a transformative effect in developing
critical thinkers who can consistently apply their skills to real-world situations.
Furthermore, comparing the scores of control groups with those of treatment
groups may provide a more complete and pragmatic understanding of the problem and
help identify the most optimum pedagogical solution for addressing the issue of college
students continuing to graduate without having skills necessary to apply critical thinking
in various situations. The notation for the quasi-experimental design is as follows, where
N represents the non-equivalent groups, O1 and O2 are the observations of pre- and post-
tests, respectively. The treatment, depicted by the letter X, represents critical thinking

62
instruction within Physics-110 course. The dashed line indicates that the treatment and
control groups were not formed by assigning participants randomly to conditions:
N O1 X O2
------------------------
N O1 O2
A quasi-experimental design for this study was the optimum research design
choice because a qualitative method would not be sufficient to capture the necessary
empirical data that would appropriately address the complex phenomenon of consistently
applying critical thinking skills to address real-world situations. Other quantitative
methods were not appropriate because the selection of participants for this study were not
completely random.
Population
This research involved the United States Air Force Academy located in Colorado
Springs, Colorado. The Academy embraces an integrated curriculum; however the
Physics-110 course is a discipline-based course that was offered nine times during the
2015 fall semester and taught by five different faculty members. The total United States
Air Force Academy student population during the 2015-2016 academic year was
approximately 4,000 enrolled cadets. The Office of the Registrar randomly assigned
freshmen and sophomore cadets to nine offerings of Physics-110 during the fall 2015
semester. Approximately 218 (total population) freshman and sophomore cadets were
enrolled in the Physics-110 course during the fall 2015 semester. Each class size
consisted of approximately 20-23 cadets. Each class of 20-23 freshmen and sophomore
cadets was representative of beginning physics students at the Academy. The

63
participants were 85 freshman and sophomore cadets enrolled in two morning and two
afternoon sessions of Physics-110 course.
Sample
The intended sample included 85 participants all enrolled in Physics-110 during
fall semester 2015 who received course credit. The intended sample size (n=85) was
determined using an a priori power analysis. The power analysis took into account the
Non-equivalent Group Design to calculate the power. Without pilot data, the expected
effect is not known therefore a literature search for a similar studies was conducted to
reveal large effect sizes in multiple populations (Butler, Dwyer, Hogan, Franco, Rivas,
Saiz, & Almeida, 2012; Butler, 2012; Marin & Halpern, 2011).
A multi-stage sampling procedure (called clustering) was used because it was
impractical to compile a list of the elements composing the population (Creswell, 2009).
For example, the researcher selected two of the morning sessions and two of the
afternoon sessions of this course to participate in this study. The researcher then selected
one of the two morning and one of the two afternoon sessions of the Physics-110 course
to be the treatment groups. The other morning and afternoon session groups were
identified as the control groups. The identities of the individuals within these clusters, or
groups, were coded to ensure anonymity.
Each of the nine sessions of Physics-110 offered consisted of 20-23 participants
who were selected at random by the Office of the Registrar. A total of 85 participants
(four groups of 20-23 participants) were the intact groups. An intact group is one that is
naturally formed (e.g., a classroom, an organization, etc.). By randomizing the selection
process, a representative sample from the population provided the ability to generalize to

64
the rest of the population. Demographic data such as gender, ethnicity, age, and
composite academic scores, were collected from the Office of the Registrar’s database.
This data was used to stratify the population before selection of the sample to reflect the
true proportion in the population of individuals with certain characteristics such as
entrance exam scores.
From the original sample of n=85, data were excluded from 49 participants
resulting in an actual sample size of n=36. Data was excluded primarily as the result of
participants not being able to access the post-test due to information technology problems
encountered at the United States Air Force Academy. Data was also excluded because
not all participants were available to complete both pre- and post- assessments on the
dates they were administered. As a result, the actual sample size of n=36 complete data
sets were obtained for this study. Of the actual sample size of n=36, 17 were assigned to
control groups and 19 were assigned to treatment groups.
Instruments
The instruments for this study were the Halpern Critical Thinking Assessment
(HCTA) and the Real-World Outcomes (RWO) inventory. Both the HCTA and the
RWO inventory have been checked for inter-rater and measurement reliability as well as
for content, construct, and criterion validity by its author, Dr. Diane F. Halpern and the
Vienna Test System Corporation. Neither instrument were compromised because access
to these assessments are controlled by login and password protections.
The HCTA measured participants’ critical thinking skills (DV1). The five
dimensions of critical thinking (verbal reasoning, argument analysis, thinking as
hypothesis testing, likelihood and uncertainty, and decision making and problem solving)

65
were scored from forced choice responses (measures recognition ability) on the Halpern
Critical Thinking Assessment (Halpern, 2012). Points were assigned to each of the five
dimensions of critical thinking for a maximum total point score of 194 points. The total
forced choice item scores ascertained the respondents’ critical thinking skills level
(HCTA, 2013). A percentile rank (PR) of 25 to 75 is a moderate critical thinking skills
score, a PR less than 25 is considered a low score, and a PR greater than 75 indicates
strong critical thinking skills (HCTA, 2013).
The quality of respondent’s decisions in numerous real-world situations (referred
to in this study as the application of critical thinking skills or DV2) were measured using
a modified version of the Butler Real-World Outcome inventory. The modified open
source inventory consisted of 25 item sets correlating to negative life events (outcomes of
critical thinking application) and antecedent decisions (Butler, 2012). Respondents were
asked to specify whether they made the decision(s) listed in the inventory of 25 decisions,
within the last six months, and whether they experienced a negative life event as a result
of making the decision. There were 28 possible negative life events listed in the
inventory. Respondents used a dichotomous rating scale (yes/no) to answer each
question on the inventory. The total Real World Outcomes Inventory (RWO) score is the
proportion of the negative outcome events the respondent reported. This study examined
both parts of the RWOI, the decision the respondents made were captured in RWOI part
A and the resulting outcome the respondents experienced as a result of the decision, were
captured in RWOI part B. The total RWOI score was computed by dividing the number
of negative outcomes experienced by the respondent by the total possible negative
outcomes on the inventory (Part B RWOI score/28 possible outcomes). The scores on

66
the RWOI range from 0 to 1 with lower scores indicating fewer negative real-world
outcomes and scores that are closer to 1 in this self-report inventory indicate less than
optimal application of critical thinking skills in real life settings (Butler, 2012). Both
parts A, the critical thinking decision and B, the outcome or application of critical
thinking, were analyzed in this study.
Operational Definition of Variables
Explicit critical thinking instruction (IV1). Explicit critical thinking instruction
is the first independent, nominal variable. Both faculty participants taught one of the two
the Physics-110 morning and afternoon sessions using the Just-in-Time Teaching, Peer-
Instruction and concept mapping approach (explicit critical thinking instruction) to the
treatment groups. Explicit critical thinking instruction is the intervention/treatment used
in this study.
No critical thinking instruction (IV2). No critical thinking instruction is the
second independent, nominal variable. Both faculty participants taught one of the two the
Physics-110 morning and afternoon sessions using Just-in-Time Teaching but did not
emphasize concept mapping as an instructional approach (no explicit critical thinking
instruction) to the control groups.
Junior Faculty Member (IV3). The junior faculty member is the third
independent, nominal variable.
Senior Faculty Member (IV4). The senior faculty member is the fourth
independent, nominal variable.
Critical thinking skills (DV1). A critical thinking skill is the first dependent,
continuous variable. The Halpern Critical Thinking Assessment, using a forced choice

67
response method to measure how well participants can recognize a rational, sound
response. Twenty-five scenarios are presented whereby participants select answers from a
list of possible alternatives. A calculated score of >75 indicates strong critical thinking
skills whereas a calculated score of 25-75 indicates average critical thinking skills.
Application of critical thinking skills (DV2). Application of critical thinking
skills is the second dependent, continuous variable. A modified version of the Butler
Real-World Outcomes inventory consisted of 25 item sets correlating to negative life
events and antecedent decisions (Butler, 2012). Respondents were asked to specify
whether they experienced a negative life event and the decision(s) preceding it within the
last six months using a dichotomous rating scale (yes/no). The total score obtained in this
measure were divided by the numbers of possible negative life events. High scores in
this self-report inventory are indicative of less than optimal application of critical
thinking skills in real life settings (Butler, 2012).
Data Collection, Processing, and Analysis
Both Northcentral University and the United States Air Force Academy’s IRBs
approved this study prior to the start of the fall 2015 semester (August 17, 2015).
Participants (n=85) from the United States Air Force Academy were selected based on
the course schedule identifying two morning and two afternoon sessions of Physics-110
course that were taught by the selected junior and senior faculty participants. All
participants were selected free from any biases, stereotypes, and prejudices and those
who opted out of taking the assessments, did not suffer any victimization or were
penalized. Participants in each session of the Physics-110 course met three times a week
and two times a week on an alternating basis for six weeks. The duration of each class

68
session was 53 minutes. To ascertain the baseline demographic data of the treatment and
control groups, information regarding academic performance, gender, and ethnicity were
collected from the Office of the Registrar and transferred to an MS Excel file which were
subsequently organized, coded, and transferred to the Statistical Package for the Social
Sciences (SPSS) software for analysis.
Respondents completed the Halpern Critical Thinking Assessment (HCTA) and
the Real-World Outcomes Inventory (RWOI) online as part of the Physics-110 course
syllabus. Specifically, the pre-tests, measuring critical thinking skills and the application
of critical thinking were administered on September 28, 2015 to all four groups (two
control and two treatment groups) and again after the intervention was completed on
November 10, 2015. This allowed for comparison of critical thinking skill levels and
application of critical thinking before and after receiving explicit critical thinking
instruction.
Administration of both measures took approximately 45 minutes to complete.
Only willing participants were granted access to the Critical-Thinking assessments link.
This ensured ethical consideration were made for data collection as only willing
volunteers who acknowledge their complete understanding of the purpose and intent of
the assessments were allowed to complete the assessments.
In addition, archival data such as participant academic performance, gender, and
ethnicity were obtained from the Office of the Registrar to stratify the population before
selecting the sample to reflect the true proportion in the population of individuals with
certain characteristics. Scores for each of the two online assessments were exported to
SPSS software for analysis. The researcher screen the scores for each of the two online

69
assessments to determine whether the participants that did not complete some of the four
assessments (and thus had several missing data values) were significantly different from
the participants that did not miss any of the assessments or only missed one assessment.
Descriptive statistics, of the sample and the study variables, include the
frequencies and percentages for the variables measured using a nominal or ordinal scale
whereas the ranges, means, and standard deviations are included for the variables
measured using an interval or ratio scale.
The first hypothesis was tested using a 2 x 2 mixed-analysis of variance
(ANOVA) procedure because the HCTA was measured twice; thus, this was the within-
subjects variable. There were two instruction groups, control and explicit instruction; this
was the between-subjects variable. To determine whether change in HCTA differed
across the instruction groups, the interaction between time and instruction was evaluated.
The second hypothesis was tested using a 2 x 2 x 2 mixed ANOVA procedure. As
in the first hypothesis, RWOI was measured twice; thus, this was the within-subjects
variable. Similarly, there were two instruction groups; this was the first between-subjects
variable. The second between-subjects variable was faculty (i.e., junior vs. senior). To
determine whether change in RWOI differed across the instruction groups, the interaction
between time and instruction was evaluated. To determine whether change in RWOI
differed across faculty, the interaction between time and faculty was evaluated.
Analysis of the data took place within the SPSS. This program allows the
researcher to produce graphical displays and statistical analysis however the
interpretation of the results remained with the researcher.

70
Assumptions
Nearly every statistical test relies on some underlying assumptions, and they are
all affected by the mix of data available. The following assumptions were made for this
study:
1. Participants will exhibit the population characteristics and their attitudes
towards critical thinking instruction within an introductory physics course are typical of
students at other universities and colleges.
2. Participants will engage in peer-instruction, concept-mapping, and provide
honest responses to the online assessments.
3. Just in Time Teaching technology will be operational throughout the study to
ensure pre-class assignments can be assessed by faculty participants prior to start of class.
4. Participants will complete both assessments on the pre-intervention and post-
intervention dates.
A quasi-experimental design was selected for this study because the outcomes of
a pre- and post- test on treatment and control groups would be compared to examine
whether or not the treatment has a positive effect on the outcome. Selection to receive
the treatment was not completely randomized; instead selection of the treatment groups
was based on participant assignment to either the morning or afternoon sessions of an
introductory physics course taught by two different faculty members at the United States
Air Force Academy.
Due to the fact the scores from pre- and post- test of multiple groups were
compared, this study was a Non-equivalent Group Design using a 2 x 2 mixed ANOVA

71
procedure for HCTA results and a 2 x 2 x 2 mixed ANOVA procedure for RWOI results
during statistical analyses.
Limitations
Although the quasi-experimental NEGD is the most optimum choice of research
design for this study, it did have limitations. For example, threats to internal validity
occur due to a) the non-random nature of participant selection and history, b) selection
and maturation, c) selection and instrumentation, and d) threats due to differential
statistical regression (Shaughnessy et al., 2014). Additionally, the duration of the
intervention was limited. A study conducted by Renaud and Murray (2008) suggested
that single interventions longer than 12 weeks tended to be more effective in increasing
participants critical thinking abilities than shorter interventions. This study implemented
the intervention for a period of six weeks. Expecting noticeable changes in thinking
abilities after such a brief exposure may not be as realistic as waiting for changes to
accumulate after a long period of exposure to explicit critical thinking instruction. For
this reason future instructional interventions of critical thinking should be (1) specifically
designed and (2) implemented through a long period of time. Expanding the scale and
duration of the intervention to be used in this study, could improve its effectiveness;
however time constraints prohibited the expansion of the intervention. Future
interventions could be incorporated into more courses, which could form a sequence that
provides students with constant exposure to explicit critical thinking instruction.
The lack of random assignment, and the potential nonequivalence between the
groups, complicates the statistical analysis of the nonequivalent groups design.
Individuals in the treatment and control groups may not have been equivalent on all

72
important characteristics such as academic performance, motivation levels, prior critical
thinking exposure, etc. A pre-test was therefore administered to both the control and
treatment groups to assess their similarity on the dependent variable, critical thinking
skills and the application of those skills. By assessing the skills and ability to apply
critical thinking of control groups, the research has a comparison of similarities or
differences between the control and treatment groups.
The findings from this study may be generalizable to the United States Air Force
Academy cadets because they have already met strict entrance requirements as compared
to other institutions of higher education. The scope of the study was delimited to only
include four of the possible nine sessions of Physics-110 offered during the fall 2015
semester.
Delimitations
A quasi-experimental design for this study was the optimum research design
choice because a qualitative method would not be sufficient to capture the necessary
empirical data that would appropriately address the complex phenomenon of consistently
applying critical thinking skills to address real-world situations. Other quantitative
methods were not appropriate because the proposed selection of participants for this
study were not completely random.
Ethical Assurances
The researcher complied with the standards for conducting research during data
collection, analysis, as well as during the data interpretation and reporting phases of this
study. Respect towards participants and the site for this research was maintained and
assured by the United States Air Force Academy’s IRB. The IRB required assessment of

73
the potential for risk, such as physical, psychological, social, economic, or legal harm
(Creswell, 2009). In addition to obtaining the Academy’s IRB approval for this study,
the researcher also received approval to conduct the study from Northcentral University’s
IRB. The researcher developed an informed consent statement and emailed the consent
statement and link to the instruments/assessments acknowledging that participants’ rights
will be protected throughout the study. Only willing participants were granted access to
the Critical-Thinking assessments link. This ensured ethical considerations were taken
into account for data collection as only willing volunteers who acknowledge their
complete understanding of the purpose and intent of the assessments were allowed to
complete the assessments.
Although the researcher is not assigned as a faculty member at the United States Air
Force Academy, the Center for Physics Educational Research designated the researcher
as a visiting scholar on January 8, 2015. The designation continued throughout the
duration of this study to assure access to cadet areas and facilitate development of the
study intervention, the explicit critical thinking lesson plan. As a visiting scholar, the
researcher established a trusting and respectful relationship with study participants before
the study began to detect any marginalization. Additionally, the two faculty participants
conveyed the researchers’ purpose and central intent of the study to student participants
at the beginning of the fall 2015 semester.
The researcher sought permission to conduct this study from the Center for
Physics Educational Research on January 8, 2015. Appendix C to this proposal contains
the official Permission Letter. On July 1, 2015 the Academy’s IRB exempted this study
from full review in accordance with 32 CRF 219.101(b), which states:

74
(1) Research conducted in established or commonly accepted educational
settings, involving normal educational practices, such as (i) research on
regular and special education instructional strategies, or (ii) research on
the effectiveness of or the comparison among instructional techniques,
curricula, or classroom management methods.
(2) Research involving the use of questionnaire procedures, interview
procedures or observation of public behavior, unless (i) information
obtained is recorded in such a manner that human subjects can be
identified, directly or through identifiers linked to the subjects; and (ii)
any disclosure of the human subjects’ responses outside the research could
reasonably place the subjects at risk of criminal or civil liability or be
damaging to the subjects’ financial standing, employability, or reputation.
The Academy’s IRB approval notification is contained within Appendix D of this
proposal. Notwithstanding the approval to continue from the Academy’s IRB, the
researcher also was granted approval from Northcentral University’s IRB to
proceed with data collection.
The anonymity of participants was protected by disassociating names
from responses during the coding and recording process. The data from the
online assessments did not have identifying information except students’ unique
identifier (email and section number), which was needed to determine who
completed the assessments and link outcomes of the assessments to academic
performance data. In addition, the researcher refrained from sharing student

75
assessment data with instructors during the study. All computer data was stored
on password-protected computers, and all paper data was stored behind two locks.
Written and digital hard drive records were available to only the
researcher and participating staff and were treated as academic testing material.
This means it was stored in a secure location behind two locks (i.e., office door
and file cabinet) when not in use during the study and for three years after the
study is complete. Any references used in publication to demonstrate specific
cases from the study will use pseudonyms or codes names (i.e., “Dr. Taylor” or
“Student 25”). The researcher thanked all participants (students and faculty) for
being a part of the study, and communicated the background, design, and
procedures of the study as well as its benefits to the physics education research
community and future physics students.
Summary
This study investigated the effects between explicit critical thinking instruction
and the ability to apply critical thinking skills to real-world situations. Quantifying
changes in these areas may facilitate an understanding for whether critical thinking
instruction within a specific discipline at the U.S. Air Force Academy has a
transformative effect in developing graduates who can apply critical thinking in a
consistent manner. A quasi-experimental, Non-equivalent Group Design was used on a
population (218) of United States Air Force Academy cadets enrolled in an introductory
physics course. The sample size of 85 cadets was determined using a power analysis and
two instruments, the HCTA and RWO inventory were administered to collect data.
Factorial mixed ANOVA analyses were conducted.

76
Quantifying any changes in these areas may facilitate an understanding for
whether critical thinking instruction within a specific discipline at the U.S. Air Force
Academy has a transformative effect in developing graduates who can apply critical
thinking in a consistent manner. Knowledge gained from the results of this study may
address the problem of why, despite all of the emphasis placed on critical thinking as a
goal of higher education, graduates still cannot reason well (Behar-Horenstien & Niu,
2011; Butler, 2012; Cassum, Profetto-McGarth, Gul, Dilshed, & Syeda, 2013; Davies,
2011; Lloyd & Bahr, 2010; Miller et al., 2009).

77
Chapter 4: Findings
The purpose of this quantitative study was to examine transformational effects, if
any, in the critical thinking skills and application of critical thinking of freshman and
sophomore cadets based on whether or not they received explicit critical thinking
instruction. The target population was U.S. Air Force Academy freshman and
sophomore cadets enrolled in Physics-110 Introductory Physics course during the fall
2015 semester. Participants from this target population completed two surveys, one
measuring critical thinking skills, and one measuring the application of critical thinking
in real-world scenarios.
This chapter first reviews descriptive statistics of the actual sample, presents the
results of the two assessments, the HCTA and RWOI, and answers each of the research
questions and hypotheses based on relevant statistical analyses. Preliminary analysis
indicates there is no statistically-significant positive correlation between explicit critical
thinking instruction and critical thinking skills and application of critical thinking in real-
world situations.
Results
The actual statistical power, effect size, and sample size varied from the intended
due to missing data completely at random. Multiple combinations of effects were
analyzed using G*Power 3.1.9.2 (Faul, Erdfelder, Buchner, & Lang, 2009) to determine
the actual statistical power for the variables of this study. Table 2 summarizes the
statistical power between the pre and post scores of both the HCTA and RWOI
assessments given an alpha of .05, a sample of 26 for the HCTA scores and of 36 for the
RWOI scores.

78
Table 2.
Actual Statistical Power
Effects Sample Size Correlation Effect Size Power
HCTA
Instruction 26 .82 .21 .20
Time 26 .82 .22 .94
Interaction between
Instruction& Time 26 .82 .07 .21
RWOI_A
Faculty 36 .47 .00 .05
Time 36 .47 2.48 1.00
Interactions between
Instruction & Faculty 36 * .22 .26
Time & Instruction 36 .47 .03 .06
Time & Faculty 36 .47 .03 .06
3-way interaction 36 .47 .16 .29
RWOI_B
Faculty 36 .45 .18 .23
Time 36 .45 2.84 1.00
Interactions between
Instruction & Faculty 36 * .03 .05
Time & Instruction 36 .45 .13 .21
Time & Faculty 36 .45 .03 .06
3-way interaction 36 .45 .08 .10
* Degrees of freedom of 1
alpha = .05
Recruitment. Recruitment of participants was not applicable to this study
because the USAF Academy falls under the IRB exemption category 32 Code of Federal
Regulations [CFR] 219.101(b). In 1991 the Federal Policy for the Protection of Human
Subjects or the “Common Rule” was published and codified in separate regulations by 15
Federal departments and agencies. The Department of Defense, for which the USAF
Academy is part of, has a separate regulation which includes in its chapter of the Code of

79
Federal Regulations [CFR] section numbers and language that are identical to those of 45
CFR part 46, subpart A.
Physics-110 is a mandatory course for all cadets; therefore there are no exclusion
criteria to determine non-eligibility. Participants (n=85) were selected based on the
course schedule identifying two morning and two afternoon sessions of Physics-110
course that were taught by the selected junior and senior faculty participants. Selection
of all four intact groups was made prior to the fall 2105 semester start date of August 6,
2015. A diagram of participants’ progress through the phases of this study is shown in
Figure 2 below.
Figure 2. Diagram of Participants’ Progress through Study Phases
Figure 2 captures the actual sample size used in this study by treatment and control
groups and assessment type.

80
Participant characteristics. Although 85 cadets were selected as the intended
sample size, three cadets did not complete the Physics-110 course during the fall 2015
semester. Therefore, only the data of 82 participants who completed the course were
included in the analysis. Additionally, due to inaccessibility to the online critical
thinking assessments, coupled with the unavailability of all participants in the sample to
complete both assessments on a pre- and post- basis, the response rate was 31.7 percent.
Only the cadets without missing data and those missing just one assessment score were
included in this description of the sample (n=36). As shown in Table 2, the sample
consisted of more males (63.9%) than females (36.1%). Approximately half of the cadets
were Caucasian (44.4%); the least represented ethnic group was American Indian (2.8%).
Only one participant in the study was a sophomore, all other participants were freshmen.
Table 3 illustrates the demographic frequencies and percentages for the sample.
Table 3.
Demographic Characteristics of Participants (N = 36)
Variables n %
Gender
Male
Female
Ethnicity
Caucasian
Black
Hispanic
Asian
American Indian
Unknown
13
23
16
6
2
5
1
6
36.1
63.9
44.4
16.7
5.6
13.9
2.8
16.7
The sample was assigned to either treatment (explicit critical thinking instruction) or
control groups taught by either a junior or senior faculty member as illustrated in Figure 3
below.

81
Figure 3. Two by Two Factorial Design
Figure 3 illustrates the major independent variables by subdivision of factors in a two by
two factorial design.
The groups were referenced by class number; either M3A, M3B, M6A, or M6B. The
number of cadets enrolled in each class designation is shown within parenthesis in Figure
3 above. Of the total intended sample size, 50.6% were assigned to the treatment groups,
taught by either a junior or senior faculty member.
Missing data. The propensity for data points to be missing was independent of
both observed and unobserved data, meaning missing data was completely at random
(MCAR). For example, 60 of the 85 intended participants completed the pre-HCTA, but
only 43 completed the pre-RWOI on the pre-intervention assessment date. Additionally,
on the post-intervention assessment date, only 32 participants completed the post-HCTA,
yet 65 participants completed the post- RWOI. Merely 12.2% of the cadet sample missed
only a single assessment, the post-HCTA. Nearly half of the sample, 48.8%, missed
between two to five assessments. Substitution was applied to the post-HCTA for seven
cases in control group M6A because there were zero complete data sets within this group.
In addition, substitution was applied to three cases in treatment group M3A because

82
participants without missing data did not differ significantly from those with missing data
in terms of academic performance, F(3,78) = .08, p = .973. Missing data was addressed
in statistical analysis shown in Table 4 below. Pre-test scores were used as post-test
scores for these ten cases.
Table 4.
Frequencies and Percentages for Participants with Missing Data (N = 82)
Levels of Missing-ness n %
No missing data
Missing data from one test
Missing data from two to five tests
Missing data from all six tests
26
10
40
6
31.7
12.2
48.8
7.3
The proportion of males to females did not differ significantly across the levels of
missing-ness, χ2
(3) = 1.88, p = .597. The proportion of ethnic categories also did not
differ across levels of missing-ness, χ2
(15) = 17.85, p = .271.
Descriptive information of independent variables.
Explicit critical thinking instruction versus no explicit critical thinking
instruction (IV1 and IV2). The intervention/treatment utilized within this study consisted
of explicit critical thinking instruction operationalized through the implementation of
Just-In-Time Teaching, Peer-Instruction, and concept mapping. Despite the elusive
nature of standardized guidelines for explicit critical thinking instruction, research has
been conducted by numerous scholars who reached similar findings regarding explicit
critical thinking instruction (Marin & Halpern, 2011). The literature revealed that
explicit critical instruction (referred to in this study as IV1) increases students’ critical
thinking skills. A variation of Marin and Halpern’s strategy for transferring critical
thinking skills is noted in an interactive engagement pedagogy used across disciplines
called Just-in-Time Teaching combined with Peer-Instruction and concept mapping.

83
The basic premise of Just-in-Time Teaching is that instructors adjust in-class
lessons based on students’ responses to pre-class, web-based “warm-up” activities
(Novak, 2011). The goal of this strategy is to optimize class time discussion among peers
to ensure a common understanding of key concepts. These discussions among peers are
the foundation of Peer Instruction. Just-in-Time Teaching combined with Peer-
Instruction and concept mapping makes attainment of knowledge an explicit outcome
primarily because of the nature of pre-class and in-class assignments. For Just-in-Time
Teaching, Peer-Instruction, and concept mapping to be successfully implemented,
instructors must hold students responsible for on-time delivery of meaningful responses
(Scott, 2013).
This study capitalized on the Just-in-Time Teaching and Peer-Instruction
pedagogical approach by introducing participants in the experimental groups to a four-
part model for learning critical thinking within the Physics-110 course warm-up activities
as well as within classroom discussion and in homework assignments. The four part
model consist of a) explicitly learning the skills of critical thinking as identified by the
Physics Department chairperson; b) developing the students dispositions for effortful
thinking and learning; c) directing learning activities in ways that increase the probability
of transformative and trans contextual transfer (structure training); and d) making
metacognitive monitoring explicit and overt (Halpern, 2014).
The goal of the Just-in-Time Teaching/Peer-Instruction/concept mapping
approach is to explicitly introduce experimental groups to a common definition of what
critical thinking is, in the context of Physics, and provide them with the description of the
critical thinking skills necessary to solve real-world Physics applications. These three

84
explicit concepts were introduced as part of the pre-class warm-up activities, repeated
during classroom discussions and assessed on a pre-test and post-test basis. During
classroom time the teacher-student team, emphasized the key concepts that some students
struggled to understand during pre-class warm-ups. Whereas students were expected to
participate in and reflect on the learning and teaching process using concept mapping
techniques, instructors fostered a community of mutual help through Peer Instruction.
Appendix E consists of a sample lesson plan introducing concept mapping and a student
concept map created during the study period.
Participants in the control groups were also afforded Just-in-Time Teaching and
Peer-Instruction but were not afforded the concept mapping portion of explicit critical
thinking instruction (no critical thinking instruction is referred to in this study as IV2);
however they completed both assessments at the beginning of the semester (September
28, 2015) and then again at the conclusion of lesson 28 of the Physics-110 course
(November 10, 2015). Faculty participants included a junior faculty member (IV3) and a
senior faculty member (IV4) who each provided critical thinking instruction to the
treatment groups and did not provide the concept mapping portion of the critical thinking
instruction to the control groups.
Faculty members (IV3 and IV4). The two faculty participants in this study
volunteered to participate from an available pool of 12 faculty members who were all
qualified to teach the Physics-110 course. The researcher, in collaboration with the
director of the Center for Physics Educational Research at the United States Air Force
Academy, proposed this study to Physics faculty members. A senior and a junior faculty
member volunteered to participate in the study.

85
The senior faculty member graduated with a Bachelor’s of Science, two Masters
of Science degrees, and a Doctor of Philosophy degree from the University of Florida and
the Air Force Institute of Technology. With over 17 years of experience teaching eight
different core science courses, including physics, this faculty participant authored or co-
authored 13 peer reviewed publications. During this study the researcher observed the
senior faculty member instruct a treatment group of participants using the Just-in-Time,
Peer-Instruction and concept mapping teaching approaches. These approaches coupled
with the senior faculty members’ emphasis on participants relationships with other
participants provided the researcher with evidence that all four factors influence
transformational learning were implemented during the study timeframe. Whereas the
treatment group demonstrated concept mapping through task-oriented problem solving
using real-world physics problems, the control group did not received concept mapping
instruction, hence the control group solved the given real-world physics problems using
individualized techniques.
Similarly, the researcher observed the junior faculty member during the treatment
and control groups’ individual class sessions. The junior faculty member earned a
Bachelor’s of Science and a Doctoral degree in Physics hand have five years teaching
experience all gained at the United States Air Force Academy’s Physics Department.
The same course material and approaches that the senior faculty member implemented
during the study timeframe were also used by the junior faculty member during the study
timeframe. As with the senior faculty member, the treatment group taught by the junior
faculty member demonstrated concept mapping through task-oriented problem solving
using real-world physics problems whereas the control group, who did not received

86
concept mapping instruction, solved the given real-world physics problems using
individualized techniques. Table 5 shows the number of cadet participants who
completed all four assessments and were taught by either the senior or junior faculty
participant.
Table 5.
Frequency Table of Faculty Participants
Frequency Percent Valid Percent Cumulative Percent
Valid Senior 17 47.2 47.2 47.2
Junior 19 52.8 52.8 100.0
Total 36 100.0 100.0
Descriptive information of dependent variables. The findings in Table 6 reveal
that Halpern Critical Thinking Assessment (HCTA) scores increased from the pre-test
date (M = 32.81, SD = 28.00) to the post test (M = 41.96, SD = 29.71) date. Real World
Outcomes Inventory – Part A (RWOI-A) scores dropped from the pre-test date (M = .55,
SD = .12) to the post-test (M = .21, SD = .14) date. Similarly, Real World Outcomes
Inventory – Part B (RWOI-B) scores dropped from the pre-test date (M = .53, SD = .12)
to the post-test (M = .15, SD = .13) date; accordingly, both the HCTA skills scores and
RWOI application of critical thinking skills improved across time.
Table 6.
Descriptive Statistics for the Dependent Variables (N = 36)
Pre-Test Post-Test
Measures Range M SD Range M SD
HCTA
RWOI- A
RWOI- B
1.00 to
91.00
.40 to
.84
.32 to
.84
32.81
.55
.53
28.00
.12
.12
1.00 to
94.00
.04 to
.75
.00 to
.61
41.96
.21
.15
29.71
.14
.13
Note. Halpern Critical Thinking Assessment N for the post-test was only 26.

87
Critical thinking skills (DV1). The histograms illustrated in Figure 4 depict an
increase in critical thinking skills from pre-intervention (HCTA1) to post-intervention
(HCTA2).
Figure 4. Histograms of Pre and Post HCTA Scores
Application of critical thinking (DV2). The histograms illustrated in Figures 5
and 6 depict an increase in the application of critical thinking from pre-intervention
(RWOI_A1, RWOI_B1) to post-intervention (RWOI_A2, RWOI_B2).

88
Figure 5. Histograms of Pre and Post RWOI-Part A Scores

89
Figure 6. Histograms of Pre and Post RWOI-Part B Scores

90
Comparative Analyses.
Assumptions of statistical tests. The researcher substituted pre-HCTA scores in
cases where participants were missing only one assessment score, the post HCTA. The
assumption made was that the academic performance of these particular cases did not
vary much therefore the pre- and post- HCTA scores would also not vary. Selection of
regression and missing data were the internal threats to validity. These threats were
addressed through completion of within-subject and between-subject multivariate tests
using the independent variable of instruction.
Research question one. The first research question sought to examine the effects
of explicit critical thinking instruction on the critical thinking skill levels of a control
group as compared to a treatment group of freshman and sophomore cadets enrolled in
the Physics-110 course at the U.S. Air Force Academy. A 2 X 2 mixed ANOVA
procedure revealed that the within-subjects variable was time period (from pre-test date
to post-test date); the between-subjects variable was the type of instruction (i.e., control
vs. explicit critical thinking instruction). As depicted in Tables 7 and 8, the change in
HCTA scores across time did not differ by groups, F(1, 24) = .12, p - .730, partial η2 =
.005. Therefore, explicit instructions to think critically did not affect critical thinking
skills.
Table 7.
Means and Standard Deviations for HCTA Scores
Pre-Test Date Post-Test Date
Instruction N M SD M SD
Control
Explicit
12
14
32.42
43.07
28.12
27.91
34.92
48.00
34.31
24.81

91
Table 8.
Mixed ANOVA Results for Change in HCTA Scores as a Function of Instruction (N = 26)
Source df MS F Partial η2
Between-subjects
Instruction
Error
Within-subjects
Time
Time x instruction
Error
1
24
1
1
24
1820.53
1501.28
178.29
19.06
155.92
1.21
1.14
.12
.048
.045
.005
*
p < 05. **
p < .01. ***
p < .001.
H10. There is no difference between the critical thinking skills scores of participants who
complete Physics-110 with explicit critical thinking instruction as compared to those who
complete Physics-110 without explicit critical thinking instruction.
Research question two. The second research question sought to examine the
effects of explicit critical thinking instruction on the application of critical thinking skills
of a control group as compared to a treatment group of freshmen and sophomore cadets
enrolled in the Physics-110 course at the U.S. Air Force Academy. A 2 x 2 x 2 mixed
ANOVA procedure was conducted to answer this question. The within-subjects variable
was time period. The first between-subjects variable was instruction (i.e., control vs.
explicit critical thinking instruction); the second between-subjects variable was rank of
faculty member (i.e., senior vs. junior).
Real World Outcome Inventory – Part A: Decision
The findings in Tables 9 and 10 reveal that the change in RWOI – Part A scores
across time did not differ across instruction groups, F(1, 32) = .00, p = .977, partial η2 =
.000. Similarly, the change in RWOI – Part A scores across time did not differ across
rank of faculty member, F(1, 32) = .00, p = .973, partial η2 = .000. Therefore, explicit

92
instructions to think critically and rank of faculty member did not affect the application of
critical thinking skills to real world problems.
Table 9.
Means and Standard Deviations for RWOI – Part A Scores
First Time Period Second Time Period
Condition N M SD M SD
Control
Senior faculty
Junior faculty
Explicit
Senior faculty
Junior faculty
11
6
6
13
.58
.56
.51
.54
.14
.16
.12
.10
.25
.18
.13
.22
.11
.11
.08
.19
Table 10.
Mixed ANOVA Results for Change in RWOI – Part A Scores as a Function of Instruction
and Faculty Rank (N = 36)
Between-subjects
Instruction
Faculty rank
Instruction x faculty rank
Error
Within-subjects
Time
Time x instruction
Time x faculty rank
Time x instruction x faculty rank
Error
1
1
1
32
1
1
1
1
32
.03
.00
.04
.03
1.98
.00
.00
.01
.01
1.11
.07
1.60
197.76
.00
.00
.86
***
.034
.002
.048
.861
.000
.000
.026
*
p < 05. **
p < .01. ***
p < .001.
Butler Real World Outcome Inventory – Part B: Outcomes
The findings in Tables 11 and 12 show that the change in RWOI – Part B scores
.019. Similarly, the change in RWOI – Part B scores across time did not differ across
rank of faculty member, F(1, 32) = .02, p = .898, partial η2 = .001. As such, explicit

93
instructions to think critically and rank of faculty member did not affect the application of
critical thinking skills to real world problems.
Table 11.
Means and Standard Deviations for RWOI – Part B Scores
Pre-Test Date Post-Test Date
Condition N M SD M SD
Control
Senior faculty
Junior faculty
Explicit
Senior faculty
Junior faculty
11
6
6
13
.53
.57
.48
.54
.11
.19
.12
.11
.16
.23
.09
.13
.08
.21
.05
.13
Table 12.
Mixed ANOVA Results for Change in RWOI – Part B Scores as a Function of Instruction
and Faculty Rank (N = 36)
Between-subjects
Instruction
Faculty rank
Instruction x faculty rank
Error
Within-subjects
Time
Time x instruction
Time x faculty rank
Time x instruction x faculty rank
Error
1
1
1
32
1
1
1
1
32
.06
.04
.00
.02
2.31
.01
.00
.00
.01
2.91
1.94
.00
251.81
.61
.02
.21
***
.083
.057
.000
.887
.019
.001
.007
*
p < 05. **
p < .01. ***
p < .001.
H20. There is no difference between the application of critical thinking skills of
participants who complete Physics-110 with explicit critical thinking instruction as
compared to those who complete Physics-110 without critical thinking instruction.
Evaluation of Findings
This study’s findings were not statistically significant as reflected by the results of
the one-way ANOVA tests and the multivariate tests. All significance levels were greater

94
than p=.05. According to Tabachnick & Fidell (2006), p values greater than .05 do not
have evidence of a significant effect. These results lend credence to the null hypotheses
this study sought to explore which was that explicit critical thinking instruction, within
the constraints experienced in this study, does not have a transformational effect on the
critical thinking skills nor the application of critical thinking to real-world scenarios.
The results of these analyses were not statistically consistent with a similar study
Butler (2012) completed where she studied the relationship between real-world outcomes
and critical thinking skill levels of community college students (n=35), state university
students (n=46), and community adults (n=50) with a total sample size of n=131. Butler
hypothesized that the respondents who scored higher on the HCTA would report fewer
negative life events in multiple domains of life. This study referred to negative life events
as the application of critical thinking. Butler concluded that critical thinking scores were
inversely proportional to the negative life events; hence the higher the critical thinking
scores the lower the scores on the RWOI.
Although the small number of respondents to this study was not significant
enough to reflect a statistically similar conclusion, the 36 cases included in this study did
show an inversely proportionate relationship between critical thinking skills and the
application of critical thinking. Furthermore, whereas the Butler study only examined the
relationship between critical thinking skills and real-world outcomes of critical thinking
using explicit critical thinking instruction as the intervention, this study incorporated
explicit critical thinking instruction using the integration of Just-in-Time, Peer
Instruction, and concept mapping techniques in efforts to optimize students’ deep
understanding of critical thinking concepts so that they can consistently apply those

95
concepts and processes to real-world situations. The integration of these approaches was
done to better define the classroom experience and have the faculty foster the all four
factors that influence transformative learning. The Butler study did not consider
transformative effects of explicit critical thinking instruction.
While this study examined the transformative effects of explicit critical thinking
instruction on critical thinking skills and application of critical thought of first year
Physics cadets, Dwyer, Hogan, and Stewart (2012) examined the effects of
argument/concept mapping on critical thinking skills of first year psychology students.
This study implemented more than merely one teaching approach; it incorporated the best
practices of three different approaches coupled with faculty involvement in a student-
centered, active learning environment. Dwyer et al., (2012) also used the HCTA as the
instrument for measuring gains in critical thinking skills, however unlike this study,
Dwyer et al., concluded that the treatment groups’ (n=43) overall critical thinking skills
were significantly enhanced after an 8-week intervention period as compared to the
control group (n=31). The intervention period for this study was 6-weeks and sample
was shorter than the Dwyer et al., study which may have negatively impacted results.
Examining one pedagogical approach towards improving critical thinking skills is not the
only disparity between current literature and this study. The instrument used in this study
has been used in a myriad of studies with varying results (Butler, Dwyer, Hogan, Franco,
Rivas, Saiz, & Almeida, 2013).
Several cross-national applications of the HCTA and RWOI were evaluated by
Butler et al., in a 2013 study to conclude that overall the HCTA is a useful instrument for
assessing critical thinking skills and predicting the application of those skills to real-

96
world situations. de Brie and Wilhelms’(2015) also examined the effectiveness of the
HCTA and RWOI instruments and found that 23% of respondents thought the HCTA
questions were too complex and difficult to comprehend the scenarios which may have
impacted their individual scores. Other respondents commented on the excessive time
the assessment took to complete with some stating it took them three hours to complete
(de Brie & Wilhelms, 2015). The de Brie and Wilhelms study may provide some insight
into how the complexity and time constrained limitations of the HCTA may have
impacted the low number of respondents within this study. This study afforded only 53
minutes to cadets to complete both the HCTA and RWOI online. In addition to the time
constraints of this study, 48.8 % of cadets experienced problems accessing the online
assessments which also contributed to the low sample size.
Findings in the context of the transformative learning theory. In light of the
theoretical framework for this study, in order for critical thinking to be transformative
and consistently applied to real-world situations and within different domains/contexts,
the learners’ experience must be addressed through appropriate classroom environments
and teaching strategies. The relationship between the cognitive psychological and the
philosophical definition of critical thinking, as they relate to the transformative learning
theory, correspond to the four factors that influence transformative learning with the
exception of the forth factor, the learning experience. The learning experience is
depicted by an asterisk in Table 13 below.

97
Table 13.
Relationship between Transformative and Critical Thinking Learning Theories
Transformational Learning Theory Critical Thinking Learning Theory
Reflection Purposeful reflection; pragmatic approach
to thinking
Rational discourse Rationality, logic, communication of
thought
Action/Task-orientation Actions of the mind, judgment, analysis
of thought
Learning Experience *
Note. *There is a gap in the current literature on critical thinking. For critical thinking
to be transformative a definition in the educational context should also address a
learners’ experience.
This study was intended to fill the gap in the critical thinking learning theory by
examining the effects of explicit critical thinking instruction on the skills and application
of critical thinking. The explicit form of critical thinking instruction used in this study
included multiple aspects of what was previously described as a positive learning
experience. For example, during the six-week study period, the researcher observed that
both faculty members fostered purposeful reflection by assigning physics problems that
allowed enough uncertainty to trigger reflection then allowed for participants to make a
judgment or take an action on a best solution. They each also emphasized rational
discourse amongst participants by encouraging Peer-Instruction and Socratic questioning
techniques. For the transformative learning theory factor of task-orientation and the
critical thinking learning theory factor of analysis of thought, faculty participants
assigned concept mapping activities as part of pre-flight assignments and in-class
problem solving activities for the treatment groups. An example of a participant’s
concept map is included in Appendix E to this manuscript.

98
The learning experience during this study can be described as a student-centered,
active learning environment with elements of team-based learning approaches. Faculty
encouraged critical thinking throughout the study however participants were not
necessarily aware of their thinking processes; they simply applied critical thinking skills
to solve given physics problems using hands-on techniques, discussing with their peers
possible solutions, and reflecting on their own reasons for arriving at their individual
solution. Halpern (2012) explained that transfer of knowledge where individuals are not
aware of their thinking processes is a generalist view of critical thinking skills. This view
also implies that critical thinking skills may be applied across disciplines because they
are skills that have been internalized to the point of becoming habits of the mind (Abu-
Dabat, 2012; Halpern, 2014).
Study limitations and validity. The primary limitations to this study were the
extremely small sample size and the fact that the population was specific to the United
States Air Force Academy freshmen cadets enrolled in a Physics-110 course during the
fall 2015 semester. These limitations might impact the generalizability of results to other
populations. The generalizability of this study’s findings to target population may not be
optimum because most universities offering introductory physics courses do not require
that all students enroll and complete introductory physics courses nor do they specify that
only freshmen or sophomore students can enroll in introductory physics courses.
The major multiple-group threats to internal validity for this study were selection-
mortality and selection regression. Selection-mortality arises when there is differential
nonrandom dropout between pretest and posttest (Tabachnick & Fidell, 2006). In this
study the researcher experienced internet link problems. Additionally, some participants

99
were not available on the specific dates the assessments were administered. This resulted
in 62% of participants who could not complete the post-HCTA and 24% who could not
complete the post-RWOI. Posttest differences may have been due to the different types
of dropouts, the selection-mortality, and not to the explicit critical thinking instruction.
Another threat to internal validity of this study was selection-regression.
Selection-regression occurs when there are different rates of regression to the mean in the
two groups (Tabachnick & Fidell, 2006). This might happen if one group is more extreme
on the pretest than the other. In the context of this study, it may be that the treatment
group is getting a disproportionate number of opportunities to practice concept mapping
because they were afforded this portion of the intervention whereas the control group was
not. Since the control group has the more extreme lower scorers, their mean will regress
a greater distance toward the overall population mean and they will appear to gain more
than their comparison group counterparts. This is not a real program gain; it's just a
selection-regression artifact.
A major weaknesses of this study, is the failure to statistically measure the effect
of the classroom experience as one of the four factors influencing transformative
learning. A qualitative measurement of the classroom experience may have consisted of
participants’ self-evaluation of their experience during the study. A self-evaluation of
this variable however would have changed the method used in this study from a
quantitative, quasi-experimental design to a mixed method design, thereby complicating
statistical analyses and subsequent reporting of findings.

100
Summary
The relationship between explicit critical thinking instruction and critical thinking
skills and application of critical thinking was examined during the study. It was
hypothesized that there would be a positive relationship between the two variables.
Statistical analyses were performed to assess the skills and ability to apply critical
thinking to various real-world outcomes of cadets enrolled in an introductory physics
course during the fall 2015 semester. The results of the study indicated there was not a
statistically significant positive relationship between explicit critical thinking instruction
and critical thinking skills and application of critical thinking (r = .85, p < .05). In other
words, although explicit critical thinking instruction did show an increase in both critical
thinking skills and the ability to apply critical thinking to real-world situations, the
available data was not statistically significant enough to validate the two alternate
hypotheses. Therefore, the null hypotheses were adopted.

101
Chapter 5: Implications, Recommendations, and Conclusions
There is a need for higher education to address how to develop critical thinkers
who are capable of applying critical thinking skills to address real-world situations
(Behar-Horenstein & Niu, 2011; Miller et al., 2009; Periklis, 2010). Critical thinking
outcomes in particular, have been a priority for all United States Air Force Academy
courses. Empirical research on the effectiveness of teaching critical thinking skills was
therefore of high importance to both the United States Air Force Academy as well as
other institutions of higher education. This study sought to provide a strategy for
enhancing critical thinking outcomes for the Physics Department and possibly other
academic departments at the United States Air Force Academy.
The purpose of this quantitative study was to examine the relationship between
teaching critical thinking using an integrated pedagogical approach and the critical
thinking skill levels as well as the application of those skills through the lens of the
transformative learning theory. The purpose of this study was achieved. The study was a
quasi-experimental non-equivalent design. Participants included 85 randomly selected
freshmen and sophomore cadets enrolled in Physics-110 course at the United States Air
Force Academy, Colorado Springs, Colorado. The sample was grouped into four classes
taught by two faculty participants using explicit critical thinking instruction for a period
of six weeks. Two of the groups were treatment groups and the other two were control
groups.
The independent variables were defined as explicit critical thinking instruction
(intervention vs. no intervention) and faculty experience (junior vs. senior). The
dependent variables were defined as critical thinking skills scores as measured by the

102
Halpern Critical Thinking Assessment and the outcomes of critical thinking, also referred
to as the application of critical thought as measured by the Real-World Outcomes
Inventory. From the original sample of n=85, data were excluded from 49 participants
resulting in an actual sample size of n=36. Data was excluded primarily as the result of
participants not being able to access the post-test due to information technology problems
encountered at the United States Air Force Academy. Data was also excluded because
not all participants were available to complete both pre- and post- assessments on the
dates they were administered. As a result, the actual sample size of n=36 complete data
sets were obtained for this study.
The first hypothesis was tested using a 2 x 2 mixed-analysis of variance
(ANOVA) procedure because the HCTA was measured twice; thus, this was the within-
subjects variable. There were two instruction groups, control and explicit instruction; this
was the between-subjects variable. To determine whether change in HCTA differed
across the instruction groups, the interaction between time and instruction was evaluated.
The second hypothesis was tested using a 2 x 2 x 2 mixed ANOVA procedure. As in the
first hypothesis, RWOI was measured twice; thus, this was the within-subjects variable.
Similarly, there were two instruction groups; this was the first between-subjects variable.
The second between-subjects variable was faculty (i.e., junior vs. senior). To determine
whether change in RWOI differed across the instruction groups, the interaction between
time and instruction was evaluated. To determine whether change in RWOI differed
across faculty, the interaction between time and faculty was evaluated.
The results of this study indicated that there is no significant difference between
explicit critical thinking instruction and the critical thinking skills and application of

103
critical thinking to real-world scenarios; however multiple limitations were encountered
during the study that may have impacted the results. The nature of the population under
study, the particular dates assessments were administered, and assessment inaccessibility
are among several disparities that may have negatively impacted the results of this study.
This final chapter expands further on the study limitations and frames the results of
statistical analyses against the relevant literature reviewed in chapter two. More
specifically, the results are described in the context of critical thinking teaching
approaches and the relationship to critical thinking skills and application of those skills
within the transformative learning framework. Implications of study results as they relate
to the purpose and existing literature on critical thinking are presented followed by future
research recommendations and study conclusions.
Limitations. Several limitations experienced during this study may have
contributed to the insignificant results. The nature of the population may have
contributed Academy located in Colorado Springs, Colorado. The United States Air
Force Academy is unique from other universities because it is a military university. As
such cadets are faced with rigorous academic and athletic schedules, they face extremely
high expectations, and strict rules of conduct (USAFA, 2015). The intent behind this
demanding program is to develop future officers with a strong sense of character who are
ready to lead the nation in the United States Air Force (USAFA, 2015). As compared to
other universities in the United States, the United States Air Force Academy offers cadets
increasing levels of responsibility over the course of their tenure.
Approximately 218 (total population) freshman and sophomore cadets were
enrolled in the Physics-110 course during the fall 2015 semester. Each class of 20-23

104
freshmen and sophomore cadets was representative of beginning physics students at the
Academy, but not representative of other universities in the United States primarily
because Academy freshmen and sophomore cadets are required to complete the
introductory physics course, Physics-110, regardless of their chosen academic major.
Most universities do not require all students to complete introductory physics courses.
Cadets are also unique as compared to the general population of students in other
universities because they volunteer to attend a military university and understand the
environment for which they are about to encounter. The unique characteristics of the
population may be a possible cause for the disparity between this research’s expected and
actual results.
The date that the post-intervention tests were administered coupled with the
technology problems encountered on that date, contributed to the low response rate. The
post-intervention tests were administered during finals week when cadets tend to be
focused on their individual academic performance within their courses. Completing the
voluntary post-intervention tests in support of this study may not have been on the
forefront of participant’s priorities. The resulting small sample size was also due to the
fact that the United States Air Force Academy’s computer network division had pushed
out computer security upgrades the week prior to the post-intervention test date. Many
participants were not able to access the post-intervention tests because their individual
computers had not received the security upgrades and therefore locked out the unfamiliar
test links. These two unforeseeable conditions contributed to only 31% respondents on
the post-test as opposed to 71% respondents to the pre-test.

105
The researcher acknowledges that the length of exposure to the intervention was
only six weeks and data was collected from only one institution of higher education. This
study implemented the intervention for a period of six weeks. According to a Renaud
and Murray (2008) study, single interventions longer than 12 weeks tended to be more
effective in increasing participants’ critical thinking abilities than shorter interventions.
Expecting noticeable changes in thinking abilities after only six weeks of exposure may
not be as realistic as waiting for changes to accumulate after a long period of exposure to
explicit critical thinking instruction. Expanding the scale and duration of the intervention
used in this study, may have improved its effectiveness; however time constraints
prohibited the expansion of the intervention. With regard to the external validity of this
study, the researcher does not suggest that the results of small cadet sample can be
generalized to students at other universities or in other geographical regions, employees
at different companies, or to other cultures.
The instrument used to measure critical thinking skills, the Halpern Critical
Thinking Assessment, was a forced choice assessment despite the availability of a
constructed response version of the assessment. Unfortunately, the constructed response
version of the HCTA would have taken more time to complete than the forced choice
version. That is, the constructed response version would have taken too much time away
from cadet’s academics, therefore it was not approved. Constructed response
assessments rely on free recall for which more cognitive effort is required (Ku & Ho,
2010). Additionally, the format may reveal more of the respondents’ disposition for
critical thinking because answering open-ended questions would demonstrate the extent

106
the respondent is willing to engage in critical thought processes (Butler, 2012; Ku & Ho,
2010).
Effects of limitations on the interpretation of results. The findings of this
study indicate a relationship between explicit critical thinking instruction and critical
thinking skills and application to real-world outcomes; however, the sample size was
statistically limited leading to the rejection of alternative hypotheses hence the null
hypotheses were adopted. One empirical study alone cannot prove a cause and effect
association; therefore, additional similar studies conducted under stricter controls may
result in a cause and effect relationship between the variables of this study.
Ethical issues. Security features embedded within the links accessing the
Halpern Critical Thinking Assessment provided anonymity and confidentiality for the
participants. In addition, the researcher sent individual participants an email with the
Real-World Outcomes Inventory attached. Participant codes were attached to each
assessment to ensure no identifying information was provided. Participants agreed to the
conditions specified in their introductory email prior to proceeding with the completion
of the online assessments. Informed consent and instructions for accessing the online
assessments were included in the introductory email. Results were presented in
aggregate form. No other ethical issues were noted throughout the conduct of this study.
Implications
Following is a restatement of each of the two research questions addressed in this
study, their associated hypothesis, and a discussion of the implications and significance
of the findings as related to existing literature.

107
Research question one. What are the effects of explicit critical thinking
instruction on the critical thinking skill levels of a control group as compared to a
treatment group of freshman and sophomore cadets enrolled in the Physics-110 course at
the U.S. Air Force Academy?
H10. There is no statistical difference between the critical thinking skills scores
compared to those who completed Physics-110 without explicit critical thinking
instruction.
H1a. There is a statistical difference between the critical thinking skills scores of
cadets who completed Physics-110 with explicit critical thinking instruction as compared
to those who completed Physics-110 without explicit critical thinking instruction.
The null hypothesis was adopted due to the results of a 2 X 2 mixed ANOVA procedure
which revealed that the within-subjects variable was time period (from pre-test date to
post-test date); the between-subjects variable was the type of instruction (i.e., control vs.
explicit critical thinking instruction). The change in HCTA scores across time did not
differ by groups, F(1, 24) = .12, p - .730, partial η2 = .005. Therefore, explicit
instructions to think critically did not affect critical thinking skills.
The field of educational leadership might be positively affected by the inquiry of
this study because there have been many unsuccessful attempts at increasing the critical
thinking skills of graduates yet the demand for critical thinkers continues to rise (Niu et
al., 2013; de Bie & Wilhelm, 2015). The significance of this study to the field of
educational leadership, in general, and critical thinking theory in particular is that of
placing an emphasis on the factors influencing transformative learning during the design

108
and implementation of curricula as well as prolonging student exposure to explicit critical
thinking instruction using an integrated teaching approach. Faculty participants played
an essential role in designing instruction that was consistent with the factors that
influence transformative learning. The literature on critical thinking interventions tend to
focus on single pedagogical approaches. This study integrated Just-in-Time Teaching,
Peer-Instruction, concept mapping, repetition, and Socratic questioning. These teaching
approaches individually have been shown to foster the transfer of knowledge to varying
scenarios as well as encourage positive critical thinking dispositions such as engagement,
cognitive maturity, and innovativeness (Ya-Ting, Yung-Hsin, & Cowan, 2014; Halpern,
2014; Khandaghi & Pakmehr, 2012; Mathews & Lowe, 2011; Ricketts & Rudd, 2004).
A logical conclusion would be that the integration of all these approaches would further
increase students’ critical thinking skills and dispositions. Although this study did not
validate this assumption, future research should incorporate this study’s integrated
teaching approach as well as implement stricter assessment controls to allow for a greater
sample size.
This study diverted from past studies on the effectiveness of explicit critical
thinking instruction by placing an emphasis on the transformative effects of critical
thinking skills on the application of critical thinking whereas most other empirical studies
focused primarily on capturing critical thinking skill levels as related to academic
achievement (Dwyer et al., 2012; Niu, Behar-Horenstein, & Garvan, 2013; Butler et al.,
2012). The goal of this study was also different from other similar studies as the goal
was to examine transformative learning effects of teaching an integrated approach
consisting of Just-in-Time Teaching, Peer-Instruction and concept mapping. These

109
combined approaches relate to the factors influencing transformative learning whereas
the existing literature on critical thinking emphasizes the use of either one pedagogical
approach or another for instilling critical thinking and improving academic achievement
(Niu, Behar-Horenstein, & Garvan, 2013; Scott, Gray, & Yates, 2013).
By taking an integrated teaching approach, combined with a lengthier exposure to
the integrated approach than this study was afforded, a transformation in student behavior
and therefore a lasting learning experience may take place; in other words a
transformative effect may be imparted. The four factors that influence transformative
learning (Carawan, Knight, Wittman, Pokorny, & Velde, 2011; Summit, 2014) correlate
to the intervention techniques the researcher asked the faculty participants to use during
this study. Figure 7 below captures the correlation between transformative learning
factors, the explicit critical thinking intervention used in this study, and the four themes
the researcher uncovered in the literature on critical thinking definitions.
Figure 7. Comparison between Themes in Defining Critical Thinking, Transformative
Learning Factors, and Explicit Critical Thinking Instruction

110
Figure 7 depicts a gap in existing literature on definitions of critical thinking. The gap is
the lack of examining the classroom experience in applying critical thought. This study
addressed all four factors influencing transformative learning through the integrated
pedagogical approach consisting of Just-in-Time Teaching, Peer-Instruction, and concept
mapping.
This study incorporated three pedagogical approaches consisting of the
transformative factors necessary for lasting effects. Data was collected from cadets
enrolled in four different Physics-110 classes (M3A, M6A, M3B, and M6B), two
treatment groups and two control groups. The four groups were taught by two faculty
participants using the same lesson plans and teaching approaches. Cadet assignment to
four groups was reasonably successful based on the statistical results of cadets’ critical
thinking skills and application scores as measured by the HCTA and RWOI assessments.
The sample size for this study, however was not reasonable (n=36) in quasi-experimental
studies (Trochim & Donnelly, 2008).
Through pre-class assignments, in class discussions and emphasis on individual
cadets’ existing knowledge, faculty participants instilled the four transformative learning
factors. The course content and processes faculty participants used, were exactly the
same. Unfortunately the timeframe allotted to this study was insufficient to reap the long
term effects of Just-in-Time Teaching, Peer-Instruction, and concept mapping. The non-
significant between subjects differences might have been caused by cadet participants’
not being able to access post-tests due to technology problems. No paper versions of the
post-tests were available during the post-test date resulting in a low response rates.
Future research should consider a longer timeframe for the intervention and paper
versions of assessments.

111
Research question two. What are the effects of explicit critical thinking
instruction on the application of critical thinking skills of a control group as compared to
a treatment group of freshmen and sophomore cadets enrolled in the Physics-110 course
at the U.S. Air Force Academy?
H20. There is no statistical difference between the application of critical thinking
H2a. There is a statistical difference between the application of critical thinking
The null hypothesis was adopted because the change in RWOI – Part A scores
.000. Similarly, the change in RWOI – Part A scores across time did not differ across
rank of faculty member, F(1, 32) = .00, p = .973, partial η2 = .000. Likewise, the change
in RWOI – Part B scores across time did not differ across instruction groups, F(1, 32) =
.61, p = .441, partial η2 = .019. The change in RWOI – Part B scores across time did not
differ across rank of faculty member, F(1, 32) = .02, p = .898, partial η2 = .001. As such,
explicit critical thinking instructions and rank of faculty member did not affect the
application of critical thinking skills to real world problems.
These results may be significant to educators because they may not have been
aware that learning that transforms individual behaviors have longer lasting experiences
than mere memorization of facts such as lists of ideal critical thinking dispositions and
skills (Herlo, 2010; Dwyer et al., 2012). Teaching students critical thinking definitions

112
without providing opportunities for students to take actions, make judgments, reflect on
alternatives, or discuss why their solutions are the correct ones, do not afford them the
hands-on practical knowledge needed to consistently apply critical thinking skills in
varying situations (Halpern, 2012; Ya-Ting, Yung-Hsin, & Cowan, 2014). Without
repeated practice of those skills in a myriad of contexts and domains, transformative
effects cannot take place (Herlo, 2010), hence students will continue to graduate lacking
the skills necessary to consistently apply critical thought to address the societal and 21st
century work force demands.
This study also uncovered that a universal assessment instrument measuring the
consistent application of critical thinking skills beyond the classroom is nonexistent.
Assessment instruments for measuring the classroom experience are also nonexistent.
The development and validation of such instruments may assist both future employers
and educators in assessing candidate’s disposition for consistently applying critical
thinking skills to various real-world situations. Although the findings of this study
differed from the expected findings supported by the literature reviewed in chapter two,
the design of this study may be replicated with the addition of stricter controls that would
address the disparities encountered during this study.
Recommendations
The insights developed in this study can positively influence other scholars or
practitioners in the context of the population of cadets studied and in the field of critical
thinking within introductory physics courses. The United States Air Force Academy
Physics Department was presented with the findings of this study and was encouraged to
continue to practice the explicit critical thinking instruction described in this study. With

113
repeated practice of this approach within all of their introductory physics courses, a
resulting gain in cadets’ critical thinking skills may be imparted. The critical thinking
assessments reviewed in chapter two of this manuscript should be adopted to measure
any gains in cadets’ critical thinking skills at the end of each introductory physics course.
Collecting and trending data from the end of course assessment of critical thinking should
be presented to other academic departments in efforts to optimize the positive effects of
explicit critical thinking instruction within those academic departments.
A key implication for developing programs and theory that are derived from this
study’s findings is for educators to place a stronger emphasis on the student’s classroom
experience as applicable to the transformative learning theory. This study uncovered that
the classroom experience may be an additional variable requiring measurement in future
studies to assure a complete depiction of the transformative effects of explicitly teaching
critical thinking. Unfortunately the literature is sparse on validated assessment
instruments to measure the classroom experience.
Finally, further information needs to be uncovered such as how different
disciplines or subjects may be impacted by this study’s intervention. Contradictions
between similar studies and this study may have occurred because of the small sample
size and other disparities as explained in this chapter. This study may have shed new
light on the topic of critical thinking because it was conducted through the lens of the
transformative learning theory and it measured participant’s ability to apply critical
thinking skills through the Real-World Outcomes Inventory.
Recommendations for education scholars. Education scholars may replicate
this study under more controlled environments to yield a statistically valid conclusion to

114
the transformational effects of explicit critical thinking instruction on the application of
critical thinking to real-world events. To achieve this, they should attempt to expand the
sample size of a future study to a more statistically significant sample size (Cohen, 1988).
Furthermore, such scholars may also optimize the inquiry of this study by expanding the
study to other domains, such as the humanities (Halpern, 1998). Ennis (1996) contended
that explicit critical thinking instruction should be tailored not only to the students but
also to the subject of the course.
Educators play an essential role in designing instruction that compels instructors
to consistently implement the four factors influencing transformative learning. To this
end educators should create faculty development standards for teaching critical thinking
that includes multiple pedagogical approaches in an active, student-centered
environment. By practicing repetition, Socratic questioning, and concept mapping in and
out of the classroom, instructors may increase the opportunity to foster students’
disposition for critical thought which consists of student engagement, cognitive maturity,
and innovativeness as explained in chapter two of this manuscript. All of these
approaches may lend support to this study’s primary inquiry of examining the
transformational effects of critical thinking instruction on the application of critical
thinking. To measure the effectiveness of the faculty development standard for teaching
critical thinking, a validated assessment instrument must first be designed and evaluated
through future research. Validated assessment instruments are required to measure the
classroom experience as well as the application of critical thinking skills beyond the
classroom. This study did not measure the classroom experience nor did it measure the
application of critical thinking skills beyond the six-week study duration. Future

115
longitudinal research is recommended to assess participants’ application of critical
thinking during their tenure at the United States Air Force Academy and a minimum of
three years beyond graduation. Transformative effects of the explicit critical thinking
instruction used in this study may be imparted throughout the United States Air Force
Academy with the development and consistent use of faculty and curricula standards that
implement the knowledge gained from this study.
Recommendations for future research. The design and conduct of this research
is repeatable and may serve as a foundation for confirmatory research on the
transformative effects of explicit critical thinking instruction on the skills and application
of critical thinking in real-world scenarios. Future research is recommended to include a
large sample across multiple disciplines and in different universities, because the unique
environment of the United States Air Force Academy may not be generalized to other
universities. A future study should include a qualitative component to examine student
participants’ perception of their experience during the study. This study did not
qualitatively assess this factor.
A future study should also incorporate strict scheduling procedures for pre-post
assessments. More specifically, pre-post assessments should not be administered during
times when participants are focused on higher academic priorities, such as during finals
week. Stricter controls would include the use of paper versions of the pre-post
assessments to serve as back-up to online assessments should technical problems be
experienced as they were in this study. Participants in a future study should also be
exposed the intervention for a longer period of time to instill a transformation of
behavior. These controls may yield larger sample sizes for statistically significant results.

116
Finally, other future research should consider addressing any cultural impacts of
individual backgrounds to the transformative effects of critical thinking interventions.
Conclusions
The primary goal of this study was to quantify the effect explicit critical thinking
instruction may have on participants’ critical thinking skills as well as their propensity for
applying critical thinking beyond the duration of this study. Critical thinking skills and
outcomes of critical thought were measured using validated assessment instruments.
Collection of data were controlled to assure anonymity and confidentiality however of the
intended sample size n=85, only 36 participants completed all pre- and post- intervention
assessments. With a low sample size all statistical tests indicated no significant
relationship between the explicit critical thinking instruction and participants’ critical
thinking skills or their ability to apply critical thought to real-world situations.
Findings were not consistent with existing literature on critical thinking skills.
With regard to the learning experience for example, Pitkaniemi and Vannienen (2012)
suggested that a positive learning experience and associated learning outcome is not only
related to the teacher’s instruction or classroom configuration, but it is also related to the
students’ motivation to learn and to the students’ metacognition. This study did not
measure student motivation nor metacognition. Additionally, the results of 61 empirical
studies analyzed by Niu et al. (2013) utilized standardized assessments of critical
thinking skills with little to no emphasis on assessing the consistent application of critical
thinking to solve real-world problems.
This study was unique with respect to the intervention that was implemented. The
explicit critical thinking instruction consisted of the integration of three pedagogical

117
approaches taught by a senior and a junior faculty participant. The literature on critical
thinking interventions tended to focus on implementing a single pedagogical approach to
foster critical thinking (Carr-Lopez et al., 2014; Gok, 2012; Marin et al., 2011; Osmond
& Goodnough, 2011; Scott et al., 2013). From the perspective of the transformative
learning theory, all four factors influencing transformative learning were implemented;
however, the duration of the intervention coupled with the small sample size may have
negatively impacted statistical results. Renaud and Murray (2008) suggested that single
interventions longer than 12 weeks tended to be more effective in increasing participants’
critical thinking abilities than shorter interventions. This study implemented the
intervention for a period of six weeks. Furthermore the small sample size contributed to
the unlikelihood of a positive relationship between the intervention and assessment
scores. Norusis (2010) stated that the rule of thumb typically used to characterize results
as significant is a probability of 5% or less. The majority of this study’s statistical tests
results yielded probabilities of greater than 5%, indicating non-significant findings. For
these reasons this study should be replicated implementing stricter controls and larger a
larger sample.
Mixed method studies are also needed to examine the relationships of
participants’ perceptions of their classroom experience and their ability to apply critical
thinking skills consistently. Knowledge gained from such future research, coupled with
the findings of this study, may help address the problem of why, despite all the emphasis
placed on critical thinking as a goal of higher education, students continue to graduate
with inadequate reasoning (critical thinking) skills.

118
References
Abu-Dabat, Z. I. (2011). Critical thinking, skills and habits. Journal of Education &
Sociology, 2(1), 21-27. doi:http://dx.doi.org/10.7813/jes.2013
Allen, K. (2013). Locke and sensitive knowledge. Journal of the History of Philosophy,
51(2), 249-266. doi:http://dx.doi.org/10.1353/hph.2013.0029
Association of American Colleges & Universities. (2010). Raising the bar: Employers’
views on college learning in the wake of the economic downturn. Retrieved from
the AAC&U website http://www.aacu.org/leap
Bareham, S. (2012). Think well and prosper: A critical thinking guide. Birmingham, AL:
Summa Publishing.
Behar-Horenstien, L. S., & Niu, L. (2011). Teach critical thinking skills in higher
education: A review of the literature. Journal of College Teaching & Learning,
8(2), 25-41. Retrieved from
http://journals.cluteonline.com/index.php/TLC/article/view/3554/3601
Bensley, D. A., & Spero, R. A. (2014). Improving critical thinking skills and
metacognitive monitoring through direct infusion. Thinking Skills & Creativity,
1255-68. doi:http://dx.doi.org/10.1016/j.tsc.2014.02.001
Bleicher, E. (2011). The last class: Critical thinking, reflection, course effectiveness, and
student engagement. Honors in Practice, 7, 39-51. Retrieved from
http://search.ebscohost.com.proxy1.ncu.edu/login.aspx?direct=true&db=ofs&AN=
60862907&site=eds-live
Boghossian, P. (2012). Critical thinking and constructivism: Mambo dog fish to the
banana patch. Journal of Philosophy of Education, 46(1), 73-84.
doi:http://dx.doi.org/10.1111/j.1467-9752.2011.00832.x
Bureau of Labor Statistics. (2010-2011). Occupational Outlook Handbook, 2010-2011
edition. Retrieved from http://www.Bls.gov/oco/ocos021.htm
Butler, H.A. (2012). Halpern critical thinking assessment predicts real-world outcomes of
critical thinking. Applied Cognitive Psychology, 26, 721-729.
doi:http://dx.doi.org/10.1002/acp.2851
Butler, H. A., Dwyer, C. P., Hogan, M. J., Franco, A., Rivas, S. F., Saiz, C., & Almeida,
L. S. (2012). The Halpern Critical Thinking Assessment and real-world outcomes:
Cross-national applications. Thinking Skills and Creativity, 7, 112-121.
doi:http://dx.doi.org/10.1016/j.tsc.2012.04.001

119
Carawan, L. W., Knight, S., Wittman, P., Pokorny, M., & Velde, B. P. (2011). On
becoming a qualitative researcher: A view through the lens of transformative
learning. Journal of Teaching in Social Work, 31(4), p387-399.
doi:http://dx.doi.org/10.1080/08841233.2011.597671
Carmel, J. H., & Yezierski, E. J. (2013). Are we keeping the promise? Investigation of
students' critical thinking growth. Journal of College Science Teaching, 42(5), 71-
81. Retrieved from
http://search.ebscohost.com.proxy1.ncu.edu/login.aspx?direct=true&db=ehh&AN
=86988366&site=eds-live
Carr-Lopez, S. M., Galal, S. M., Vyas, D., Patel, R. A., & Gnesa, E. H. (2014). The utility
of concept maps to facilitate higher-level learning in a large classroom setting.
American Journal of Pharmaceutical Education, 78(9), 1-7. Retrieved from
http://eds.a.ebscohost.com.proxy1.ncu.edu/eds/pdfviewer/pdfviewer?sid=3d225aee
-4c36-4888-98d6-6d91590467e2%40sessionmgr4001&vid=7&hid=4210
Cassum, S. H., Profetto-McGrath, J., Gul, R. B., Dilshad, A., & Syeda, K. (2013).
Multidisciplinarity of critical thinking: A holistic perspective from
multidisciplinary educators in Karachi, Pakistan. Journal of Nursing Education
and Practice, 3(7), 9-23. doi:http://dx.doi.org/10.5430/jnep.v3n7p9
Celuch, K., Kozlenkova, I., & Black, G. (2010). An exploration of self-efficacy as a
mediator of skill beliefs and student self-identity as a critical thinker. Marketing
Education Review, 20(3), 70-80. doi:http://dx.doi.org/10.2753/MER1052-
8008200306
Cleveland, B., & Fisher, K. (2014). The evaluation of physical learning environments: A
critical review of the literature. Learning Environments Research, 17(1), 1.
doi:http://dx.doi.og/10.1007/s10984-013-9149-3
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd
ed.). Hillside:
Lawrence Erlbaum Associates, Inc.
Davies, M. (2011). Introduction to the special issue on critical thinking in higher
education. Higher Education Research & Development, 30(3), 255-260.
de Bie, H., Wilhelm, P., & van der Meij, H. (2015). The Halpern Critical Thinking
Assessment: Toward a Dutch appraisal of critical thinking. Thinking Skills and
Creativity, 1733-44. doi:http://dx.doi.org/10.1016/j.tsc.2015.04.001
Deksissa, T., Liang, L. R., Behera, P., & Harkness, S. J. (2014). Fostering significant
learning in sciences. International Journal for the Scholarship of Teaching &
Learning, 8(2), 1-25. Retrieved from

120
http://eds.a.ebscohost.com.proxy1.ncu.edu/eds/pdfviewer/pdfviewer?sid=48b719c
3-db41-4b30-a615-615c62f5b1dd%40sessionmgr4003&vid=10&hid=4105
Dewey, J. (1933). How we think: A restatement of the relation of reflective thinking to the
education process. Boston, MA: Heath.
Dondlinger, M. J., & Wilson, D. A. (2012). Creating an alternate reality: Critical,
creative, and empathic thinking generated in the global village playground
capstone experience. Thinking Skills and Creativity, 7(3), 153-164.
doi:http://dx.doi.org.proxy1.ncu.edu/10.1016/j.tsc.2012.02.001
Drummond, C.K. (2012). Team-based learning to enhance critical thinking skills in
entrepreneurship education. Journal of Entrepreneurship Education 15, 57-63.
Retrieved from
http://eds.a.ebscohost.com.proxy1.ncu.edu/eds/pdfviewer/pdfviewer?sid=d569073
a-de39-426c-a8db-9ed5f9dbaa34%40sessionmgr4005&vid=28&hid=4213
Dwyer, C., Boswell, A., & Elliott, M. (2015). An evaluation of critical thinking
competencies in business settings. Journal of Education for Business, 90(5), 260-
269. doi:http://dx.doi.org/10.1080/08832323.2015.1038978
Dwyer, C., Hogan, M., & Stewart, I. (2012). An evaluation of argument mapping as a
method of enhancing critical thinking performance in e-learning environments.
Metacognition & Learning, 7(3), 219-244. doi:http://dx.doi.org/10.1007/s11409-
012-9092-1
Ennis, R. H. (1996). Critical thinking dispositions: Their nature and assessability.
Informal Logic, 18(2 & 3), 165-182. Retrieved from
http://ojs.uwindsor.ca/ojs/leddy/index.php/informal_logic/article/view/2378/1820
Estes, T. H., Gunter, M. A., & Mintz, S. L. (2011). Instruction: A models approach. (6th
ed.). Boston, MA:Pearson.
Facione, P. A. (2013). Critical thinking: what it is and why it counts. Insight Assessment,
1-27. Retrieved from website:http://www.insightassessment.com/CT-
Resources/Critical-Thinking-What-It-Is-and-Why-It-Counts
Facione, P. A., Facione, N. C., & Giancarlo, C. A. (2000). The disposition toward critical
thinking: Its character, measurement, and relationship to critical thinking skill.
Informal Logic, 20(1), 61-84. Retrieved from
http://ojs.uwindsor.ca/ojs/leddy/index.php/informal_logic/article/view/2254
Fahim, M., & Bagheri, M. B. (2012). Fostering critical thinking through Socrates'
questioning in Iranian language institutes. Journal of Language Teaching &
Research, 3(6), 1122-1127. doi:http://dx.doi.org/10.4304/jltr.3.6.1122-1127

121
Fahim, M., Barjesteh, H., & Vaseghi, R. (2012). Effects of critical thinking strategy
training on Male/Female EFL learners' reading comprehension. English Language
Teaching, 5(1), 140-145. Retrieved from
http://search.proquest.com.proxy1.ncu.edu/docview/993153634?accountid=28180
Fahim, M., & Ghamari, M. R. (2011). Critical thinking in higher education: Globally
developed and locally applied. Theory and Practice in Language Studies, 1(11),
1632-1638. doi:http://dx.doi.org/10.4304/tpls.1.11.1632-1638
Fahim, M., & Masouleh, N. (2012). Critical thinking in higher education: A pedagogical
look. Theory & Practice in Language Studies, 2(7), 1370-1375.
doi:http://dx.doi.org/10.4304/tpls.2.7.1370-1375
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analysis
using G*Power 3.1: Tests for correlation and regression analyses. Behavior
Research Methods, 41, 1149-1160. doi:http://dx.org/10.3758/BRM.41.4.1149
Flores, K. L., Matkin, G. S., Burbach, M. E., Quinn, C. E., & Harding, H. (2012).
Deficient critical thinking skills among college graduates: Implications for
leadership. Educational Philosophy and Theory, 44(2), 212-230.
Foundation for Critical Thinking (FCT). (2013). The critical thinking community.
Retrieved from http://www.criticalthinking.org/pages/international-critical-
thinking-basic-concepts-amp-understan/782
Fraser, J. M., Timan, A. L., Miller, K., Dowd, J. E., Tucker, L., & Mazur, E. (2014).
Teaching and physics education research: bridging the gap. Reports on Progress
in Physics, 77(3), 032401-032417. doi:http://dx.doi.org/10.1088/0034-
4885/77/3/032401
Frodeman, R., Klein, J. T., & Mitcham, C. (2012). The Oxford handbook of
interdisciplinarity. (2nd ed.). New York, NY: Oxford University Press.
Gok, T. (2012). The impact of peer instruction on college students’ beliefs about physics
and conceptual understanding of electricity and magnetism. International Journal
of Science and Mathematics Education, 10(2), 417-436.
doi:http://dx.doi.org/10.1007/s10763-011-9316-x
Gok, T. (2014). Peer instruction in the physics class room: effects on gender difference
performance, conceptual learning, and problem solving. Journal of Baltic Science
Education, 13(6), 776-788. Retrieved from

122
Halpern, D. F. (1998). Teaching critical thinking for transfer across domains. American
Psychological Association, 55(4), 449-455. Retrieved from
http://eds.b.ebscohost.com.proxy1.ncu.edu/eds/pdfviewer/pdfviewer?vid=2&sid=3
76b36c6-6e06-40c7-ab7a-9d826400bd2d%40sessionmgr115&hid=104
Halpern, D. F. (2012). Halpern Critical Thinking Assessment: Test Manual. Mödling,
Austria: Schuhfried GmbH. Retrieved from
http://www.schuhfried.com/viennatestsystem10/tests-test-sets/all-tests-from-a-
z/test/hcta-halpern-critical-thinking-assessment-1/
Halpern, D. F. (2014). Thought and knowledge: An introduction to critical thinking. (5th
ed.). New York, NY: Psychology Press
Hatcher, D. L. (2011). Which test? Whose scores? Comparing standardized critical
thinking tests. New Directions for Institutional Research, (149), 29-39.
doi:http://dx.doi.org/10.1002/ir.378
Heijltjes, A., Gog, T., & Paas, F. (2014). Improving students' critical thinking: Empirical
support for explicit instructions combined with practice. Applied Cognitive
Psychology, doi:http://dx.doi.org/10.1002/acp.3025
Herlo, D. (2010). Implement the transformative learning theory through Dolceta Project.
Journal Plus Education/Educatia Plus, 6(2), 107-114. Retrieved from
http://uav.ro/files/educatie/plus/EPvol8no2.pdf
Hodge, K. A., & Lear, J. L. (2011). Employment skills for 21st Century workplace: The
gap between faculty and student perceptions. Journal of Career and Technical
Education, 26(2), 28-41. Retrieved from
http://www.eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ974462
Holley, K. A. (2009). Special issue: Understanding interdisciplinary challenges and
opportunities in higher education. ASHE Higher Education Report, 35(2), 1-131.
doi:http://dx.doi.org/10.1002/aehe.3502
Hung, W. (2013). Problem-Based Learning: A learning environment for enhancing
learning transfer. New Directions for Adult & Continuing Education, 2013(137),
27-38. doi:http://dx.doi.org/10.1002/ace.20042
Jones, A. (2009). Redisciplining generic attributes: The disciplinary context in focus.
Studies in Higher Education, 34(1), 85-100.
doi:http://dx.doi.org/10.1080/03075070802602018
Jones, R. (2012). What were they thinking? Science Teacher, 79(3), 66-70. Retrieved
from
http://www.nsta.org/publications/browse_journals.aspx?action=issue&id=10.2505
/3/tst12_079_03

123
Kennedy, M., Fisher, M. B., & Ennis, R. H. (1991). Critical thinking: Literature review
and needed research. In L. Idol, B. Jones (Eds.), Educational values and cognitive
instruction: Implications for reform (pp. 11-40). Hillsdale, NJ England: Lawrence
Erlbaum Associates, Inc.
Kennedy, N. (2012). Lipman, Dewey, and philosophical inquiry in the mathematics
classroom. Education and Culture, 28(2), 81-94.
doi:http://dx.doi.org/10.1353/eac.2012.0012
Khandaghi, M. A., Pakmehr, H., & Amiri, E. (2011). The status of college students’
critical thinking disposition in humanities. Procedia - Social and Behavioral
Sciences, 15(0), 1866-1869.
doi:http://dx.doi.org.proxy1.ncu.edu/10.1016/j.sbspro.2011.04.017
Khandaghi, M. A., & Pakmehr, H. (2012). Critical thinking disposition: A neglected loop
of humanities curriculum in higher education. Cypriot Journal of Educational
Sciences, 7(1), 1-13. Retrieved from http://www.world-education-
center.org/index.php/cjes/issue/view/38
Kiener, M., Ahuna, K. H., & Tinnesz, C. G. (2014). Documenting critical thinking in a
capstone course: moving students toward a professional disposition. Educational
Action Research, 22(1), 109-121.
Kim, K., Sharma, P., Land, S., & Furlong, K. (2013). Effects of active learning on
enhancing student critical thinking in an undergraduate general science course.
Innovative Higher Education, 38(3), 223-235.
doi:http://dx.doi.org/10.1007/s10755-012-9236-x
Kong, L. N., Qin, B., Zhou, Y., Mou, S., & Gao, H. (2014). The effectiveness of problem-
based learning on development of nursing students’ critical thinking: A systematic
review and meta-analysis. International Journal of Nursing Studies, 51, 458-469.
doi:http://dx.doi.org/10.1016/j-ijnurstu.2013.06.009
Ku, K. Y. L. & Ho. I. T. (2010). Metacognitive strategies that enhance critical thinking.
Metacognitive Learning, 5, 251-267. doi:http://dx.doi.org/10.1007/s11409-010-
9060-6
Ku, K. L., Ho, I. T., Hau, K., & Lai, E. M. (2014). Integrating direct and inquiry-based
instruction in the teaching of critical thinking: An intervention study. Instructional
Science, 42(2), 251-269. doi:http://dx.doi.org/10.1007/s11251-013-9279-0
Lai, E. R. (2011). Critical thinking: A literature review. Pearson's Research Reports, 6,
40-41. Retrieved from http://www.studymode.com/essays/Critical-Thinking-
Literature-Review-822786.html

124
Lampert, N. (2011). A study of an after-school art programme and critical thinking.
International Journal of Education through Art, 7(1), 55-67. Retrieved from
http://arts.vcu.edu/arteducation/files/2012/02/Lampert_IJETA.pdf
Lansiquot, R. D., Blake, R. A., Liou-Mark, J., & Dreyfuss, E. E. (2011). Interdisciplinary
problem-solving to advance STEM success for all students. Peer Review, 13(3),
19-22. Retrieved from http://www.aacu.org/peerreview/pr-su11/index.cfm
Lloyd, M., & Bahr, N. (2010). Thinking critically about critical thinking in higher
education. International Journal for the Scholarship of Teaching and Learning,
http://digitalcommons.georgiasouthern.edu/ij-sotl/vol4/iss2/9
Loes, C., Pascarella, E., & Umbach, P. (2012). Effects of diversity experiences on critical
thinking skills: Who benefits? Journal of Higher Education, 83(1), 1-25. Retrieved
from
Macke, C., Taylor, J., & Taylor, J. (2013). Using team-based learning to teach social work
research, human behavior, and policy. Journal of Baccalaureate Social Work, 18,
63-75. Retrieved from
http://search.ebscohost.com.proxy1.ncu.edu/login.aspx?direct=true&db=ofs&AN=
89596239&site=eds-live
Marin, L. M., & Halpern, D. F. (2011). Pedagogy for developing critical thinking in
adolescents: Explicit instruction produces greatest gains. Thinking Skills and
Creativity, 6(1), 1-13. doi:http://dx.doi.org/10.1016/j.tsc.2010.08.002
Martin, J. (2013). Refreshing information literacy. Communications in Information
Literacy, 7(2), 114-127. Retrieved from
http://eds.a.ebscohost.com.proxy1.ncu.edu/eds/pdfviewer/pdfviewer?sid=c112bdd
2-630e-482e-b350-02a04fd07823%40sessionmgr4001&vid=8&hid=4208
Mathews, S., & Lowe, K. (2011). Classroom environments that foster a disposition for
critical thinking. Learning Environments Research, 14(1), 59-73.
doi:http://dx.doi.org/10.1007/s10984-011-9082-2
Mazur, E. (1997). Peer instruction: A user’s manual. Upper Saddle, NJ: Prentice Hall.
McArthur, J. (2010). Time to look anew: Critical pedagogy and the disciplines within
higher education. Studies in Higher Education, 35(3), 301-315.

125
Meltzer, D. E., & Thornton, R. K. (2012). Resource letter ALIP-1: Active-learning
instruction in physics. American Journal of Physics, 80(6), 478-496.
doi:http://dx.doi.org/10.1119/1.3678299
Miller, S., Hall, D., & Tice, C. (2009). Assessing critical thinking: The use of literature in
a policy course. Journal of Baccalaureate Social Work, 14(2), 89-104. Retrieved
from https://www.metapress.com/content/4425p5004t361212/resource-
secured/?target=fulltext.pdf
Moore, T. J. (2011a). Critical thinking and disciplinary thinking: a continuing debate.
Higher Education Research and Development, 30(3), 261-274.
Moore, T.J. (2011b). Critical thinking: seven definitions in search of a concept. Studies in
Higher Education, 38(4), 506-522.
Myo-Kyoung, K., Patel, R. A., Uchizono, J. A., & Beck, L. (2012). Incorporation of
Bloom's Taxonomy into multiple-choice examination questions for a
pharmacotherapeutics course. American Journal of Pharmaceutical Education,
http://eds.a.ebscohost.com.proxy1.ncu.edu/eds/pdfviewer/pdfviewer?sid=d569073
a-de39-426c-a8db-9ed5f9dbaa34%40sessionmgr4005&vid=23&hid=4213
Nelson, L. P., & Crow, M. L. (2014). Do active-learning strategies improve students'
critical thinking? Higher Education Studies, 4(2), 77.
doi:http://dx.doi.org/10.5539/hes.v4n2p77
Niu, L., Behar-Horenstein, L. S., & Garvan, C. W. (2013). Review: Do instructional
interventions influence college students’ critical thinking skills? A meta-analysis.
Educational Research Review, 9114-128.
doi:http://dx.doi.org/10.1016/j.edurev.2012.12.002
Norusis, M. J. (2010). PASW statistics 18 guide to data analysis. Upper Saddle River,
NJ: Prentice-Hall, Inc.
Novak, G. M. (2011). Just-in-time teaching. New Directions for Teaching and Learning,
2011(128), 63-73. doi:http://dx.doi.org/10.1002/tl.469
Novak, G .M., Patterson, E. T., Gavrin, A. D., & Christian, W. (1999). Just-in-time
teaching: Blending active learning with web technology. Upper Saddle River,
NJ: Prentice-Hall, Inc.
Novak, J. D. (1990). Concept mapping: A useful tool for science education. Journal of
Research Science Teaching, 27(10), 937–949.
doi:http://dx.doi.org/10.1002/tea.3660271003

126
Ofstad, W., & Brunner, L. J. (2013). Team-Based Learning in Pharmacy Education.
American Journal of Pharmaceutical Education, 77(4), 1-11. Retrieved from
http://search.ebscohost.com.proxy1.ncu.edu/login.aspx?direct=true&db=ehh&AN
=87627859&site=eds-live
Osmond, P., & Goodnough, K. (2011). Adopting Just-in-Time Teaching in the context of
an elementary science education methodology course. Studying Teacher
Education: Journal of Self-Study of Teacher Education Practices, 7(1), 77-91.
Parmelee, D. X., & Michaelsen, L. K. (2010). Twelve tips for doing effective Team-
Based Learning (TBL). Medical Teacher, 32(2), 118-122.
Pascarella, E., Wang, J., Trolian, T., & Blaich, C. (2013). How the instructional and
learning environments of liberal arts colleges enhance cognitive development.
Higher Education, 66(5), 569-583. doi:http://dx.doi.org/10.1007/s10734-013-
9622-z
Paul, R. (2013). The critical thinking community. Retrieved from
http://www.criticalthinking.org
Periklis, P. (2010). Critical thinking as dialectics: A Hegelian-Marxist approach. Journal
for Critical Education Policy Studies, 8(2), 74-102. Retrieved from
http://eds.b.ebscohost.com.proxy1.ncu.edu/eds/pdfviewer/pdfviewer?sid=8cb5b01
b-349c-4c53-a028-72bbd11e6f05%40sessionmgr198&vid=3&hid=116
Pitkäniemi, H., & Vanninen, P. (2012). Learning attainments as a result of student
activity, cognition and the classroom environment. Problems of Education in the
21St Century, 4575-86. Retrieved from
http://www.jbse.webinfo.lt/Problems_of_Education_Volumes.htm
Ravitch, D. (2008). Chapter 3: Education and democracy: The United States of America
as a historical case study. Yearbook of the National Society for the Study of
Education (Wiley-Blackwell), 107(1), 42-57. doi:http://dx.doi.org/10.1111/j.1744-
7984.2008.00129.x
Renaud, R. D., & Murray, H. G. (2008). A comparison of a subject-specific and a general
measure of critical thinking. Thinking Skills and Creativity, 3, 85-893.
doi:http://dx.doi.org/10.1016/j.tsc.2008.03.005
Ricketts, J. C., & Rudd, R. (2004). The relationship between critical thinking dispositions
and critical thinking skills of selected youth leaders in the national FAA
organization. Journal of Southern Agricultural Education Research, 54(1), 21-33.
Retrieved from http://jsaer.org/archive.html

127
Riggs, L., & Hellyer-Riggs, S. (2010). An interdisciplinary/developmental approach to
defining, and teaching. The International Journal of Interdisciplinary Social
Sciences, 5(6), 63-71. Retrieved from
http://iji.cgpublisher.com/product/pub.88/prod.1167
Roy, R. B., & McMahon, G. T. (2012). Video-based cases disrupt deep critical thinking in
problem-based learning. Medical Education, 46, 426-435.
Rué, J., Font, A., & Cebrián, G. (2013). Towards high-quality reflective learning amongst
law undergraduate students: Analysing students’ reflective journals during a
problem-based learning course. Quality in Higher Education, 19(2), 191-209.
Sammut, K. (2014). Transformative learning theory and coaching: Application in practice.
International Journal of Evidence Based Coaching & Mentoring, 39-53. Retrieved
from
http://eds.a.ebscohost.com.proxy1.ncu.edu/eds/pdfviewer/pdfviewer?sid=3d225aee
-4c36-4888-98d6-6d91590467e2%40sessionmgr4001&vid=2&hid=4210
Scott, T., Gray, A., & Yates, P. (2013). A controlled comparison of teaching methods in
first year university physics. Journal of the Royal Society of New Zealand, 43(2),
88-99. doi:http://dx.doi.org/10.1080/03036758.2012.658816
Shaughnessy, J. J., Zechmeister, E. B., & Zechmeister, J. S. (2014). Research methods in
psychology. (10th
ed.). New York, NY: McGraw-Hill Education.
Shimic, G., & Jevremovic, A. (2012). Problem-based learning in formal and informal
learning environments. Interactive Learning Environments, 20(4), 351-367.
Simon, B., & Cutts, Q. (2012). Peer instruction: A teaching method to foster deep
understanding. Communications of the ACM, 55(2), 27-29.
doi:http://dx.doi.org/10.1145/2076450.2076459
Stedman, N. P., & Adams, B. L. (2014). Getting it to click: Students self-perceived critical
thinking style and perceptions of critical thinking instruction in face-to-face and
online course delivery. NACTA Journal, 58(3), 236-243. Retrieved from
http://eds.a.ebscohost.com.proxy1.ncu.edu/eds/pdfviewer/pdfviewer?sid=696b7a6
2-6090-4b3f-937a-f21a9ac8e1c3%40sessionmgr4001&vid=17&hid=4210
Stedman, N. P., & Adams, D. L. (2012). Identifying faculty’s knowledge of critical
thinking concepts and perceptions of critical thinking instruction in higher
education. North American Colleges and Teachers of Agriculture Journal, 56(2),

128
http://eds.b.ebscohost.com.proxy1.ncu.edu/eds/pdfviewer/pdfviewer?sid=6270a2fe
-3f0e-4d4e-b4ab-9ca546e8c023%40sessionmgr110&vid=11&hid=116
Sternberg, R. J. (1986). Critical thinking: Its nature, measurement, and improvement.
National Institute of Education. Retrieved from
http://eric.ed.gov/PDFS.ED272882.pdf.
Stuart, M. (2010). Having Locke's ideas. Journal of the History of Philosophy 48(1), 35-
59. Retrieved from
http://search.proquest.com.proxy1.ncu.edu/docview/210626970?accountid=28180
Sun, J. C.-Y., Martinez, B., & Seli, J. (2014). Just-in-time or plenty-of-time? Different
electronic feedback devices and their effect on student engagement. Educational
Technology & Society, 17(2), 234-244. Retrieved from
http://eds.a.ebscohost.com.proxy1.ncu.edu/eds/pdfviewer/pdfviewer?vid=1&sid=4
8b719c3-db41-4b30-a615-615c62f5b1dd%40sessionmgr4003&hid=4105
Tabachnick, B. G., & Fidell, L. S. (2006). Using multivariate statistics (5th
ed.). Needham
Heights, MA: Allyn & Bacon.
Taylor, E. W. (2007). An update of transformative learning theory: A critical review of the
empirical research (1999-2005), International Journal of Lifelong Education,
26(2), 173-191. doi:http://dx.doi.org/10.1080/02601370701219475
Thomas, I. (2009). Critical thinking, transformative learning, sustainable education, and
problem-based learning in universities. Journal of Transformative Education, 7(3),
245-264. doi:http://dx.doi.org/10.1177/1541344610385753
Tishman, S., & Jay, E. (1993). Teaching thinking dispositions: from transmission to
enculturation. Theory into Practice, 32(3), 147. Retrieved from
http://eds.b.ebscohost.com.proxy1.ncu.edu/eds/pdfviewer/pdfviewer?sid=4cb6870
4-50b9-4552-8774-e94460d92c0d%40sessionmgr115&vid=35&hid=109
Tomczak, M., Tomczak, E., Kleka, P., & Lew, R. (2014). Using power analysis to
estimate appropriate sample size. Trends in Sports Sciences, 21(4), 195-206.
Trochim, W. M. K., & Donnelly, J. P. (2008). The research methods knowledge base, (3rd
ed.). Mason, OH: Cengage Learning.
United States Air Force Academy (2015). Re: It’s more than a university. It’s a way of
life (Online admissions page). Retrieved from
http://www.academyadmissions.com/what-to-expect/
Weiner, J. (2011). Is there a difference between critical thinking and information
literacy? A systematic review 2000-2009. Journal of Information Literacy, 5(2),

129
Wilson-Mulnix, J. (2012). Thinking critically about critical thinking. Educational
Philosophy and Theory, 44(5), 464-479. doi:http://dx.doi.org/10.1111/j.1469-
5812.2010.00673.x
Ya-Ting, C., Yung-Hsin, C., & Cowan, J. (2014). Concentrating on affective feedforward
in online tutoring. British Journal of Educational Technology, 45(4), 694-706.
doi:http://dx.doi.org/10.1111/bjet.12077
Yeo, C. M. (2014). Concept mapping: A strategy to improve critical thinking. Singapore
Nursing Journal, 41(3), 2-7.
Yew, E., & Schmidt, H. (2012). What students learn in problem-based learning: a process
analysis. Instructional Science, 40(2), 371-395.
doi:http://dx.doi.org/10.1007/s11251-011-9181-6
Yin Yin, K., Kanesan Abdullah, A. G., Alazidiyeen, N. J. (2011). Collaborative problem
solving methods towards critical thinking. International Education Studies, 4(2),
58-62. doi:http://dx.doi.org/10.5539/ies.v4n2p58
Zapatero, E. G., Maheshwari, S. K., & Chen, J. (2012). Effectiveness of active learning
environment: Should testing methods be modified? Academy of Educational
Leadership Journal, 16(4), 101-114. Retrieved from
Zhang, L. (2003). Contributions of thinking styles to critical thinking dispositions. The
Journal of Psychology, 137(6), 517-544. Retrieved from
Zimmerman, H., & Land, S. M. (2014). Facilitating Place-Based Learning in Outdoor
Informal Environments with Mobile Computers. Techtrends: Linking Research &
Practice to Improve Learning, 58(1), 77-83. doi:http://dx.doi.org/10.1007/s11528-
013-0724-3

130
Appendixes
Appendix A: Halpern Critical Thinking Assessment Manual

131
Appendix B: Real-World Outcomes Inventory
Think about your day to day experiences over the last three months. Place a “Y” (yes) or
an “N” (no) next to each statement to show if you have experienced the statement (Y) or
if you have not experienced that statement (N) within the last three months.
1a Rented a movie ___
1b Returned a movie you rented without having watched it at all ___
1c Had to pay late fee because you returned it too late___
2a Bought new clothes or shoes___
2b Bought new clothes or shoes you never wore___
3a Gone shopping for food or groceries___
3b Threw out food or groceries you had bought because they went bad___
4a Done your own laundry___
4b Ruined your clothes because you didn’t follow the laundry instructions on the
label___
5a Been enrolled in any kind of school___
5b Missed a class because you slept through your alarm or forgot to set your
alarm___
5c Pulled an “all-nighter”___
5d Forgotten to do a class assignment___
5e Arrived to class only to realized that you had forgotten about an exam that day___
6a Taken a trip by airplane___
6b Missed a flight___
7a Had any form of ID (driver’s license, passport, birth certificate)___
7b Had your ID replaced because you lost it___
8a Carried a key to your dorm room___
8b Had the key to your dorm room replaced because you lost it___
8c Locked yourself out of your dorm room___
8d Unintentionally left the door to your dorm room unlocked___
9a Had a credit card___
9b Had more than $5,000 in credit card debt___
10a Loaned more than $50 to someone___
10b Loaned more than $50 to someone and never got it back___
11a Borrowed more than $50 from someone___
11b Borrowed more than $50 from someone and never paid it back___

132
12a Been out in the sun___
12b Got blisters from sunburn___
13a Ate fast food at least once a week over the last month___
13b Gained 1-2 pounds over last month___
13c Felt sick to the stomach at least once over the last month___
14a Played a video game with a friend___
14b Friend scored higher on a video game___
15a Had to lead your flight in drill commands___
15b Forgot how to execute multiple commands___
15c Was relieved from leading the flight in drill commands___
16a Had to lead a group discussion___
16b Group discussion did not yielded an agreed upon outcome___
17a Volunteered to lead a complex project___
17b Team had difficulty understanding roles and responsibilities___
18a Had to tutor a classmate___
18b Classmate's academic performance did not improve___
19a Played a sport without using protective gear___
19b Broke a bone, sprained a muscle or got hurt___
20a Memorized a physics formula___
20b Got wrong answer despite having memorized and used a physics formula___
21a Purchased herbal remedies that enhance thinking or memory___
22a Own a lucky object? (e.g., rabbit foot, etc.)___
23a Paid to speak to a Psychic (i.e., in person or over-the-phone)___
24a Purchased Airborne___
25a Owned an object with healing properties (e.g, healing crystals, magnetic bracelets,
mystical stones, etc.)___

133
Appendix C: Permission Letter

134
Appendix D: United States Air Force Academy IRB Approval

135
Appendix E: Sample Lesson Introducing Concept Maps
Lesson 18
Kinetic Energy and Power
[Obj 33] Explain the concept of kinetic energy and its relation to work.
[Obj 34] Explain the relation between energy and power.
[Obj 39] Solve problems by applying the work-energy theorem, conservation of
mechanical energy, or conservation of energy.
Participant’s Use of Concept Mapping to
Solve a Physics Problem

Grieco_Luz gradworks.proquest.com 10103879

More Related Content

Viewers also liked (10)

Similar to Grieco_Luz gradworks.proquest.com 10103879 (20)

Grieco_Luz gradworks.proquest.com 10103879