CIA Studies in Intelligence Soviet Deception in the Cuban Missi.docx

CIA Studies in Intelligence: Soviet Deception in the Cuban
Missile Crisis
Available on-line at: https://www.cia.gov/library/center-for-the-
study-of-intelligence/csi-publications/csi-
studies/studies/vol46no1/article06.html
M
a
n
a
g
in
g
t
h
e p
r
iv
at
e sp
ies
PCN 5193

The National Defense Intelligence College supports
and encourages research on intelligence issues that
distills lessons and improves Intelligence Community
capabilities for policy-level and operational consumers
Critical Thinking and Intelligence Analysis, Second Printing
(with revisions)
David T. Moore
This series of Occasional Papers presents the work of faculty,
students
and others whose research on intelligence issues is supported or
otherwise
encouraged by the National Defense Intelligence College
(NDIC) through
its Center for Strategic Intelligence Research. Occasional
Papers are
distributed to Department of Defense schools and to the
Intelligence
Community, and unclassified papers are available to the public
through
the National Technical Information Service (www.ntis.gov).
Selected
papers are also available through the U.S. Government Printing
Office
(www.gpo.gov).
This work builds on earlier publications in this series,
particularly
Occasional Papers Two, Six, and Seven; Getting Intelligence
Right: The Power
of Logical Procedure, Intelligence Essentials for Everyone, and
Intelligence Analysis
in Theater Joint Intelligence Centers: An Experiment in

Applying Structured Methods.
The author’s insights have been developed through years of
creative
interaction with the Community’s leaders in analytic
methododology,
and tested for practicality by successive waves of students in
training
and education courses throughout the Community.
Proposed manuscripts for these papers are submitted for
consideration
to the NDIC Press Editorial Board. Papers undergo review by
senior
officials in Defense, Intelligence and civilian academic or
business
communities. Manuscripts or requests for additional copies of
Occasional
Papers should be addressed to Defense Intelligence Agency,
National
Defense Intelligence College, MC-X, Bolling AFB, Washington,
DC
20340-5100.
This publication has been approved for unrestricted distribution
by the
Office of Security Review, Department of Defense.
Russell [email protected], Editor and Director
Center for Strategic Intelligence Research
CritiCal thinking
and intelligenCe analysis
Occasional Paper Number Fourteen

david t. Moore
National Defense Intelligence College
Washington, DC
March 2007
The opinions expressed in this paper are those of the author.
They do
not represent the official position of the National Security
Agency, the
Department of Defense, or any other agency of the U.S.
Government.
– iii –
Contents
Figures and Tables
........................................................................ v
Preface
......................................................................................... vii
Foreword
Mark M. Lowenthal ...............................................................
ix
Commentary
Jeffrey R. Cooper .................................................................
xiii

Francis J. Hughes..................................................................
xvi
Gregory F. Treverton ...........................................................
xvii
Acknowledgments....................................................................
... xix
Abstract
......................................................................................... 1
Definitions
..................................................................................... 2
Introduction: How Do People Reason?
........................................ 3
What Is Critical
Thinking?............................................................ 8
Defining Critical Thinking ................................................ 8
Standards for Critical Thinking ...................................... 10
Skill-Based Definitions .................................................... 13
A Disposition to Think Critically .................................... 15
The Role of Questions .................................................... 17
Pseudo-Critical Thinking ................................................ 18
What Can Be Learned from the Past?
Thinking Critically about Cuba ............................................
20
Deploying the Missiles..................................................... 20
Assessing the Implications ............................................... 24
Between Dogmatism and Refutation .............................. 26
Lacking: Disconfirmation ................................................ 29
The Roles of Critical Thinking in the Cuban Crisis ...... 33

Winners and Losers: The Crisis in Context .................... 37
Ten Years Later, They Meet Again ................................. 45
Judgment: Critical Thinking Would
Have Made a Difference ........................................... 46
How Can Intelligence Analysts Employ Critical Thinking?
........48
The Poor Record ..............................................................48
Assessing Evidence ...........................................................51
Facilitating Evidentiary Assessment ................................ 56
Embracing a Methodology ............................................. 57
Creating Better Inferences .............................................. 58
Producing Intelligence .................................................... 59
How can Analysts be Taught to Think Critically?
...................... 61
Critical Thinking Education
Outside the Intelligence Community ......................... 61
Inside the Intelligence Community ............................ 64
Implications of Teaching Critical Thinking ................... 66
Evaluating Teaching Models ........................................... 68
Encouraging Analysts to Think Critically ....................... 69
Persuading to Improve Analysis ...................................... 70
How Does Critical Thinking Transform?
................................... 74
Transforming Intelligence Corporations ......................... 74
Learning from Early Adopters ........................................ 75
The Costs and Benefits of Thinking Critically ............... 76
Validation ........................................................................ 84
What Other Points of View Exist?

.............................................. 86
What Does the Future Hold?
...................................................... 90
Conclusion
.................................................................................. 95
Appendix: NSA’s Critical Thinking
and Structured Analysis Class Syllabus .................................
97
Bibliography
.............................................................................. 117
About the Author
...................................................................... 133
Colophon
.................................................................................. 134
– iv –
Figures and tables
Figure 1: A Comparison of Inductive,
Deductive, and Abductive Reasoning .....................................
5
Figure 2: Elements of Thought.
.................................................... 9
Figure 3: A Continuum of Intellectual Standards.
..................... 10

Figure 4: Levels of Questioning:
Critical Thinking in a Social Setting. ....................................
17
Figure 5: Respective Ranges of Soviet SS-4 MRBM
and SS-5 IRBM Missiles.
......................................................... 22
Figure 6: A Place for Analysis between
Dogmatism and Criticism .....................................................
28
Figure 7: Detail of a U-2 Photograph of an SS-4 MRBM
Launch Site, San Cristobal, Cuba, 14 October 1962 .......... 30
Figure 8: U-2 Tracks over Cuba, 4–14 October 1962 ................
32
Table 1: A Comparison of Different Sets of
Critical Thinker’s Competencies ..........................................
14
Table 2: Applying Paul and Elder’s Critical Thinking Model
to the Situation in Cuba, August–September 1962 .............. 27
Table 3: A Comparative Assessment of Rival Motivations
.............. 38
Table 4: Goals and Outcomes in the Cuban Missile Crisis.........
44
Table 5: How Analysts Decide
.................................................... 50
Table 6: Cooper’s Analytic Pathologies

...................................... 82
Table 7: Analysis: Past, Present and Future
................................ 92
– v –
PreFaCe
The world in which intelligence analysts work has changed
dramatically over the 67 years since the beginning of the
Second
World War. Adversaries have shifted from large armies arrayed
on
battlefields to individuals lurking in the shadows or in plain
sight.
Further, plagues and pandemics, as well as floods and famines,
pose
threats not only to national stability but even to human
existence. To
paraphrase a Chinese curse, we certainly live in interesting
times.
Our times demand fresh, critical reasoning on the part of those
tasked to assess and warn about threats as well as those tasked
to act
on those threats. Education in the bases and practices of
intelligence
foraging and sensemaking – often called intelligence collection
and
analysis – is a means by which this can be accomplished.
Indeed, the

Intelligence Reform and Terrorism Prevention Act of 2004
legislates
improved education for intelligence analysis. But, that
education is
not specifically defined. This volume provides a framework for
one
area of the act’s educational requirement: improving how
analysts
think – and by extension, how policymakers act. It asserts that
people who are skilled critical thinkers are better able to cope
with
interesting times than those who are not.
The model for thinking developed here also provides specific
tools for coping with accelerating disruptive technologies. Such
technologies routinely appear in the hands of adversaries.
They
also offer intelligence professionals capabilities to counter
adversaries
in novel ways. The key is knowing which technologies are truly
disruptive in advance, which pose threats, and which can be
harnessed to mitigate threats. Critical thinking – as it is here
defined
and developed – provides part of the solution as it encourages
careful consideration of the available evidence, close
examination of
presuppositions and assumptions, review of the alternate
implications
of decisions, and finally, discussion of alternative solutions
and
possibilities. In short, it equips intelligence professionals with
an
essential tool for their work.
– vii –

– viii –
Foreword
Mark M. Lowenthal
Former Assistant Director of Central Intelligence
for Analysis and Production
President, Intelligence & Security Academy, LLC
Some years ago, when I worked at the Congressional Research
Service, my colleague next door had a sign in his office:
“Thinking
is hard work. That’s why so few people do it.” Indeed. As
sentient
beings we think all the time. Our waking, conscious hours are
full
of thoughts: what to wear; what to eat; how to respond to
someone.
These all take thought. One might venture to say that we think
almost unconsciously through most of our day.
And then there is that other thinking: Thinking about the
conundrums that we face, the alternatives and choices we have
to
make on larger issues, the dilemmas we wish to resolve. This is
hard
work. And if the work you do is largely intellectual in nature,
then
you are thinking that much more often about these more
difficult
problems. This is not to suggest that those who work with their

hands or in crafts or industry do not think. Of course they do.
But
these people, in the end, have something physical to show for
their
thought and its execution. Intellectuals only have their thoughts
to
show, presumably expressed for others in some way.
Intelligence officers are engaged in an intellectual pursuit.
They
are trying to solve puzzles, resolve uncertainties, discover the
nature
and meaning of things that others would keep secret. They
must have
an entire intellectual apparatus to help them identify the
problem,
assess the parts they know and the parts they do not, come up
with
an explanation of what is going on and then express it in a way
that
others – including an audience not steeped in their own
techniques
– can understand.
– ix –
– x –
As one would expect, the Intelligence Community has spent a
fair amount of time thinking about how it does analysis. But
most
of this has been spent on techniques, presentation and
outcomes;

much less has been spent on what happens intellectually while
you are
thinking. That, indeed, is what critical thinking is about: the
ability
to step out of one’s thoughts, if you will, to examine them and
the
process that brought them about even while you are thinking
them.
It does not require some higher level of cosmic consciousness
or a
split personality. It does require training and thought! So now
we
are thinking about our thinking while we are thinking – a mental
triple play, if you will. This is no mean feat and it is a very
crucial
skill for intelligence analysts.
John McLaughlin, a former Deputy Director of Central
Intelligence, has observed that one of the major perceptions
that
separates intelligence officers from the policy makers is that
policy
makers tend to be optimists, believing they can get things done,
while intelligence officers tend to be skeptics, knowing that
little
is known with certainty. Like everyone else, however,
intelligence
analysts can be adamant about their conclusions, understanding
the
path they took to reach them, and well-satisfied that they have
done
the right work and come to the right end. That is where critical
thinking comes in. The ability to examine how you came to
your
conclusion is an important adjunct to your other intellectual
work.

You may be certain or you may not – which is also acceptable.
But
you owe it to yourself and to your readers to have examined
how
you got there.
This is not as easy as it sounds for intelligence analysts for the
simple reason that they are usually very pressed for time. There
are not a lot of lulls in the analyst’s work week, which makes it
very
difficult to be introspective. This means that critical thinking
ought
to be something that is ingrained in analysts as part of their
training,
so that it becomes reflexive. But to do that, one must first
understand
critical thinking – what it is, how to do it, how to teach or learn
it.
That is the outstanding value of this monograph by David
Moore.
The Intelligence Community is filled with people happily
working away under a cloak of anonymity, satisfied in making
their
contribution to part of a larger activity about which very few
beyond
their immediate circle of co-workers will know. David Moore
has
long been one of these contributors. In addition to his duties as
an
intelligence officer at the National Security Agency, David
Moore
has devoted many fruitful hours to the intellectual
underpinnings of

intelligence, especially to what makes analysts and what makes
better
analysts. His work on analytic core competencies, written with
his
colleague Lisa Krizan, is a fascinating dissection of the
intellectual
attributes that make up an intelligence analyst. It was also a
very
influential work, becoming a prime motivator in the creation of
the Analytic Resources Catalog created under Director of
Central
Intelligence George Tenet and now much touted by Director of
National Intelligence John Negroponte.
This new monograph on critical thinking will serve a similar
purpose. Much has been written in the last five years about
intelligence
and about the errors that it can make. Some of this has been
diagnostic; much of it has been blithely condemnatory.
Prescriptions
about connecting dots better (an absurd metaphor), about the
need to
share information better and alarms about groupthink do not get
us
closer to improvements in the art, craft and profession of
intelligence.
New ways of thinking about our profession, new ways of doing
our
work and new ways of assessing it will be useful. Critical
thinking
needs to be at the top of this list of desiderata.
I would like to believe that we are on the verge of a
renaissance in
intelligence studies, and more specifically in the intellectual
basis of

intelligence analysis. There are many straws in the wind:
discussions
about a national intelligence university and a lessons learned
center,
more schools offering serious courses about intelligence and
books
like this. Critical thinking is and should be critical for all
analysts
and they will profit from this book.
– xi –
– xii –
CoMMentary:
Jeffrey R. Cooper
SAIC Technical Fellow
Science Applications International Corporation
David Moore has written an elegant and largely persuasive
argument for the Intelligence Community to move forthrightly
and adopt critical thinking in order to improve the quality both
of
its analytic processes and its intelligence judgments. Since I
have
recently written about the need for the Community to identify
and
cure deep-seated and systemic “analytic pathologies,” I am
obviously
sympathetic to the general case for incorporating more rigorous

analytic processes—among which “critical thinking” is one
attractive
approach.1 Along with Moore, I also believe strongly that the
traits
often associated with critical thinking need to become
fundamental
characteristics of every analyst and to be practiced consistently
in
their work habits. Moore discusses these aspects at some length
and
highlights their value to making sounder judgments.
In doing so, he makes a compelling case that critical thinking
can help analysts reshape their methods in ways that help to
avoid
damaging errors and enhance judgment. He and I agree with
Rieber
and Thomason that we should validate the potential
improvement
before widespread deployment.2 The need for validation raises
some related points concerning the criteria for evaluation and
measurement. As Philip Tetlock has noted, on the one hand,
one
can do evaluation by assessing whether a process (assumed to
be
good) has been followed; one the other hand, one can also
assess
1 Jeffrey R. Cooper, “Curing Analytic Pathologies: Pathways
to Improved
Intelligence Analysis” (Washington, DC: Central Intelligence
Agency, Center for
the Study of Intelligence, 2005). Cited hereafter as Cooper,
Pathologies.
2 Steven Rieber and Neil Thomason, “Toward Improving

Intelligence
Analysis: Creation of a National Institute for Analytic
Methods,” Studies in Intelligence
49, no. 4 (Winter 2006), 71.
– xiii –
– xiv –
quality by looking at the products themselves.3 However,
following
a good process does not guarantee good products, nor does a
good
judgment necessarily validate the quality of the analysis or the
analyst.
Both approaches have flaws as methods of validation.
I urge the leadership of the Intelligence Community to place
far more emphasis on structured analytic methods. In my view,
the transformation of the intelligence enterprise demands a
more curious, more agile, and more deeply thoughtful cadre of
intelligence analysts—but it should also require the same traits
among its intelligence organizations and the intelligence
enterprise
as a whole. Moore notes that “Investment in critical thinking as
part of the analysis process minimizes the likelihood of
specific
failures” (page 81). However, from my perspective, critical
thinking
(and other structured methods) are more important for changing
the
organization’s overall approach to analysis, rather than in
improving
specific judgments or preventing particular failures.

I believe such methods are crucial in preventing systemic
analytic
pathologies from developing, exactly because the Community
lacks
many of the desirable self-corrective mechanisms found in
science.
Second, while Moore focuses on the role of critical thinking in
improving an individual analyst’s ability to make good
judgments,
my view emphasizes the importance of more rigorous processes
for
the organization as a whole if it is to improve its capacity to
meet
user needs. Indeed, given my emphasis on the systemic nature
of
the pathologies that afflict intelligence analysis, structured
analytic
methods become a first line of defense in preventing networks
of
errors—they are like “ripstops” that keep problems from
propagating
into wider “error-inducing systems,” in Perrow’s terms. The
quality
of “mindfulness” and a more self-reflective process are
essential if
the intelligence organizations are to acquire some of the
desirable
characteristics of high-reliability organizations. While critical
thinking
can clearly assist individual analysts, and these “memes” can be
spread through viral dissemination, I would place far more
emphasis
on fomenting social and organizational changes as remedies.
3 Philip E. Tetlock, Expert Political Judgment: How Good Is

It? How Can We
Know? (Princeton, NJ: Princeton University Press, 2005).
Finally, exactly because Moore focuses so tightly on the value
of
critical thinking to improve how individual analysts reason, he
largely
ignores the important role such habits should, and could, play in
the
“social construction” of intelligence analysis and its
communication to users
of that intelligence. The use of more structured techniques “to
present
more effective assessments” should go beyond simply
convincing
policymakers “to question their own assumptions on the issues”
(page
80). By restructuring both the analytic process and the
modalities of
interactions with policy users, critical thinking techniques can
draw
policy users into “co-creating” important judgments and, by
doing
so, get them to adopt some of the important attributes of
critical
thinking. This internalization would be more powerful than
hoping
the exquisite brilliance of the improved analytic judgments
might
make them more palatable.
These minor quibbles aside, David Moore has highlighted for
the Intelligence Community and its users some tangible methods

for improving intelligence analysis and addressing systemic
analytic
pathologies.
– xv –
– xvi –
CoMMentary
Francis J. Hughes
Professor, National Defense Intelligence College
The truly difficult problems of the information age are not
technological; rather, they concern ourselves – what it is to
think, to reason, and to engage in conversation [using] …
new analytic techniques, new conceptual tools with which
to analyze and understand the workings of the human
mind.
—Keith Devlin, Goodbye Descartes: The End of Logic
and the Search for a New Cosmology of the Mind, 1997
David Moore has undertaken a valuable initiative on behalf
of the Intelligence Community by producing this monograph on
critical thinking. In doing so, he has addressed the concerns of
Keith Devlin and many other scholarly writers on the subject of
human reasoning. By contextualizing the concept and practices
of critical thinking within intelligence analysis, he has set forth
a
much-needed cognitive linkage between intelligence analysis
and
the human thought process, which, as he states, remains
otherwise

“poorly understood.”
Mr. Moore has developed a methods-of-thinking course,
outlined
in the Appendix to this paper, which should guide the collection
of evidence, the reasoning from evidence to argument, and the
application of objective decision-making judgment. Critical
thinking
in intelligence depends primarily on a conscious application of
suitable habits of thought, and the course certainly promises to
advance the education of intelligence professionals.
CoMMentary
Gregory F. Treverton
RAND Corporation
Former Assistant Director of Central Intelligence for
Analysis and Production
and Vice-Chairman, National Intelligence Council
David Moore has added his powerful voice to those calling for
America’s intelligence analysts to be more self-conscious about
their
methods and more venturesome in applying more formal
methods.
His starting point is critical thinking, but his strong case
embraces
a range of techniques from Paul and Elder’s checklist, to
analysis
of competing hypotheses (ACH), to Factions analysis, which is
really

a way to be more systematic in aggregating subjective
judgments,
to techniques such as those developed by the Novel Intelligence
from Massive Data (NIMD) project. These help analysts search
for
patterns and test hypotheses. Moore’s discussion pays
particular
attention to critical thinking about evidence – that is perhaps
natural
given that the National Security Agency is his home agency, but
it
is also a welcome emphasis.
My own explorations of the analytic agencies over the last few
years confirm my own experience managing the National
Intelligence
Council (NIC): In general, U.S. intelligence analysis has been
neither
very self-conscious about its methods, nor made much use of
machines
and formal analytic tools. That is Moore’s point of departure,
and it
is a case made graphically by Rob Johnston, on whose work
Moore
draws. The state of affairs Moore and others describe is
changing,
and his work should contribute to accelerating that change, in a
variety of experiments and pilot projects.
Moore’s paper raises at least two intriguing questions at its
edges.
He rightly takes issue with Malcolm Gladwell’s book about the
power of unconscious thought or “deliberation without
attention.”
– xvii –

– xviii –
That said, clever defense lawyers often discredit ballistic
experts by
asking them to be explicit about how they reached their
conclusions.
Likewise, chess masters can be reduced to middle-weights by
having
to explain the logic of their moves. In both cases, the
decisions result
from patterns as much sensed as seen, the result of thousands
of
previous cases. I sometimes worried at the NIC that the more
we
required analysts to be explicit about their methods, the more
we
risked turning them into middle-weights.
Finding ways to incorporate not just expertise, but also hunch
and sense into intelligence analysis is a challenge that lies
ahead. The
Defense Advanced Research Projects Agency’s ill-fated
experiment
a few years ago seeking to establish a terrorism futures market
was
testimony to how hard the challenge will be. When the idea
became
public, every politician – right, left and center – quickly
denounced
it as borderline immoral. Over the ensuing days, however, a
wide
range of analysis commented on what a good idea it might have
been when information is in short supply and there is a premium

on sensing patterns before they become facts.
The second issue is consumers of intelligence, who figure only
at
the margins of Moore’s discussion. We can only hope he is
right that
better analysis will find, and perhaps help create, a better
audience.
But the unboundedness of the terrorist threat we now face
means
that intelligence and policy no longer share a “story”
comparable
to that about state threats, like the Soviet Union. In principle,
that
would suggest that policy should accord more time and attention
to intelligence, but that is not likely to happen in Washington’s
whirligig.
Thus, intelligence will need to be as creative in finding new
ways
to work and communicate with policy officials as it is in
improving
its analytic techniques. In the process, it will need to rethink
what
it regards as its product. I came to think at the NIC that our
product
was not really National Intelligence Estimates but, rather,
National
Intelligence Officers – people, not paper, and people in a
position to
come to know their policy counterparts and to make their
expertise
available informally, in meetings or at lunch.

aCknowledgMents
The first notes for this paper were begun a year prior to the 11
September 2001 terrorist attacks in New York and Virginia as
part
of research conducted with a peer, Lisa Krizan, on a set of
core
competencies – knowledge, skills, abilities, and an associated
set of
personal characteristics – for successful intelligence analysis.
At the
time, the notion of critical thinking was at best vaguely
understood by
members of the Intelligence Community, and the National
Security
Agency (NSA) was no exception. Six years later critical
thinking is
a recognized key skill by which intelligence readiness is defined
at
NSA. Similar growth in its importance is recognized elsewhere
in
the Community. This recognition is due in part to the work of
Dr.
Linda Elder, Dr. Gerald Nosich and Dr. Richard Paul, all of the
Foundation for Critical Thinking. They have created a practical
model for critical thinking that is both effective and easy to
teach.
That the Paul and Elder model is known within the Intelligence
Community is due in no small part to Professor Francis J.
Hughes
of the National Defense Intelligence College (NDIC). It was
he who
introduced the author to the model and encouraged attendance at
Paul and Elder’s annual conference on critical thinking. He
kindly

also provided a commentary on the paper. Dax Norman, of the
NSA Associate Directorate for Education and Training, a
number
of NSA colleagues, and approximately 250 participants in the
NSA
critical thinking and structured analysis course helped make that
course a reality and additionally provided valuable insights into
the
nuances of critical thinking. Comments from LT Robert D.
Folker,
USAF, also provided insight into the teaching of critical
thinking. LT
Folker developed a similar course while a student at the NDIC
that
unfortunately did not remain in the curriculum.
As the paper progressed, a number of colleagues and peers
provided feedback, comments, and challenges to the arguments
presented. They include, in alphabetical order, Solveig
Brownfeld,
NDIC; James Bruce, SAIC; Jan Goldman, NDIC; Mary Rose
Grossman, the John F. Kennedy Presidential Library and
Museum;
Alexandra Hamlet; Noel Hendrickson, James Madison
University;
– xix –
Robert Heibel, Mercyhurst College; Richards J. Heuer, Jr.,
Central
Intelligence Agency (retired); Norval J. Hilmer, Defense
Intelligence
Agency; LTCOL James Holden-Rhodes, USMC (retired);
Morgan D.

Jones, Central Intelligence Agency (retired); Hugo Keesing,
Defense
Intelligence Agency (retired); Mark Marshall, NDIC; Stephen
Marrin,
University of Virginia; Montgomery McFate, Institute for
Defense
Analysis; William Reynolds, Least Squares Software; the staff
of the
Central Intelligence Agency’s Sherman Kent Center; and the
staff
of the John F. Kennedy Presidential Library and Museum.
Several
other colleagues at the National Security Agency – who wish to
remain anonymous – also offered valuable insight and
suggestions.
Three colleagues, Mark Lowenthal, president of the
Intelligence
& Security Academy, LLC; Jeffrey Cooper, Science
Applications
International Corporation; and Gregory Treverton, Rand
Corporation, also deserve mention. Mark kindly agreed to write
the foreword and offered valuable suggestions. His quip about
really
good analysts set up much of the arguments presented herein.
Both
Jeff and Greg provided critical comments and suggestions late
in
the game. They also kindly agreed to write commentaries on
the
paper.
Russell Swenson, Director of NDIC’s Center for Strategic
Intelligence Research and NDIC Press editor, did what a really
good
editor is supposed to do: His input and recommendations

ensured
that the arguments that follow were focused and strong. The
Center’s
William Spracher also provided valuable technical editing
services.
Colleague Lisa Krizan provided a final review of the text that
proved
extremely valuable. This paper would not have been published
if
not for their dedicated assistance to the author.
Acknowledgment is due to four other people. Peter and Mary
Moore encouraged their son to be inquisitive and to always look
beyond the obvious for answers. Anthropologist John McDaniel
first introduced the author – then a student at Washington and
Lee
University – to an analogy for intelligence sensemaking:
science-based
field archaeology. Finally, Elizabeth Moore provided
tremendous
encouragement. She also graciously tolerated the many nights
and
weekends spent writing, listening to, and commenting on, ideas
as
– xx –
they occurred, sharpening the resulting arguments. It is to these
people that this work is dedicated.
While the author was privileged to receive assistance of others,
this remains his work and any errors are his. The author may be
reached via electronic mail at [email protected]
– xxi –

– � –
CRITICAL THINKING
AND INTELLIGENCE ANALYSIS
DAviD T. Moore
Examples included in this paper were selected to illustrate
points raised by the author. No interest by the National
Security
Agency, the Department of Defense, or any other agency of the
U.S. Government should be inferred from their inclusion.
AbsTrAcT
Analysts and analysts alone create intelligence. Although
technological marvels assist analysts by cataloguing and
presenting
data, information and evidence in new ways, they do not do
analysis.
To be most effective, analysts need an overarching, reflective
framework to add structured reasoning to sound, intuitive
thinking.
“Critical thinking” provides such a framework and goes further,
positively influencing the entire intelligence analysis process.
Analysts
who adopt critical thinking stand to improve their analyses.
This
paper defines critical thinking in the context of intelligence

analysis,
explains how it influences the entire intelligence process,
explores
how it toughens the art of intelligence analysis, suggests how it
may
be taught, and deduces how analysts can be persuaded to adopt
this
habit.
– � –
DefiNiTioNs
Thinking – or reasoning – involves objectively connecting
present
beliefs with evidence in order to believe something else.
Critical Thinking is a deliberate meta-cognitive (thinking about
thinking) and cognitive (thinking) act whereby a person reflects
on the
quality of the reasoning process simultaneously while
reasoning to
a conclusion. The thinker has two equally important goals:
coming
to a solution and improving the way she or he reasons.
Intelligence is a specialized form of knowledge, an activity,
and an organization. As knowledge, intelligence informs
leaders,
uniquely aiding their judgment and decision-making. As an
activity,
it is the means by which data and information are collected,
their
relevance to an issue established, interpreted to determine likely

outcomes, and disseminated to individuals and organizations
who
can make use of it, otherwise known as “consumers of
intelligence.”
An intelligence organization directs and manages these
activities to
create such knowledge as effectively as possible.
– � –
CRITICAL THINKING
AND INTELLIGENCE ANALYSIS
iNTroDucTioN: How Do PeoPle reAsoN?
The best analytical tool remains a really good analyst.
—Mark Lowenthal, Former Assistant Director of
Central Intelligence for Analysis and Production
To create intelligence requires transformations resulting from
an
intellectual endeavor that sorts the “significant from [the]
insignificant,
assessing them severally and jointly, and arriving at a
conclusion by
the exercise of judgment: part induction, part deduction,” and
part
abduction.
�
That endeavor is known as thinking, or “that operation

in which present facts suggest other facts (or truths).”
�
Thinking
– or as it is sometimes known, reasoning – creates an “objective
connection” between our present beliefs and “the ground,
warrant,
[or] evidence, for believing something else.”
�
These three reasoning processes trace the development of
analytic
beliefs along different paths. Whereas inductive reasoning
reveals
“that something is probably true,” deductive reasoning
demonstrates
“that something is necessarily true.”
�
However, both are limited:
� William Millward, “Life in and out of Hut �,” in F. H.
Hinsley and
Alan Stripp, Codebreakers: The Inside Story of Bletchley Park
(Oxford, UK: Oxford
University Press, �99�), ��. The author adds “abduction” for
reasons that will
shortly become evident.
� John Dewey, How We Think: A Restatement of the Relation
of Reflective Thinking
to the Educative Process (New York, NY: D.C. Heath and
Company, �9�0), ��. Cited
hereafter as Dewey, How We Think.

� Dewey, How We Think, ��.
� David A. Schum, “Species of Abductive Reasoning in Fact
Investigation
in Law,” Cardozo Law Review ��, nos. �–�, July �00�,
��, emphasis added. Cited
– � –
inductive reasoning leads to multiple, equally likely solutions
and
deductive reasoning is subject to deception. Therefore, a third
aid to judgment, abductive reasoning, showing “that something
is plausibly true,” can offset the limitations of the others.
�
While
analysts who employ all three guides to sound judgment stand to
be the most persuasive, fallacious reasoning or
mischaracterization
of rules, cases, or results in any of the three can affect
reasoning
using the others.
Inductive reasoning, moving from the specific to the general,
suggests
many possible outcomes, or the range of what adversaries may
do in the
future. However, inductive reasoning lacks a means to
distinguish among
each outcome – all are possible. An analyst has no way of
knowing
whether a solution is correct.

Deductive reasoning on the other hand, moving from the general
to
the specific, addresses questions about adversarial behavior and
inten-
tions. Deductive reasoning becomes essential for warning.
Based on
past perceptions, certain facts indicate specific outcomes. If,
for exam-
ple, troops are deployed to the border, communications are
increased,
and leadership is in defensive bunkers, then war is imminent.
However,
if leadership remains in the public eye then these preparations
indicate
that an exercise is imminent.
Abductive reasoning reveals plausible outcomes to the
intelligence
analyst. When an adversary’s actions defy accurate
interpretation
through existing paradigms, abductive reasoning generates
novel means
of explanation. In the case of intelligence warning, an
abductive process
presents policy–making intelligence consumers with an
“assessment
of probabilities.” Although abduction provides no guarantee
that the
analyst has chosen the correct hypothesis, the probative force of
the ac-
companying argument indicates that the most likely hypothesis
is known
and that elusive, actionable intelligence is on tap.
hereafter as Schum, “Species.” American mathematician,

logician, semiotician,
and philosopher Charles Sanders Peirce (��9–�9��)
developed the concept of
abductive reasoning. See Joseph Brent, Charles Sanders Peirce:
A Life (Bloomington,
IN: Indiana University Press, �99�).
� Schum, “Species,” ��.
– � –
Figure � models analysts’ thinking about HUMINT received
during the summer of �9�� regarding reports of foreigners in
Cuba.
An initial (fallacious) conclusion derived from inductive
reasoning
apparently was reapplied through deductive reasoning as
explained
in the case study incorporated in this paper.
9
Something else was
needed – but not considered – to challenge the initial conclusion
that “All HUMINT from Cuba was false.”
Figure 1: A Comparison of Inductive, Deductive, and
Abductive Reasoning
Sources: Adapted from Thomas A. Sebeok, “One, Two, Three
Spells UBER-
TY,” in Umberto Eco and Thomas A. Sebeok, The Sign of
Three: Dupin, Hol-
mes, Pierce (Bloomington, IN: Indiana University Press,

�9��), �; and James H.
Hansen, “Soviet Deception in the Cuban Missile Crisis,” Studies
in Intelligence,
��, no. � (�00�), ��.
If analysts can model an adversary’s intentions based on
observable
actions, adversaries also can deduce how those analysts will
have
9 While there is no evidence it was applied in the Cuban case,
abductive
reasoning is included in the figure for purposes of illustration.
– � –
moved toward these conclusions. Under these circumstances,
the
adversary then can engage in deceptive practices that can lead
an
intelligence service to misinterpret that adversary’s intentions.
�0
For example, the Indian government deceived the U.S.
Intelligence
Community before the former’s �99� nuclear test. Through a
�99� demarche from the U.S. government, the Indian
government
knew what indicators the U.S. sought and was able to obscure
them
until after the test. Then, the Indian government boasted of its
success.

��
Abduction forces a close consideration of the evidence at hand.
Given competing estimates – such as “war is imminent” or “an
exercise will occur” – the body of facts and inferences must be
examined to determine “whether there are facts which falsify
one
of the estimates.”
��
Only then can an assessment of accuracy be
determined. Even then, the selected estimate remains only the
“most likely.”
��
Despite their individual limitations, induction, deduction, and
abduction taken together offer a means of thoroughly
examining
evidence in order to arrive at accurate intelligence conclusions.
However, as becomes obvious from studying figure 1, in order
to
be successful, intelligence analysis requires something more: an
overarching framework is needed to ensure reasoning relies on
valid
assertions, is not fallacious, and is self–consciously objective.
Critical
thinking provides that framework by ensuring that each form of
reasoning is appropriately used. Critical thinking extends to the
entire
intelligence analysis process. The claim here is that analysts
who

10 Isaac Ben-Israel explores the duality of estimation in
considerable detail.
See “Philosophy and Methodology of Intelligence: The Logic
of the Estimate
Process, Intelligence and National Security �, no. � (October
�9�9), ��0–��. Cited
hereafter as Ben-Israel, “Logic of the Estimate Process.”
�� Paul J. Raasa, “The Denial and Deception Challenge to
Intelligence,”
in Roy Godson and James J. Wirtz, Strategic Denial and
Deception (New Brunswick,
NJ: Transaction Publishers, �00�), ��, ��–��.
�� Ben-Israel, “Logic of the Estimate Process,” ��.
�� Cynthia M. Grabo, Anticipating Surprise: Analysis for
Strategic Warning
(Washington, DC: National Defense Intelligence College,
�00�), ��.
– � –
become better critical thinkers will improve their analyses,
helping to
lessen the likelihood of intelligence failures. In the example
above,
critically thinking analysts would have questioned the
generalizing
of their initial conclusion by asking, “Was all the HUMINT
from
Cuba really false?” To avoid the tautological problem of
equating
quality analysis with the concept of “critical thinking” requires

a
careful assessment of cognition in the intelligence
environment. The
underlying question is “How can the intelligence analysts to
whom
Lowenthal’s epigraph refers be ‘really good’?”
– � –
wHAT is criTicAl THiNkiNg?
Defining Critical Thinking
Defining critical thinking is a first step to understanding how it
contributes to intelligence analysis. Richard Paul and Linda
Elder,
of the Foundation for Critical Thinking, consider it to be
that mode of thinking – about any subject, content, or problem
– in which the [solitary] thinker improves the quality of his
or her thinking by skillfully taking charge of the structures
inherent in thinking and imposing intellectual standards
upon them.
��
In other words, critical thinking is both a deliberate meta-
cognitive
(thinking about thinking) and cognitive (thinking) act whereby a
person
reflects on the quality of the reasoning process simultaneously
while
reasoning to a conclusion. The thinker has two equally

important
goals: improving the way she or he reasons and coming to a
correct
solution.
��
To accomplish these goals, the reasoner requires assisting
structures. Paul and Elder define eight elements of reasoning,
shown
in figure 2. These elements lead thinkers to ask focused
questions
about the topic being considered and the thinking process itself.
Paul and Elder assert that whenever people reason, they do so
for a
purpose. This reasoning exists within a point of view and is
shaped
by both conscious and unconscious assumptions. Reasoning
involves
the creation of inferences based on conceptual frameworks
about
reality. These inferences are generated as people consider
evidence
necessary to answer questions or solve problems. Further,
reasoning
leads to decision points with implications – things that might
happen
�� Richard Paul and Linda Elder, The Miniature Guide to
Critical Thinking
Concepts and Tools, �th Edition (Dillon Beach, CA: The
Foundation for Critical
Thinking, �00�), �. Cited hereafter as Paul and Elder,
Concepts and Tools.
15 Exactly what is the “correct solution” is issue and context

dependent.
What is important here is that the process of reasoning is
enhanced.
– 9 –
– and consequences – things that do happen once the decision
has
been made. Finally, Paul and Elder insist that “learning to
analyze
thinking requires practice in identifying the structures in use.”
��
Figure 2: Elements of Thought
Source: Derived from Richard Paul and Linda Elder, The
Miniature Guide to
Critical Thinking Concepts and Tools, �th Edition (Dillon
Beach, CA: The Foun-
dation for Critical Thinking, �00�), �.
�� Linda Elder and Richard Paul, The Foundations of
Analytic Thinking: How
to Take Thinking Apart and What to Look for When You Do
(Dillon Beach, CA: The
Foundation for Critical Thinking, �00�), �. Cited hereafter as
Elder and Paul,
Analytic Thinking. Paul and Elder, Concepts and Tools, �.
– �0 –

Standards for Critical Thinking
Figure 3: A Continuum of Intellectual Standards
Source: Derived from Paul and Elder, Concepts and Tools, 9.
Paul and Elder also establish a set of intellectual standards that
offer criteria for assessing the level and quality of thinking.
These
form a continuum as shown in figure 3.
��
Active – or Socratic
– questioning in a social setting becomes the means of
critically
exploring a topic or discipline as well as assessing the thinking
on that
topic. Questions assessing the issue delve into purpose,
assumptions,
inferences, as well as points of view on a topic.
��
Paul and Elder’s
�� Elder and Paul, Analytic Thinking. �–��.
�� Paul and Elder, Concepts and Tools, �–9.
– �� –
approach ensures that a topic or issue is thoroughly, logically,
and
objectively developed. Their continuum provides qualitative

metrics
to inform the thinker and others who may evaluate the object of
the
thinking along with the process employed. Each standard is
related
to but independent of the others. The object of consideration
might
be vaguely or clearly presented. But it also might be
inaccurate. An
example of this would be “the world is flat.” This clear
statement
of reality is false.
The reasoning or its object might be clearly and accurately
presented but remain imprecise, the level of detail required
depending
on the issue itself. Precision becomes important in assessing
the
location of an adversary or a threat as in “terrorists placed the
improvised explosive device (IED) near the highway.”
It is possible to create clear, accurate, and precise answers that
are irrelevant. If one is considering the IED along the highway
but
the issue is how best to fly the troops in question from one
location
to another then the answer is irrelevant. Similarly, topics and
their
accompanying reasoning need to consider both the complexities
involved (depth) and other points of view (breadth).
�9
Such reasoning
becomes essential when systems must be assessed because in
systems

“chains of consequences extend over time and many areas: The
effects of action are always multiple.”
�0
Intelligence often must assess
such systems – terrorist networks are but one example.
Additionally,
as sociologist Emile Durkheim observes, the combinations of
elements
related to an issue “combine and thereby produce, by the fact of
their combination, new phenomena.”
��
When the correct phenomena or questions – and related
alternative explanations – are not fully considered, intelligence
failures
occur. For instance, the Senate Select Committee on
Intelligence
�9 Elder and Paul, Analytic Thinking, �.
�0 Robert Jervis, System Effects: Complexity in Political and
Social Life (Princeton,
NJ: Princeton University Press, �99�), �0. Cited hereafter as
Jervis, System
Effects.
�� Emile Durkheim, The Rules of Sociological Method
(Glencoe, IL: Free Press,
1938), xlvii.

– �� –
found such reasoning to be among the causes of the
Intelligence
Community’s failure to assess accurately Iraq’s alleged
programs to
develop weapons of mass destruction (WMD). The senators
noted
that “Intelligence Community managers…did not encourage
analysts
to challenge their assumptions, fully consider alternative
arguments,
accurately characterize the intelligence reporting, or counsel
analysts
who lost their objectivity.”
��
Logic and significance also come into play. If the evidence
presented does not imply the conclusions then the results are
illogical. Focusing on appropriate evidence maintains
significance.
For example, evidence that terrorists who place an IED also
have
Stinger surface–to–air missiles bears on how troops can move
across
hostile territory.
Fairness deals with the agendas – hidden and expressed – of the
thinker, her collaborative peers, and her customers. Knowing
who
has a stake in an issue as well as the stakes themselves helps
ensure
that issues are fairly reasoned and presented. Such
considerations also
reflect the analyst’s own biases or opinions. In considering this

point,
Paul and Elder offer the evaluative question, “Am I
misrepresenting
a view with which I disagree?”
��
Finally, while getting to the right-hand side of the spectrum on
each of these standards in figure 3 is highly desirable, the
standard
of thinking on an issue will vary based on the skills of the
thinkers
and the issues under scrutiny. Assessing the resulting shortfalls
in
thinking reveals gaps that can be corrected. The continuum
therefore
provides a detailed assessment by which thinking on issues can
be
improved.
�� Senate Select Committee On Intelligence, Report on the
U.S. Intelligence
Community’s Prewar Intelligence Assessments on Iraq, United
States Senate, �0�th Congress,
� July �00�, ��. Cited hereafter as SSCI, Iraq.
�� Elder and Paul, Analytic Thinking,, �.
– �� –
Skill-Based Definitions
Other approaches to defining critical thinking focus on the
specific

skills. For example, Diane Halpern considers that
[critical] thinking is the use of those cognitive skills or
strategies
that increase the probability of a desirable outcome. It is…
thinking that is purposeful, reasoned, and goal directed – the
kind of thinking involved in solving problems, formulating
inferences, calculating likelihoods, and making decisions,
when the thinker is using skills that are thoughtful and effective
to the particular context and type of thinking task.
��
Additional advocates of skills-based critical thinking include
Edward Glaser and Alec Fisher, and their sets of critical
thinking
skills include a number of elements common to those identified
by Halpern. The overlapping competencies of critical thinkers
as
advanced by Paul and Elder and these three other proponents are
summarized in table �. The comparison reveals the
completeness
of the Paul and Elder model.
These competencies assist intelligence analysts who contribute
to
the solution of threats to national security by ensuring the
formulation
of sound inferences about adversarial capabilities and
intentions.
Resulting findings are disseminated in reports that are often
referred
to as “assessments.” At their best, they offer choices to
decision-
making consumers, as well as a clear outline of the
implications of

those choices.
��
�� Diane Halpern, Thought and Knowledge: An Introduction
to Critical Thinking,
�th Edition (Mahwah, NJ: Lawrence Erlbaum Associates,
Publishers, �00�), ��.
Cited hereafter as Halpern, Thought and Knowledge.
�� The term “assessment” describes the result of an
intelligence production
process but is not universally used within the Intelligence
Community. Assessments
are created at the Central Intelligence Agency and at the
Defense Intelligence
Agency. Other names for the reports that analysts produce
include “products” at
the National Security Agency (NSA) and “estimates” at the
National Intelligence
Council (NIC). Regardless of how they are named, they record
and disseminate
the results of analysts’ work.
– �� –
Competencies of Critical
Thinkers
Paul and
Elder
Fisher Glaser Halpern

Recognize problems or
questions and find effective
means of solution
3 3 3
Engage in meta-cognitive
activities that identify assump-
tions, biases, and performance
as solutions are developed
3 3 3 3
Interpret data, appraise
evidence, and evaluate state-
ments in order to recognize
logical relationships between
propositions
3 3 3
Infer warranted conclusions
and generalizations from
evidence
3 3 3 3
Test generalizations and
conclusions by seeking out
contradictory evidence that
enables them to judge the
credibility of claims
3 3 3 3
Convey sound, well-reasoned
arguments 3 3 3 3
Focus on the process of rea-

soning with the intention of
improving the process
3
Table 1: A Comparison of Different Sets of Critical
Thinker’s Competencies
Sources: Compiled by author from Linda Elder and Richard
Paul, The Founda-
tions of Analytic Thinking: How to Take Thinking Apart and
What to Look for When You
Do (Dillon Beach, CA: The Foundation for Critical Thinking,
�00�); Diane
Halpern, Thought and Knowledge: An Introduction to Critical
Thinking, �th Edition
(Mahwah, NJ: Lawrence Erlbaum Associates, Publishers,
�00�), ��; Edward M.
Glaser, An Experiment in the Development of Critical Thinking
(New York, NY: AMS
Press, �9��), �; and Alec Fisher, Critical Thinking: An
Introduction (Cambridge,
UK: Cambridge University Press, �00�), �.
– �� –
A Disposition to Think Critically
It is not enough merely to know the skills needed for critical
thinking. To be successful, analysts as critical thinkers also
need
“certain attitudes, dispositions, passions, [and] traits of mind.”
��

Actively thinking critically hones the skills; practice yields
proficiency.
But in order to gain mastery, willingness to reason in this
manner
becomes essential. The importance of this disposition to
critically
think cannot be over-emphasized. According to Peter A.
Facione,
Noreen C. Facione, and Carol A. Giancarlo, “Empirical
studies…at
multiple sites indicate that for all practical purposes” both
critical
thinking skills and the disposition to critically think are
essential.
��
There are a number of characteristics associated with the
disposition to think critically. The Faciones and Giancarlo
identify
“seven characterological attributes or habits of mind… truth-
seeking,
open-mindedness, analyticity, systematicity, critical thinking,
self
confidence, inquisitiveness, and maturity of judgment.”
��
According
to Richard Paul and Gerald Nosich, the characteristics of
critical
thinkers include
thinking independently, exercising fair-mindedness,

developing insight into egocentricity and sociocentricity,
developing intellectual humility and suspending judgment,
developing intellectual courage, developing intellectual good
faith and integrity, developing intellectual perseverance,
�� Richard W. Paul and Gerald Nosich, A Model for the
National Assessment of
Higher Order Thinking, (Dillon Beach, CA: Foundation for
Critical Thinking, n.d.),
�0. Cited hereafter as Paul and Nosich, National Assessment.
�� Peter A. Facione, Noreen C. Facione, and Carol A.
Giancarlo, “The
Disposition Toward Critical Thinking: Its Character,
Measurement, and
Relationship to Critical Thinking Skill,” Informal Logic �0, no.
� (�000): ��–��.
Reprinted by Insight Assessment, URL:
<http://www.insightassessment.com/pdf_
files/J_Infrml_Ppr%20_2000%20–
%20Disp%20&%20Skls.PDF>, last accessed
�� March �00�. Cited hereafter as Facione, “Disposition.”
�� Peter A. Facione, Noreen C. Facione, Carol A. F.
Giancarlo, Professional
Judgment and the Disposition Toward Critical Thinking
(Milbrae, CA: California Academic
Press, 2002), URL: <
http://www.calpress.com/pdf_files/Prof_jmn.pdf>, last
accessed �� March �00�.
– �� –
developing confidence in reason, exploring thoughts

underlying feelings and feelings underlying thoughts,
developing intellectual curiosity.�9
Both sets closely match the characteristics of successful
intelligence
analysts identified by Lisa Krizan and the author in their work
on
intelligence analysts’ core competencies. Krizan and the author
observe that successful intelligence analysts are insatiably
curious.
Fascinated by puzzles, their high levels of self-motivation lead
them
to observe and read voraciously, and to take fair-minded and
varied
perspectives. This helps them to make the creative connections
necessary for solving the hardest intelligence problems.
Finally, the
emotional tensions created by problems, and the cathartic
release
at their solution, powerfully motivate analysts.
�0
In addition, emotions play a significant part in the process
of critical thinking. As adult learning expert Stephen
Brookfield
observes, “[emotive] aspects – feelings, emotional responses,
intuitions,
sensing – are central to critical thinking.”
��
He argues that the
consideration of alternatives “to one’s current ways of
thinking”

characterizes the approach.
��
Indeed, the consideration of these
alternatives requires creativity, which he considers a non–
rational
form of thought.
��
Creative reasoning generates hypotheses; it has
contributed significantly to the development of intellectual,
scientific,
and technological knowledge.
�9 Paul and Nosich, National Assessment.
�0 David Moore and Lisa Krizan, “Intelligence Analysis: Does
NSA Have
What it Takes,” reprint NSA Center for Cryptologic History,
Cryptologic Quarterly
�0, nos. �/� (Summer/Fall �00�), �–��. Cited hereafter as
Moore and Krizan,
“NSA,”
31 Stephen D. Brookfield, Developing Critical Thinkers:
Challenging Adults to Explore
Alternative Ways of Thinking and Acting (San Francisco, CA:
Jossey–Bass Publishers,
1987), 12. Cited hereafter as Brookfield, Developing Critical
Thinkers.
32 Brookfield, Developing Critical Thinkers, ��.
33 Brookfield, Developing Critical Thinkers, ��.

– �� –
The Role of Questions
Greg Treverton observes that in intelligence the right questions
rarely get asked.
��
Yet, critical thinking offers a means by which
at least appropriate questions may be raised. The combination
of
critical thinking skills and the disposition to think critically
focuses
and directs inductive, deductive, and abductive reasoning to
solve
problems. This is an interrogative paradigm. Critical thinking
involves questioning that forces broader consideration of issues
and
problems as well as the various means of resolving or
answering
them. Such questioning happens at both the individual and
collective
levels as shown in figure 4.
Figure 4: Levels of Questioning: Critical Thinking in a
Social Setting
Source: Created by the author with contributions by Russell G.
Swenson.
Questioning is either personal or impersonal and ranges from
the

musings of the individual critical thinker to that of a global
discourse.
As an individual ponders problems and issues, she abduces the
most likely explanations. Moving up a level, questioning
becomes a
�� Greg Treverton, conversation with the author, �� April
�00�.
– �� –
hierarchical dialogue within dyads or larger assemblies. A
critically
thinking questioner ferrets out answers from others within the
group.
The process generates new questions that are answered in turn –
and
in turn, raise further questions. At the level of a community
debate,
critical thinkers explore what is possible politically and within
the
community’s purview. Ideally, the relationship between
thinkers is
peer-to-peer. Finally, questions addressed within the context of
a
global discourse take into account the entire biological
landscape
(noösphere).
��
Such questioning produces new and creative thinking, analogous
to switching from Newtonian to quantum physics. In this
analogy,

an atomistic approach to analysis gives way to the associative
impulse
of synthesis as structured thoughts facilitate a leap to new
questions,
ideas, and possibilities. This overarching framework of
structured
reasoning frames questions to help analysts decide the best
means
or best combination of means that are suited to solving specific
intelligence problems. The questions serve to alert an analyst
to the
notion that she might be deceived, or that she needs to employ
some
other means of reasoning to determine which of several
alternative
outcomes is most likely. These questions provide a formal
means
by which an analyst confronts her biases about likely outcomes.
Therefore, in the context of intelligence analysis, critical
thinking
becomes one of the most – if not the most – important skills of
the
analyst. Facing symmetric and asymmetric threats, analysts
have a
potent and powerful tool to facilitate their asking the right
questions
in the process of improving their understanding of the problem
at
hand as well as their own reasoning about that problem.
Pseudo-Critical Thinking
It is important to understand what critical thinking is not.
Critical
thinking, as has been noted, focuses on both the process and the
results of reasoning. However, the term is also used in

reference
to reasoning that is not reflective. The application of formal
logic
is sometimes (incorrectly) equated to critical thinking. So too
are
35 Wikipedia defines the noösphere as the domain of human
thought.
Wikipedia, entry under “noösphere,” accessed �� February
�00�.
– �9 –
problem solving and structured methods of analysis.
Developers of
school curricula and other exponents of “sound thinking” often
lay
claim to the mantel of critical thinking but are really
leveraging their
coverage of logic or problem solving to capitalize on an
uncritical
understanding of what critical thinking is. Problem solving,
for
example, focuses on answers and not on the process by which an
answer is obtained. Additionally, logic or problem solving,
being
goal oriented, offer little means by which a person can improve
the
process of her thinking. The following problem is a typical
example
of pseudo–critical thinking:
Two cyclists race along a straight course. The faster of the
pair maintains an average speed of �0 mph. The slower

cyclist averages �� miles per hour. When the race ends, the
judges announce that the faster cyclist crossed the finish line
one hour before the slower racer. How many miles long was
the racing course?
��
This example (and others like it) focuses on the answer,
provides no
guidance on the process, and ignores any improvement in
reasoning
skills. Solvers either figure it out through trial and error or by
following a rule-based strategy learned by rote. Therefore it
fails to
be critical thinking.
Unless the process, and the means, to improve a person’s
reasoning
are emphasized, then at best such examples teach structured
problem
solving. At worst, they deal merely in criticism; they fail to
assist
people to learn to reason better. Central to critical thinking are
the
twin foci on the way in which a person is reasoning and the goal
of
improving that process. One could easily infer that
misconceptions
about critical thinking could be associated with what is taught
as
critical thinking in American educational institutions.
��
�� Michael A. DiSpezio, Classic Critical Thinking Puzzles
(New York, NY: Main

Street, �00�), ��. The answer, by the way, is ��0 miles
(page ��).
�� Michael R. LeGault reaches this same conclusion in Think:
Why Crucial
Decisions Can’t Be Made in the Blink of an Eye (New York,
NY: Threshold Editions,
�00�), ��. Cited hereafter as LeGault, Think.
– �0 –
wHAT cAN be leArNeD froM THe PAsT?
THiNkiNg criTicAlly AbouT cubA
Examining past intelligence successes and failures in light of
new developments
provides a means to reassess what happened and how outcomes
could have been
different. The case of the Soviet missiles in Cuba in the
summer and fall of
1962 provides a means of examining how critical thinking and
structured
methodologies could have made a difference.
Deploying the Missiles
During the summer of �9��, CIA analysts received a spate of
potentially alarming reports about Russians being seen in Cuba.
The
reports, however, were only part of a stream of similar,
“farfetched
tales of African troops with rings in their noses, lurking
Mongolians,
and even Chinese troops” on the island.

��
Most or all of these reports
were discounted by analysts who were inured to spurious
reports of
Soviet equipment secreted away in caves.
�9
James Hansen – who worked in both the Central Intelligence
Agency (CIA) and Defense Intelligence Agency (DIA) – posits
that
the U.S. Intelligence and Policy Communities were the victims
of a
concerted Soviet campaign of denial and deception that masked
the
deployment of Soviet forces and missiles into Cuba.
�0
The deception
campaign even included “accurate information about the
deployment
[leaked] so as to mask it.”
��
As Raymond Garthoff relates, “there
were literally thousands of reports of missiles in Cuba in the
period
before any missiles were actually brought there.”
��

�� James H. Hansen, “Soviet Deception in the Cuban Missile
Crisis,” Studies in
Intelligence ��, no. � (�00�), ��. Cited hereafter as
Hansen, “Soviet Deception.”
�9 Hansen, “Soviet Deception,” ��.
�0 Hansen, “Soviet Deception,” �9–��.
�� Domingo Amuchastegui, “Cuban Intelligence and the
October Crisis,”
in James G. Blight and David A. Welch, Eds., Intelligence and
the Cuban Missile Crisis
(London, UK: Frank Cass, �99�), �0�.
�� Raymond L. Garthoff, “US Intelligence in the Cuban
Missile Crisis,” in
– �� –
In fact, the Soviets were able to deploy more than the offensive
nuclear missiles that became the centerpiece of the subsequent
crisis
with the United States. While U.S. analysts and policymakers
knew
of the conventional weapons, they were blind to the presence of
SS-
� Medium Range Ballistic Missiles (MRBM) and SS-�
Intermediate
Range Ballistic Missiles prior to the U-2 overflights of 14
October.
Additionally, it seems they never discovered the presence of
approximately 100 tactical nuclear weapons deployed on the

island.
��
There is debate whether U.S. Intelligence also underestimated
the
number of Soviet troops deployed to Cuba. Garthoff reports
that
one CIA unit concluded there were between ��,000 and �0,000
Soviet
troops in Cuba (the actual number was about 42,000) but the
official
estimate was between �,�00 and �,000 prior to the crisis.
��
The Soviets employed an elaborate campaign of denial and
deception that took advantage of American points of view
about
the likelihood of Soviet weapons being located in Cuba. As
Robert
Jervis makes clear,
the U.S. did not expect the Russians to put missiles into
Cuba or Japan to attack Pearl Harbor because American
officials knew that the U.S. would thwart these measures if
they were taken. These judgments were correct, but because
other countries saw the world and the U.S. less accurately,
the American predictions were inaccurate.
��
It could have been worse. As Gil Merom writes, the Soviets
might

James G. Blight and David A. Welch, Eds., Intelligence and the
Cuban Missile Crisis
(London, UK: Frank Cass, �99�), ��. Emphasis in original.
Cited hereafter as
Garthoff, “US Intelligence.”
�� Garthoff, “US Intelligence,” �9.
�� Garthoff, “US Intelligence,” ��, ��. U.S. Intelligence
never did reach
a correct conclusion. The closest they got was an estimate of
��,000 troops in
early �9�� (Garthoff, “U.S. Intelligence,” ��).
�� Robert Jervis, System Effects: Complexity in Political and
Social Life (Princeton,
NJ: Princeton University Press, �99�), ��. Jervis draws on
the work of Klaus
Knorr. See Klaus Knorr, “Failures in National Intelligence
Estimates: The Case
of the Cuban Missiles,” World Politics �� (April �9��):
��–��.
– �� –
have completed the bases and “threatened major U.S. cities with
extinction.”
��
The threat is brought home by figure 5.
What caused U.S. analysts to get it wrong? Apparently the U.S.
analysts concluded that because some of the HUMINT evidence

was ludicrous, all of it was. This inductive conclusion then led
them to discount further evidence. For example, having
concluded
that all the previously considered Cuban HUMINT was false,
each
new piece, since it came from Cuba, also had to be false. Thus,
a false
inductive conclusion led subsequently to false deductive
conclusions.
��
It does not appear that abductive reasoning strategies were ever
employed.
Figure 5: Respective Ranges of Soviet SS-4 MRBM and
SS-5 IRBM Missiles
Source: Derived from Hansen, “Soviet Deception,” �9.
�� Gil Merom, “The �9�� Cuban Intelligence Estimate: A
Methodological
Perspective,” Intelligence and National Security ��, no. �
(Autumn �999), �9. Cited
hereafter as Merom, “Estimate.”
47 Inductive and deductive reasoning were illustrated in figure
1.
– �� –
It should be noted that the U.S. Intelligence Community was
not blind to the possibility that the Soviets might conduct a

military
buildup in Cuba. In fact, there were two theories being debated:
One, that the Soviets would emplace defensive weapons, and the
other
that they would emplace offensive weapons. Senior analysts in
the
Intelligence Community held the former theory while John
McCone,
then Director of Central Intelligence, favored the latter.
Part of McCone’s reasoning had to do with the cost and utility
of the installation of SA-� air defense missiles. McCone
apparently
reasoned that the purpose of installing such expensive missiles
had
to be greater than merely denying the United States overflight
capabilities (the SA-� could shoot down – as Francis Gary
Powers
discovered – a U-�).
��
This led McCone to come “up with an answer
that no one wanted to hear: the SA-�s were on the island to
deny
the United States the capability to see the construction of
offensive
missile installations.”
�9
Unfortunately, McCone was unable to dissuade the majority
from their point of view. This may have stemmed, as James
Blight
and David Welch write, from the realization that while

McCone’s
inference seemed reasonable when viewed in hindsight, in
foresight
it was faulty because it failed to sufficiently cover the
alternatives.
�0
Blight and Welch observe that
the Soviets had also deployed SA-� missiles to Egypt,
Syria, and Indonesia, and in none of those cases had they
also deployed strategic nuclear weapons. Indeed, the US
intelligence community [sic] expected that the Soviet Union
�� Thomas R. Johnson and David A. Hatch, NSA and the
Cuban Missile Crisis
(Fort Meade, MD: National Security Agency Center for
Cryptologic History,
1998), URL:
<http://www.nsa.gov/publications/publi00033.cfm>, last
accessed
�� April �00�. Cited hereafter as Johnson and Hatch, NSA.
�9 Johnson and Hatch, NSA.
�0 James G. Blight and David A. Welch, Eds., Intelligence and
the Cuban Missile
Crisis (London, UK: Frank Cass, �99�), �. Cited hereafter as
Blight and Welch,
Intelligence and the Cuban Missile Crisis.
– �� –

would deploy SA-� missiles to Cuba, precisely because it had
done so elsewhere.
��
Even though history proved McCone to be correct (for the
wrong
reasons), the defensive weapons theory predominated.
��
The Soviets took advantage of the American beliefs and faulty
reasoning. Capitalizing on the idea that it is easier to lead a
target
astray than to try to change his mind, they successfully placed
the
nuclear missiles in Cuba.
��
Heuer observes that
[deceptions] that follow this principle seldom fail, for the
odds are then strongly in favor of the deceiver. The human
capacity to rationalize contradictory evidence is easily
sufficient to outweigh the pernicious effects of security leaks
and uncontrolled channels of information that planners of
deception…might compromise their efforts.
��
Assessing the Implications
Exactly what happened in the Cuban case and does it apply to
contemporary issues? Roberta Wohlstetter notes in retrospect,
“We

would like to know not only how we felt, but what we did and
what we
might have done, and in particular what we knew or what we
could
have known.”
��
Wohlstetter’s musings lead to two key questions for
analysts: How could this successful deception campaign have
been
�� Blight and Welch, Intelligence and the Cuban Missile
Crisis, �.
�� Merom, “Estimate,” ��.
�� Richards J. Heuer, Jr., “Strategic Deception and
Counterdeception:
A Cognitive Process Approach,” International Studies Quarterly
��, no. � (June
�9��), �00. Cited hereafter as Heuer, “Strategic Deception.”
It is interesting to
speculate whether the Soviets had a feedback channel that
informed them of the
predominant theory.
�� Heuer, “Strategic Deception,” �00.
�� Roberta Wohlstetter, “Cuba and Pearl Harbor: Hindsight
and Foresight,”
Foreign Affairs ��, no. � (July �9��), �9�. Cited hereafter
as Wohlstetter, “Cuba.”

– �� –
thwarted? What can be learned to advise analysts about current
adversarial denial and deception?
Simply looking at additional evidence is sometimes promoted
as a means of detecting denial and deception by adversaries.
In the Cuban case, however, analysts had already processed a
superabundance of evidence. Wohlstetter suggests that such
riches
can be “embarrassing.”
��
This is because even as “signals” point
to the action or to an adversary’s intention to undertake it,
“noise” or a background of irrelevant or inconsistent signals,
[and] signs pointing in the wrong directions…tend always
to obscure the signs pointing the right way.
��
HUMINT assets overwhelmed analysts’ abilities to distinguish
signals
from noise. Heuer and Hansen agree that once “locked in,”
analysts
resist changing their minds. More evidence alone fails to change
an
analyst’s mind because “[new] information is assimilated to
existing
images.”
��
Yet, analysts stand to benefit from changing their opinion
in the face of disconfirming evidence.

The limitations of hindsight analysis notwithstanding, if
analysts
had employed critical reasoning in �9�� they could have
detected
the Soviet maskirovka in Cuba. For example, by applying Paul
and
Elder’s model of critical thinking, analysts would have had a
means
to question their assumptions and points of view, and
subsequently
might have questioned the purpose of the apparent “noise” they
were discounting as well as their assumptions about the
prevalent
point of view that the Soviets would not place missiles on the
island
because we would react. Gil Merom suggests that some
appropriate
questions would have included: “How aware were the analysts
of
their own assumptions? Were these assumptions sensible and
was
it sensible to adhere to them?”
�9
Other considerations that critical
�� Wohlstetter, “Cuba,” �9�.
�� Wohlstetter, “Cuba,” �9�.
�� Heuer, Psychology, �0–��.
�9 Merom, “Estimate,” ��.

– �� –
thinking about the crisis might have raised are included in table
�.
Applying Paul and Elder’s model reveals other considerations
that
conceivably would have led to an earlier detection of the
likelihood
of the deployed weapons being offensive.
Would this approach have allowed detection of what the
Soviets
were doing in Cuba before mid-October �9��? One could
argue
counterfactually that it might have enhanced analysis of
collection
from other sources.
�0
SIGINT assets capable of collecting Russian
communications were in place near the island.
��
Imagery assets might
have been tasked differently prior to �� October. Sources less
subject
to generating noise would have introduced inconsistencies to an
analysis of competing hypotheses that was solely HUMINT-
based.
Between Dogmatism and Refutation
A systematic approach to directed collection also would have
moved analysts toward a disconfirmatory analytic approach.

By contrast, a confirmatory approach, often characterized as
confirmation bias, leads analysts to accept what they set out to
confirm.
��
A disconfirmatory approach aims to refute alternative
60 Counterfactual reasoning explores differing outcomes
arising from
alternative causes. Counterfactuals are commonly used in
whenever alternate
hypotheses are explored. They are essential for both post-
mortem reviews and
futures scenario exercises. See Philip E. Tetlock, and Aaron
Belkin, Counterfactual
Thought Experiments in World Politics: Logical,
Methodological, and Psychological Perspectives
(Princeton, NJ: Princeton University Press, �99�), �–��. In
the same volume,
Richard Ned Lebow and Janice Gross Stein conduct a
counterfactual review
of the Cuban missile crisis. See Richard Ned Lebow, and
Janice Gross Stein,
“Back to the Past: Counterfactuals and the Cuban Missile
Crisis,” in Philip E.
Tetlock, and Aaron Belkin, Counterfactual Thought
Experiments in World Politics: Logical,
Methodological, and Psychological Perspectives (Princeton, NJ:
Princeton University
Press, �99�), ��9–��.
�� Johnson and Hatch, NSA. According to the authors,
SIGINT played a key
role in determining the operational status of the SA-� air
defense missiles. However,
SIGINT failed to detect the delivery and installation of the

nuclear missiles.
62 Wikipedia defines confirmation bias as “a type of cognitive
bias toward
confirmation of the hypothesis under study.” (Wikipedia, entry
under “confirmation
bias”).
– �� –
Purpose
· Determine what is going on in Cuba as a part of the U.S.
“war” against Castro and the Cuban communists.
Key Questions
· If this really is a Soviet military buildup, what kinds of
weapons are being deployed: offensive weapons or defen-
sive weapons?
· Why is an expensive missile system like the SA–2 being
installed in Cuba?
· What is the SA–2 really protecting?
· Does a crisis exist?
Evidentiary
Considerations
· If defensive weapons are being deployed, what evidence
should be observed?
· If offensive weapons are being deployed, what evidence

should be observed?
· What is not being seen?
Inferences
· What is inferred from the observed and collected
evidence?
Assumptions
· What is being assumed about the evidence?
· What is being assumed about the sources of the evidence?
· What is being assumed about why the Soviets would
deploy weapons to Cuba?
Concepts
· How does human perception affect the analysis?
· How reliable are the sources of evidence? (Could also be
an assumption.)
Implications
and
Consequences
· If conclusions are incorrect about the Soviet build up,
what might occur next?
· If conclusions are correct about the Soviet build up, what
can be expected?
· If conclusions are incorrect about offensive weapons be-
ing deployed, what happens next?

· If conclusions are incorrect about defensive weapons be-
ing deployed, what happens next?
Points of View
· What other points of view about what is going on in Cuba
exist?
Table 2: Applying Paul and Elder’s Critical Thinking
Model to the Situation in Cuba, August–September 1962.
Source: Developed by author.
– �� –
hypotheses. Whatever was left at the end of the process would
be the
most likely explanation. Would this guarantee a correct
“answer”?
Ben-Israel concludes that although the answer has to be “no,”
the
approach does allow for narrowing “the margin of error.”
��
It does
so by moving the analyst away from a “pole of dogmatic
tenacity”
toward a “pole of refutation” (summarized in figure 6).
��

Figure 6: A Place for Analysis between Dogmatism and
Criticism
Source: Derived from Ben-Israel, “Methodology of
Intelligence,” ��9.
It should be noted that operating too closely to a pole of
refutation
can also create problems. For example, analysts might be
unable to
distinguish between real threats and false positives; they might
react
to each one regardless of its validity. In the Cuban case, this
problem
would have manifested itself in goading continual U.S. military
reactions to every perceived Soviet and Cuban threat. Here
again,
�� Ben-Israel, “Logic of the Estimate Process,” ��9
�� Ben-Israel, “Logic of the Estimate Process,” �9�.
– �9 –
critical thinking could moderate this Cassandrian approach.
��
The
questioning of assumptions and points of view in critical
thinking
allows analysts to discount certain evidence. Based on his own
Israeli
intelligence analyst experiences, Ben-Israel asserts that
anchoring

oneself some distance from a pole of refutation, and not far
from
the center, is probably best.
��
In other words, enlightened skepticism
probably works as a method for distinguishing between real and
perceived threats. Critical thinking leads one to assess the
processes
one is using and the point on the continuum from which to
work.
Lacking: Disconfirmation
Merom confirms that a disconfirmatory process did not occur
on the Cuban missile problem: “Information that was
inconsistent
with the prevailing conservative theory was not considered as
alarming and necessitating revision, but rather was
‘rehabilitated’
and rendered ‘harmless’ on the basis of ad hoc statements.”
��
Instead,
inductive reasoning generally led analysts to “prove” their
theory and
subsequently to adhere to it “until it was proven wrong by
conclusive
hard evidence” – the U–2 photos, to be exact (of which a detail
is
reproduced in figure 7).
��

�� Cassandra was the daughter of Priam, King of Troy. She
was blessed and
cursed by the god Apollo to accurately predict the future yet
never be believed. In
intelligence, a Cassandrian approach is characterized as one that
emphasizes dire
outcomes and worst–case scenarios. Analysts who produce such
assessments are
often discounted and rarely thanked when they are correct. For
a real account of
how such warnings are perceived (at least at the time they are
made), see Charles
E. Allen, “Intelligence: Cult, Craft, or Business?” in Seminar
on Intelligence, Command,
and Control, Guest Presentations, Spring �000 (Cambridge,
MA: Harvard University
Program on Information Resources Policy, I–0�–�, April
�000), URL: <http://
www.pirp.harvard.edu/pubs.html>, last accessed 11 January
2006.
�� Ben-Israel, “Logic of the Estimate Process,” ��9.
�� Merom, “Estimate,” �9.
�� Merom, “Estimate,” �9. Italics in original.
– �0 –
Figure 7: Detail of a U-2 Photograph of an SS-4 MRBM
Launch Site, San Cristobal, Cuba, 14 October
1962. This evidence confirmed that Soviet missiles
were being installed in Cuba.

Source: U.S. Department of Defense, photograph in the John
Fitzgerald
Kennedy Library, Boston, MA, PX ��–�0:� �� October
�9��.
What tipped off the overhead surveillance were two HUMINT
reports of “a Soviet truck convoy that appeared to be towing
ballistic
missiles toward the San Cristobal area.”
�9
That these reports were
taken seriously is one of the curious serendipities that occur
from time
to time in intelligence analysis (and other research-based
domains).
What prompted the CIA and DIA analysts to take these reports
seriously while earlier accounts had been dismissed remains a
mystery.
Garthoff asserts that it was new information taken in context
with
the observed “pattern of SA-� surface-to-air missile sites in
Western
�9 Graham Allison and Philip Zelikow, Essence of Decision:
Explaining the Cuban
Missile Crisis, �nd Edition (New York, NY: Longman, �999),
��0. Cited hereafter
as Allison and Zelikow, Essence of Decision. Raymond
Garthoff also notes this to
be the case. See Garthoff, “US Intelligence,” ��.
– �� –

Cuba” that led to the tasking of the U-2 flight on 14 October
(the
track of which is shown in figure 8).
�0
Also contributing to the determination that the deployed
missiles
were SS-� MRBMs was information transmitted to the British
and
American intelligence services by Oleg Penkovsky. While
Penkovsky’s
espionage apparently did not warn the Americans that the
Soviets
were about to deploy offensive missiles in Cuba, he is credited
with
– among other things – providing technical information about
the
missiles.
��
Len Scott observes that Penkovsky provided the technical
information that allowed the determination that the missiles
were
SS-�s instead of SS-�s, as well as information that allowed
accurate
assessment by the Americans of their readiness. Scott writes
that
this was important because
the SS-� had a range which US intelligence estimated at ��0
nautical miles (nm), enabling coverage of seven Strategic Air
Command (SAC) bomber/tanker bases; the �0�0nm SS-� could
target 18 bomber/tanker bases (plus one ICBM complex), and

some �� cities with a population of over �00,000, including
Washington, accounting for 9� million people.
��
Shortly thereafter Penkovsky was detected, arrested, tried, and
executed by the Soviets for his espionage.
�0 Garthoff, “US Intelligence,” ��.
�� Len Scott, “Espionage and the Cold War: Oleg Penkovsky
and the Cuban
Missile Crisis,” Intelligence and National Security ��, no. �
(Autumn �999), ��. Cited
hereafter as Scott, “Penkovsky.”
�� Scott, “Penkovsky,” ��. In making this assertion Scott
draws on work
previously published by Allison and Zelikow in Essence of
Decision and by Mary
S. McAuliffe, ed., CIA Documents on the Cuban Missile Crisis,
1962 (Washington DC:
Central Intelligence Agency, �99�).
– �� –
Figure 8: U–2 Tracks over Cuba, 4–14 October 1962.
Sources: Map, Central Intelligence Agency. Tracks derived
from a map
in Mary S. McAuliffe, ed., CIA Documents on the Cuban
Missile Crisis, 1962
(Washington DC: Central Intelligence Agency, �99�), �.
Hereafter McAu-

liffe. Missile locations derived from Arthur C. Lundahl,
“Additional Informa-
tion – Mission ��0�,” Memorandum for Director of Central
Intelligence
and Director, Defense Intelligence Agency, �� October �9��,
in McAuliffe,
��–��; and Arthur C. Lundahl, “Additional Information –
Mission ��0�,”
Memorandum for Director of Central Intelligence and Director,
Defense
Intelligence Agency, �9 October �9��, in McAuliffe, �09.
Track and missile
locations are approximations. Lundahl was the Director,
National Pho-
tographic Interpretation Center. The �9 October memo reports
that a �
September U–2 overflight did not detect the latter two missile
sites. How-
ever, a review of the � September track (as presented in
McAuliffe) shows the
aircraft only came near the Sagua La Grande site. It should also
be noted
that the track of the U-2 – as flown on 14 October – deviated
from the origi-
nal planned route (which was farther to the west). The track as
flown took
the aircraft closer to the San Cristobal site. What might have
happened had
the original track been flown?
Despite their efforts at denial and deception, there was an
evident
lack of concealment of the missiles by the Soviets prior to
their
“discovery” on �� October. General Anatoli Gribkov, a
member

of the Soviet General Staff in Cuba at the time relates,
– �� –
A missile–launching complex is not easily disguised…[Such]
an installation…could be hidden from ground-level view.
But from above, however, it could and did stick out like a
sore thumb.
��
Allison and Zelikow note that the Soviets only began to
camouflage
the sites after “the U.S. announced the discovery of the
missiles and
announced the blockade.”
��
They conclude that the Soviet forces
building the bases lacked personnel and resources to conceal
them
– a situation that apparently changed only after the missiles had
been discovered.
��
The Roles of Critical Thinking in the
Cuban Crisis
Merom’s critique of the Intelligence Community’s estimate of
Soviet intentions vis-à-vis Soviet weapons in Cuba reveals areas

that would be well-served by critical thinking. First, when a
critical
thinking paradigm is in control, intelligence foraging and
gathering
are efficiently oriented. Questions are raised about the existing
evidence – both anomalous and consistent – as well as where
new
disconfirming and confirming evidence might be discovered.
Second,
alternative – what in the context of Cuba, Merom calls
revolutionary
– theories are considered.
��
A structured process also speeds up the
process of analytic sensemaking: estimates are crafted earlier.
��
Third,
a methodological process opens analysis to “guiding principles
of
�� Anatoli Gribkov, “The View from Moscow and Havana,” in
Anatoli
Gribkov and William Y. Smith, Operation ANADYR: U.S. and
Soviet Generals Recount
the Cuban Missile Crisis (Chicago, IL: Edition Q , �99�), �9.
Cited hereafter as
Gribkov, “The View.”
�� Allison and Zelikow, Essence of Decision, �0�.
�� Allison and Zelikow, Essence of Decision, ��.

77 Structured processes provide a framework that reduces the
flailing around
as analysts seek to find a starting point for their analyses.
Morgan Jones discusses
this at some length. See Jones, Thinker’s Toolkit, xi–xvi.
– �� –
research,” enriching both the process and the results.
��
However, there are dangers to such a paradigm, especially as
“science” commonly is understood. As Jeffrey Cooper notes, “a
‘science of analysis’ is a conceit, partly engendered by
Sherman
Kent’s dominating view of intelligence as a counterpart of the
scientific method.”
�9
Mark and Barbara Stefik argue that “the working
methods of science and invention leaves [sic] out imagination.
This
makes it both boring and misleading.”
�0
Analysts must focus on both
evidence and inferences; otherwise they can “get the details
right at
the cost of ignoring important inferential judgments that need
to

be conveyed in order to provide a true sense of the
uncertainties of
both evidence and judgment.”
��
Here again, critical thinking, with
its emphasis on creative questioning, moderates the process.
Such
questioning opens the way for imaginative thinking.
“[Intuition],
curiosity, and a thirst for discovery – all essential elements of
good
science” could have alerted analysts working in �9�� (or at
any other
time) to the possibility that they were being denied or deceived.
��
An interesting intersection exists between critical thinking
and analogy, or drawing comparisons to derive new patterns and
explanations. This intersection lies at the heart of creativity.
��
According to Keith Holyoak and Paul Thagard, there are
commonly
four steps to analogical problem solving:
Often a problem solver will select a source analog by retrieving
information about it from memory (selection), map the source
to the target and thereby generate inferences about the target

�9 Cooper, Pathologies, ��..
80 Mark Stefik, and Barbara Stefik, Breakthrough: Stories and
Strategies of Radical
Innovation (Cambridge, MA: MIT Press, �00�), ��0.
�� Cooper, Pathologies, ��.
�� Cooper, Pathologies, ��.
�� This statement illustrates just how pervasive and powerful
the use of
analogy is in human reasoning and discourse. Taken literally,
creativity has no
heart (nor any other organs) since it is a notion or a concept, not
a living animal.
Yet the analogy to a living being aids understanding.
– �� –
(mapping), evaluate and adapt the inferences to take account
of unique aspects of the target (evaluation), and finally learn
something more general from the success or failure of the
analogy (learning).
��
Since analogy is such a powerful element in human reasoning,
how can critical thinking outwit or control it? As developed in
this paper, it does so by imposing a structure on the thinking.
By
examining inferences and implications as well as alternative
points

of view, critical thinking calls into question the
appropriateness of
the analogies in use. Recent work by Paul and Elder reveals
how this
works. They request, for example, that reasoners state and
restate
explanations, add examples, and then include an analogy.
��
While
initial statements about a phenomenon do not imply
understanding,
restatement, examples, and analogies do and further, provide
measures
by which comprehension can be fixed and assessed; they make
knowledge explicit – something that can be thought about.
��
When
knowledge and reasoning are explicit, assumptions are revealed.
According to Richard Neustadt and Ernest May, it is known that
analogy is a tool commonly invoked by policymakers.
��
Critically
thinking analysts can add substantively to the policymakers’
options
by constructively challenging the tendency to rely upon analogy
as a
way of addressing situations. For example, Neustadt and May
observe
that in coping with the missile crisis in Cuba, President

Kennedy and
his advisers relied on – among other things – an analogy to
Pearl
Harbor to justify why a “surprise” bombing of the Soviet
missile
�� Keith J. Holyoak and Paul Thagard, Mental Leaps:
Analogy in Creative
Thought (Cambridge, MA: The MIT Press, �99�), ��.
Emphasis in original. Cited
hereafter as Holyoak and Thagard, Mental Leaps.
�� Richard Paul and Linda Elder, “Instructions to
Participants,” ��th Annual
Conference on Critical Thinking, Berkeley, California, 9 July
2005. The first three
elements were developed previously; analogy was new in �00�.
�� Holyoak and Thagard, Mental Leaps, 20–22.
�� Richard E. Neustadt and Ernest R. May, Thinking in Time:
The Uses of
History for Decision Makers (New York, NY: The Free Press,
�9��). Cited hereafter
as Neustadt and May, Thinking in Time.
– �� –
bases was not a satisfactory option.
��
Kennedy’s advisers, acting as
analysts of the situation, critically evaluated the
appropriateness of

the analogy, pointing out its weaknesses and strengths.
Kennedy
concluded that sneak attacks were not a tactic lying within the
U.S.
tradition.
�9
In the Cuban case, getting analysts to restate their conclusions,
provide examples, and make analogies would have revealed the
strengths and weaknesses of their arguments about what they
observed from the HUMINT sources. It would have shown their
comprehension of what they wanted to conclude, revealed their
assumptions and, in so doing, opened an avenue for
“alternative”
assessments of the issue. It would have done so by causing the
analysts
to question how it was that they failed to notice or ignore
things.
90
The stage would have been set for an earlier, less risky,
defusing of
the impending crisis.
Another analogy employed in the Cuban missile crisis was the
comparison to the Soviet position vis-à-vis the presence of
American
MRBMs in Turkey. A critical review reveals that the Soviet
Union had
tolerated the presence of these missiles – which had a longer
range
than those placed in Cuba – since �9��.

9�
If the U.S. demanded
the removal of the Soviet missiles from Cuba, was not a similar
withdrawal of the American missiles from Turkey appropriate?
Ultimately, this analogy provided a face-saving solution for the
Soviet Union in the negotiations that followed. As a “secret”
part
of the agreement, the American Jupiter MRBMs were removed
from
Turkey five months after the Soviet Union removed its missiles
from
Cuba.
9�
�� Neustadt and May, Thinking in Time, �.
�9 Neustadt and May, Thinking in Time, �.
90 Margaret A. Boden, The Creative Mind: Myths and
Mechanisms (New York,
NY: Basic Books, �990), 25.
9� Neustadt and May, Thinking in Time, 9.
9� Neustadt and May, Thinking in Time, ��.
– �� –
Winners and Losers: The Crisis in Context
A Critical Assessment of the Leaders. In any confrontation

there are winners and there are losers. Thinking critically about
the confrontation over the Soviet missiles also involves
determining
who actually won as well as what the other outcomes might
have
been. Each of the three actors, Castro, Khrushchev, and
Kennedy,
had a number of interests at stake. In a number of
circumstances
each stood to gain at the expense of the others. What solutions
were
the most advantageous to each of the three actors? What were
the
motivating factors? A speculative – and admittedly simplified –
critical comparison, again using Paul and Elder’s model
(summarized
in table �) illustrates how critical thinking also reveals much
about
who stood to gain and who stood to lose in the crisis.
Castro. Castro found his country under attack from the United
States – both directly through economic sanctions and indirectly
by
Batista loyalists living in Florida.
9�
The previous year the United States
had launched an invasion attempt at the Bay of Pigs – which
was
repulsed. However the attempts by the United States to
destabilize
or overthrow the Cuban government did not end with their
defeat.
As Raymond Garthoff observes, “by the spring of �9�� the
United

States had embarked on a concerted campaign of overt and
covert
political, economic, psychological, and clandestine operations
to weaken the Castro regime…including attempts to assassinate
Castro.”
9�
In short, the United States was doing everything except
conventional warfare to destroy Castro and his regime.
Castro’s point of view was that the United States was a real
enemy.
He could reasonably assume that in light of its other activities
and
having attempted an invasion once, the United States would
repeat its
actions and this time might succeed (an implication). Another
possible
outcome that probably was considered was that war,
conventional
or nuclear, was likely. Castro must have possessed evidence of
the
military capabilities of the United States and probably inferred
that
9� Raymond L. Garthoff, Reflections on the Cuban Missile
Crisis (Washington,
DC: The Brookings Institution, �9�9), �. Cited hereafter as
Garthoff, Reflections.
9� Garthoff, Reflections, 9.
– �� –

Element of
Thinking
Considerations Considerations
as seen from Castro’s
Point of View
as seen from Khrushchev’s Point
of View
as seen from Kennedy’s
Point of View
Purpose · Preserve Regime
· Protect Cuba from United States
· Force concessions from United
States
· Remove U.S. and NATO from Berlin
· Spread Communism
· Preserve Regime
· Get the missiles out of Cuba
· Defend U.S. and allies against Com-
munism
· Preserve Regime
Point of View · United States threatens regime
· United States a threat to Commu-
nism
· Cuba threatened with invasion

· Communism threatens United States
· Missiles are a threat
Assumptions · A second invasion will be attempted
· Missiles in Cuba bargaining points
· Missiles in Cuba protect Castro
· Effort will be successful.
· Small Soviet Force
· Strategy of removal is possible
Implications
· Invasion might succeed
· War: conventional or nuclear
· U.S. concessions
· Successful removal
· Escalation of measures
Evidence · U.S. armed forces formidable · U.S. armed forces
formidable
· Administration politically vulnerable
over crisis
· Khrushchev is a formidable adversary
· Khrushchev is bluffing
Inferences
· U.S. can overcome unaided Cuban
forces.

forces.
· Missiles provide necessary aid
· United States will concede.
· Khrushchev will back down
· Military option not required
Concepts
· Military strategy and doctrine, commu-
nist and capitalist theories
· Military strategy and doctrine, com-
munist and capitalist theories
· Monroe Doctrine
Key Questions
· How to deter United States from
invading island
invading Cuba
· How to dominate the United States
· How to get the Soviets and their
missiles off of Cuba

– �9 –
Element of
Thinking
Considerations Considerations
as seen from Castro’s
Point of View
as seen from Khrushchev’s Point
of View
as seen from Kennedy’s
Point of View
Purpose · Preserve Regime
· Protect Cuba from United States
· Force concessions from United
States
· Remove U.S. and NATO from Berlin
· Spread Communism
· Preserve Regime
· Get the missiles out of Cuba
· Defend U.S. and allies against Com-
munism

· Preserve Regime
Point of View · United States threatens regime
· United States a threat to Commu-
nism
· Cuba threatened with invasion
· Communism threatens United States
· Missiles are a threat
Assumptions · A second invasion will be attempted
· Missiles in Cuba bargaining points
· Missiles in Cuba protect Castro
· Effort will be successful.
· Small Soviet Force
· Strategy of removal is possible
Implications
· Invasion might succeed
· U.S. concessions
· Successful removal
· Escalation of measures
Evidence · U.S. armed forces formidable · U.S. armed forces
formidable
· Administration politically vulnerable
over crisis

· Khrushchev is a formidable adversary
· Khrushchev is bluffing
Inferences
forces.
forces.
· Missiles provide necessary aid
· United States will concede.
· Khrushchev will back down
· Military option not required
Concepts
· Military strategy and doctrine, commu-
nist and capitalist theories
· Monroe Doctrine
Key Questions
invading island

invading Cuba
· How to dominate the United States
· How to get the Soviets and their
missiles off of Cuba
– �0 –
unaided, his own forces were no match for the U.S. forces.
Underlying
Castro’s considerations were a number of ideas about military
strategy and doctrine, as well as communist and capitalist
theories.
A corollary consideration commonly believed by many
Americans
(at the time and still today) might have been how Communism
could
have been spread across the Western Hemisphere, especially the
United States. Castro’s key question was, “How can I prevent
the
United States from invading again?”
Khrushchev. What motivated Khrushchev remains
controversial.
His motives certainly were more complex, as various
explanatory
hypotheses reflect. First, there is the argument that Khrushchev
had
an ally “in the faith” to protect. He too believed that the United

States would invade Cuba a second time.
9�
However, as Allison and
Zelikow counsel, the troop buildup itself and not the nuclear
missiles
was what Khrushchev did to offset the perceived threat to Cuba
from the United States.
9�
The deployment of the nuclear missiles
was related to some other issue.
Next, the situation posed an opportunity to force concessions
from the United States – perhaps about Berlin, or the United
States’
own offensive missiles located in Europe and Turkey. Allison
and
Zelikow find sufficient evidence to lead them to believe that
removing
the U.S. and NATO troops from Berlin was a key factor
motivating
Khrushchev to deploy the missiles to Cuba.
9�
Securing his borders
from the American’s Jupiter missiles may have been an
additional
motivating factor.
Additionally, there was Khrushchev’s avowed purpose of
spreading
Communism across the globe. In this context the deployment

of weapons lies within the context of “a great power rivalry…
between the U.S. and the values and interests it
represented…and
9� James J. Wirtz, “Organizing for Crisis Intelligence: Lessons
form the
Cuban Missile Crisis,” in Blight and Welch, Intelligence and the
Cuban Missile Crisis,
��0. Cited hereafter as Wirtz, “Organizing.”
9� Allison and Zelikow, Essence of Decision, ��.
9� Allison and Zelikow, Essence of Decision, 99–�09.
– �� –
the Soviet Union’s communist agenda.”
9�
At a reception for foreign
ambassadors in Moscow in �9��, Khrushchev threatened,
“Whether
you like it or not, history is on our side. We [the communists]
will
bury you!”
99
This remark (among others) also reveals an additional
purpose: Khrushchev, like all dictators, had to appear stronger
than
his adversaries – his political survival depended on it.
Khrushchev’s point of view was that the United States (and its

allies) were a threat to Communism everywhere and needed to
be
contained. He was presented with a confluence of
opportunities and
responded with the military buildup in Cuba and the deployment
of the missiles. He apparently assumed that if he could get
nuclear
missiles into Cuba he’d have bargaining points useful in such a
containment strategy. Later, Khrushchev believed he could
effect a
change in the balance of power between the two nations.
�00
Possible
outcomes included the prospect of further U.S. concessions,
protection
of Cuba from invasion, and either conventional or nuclear war.
Khrushchev had direct evidence drawn from the Korean War
and possibly other evidence from spies operating in the United
States about the capability of the U.S. military. Given the
nuclear
missiles he actually deployed as well as his assignment
(temporary) of
operational control of those missiles to the Soviet Group of
Forces
commander, General Issa Pliyev, Khrushchev seems to have
inferred
the U.S. forces were formidable.
�0�
He also apparently concluded that
the missiles – and the other forces – would be adequate to the
task

of successfully satisfying his purposes; the United States, faced
with
9� Allison and Zelikow, Essence of Decision, ��.
99 Nikita Khrushchev, “Speech to Ambassadors at Reception,
�� November
�9��,” in James Beasley Simpson, compiler, Simpson’s
Contemporary Quotations:
The most Notable Quotes since 1950 (Boston, MA: Houghton
Mifflin Company,
1988), online edition, URL:
<http://www.bartleby.com/63/83/183.html>, last
accessed �0 April �00�. Ironically, Khrushchev was wrong.
History sided with
his adversaries.
�00 Cited in Garthoff, Reflections, ��.
�00 Gribkov, “The View,” �. Control reverted back to
Moscow in
September.
– �� –
operational missiles in Cuba, would concede.
�0�
The concepts on
which Khrushchev relied probably were the same as those on
which
Castro relied: military strategy and doctrine as well as
communist
and capitalist theories. Again, Khrushchev’s key question

mirrored
that of Castro: How to keep the Americans from invading Cuba.
Additionally, there could have been a second question on how
to
dominate, or at least gain concessions from, the United States.
Kennedy. Finally, there was Kennedy. While Kennedy’s exact
purpose vis-à-vis Cuba prior to the discovery of the missiles
also
remains unclear, it is known that he did not plan to invade the
island during the summer and fall of �9��.
�0�
However, �9�� was
an election year and the Kennedy administration was vulnerable
with respect to Cuba.
�0�
Kennedy could not afford to appear soft
on Communism. Kennedy also had an interest in keeping the
U.S.
and NATO forces in Berlin.
All this changed once the missiles were discovered. Kennedy’s
deliberations revolved around removing the missiles as
expeditiously
as possible. He no doubt realized that if his administration
bungled
the crisis they might lose at the polls in November.
�0�
Therefore, it
is reasonable to conclude that Kennedy’s goals included

retaining
control of Congress – he too had a regime to preserve.
Kennedy’s
point of view was clear: the deployed Soviet strategic nuclear
missiles
were a threat to the United States and must be removed. He
assumed
that such a strategy could be developed. He apparently also
assumed
that the Soviets had placed a small force on the island. Possible
outcomes included successful removal of the missiles and their
�0� Wirtz, “Organizing,” ��. Wirtz argues that the Soviets
were not only
wrong about the U.S. reaction, they were surprised!
�0� Wirtz, “Organizing,” ��0.
�0� Fen Osler Hampson, “The Divided Decision-Maker:
American Domestic
Politics and the Cuban Crises,” International Security 9, no �
(Winter �9��/��), ��.
Cited hereafter as Hampson, “Divided.”
�0� Hampson, “Divided,” ��.
– �� –
supporting forces, an escalation of measures to prompt that
removal,
and the possibility of conventional or nuclear war.
�0�

Kennedy had evidence of his political vulnerability He also had
evidence – Khrushchev’s own actions included – that the Soviet
leader was a formidable adversary. But based on Penkovsky’s
evidence he inferred that Khrushchev might be persuaded to
back
down. As Scott concludes, “various writers contend that
Penkovsky’s
intelligence…[guided] Kennedy’s handling of the crisis from a
position of strength.”
�0�
This occurred in part because “the KGB’s
discovery of Penkovsky’s espionage alerted Khrushchev to the
fact
that Kennedy now knew he was bluffing.”
�0�
As has been noted,
Kennedy lacked accurate evidence about what was really on the
island. Nevertheless as has been noted, he inferred that military
strikes were not – at least at that time – an option. Kennedy
probably
relied on the same concepts on which Castro and Khrushchev
relied.
In addition, the concepts embodied in the Monroe Doctrine were
also probably a factor. Kennedy’s key question was, “How to
get
the Soviets and their missiles out of Cuba?”
Foresight. Viewed with foresight, there were a number of
possible
outcomes to the crisis based on the inferrable goals that could
be
associated with any of the three leaders. A win for one of the

leaders
was not necessarily a loss for the others as table � shows. In
some
cases, achieving one’s goal was a win for the leader, but a
failure to
achieve it was not necessarily a loss. “Winning” the crisis
should have
involved attaining all or most of one’s goals. However, since
not all
goals were of equal importance, failing to achieve one could
mean
that one lost in the larger crisis. Similarly, achieving one’s
goals did
not guarantee winning the larger crisis. This becomes evident
as
the actual winners and losers are considered.
�0� Neustadt and May, Thinking in Time, �-��. Given the
presence of the
Soviet tactical nuclear weapons on Cuba it is probable that any
invasion that began
with conventional forces would have escalated to nuclear
scenarios.
�0� Scott, “Penkovsky,” ��.
�0� Scott, “Penkovsky,” ��.
– �� –
And the Winner is? Who actually won the Missile Crisis?
Critical
assessments of the three leaders involved and the attainment of
their

goals reveals the answers. First, Castro certainly got what he
wanted:
The United States never again invaded Cuba.
What did Khrushchev get out of the resolution of the crisis?
From the standpoint of his first goal, to protect Cuba from the
United
States, he was successful. The subsequent removal of the
Jupiter
Goal Outcome
Leader
Castro Khrushchev Kennedy
Prevent U.S. Inva-
sion of Cuba
Achieve Win Win Neutral
Fail Lose Lose Win
Get the Missiles
out of Cuba
Achieve Neutral Lose Win
Fail Win Win Lose
Spread Commu-
nism
Achieve Win Win Lose
Fail Neutral Lose Win
Get U.S. and Allies

out of Berlin
Achieve Neutral Win Lose
Fail Neutral Lose Win
Force Concessions
from United States
Achieve Win Win Lose
Fail Lose Lose Win
Force Concessions
from Soviet Union
Achieve Neutral Lose Win
Fail Neutral Win Lose
Force Concessions
from Cuba
Achieve Lose Neutral Win
Fail Win Win Neutral
Preserve Regime
Achieve Win Win Win
Fail Lose Lose Lose
Table 4: Goals and Outcomes in the Cuban Missile
Crisis

missiles in Turkey appeared to increase the security of his
borders – an
apparent concession from the United States.
�09
However, Khrushchev
failed to spread Communism further. And Berlin remained
partially
109 The issue of the Jupiter missiles is complex. They were
not included
in the formal agreement but were offered up in a secret
unilateral assurance from
Kennedy (Garthoff, Reflections, ��).
– �� –
in the hands of the West. If, as Allison and Zelikow postulate,
this was the real motivation behind the nuclear deployment then
Khrushchev’s failure was highly significant. Additionally, his
regime
was embarrassed by the revelation in the United Nations of the
presence of the missiles. Ultimately, he was removed from
power
two years later and placed under house arrest until his death in
�9��. So, Khrushchev, while he won a number of his goals,
wound
up the overall loser.
Kennedy is the other winner in the crisis. He gets the Soviets
missiles removed from Cuba while avoiding war. He is later
able

to retire an obsolete missile system (the Jupiters). He prevents
the
Soviet Union from seizing control of Berlin. Finally, his
popularity
increases and the Democrats retained power in the �9��
elections.
Ten Years Later, They Meet Again
It is worth noting that Soviet denial and deception in support
of military deployments and operations in Cuba did not end in
the
early winter of �9��. In �9�0 Henry Kissinger stormed into
the
office of H. R. Haldeman and demanded to see President
Nixon.
Aerial reconnaissance had brought news. “The Cubans are
building
soccer fields,” Kissinger said. “Cubans play baseball. Russians
play
soccer.”
��0
The Soviets it seems had been discovered building a
submarine base at Cienfuegos. Prior to a particular overflight,
that
fact had been denied to the United States. The soccer fields
were
for the recreation of Soviet troops.
��0 H.R. Haldeman, with Joseph DiMona, The Ends of Power
(New
York, NY: Times Books, �9��), ��–��, in Patrick J. Haney,
“Soccer Fields and
Submarines in Cuba: The Politics of Problem Definition,”

Naval War College
Review L, no �, sequence ��0 (Autumn �99�), URL:
<http://www.nwc.navy.
mil/press/Review/1997/autumn/art5–a97.htm>, last accessed 6
April 2006.
Emphasis in original.
– �� –
Judgment: Critical Thinking Would
Have Made a Difference
As this case study makes clear, there are a number of junctures
where critical thinking or structured analytic methods could
have
made a difference in analyzing the Soviet missile deployment to
Cuba. While it is true that Sherman Kent argued for a
scientific
approach to analysis, it does not seem to have been widely
practiced
at the time.
��
As of �9��, most studies that we now have in hand
on human reasoning in intelligence had not been completed.
The
champions of structured intelligence analysis methods had not
yet
developed their techniques.
Nevertheless, a study of the crisis is germane because the same
kinds of errors repeat themselves again and again. The errors
seen in the

Cuban case – failure to question assumptions, to take seriously
the
evidence and the patterns they imply, to counterfactually
examine
analytic implication and consequences, in short to make
reflective
judgments – also were cited by the Senate in its critique of the
�00� Iraqi WMD estimate.
��
In both cases, deceptions confounded
analysts, leading them to wrong conclusions.
��
In the Cuban case
the analysts eventually figured it out. This did not occur in the
�� Sherman Kent, Strategic Intelligence for American World
Policy (Princeton,
PA: Princeton University Press, �9�9), ��–�0�.
�� In the Cuban situation, the debate between McCone and
analysts over
the nature of the weapons the Soviets might emplace came
close to overcoming
these errors. Regarding the Iraqi WMD estimate, the debate
between the CIA
and the Department of Energy over the use of the aluminum
tubes also came
close. Unfortunately, the incipient debate focused on the
results, not the process.
In both cases, the side that was eventually proved right failed to
make the now-
accepted case.

�� Sherman Kent challenged this assertion in his defense of
the original
estimate and the process by which it was created. He believed
that an earlier
revelation of the Soviet actions would have led to an
intractable situation. However,
an earlier revelation – say in August – would have offered
President Kennedy
and his advisers more options, particularly more diplomatic
options. Kent also
failed to observe that denial and deception occurred – at least
for the record.
See Sherman Kent, “A Crucial Estimate Relived,” Reprint,
Studies in Intelligence
��, no. � (�99�): ��–��9.
– �� –
most recent failure. Critical thinking skills went unused among
both
generations of analysts.
– �� –
How cAN iNTelligeNce ANAlysTs
eMPloy criTicAl THiNkiNg?
The Poor Record
Critical thinking is what consumers of intelligence appear to
expect when they task producers to examine issues. Corporate
consumers require analysts creating intelligence to “[evaluate] a

situation, problem, or argument and [choose] a path of
investigation
that leads to finding the best possible answers.”
��
In the national
security field, strategic intelligence pioneer Washington Platt
notes that
“[intelligence] is a meaningful statement derived from
information
which has been selected, evaluated, interpreted, and finally
expressed
so that its significance to a current national policy problem is
clear.”
��
Derived from strategic intelligence, “best answers” should
clearly
express what is significant to national policy problems. They
may
also support warfighters with essential operational and tactical
intelligence. Critical thinking leads to the best answers for the
specific context at hand.
Unfortunately, analysts’ biases and mindsets repeatedly obscure
best questions and answers. From well before the Japanese
surprise
attack on Pearl Harbor to the �00� estimate on Iraqi weapons
of
mass destruction, a failure to think critically about potential
crises
contributed to repeated intelligence failures.

��
“Expert analysis”
�� Daniel Feldman, Critical Thinking: Strategies for
Decision Making (Menlo
Park, CA: Crisp Publications, Inc, �00�), �.
�� Washington Platt, Strategic Intelligence Production:
Basic Principles (New York,
NY: Frederick A. Praeger, �9��), �.
�� George S. Pettee, The Future of American Secret
Intelligence (Washington, DC:
Infantry Journal Press, �9��). Chapter � provides a summary
of U.S. intelligence
failures during the first half of the 20th Century. While
hindsight is an imperfect
mirror for reviewing the past, one conclusion to be drawn from
a review of the
evidence is that critical thinking could have minimized many of
the ensuing
crises.
– �9 –
was not enough.
��
Biases and mindsets too often converted subject-
matter confidence into arrogance; false assumptions blinded
analysts
to their target’s true intentions. For example, at least one CIA
Iraq
analyst acknowledged that the �990 invasion of Kuwait,

was an intelligence failure…We were guilty of a kind of
mindset…The idea that a country [Iraq] would march up
to the border, put �00,000 troops there, go in and do what
they’ve done; I don’t think anybody thought they’d do it.
��
In 2002, Intelligence Community analysts failed to definitively
assess whether the Iraqi government still possessed weapons of
mass
destruction. The Senate noted in its review of the failure that
[rather] than thinking imaginatively, and considering
seemingly unlikely and unpopular possibilities, the Intelligence
Community instead found itself wedded to a set of
assumptions
about Iraq, focusing on intelligence reporting that appeared
to confirm those assumptions.
��9
The mistakes of �00� also occurred in �990 – and for that
matter
in �9��. Analysts failed to question assumptions widely held
at the
time. Instead they chose the first answer that satisfied the
situation,
a phenomenon known as “satisficing.”
��0
Other means by which
intelligence analysts and policymaking customers reasoned
poorly are
listed in table �. According to Ephraim Kam, the problem is so

great
that the intelligence analysis process is “consistently biased,
and…bias
117 Anthropologist Rob Johnston explores this paradox. See
Dr. Rob
Johnston, Analytic Culture in the U.S. Intelligence Community:
An Ethnographic Study
(Washington, DC: Center for the Study of Intelligence, �00�),
��–��. Cited
hereafter as Johnston, Analytic Culture.
�� Anonymous CIA Analyst, �990, in Don Oberdorfer,
“Missed Signals in
the Middle East,” Washington Post Magazine, �� March
�99�, �0.
��9 The Commission on the Intelligence Capabilities of the
United States
Regarding Weapons of Mass Destruction, Report to the
President of the United States,
March ��, �00�, URL: <http://www.wmd.gov/report/
index.html>, last accessed
�� July �00�, ��. Cited hereafter as WMD Commission,
Report.
��0 Morgan D. Jones, conversation with the author, ��
December �00�.
– �0 –
is the cornerstone of intelligence failures.”
��

Paul and Elder claim
that much thinking is “biased, distorted, partial, uninformed, or
down-right [sic] prejudiced.”
��
There are repeated failures to think
critically; but could critical thinking about these situations
prevent
the failures?
Means by Which Decisions are Made
Select first answer that appears “good enough.”
Focus on narrow range of alternatives, ignoring need for
dramatic change
from existing position.
Opt for answer that elicits the greatest agreement and support.
Choose the answer that appears most likely to avoid some
previous error or
duplicate a previous success.
Rely on a set of principles that distinguish “good” alternatives
from “bad”
alternatives.
Table 5: How Analysts Decide
Source: Excerpted from Alexander George, Presidential
Decisionmaking in Foreign
Policy: The Effective Use of Information and Advice (Boulder,
CO: Westview Press,
�9�0), Chapter �.

Critical thinking helps mitigate the effects of mindsets and
biases
by invoking skillful examination of evidence both for and
against an
issue, as well as consideration of obvious and less obvious
alternative
explanations. In the 1990 example, thinking critically would
have
raised other possible explanations for why Hussein’s troops
were on
the Kuwaiti border and what he intended for them to do.
��
That the
Iraqi troops were trained by the former Soviet Union and
followed
its tactics might have been an indicator of future intentions. In
the
case of the weapons of mass destruction (WMD), analysts
might have
�� Ephraim Kam, Surprise Attack: The Victim’s Perspective
(Cambridge,
MA: Harvard University Press, �990), ��.
�� Paul and Elder, Concepts and Tools, �.
�� A consideration of who trained Hussein’s troops – in
this case, the
Soviet Union – might have led to an examination of military
doctrine. Thus, it
quickly would have become clear that troops mobilized on a
border were going
to cross that border. Analysts would then have known that an
invasion of Kuwait

was imminent.
– �� –
asked, as they realized they had no new evidence of the
weapons,
“How can we prove that the weapons of mass destruction
definitely
are not there?” or “What would we expect to see if Saddam
Hussein
had gotten rid of his weapons of mass destruction?”
��
We must assume that analysts can be encouraged to think in
this manner. Modern strategic intelligence pioneer Sherman
Kent
believed this to be the case when he wrote nearly �0 years ago
that
intelligence analysts “are supposed to have had more training in
the
techniques of guarding against their own intellectual frailties”
than
the larger populace.
��
Assessing Evidence��
Understanding how evidence is assessed is a necessary first step
in understanding how analysts can better reason with and about
it. The incomplete and ambiguous information and data with
which intelligence professionals work compounds their

reasoning
process.
��
Sources are unreliable and contradictory; adversarial
denial and deception increase doubt. When information and
data are
questionable, derived evidence is even more uncertain, and
inferences
arising from that evidence may be highly suspect. One way that
analysts can reduce this uncertainty is by building defensible
inferences
�� These analyses are those of the author. The last
question was jointly
arrived at in a conversation between the author and Mark
Lowenthal, �� July
�00�.
�� Sherman Kent, Strategic Intelligence for American World
Policy (Princeton,
NJ: Princeton University Press, �9�9), �99.
�� This section is based (with additions) on materials
developed by Francis
J. Hughes and the author for a structured analytic methods
course taught at the
National Defense Intelligence College. For a deeper
examination of evidence
assessment see Francis J. Hughes and David A. Schum,
Credibility Assessment: A
First Step in Intelligence Analysis, unpublished tutorial,
National Defense Intelligence
College, April �00�.

�� Information, data, and evidence are different kinds of
knowledge.
Pieces of information comprise data, which in turn comprise
evidence. Evidence
is further distinguished from data by the fact that it contributes
to discrimination
among alternative end states. Data, while relevant, do not
necessarily allow
discrimination.
– �� –
from the evidence. At their most basic level, these inferences
depend
on credibility (can it be believed), relevance (does the evidence
bear on
the issue), and inferential or probative force (how compelling is
it in
answering the questions raised by the issue). Unfortunately, no
mass
or body of evidence – in intelligence or anywhere else – comes
with
these three properties already established. Establishment of
these
properties to abet uncertainty reduction through inference
occurs
only through a process of argument, creative hypothesis
generation,
and the development of chains of reasoning.
Authenticity, accuracy, and reliability represent criteria for
establishing the credibility of tangible evidence.
��

An analyst striving
�� “Tangible evidence” is a technical term describing things
that bear
relevance to an issue under scrutiny. It is contrasted with
“testimonial evidence”
TESTIMONIAL: APPLYING CRITICAL THINKING
This is a true story. All the characters are real and really naive.
As a fresh graduate of NSA’s critical thinking and structured
analysis class,
I attended an Intelligence Community class on
counterintelligence. My fellow
students included folks from all over the community, including
CIA and FBI case
officers. During the class, the instructor put forth a case study
for students
to think about and decide how best to “analyze and investigate”
to find the
mole. Differing opinions surfaced, but a common thread
appeared among the
case officers: follow your “gut” feeling and collect evidence to
support that
assumption, no matter how long it took.
The instructor, enjoying the collective mindset and the
opportunity to
shatter paradigms, concentrated on how information was
collected and analyzed.
Again, the case officers agreed the best solution was to continue
gathering
information until you “proved your case.” I, the lone NSA
token student, raised
my objection, “How long is long enough? Until you’ve ruined
the career of

innocent, hardworking persons in the IC, or until you find
another answer you
like?” Although it did not earn me friends, it did open up a
window to inform
these case officers and other attendees about the intricacies of
analysis.
Critical thinking is not just about putting information together,
finding
a pattern, then choosing an answer, it is about reducing bias,
considering all
options available, and presenting options to a decision–maker.
And critical
thinking is about paying attention to what and how you are
doing it. I reasoned
that, since the investigators in the case study concentrated on
only one
Source: A mid–level intelligence professional at the National
Security Agency
– �� –
to determine authenticity may ask, “Is the evidence what it
seems
to be?” In determining accuracy, the key question is whether
the
piece of evidence or the system that produced it has the
resolution
to reveal what the analyst believes the event or record should
reveal.
Assessing reliability involves determining whether different
means
of collection produce the same results.

If, on the other hand, the evidence is testimonial, different
criteria
apply. The first thing to be established is whether the source is
being
truthful. Truthfulness is not absolute. Rather, it is time- and
context-
dependent. A source may believe he is being truthful about an
issue
or may have legitimate reasons to be untruthful about that issue.
In
– what people say about the issue.
person, the analysis was faulty because somewhere along the
line they limited
their suspects (introduced bias) regardless of reason (access,
family connections,
angel–like qualities, etc.). Although it would take time, the
investigators should
conduct quantitative and qualitative analysis, make a decision
tree (options open
to a mole in a heavy security environment), and then play
Devil’s Advocate.
In addition, analysis never reveals one “solution” or “smoking
gun”; it leads to
two or three “options” which can be investigated in detail.
The instructor, pleased that someone in the class knew what the
“Analysis
of Competing Hypotheses” was, revealed the answers of the
real–life case study:
after three years of investigating, the case officers had the
wrong person due
to incorrect information in old reports and limiting their suspect
list. Instead

of going back to the original source information, the officers
read old reports
that were unfortunately biased by Agency politics and external
societal events
(1950–60s). The real mole was discovered over a decade later;
he was the
son of a former Agency chief, well educated and well liked, but
did not have
access to the information. The individual “borrowed”
interesting reports from
friends who had access, covered his tracks, and continued
spying against the
U.S. for several years. Just as the authorities were about to
arrest him, he
was found dead in a hotel room from a reported suicide, but the
cops could
not figure out how he had two bullets in his chest and one in the
back of his
head. Miracle suicide.
What is the moral of the story? Conduct a thorough analysis
right the
first time and you can catch a mole anytime. I only hope the
attending case
officers got the message, especially from a non–gun toting NSA
analyst.
who wishes to remain anonymous, email to the author, 9 March
2006.
– �� –
another time and about another issue, these impediments to
veracity

may not exist for that source. Therefore, establishing the
truthfulness
of a source can pose a significant challenge to the analyst.
In an ideal situation, a review of evidentiary relevance causes
an analyst to examine the likelihood of any potential answer to
the
problem or question – a hypothesis – with an eye to the
modification
of existing hypotheses or even the invention of new ones. In
other
words, an analyst might theorize that a certain bit of evidence
will
indicate that an individual or a group of individuals will
engage in
nefarious activity. Yet, if none of the evidence at hand bears
on the
issue, then the analyst may need to reconsider – or even reject –
this
evidence based on its lack of relevance. The analyst also
should
consider that the individual may not engage in the activity.
What is true about the future is also true about the present and
the past. A lack of evidence relevant to an issue should prompt
analysts to reassess their theories about the issue at hand. For
example, an unnamed FBI investigator in the 2001 anthrax case
noted, “[Reasoning] says that if you think a person is your guy,
but
you can’t find anything to put him in the game, you’ve got to
keep
looking at others.”
��9
The failure of the FBI to implicate its principal

suspect forced it to consider other explanations as to who sent
the
anthrax-laden letters in September 2001 to political leaders and
media figures.
The analyst also is concerned with how strongly the evidence
129 Allan Lengel and Guy Gugliotta, “Md. Pond Produces No
Anthrax
Microbes: FBI Sought Clues In Deadly Attacks,” Washington
Post, � August �00�,
A0�. Cited hereafter as Washington Post, “No Anthrax.”
While this is a law-
enforcement example, it illustrates good critical questioning
about an apparent
lack of evidence. Criminal investigation and intelligence
analysis similarly assess
events and evidence with the goal of description, explanation,
and interpretation.
The principal difference is that intelligence analysis endeavors
to do so before the
event occurs – in other words, to estimate. It is worth noting
that over two years after
the pond was drained the case remains unsolved; the individual
under suspicion
at the time was never charged with the crime. The FBI was
unable to link the
evidence to the individual. See for example, Allan Lengel,
“Little Progress In
FBI Probe of Anthrax Attacks: Internal Report Compiled As
Agents Hope for a
Break,” Washington Post, �� September �00�, A0�.
– �� –

undermines or supports the particular hypotheses under
examination
– the probative force of the evidence. Certain evidence,
coming
from certain kinds of sources, persuades more strongly than
does
other evidence drawn from other sources. Tangible evidence
might
have greater probative force than testimonial evidence.
Consider, for
example, a hypothetical biological weapons issue. Traces of
certain
toxic substances found near an alleged biological weapons lab
carry
greater probative force than the testimonial denials of the
government
of the country in which the lab and the samples were found.
The “ideal” analyst also assesses the objectivity, observational
sensitivity, and competence of sources. One question suitable
for this
assessment is how biases may have corrupted a source’s
objectivity.
Did the source see or hear the evidence directly, and under what
conditions did this occur? A reality check also is made at this
point.
Does it makes sense that a particular source claims to have been
in
a position to make the observation or have access to the source
of
information?
Another consideration involves denial and deception. In
assessing
the evidence, the analyst should ask, “What is the likelihood

that I
am being deceived? This question is surprisingly difficult to
answer
accurately. Given that anchoring biases and confirmation
heuristics
cause analysts to find what they seek to find, a denial and
deception
hypothesis is often easily proved.
��0
Critical thinking challenges
this by forcing an examination of alternative points of view.
For
intelligence analysts, proactive, focused and surreptitious
collection
of information about the often minimal disconfirming evidence
pays rich dividends.
��
In this case, the question becomes, “What is
the likelihood that I am not being deceived?” Thus engaged,
really
good intelligence analysts create valuable knowledge.
��0 Author’s notes, National Defense Intelligence College,
Denial and
Deception Advanced Studies Program, Spring �00�. The
author is a National
Intelligence Council-sponsored participant in the Foreign
Denial and Deception
Committee’s Denial and Deception Advanced Study Program at
the NDIC.
�� Ben-Israel, “Logic of the Estimate Process,” �0�–�09.

– �� –
Facilitating Evidentiary Assessment
The questions that need to be asked about each piece of
evidence
are the same as those employed in critical thinking. In making
her
evaluation of what constitutes evidence, the analyst repeatedly
asks
“why”: “Why do I believe this information is relevant to the
question
at hand (either against or for) and therefore exists as evidence?”
“Why
do I believe the source to be credible?” Additional questions
about
the analyst’s own thinking processes might arise, such as:
“What are
my biases and why do they lead me to think this way?”
For example, in assessing the intentions of former Iraqi
dictator
Saddam Hussein in light of the U.S. destruction of his
intelligence
headquarters in �99�, an analyst might ask whether Hussein’s
declarations not to retaliate against the United States were
credible.
A corollary question might be, “What does Saddam Hussein
gain by denial and deception?” Based on Hussein’s statement, a
determination that additional evidence was needed would
stimulate
collection of similar statements from other episodes. By
analyzing

what Hussein did in those instances, the analyst might
determine
that he was usually untruthful. Therefore, this piece of
evidence
would be deemed to be of low credibility despite its probative
force,
and relevance to the determination whether Hussein would or
would
not retaliate.
��
In other words, Hussein’s statement might reflect an
ongoing deception campaign. To explore this hypothesis
further,
the analyst might seek to determine whether increases in Iraqi
agent
communications were relevant to the issue and whether
additional
collection of such broadcasts was warranted.
��
In practice, these and similar questions can be answered
quickly.
Analysts often answer some of them unconsciously as they
struggle
132 Heuer develops this scenario and evidence in an example
of the analysis
of competing hypotheses. See Heuer, Psychology, �0�–�0�.
�� Heuer, Psychology, �0�–�0�. For a recent detailed
look at Saddam Hussein
and deception see Kevin Woods, James Lacey, and Williamson
Murray, “Saddam’s

Delusions: The View from the Inside,” Foreign Affairs ��, no.
� (May/June �00�).
URL:
<http://www.foreignaffairs.org/�00�0�0�faessay��0�/kevi
n-woods-
james-lacey-williamson-murray/sadda-s-delusions-the-view-
from-the-inside.html>,
last accessed �� March �00�.
– �� –
to meet short deadlines. However, as a function of not
employing
scientific methods or other structured and critical thinking
techniques,
their thinking is largely intuitive. Analysts choose to ignore
how
mindsets and assumptions impede their judgments. Conversely,
critical thinking ensures that the reasoning process is a self-
conscious
one. By making the unconscious conscious, analysts reveal
where
they may be biased, helping
ensure that questions they
address are thoroughly and
fairly considered.
An analyst committed to
critical thinking continually
asks questions while developing
the mass of evidence needed to
assess an issue. Evidence arises from questions answered
satisfactorily.
Information for which the questions cannot satisfactorily be

answered
is excluded – but only after the analyst reflects on the process
of
evidence development. Consideration is owed to the question of
how
biases, and possibly active denial and deception by an
adversary, have
influenced the selection of both questions and answers. One
way of
revealing bias is by asking questions such as, “If the opposite
outcome
is actually true, what other evidence would I expect to see?”
The marshaling of evidence refers to a questioning process by
which data and information are assessed and evidence is
created. It
may be done in solitary fashion, Socratically with a teammate,
or
collegially among Intelligence Community focus groups. It is a
vital
ingredient of productively imaginative intelligence analysis.
Each
question an analyst asks not only becomes a device for
attracting
existing evidence, but also generates new evidence not yet
visible.
Identifying new evidence increases the thoroughness by which
the
issue is evaluated and increases the probability that the correct
solution has been discovered.
Embracing a Methodology
Once a relevant mass of evidence is established, the analyst
evaluates which method or methods of analysis may best
develop

Each method of analysis uniquely
interacts with the intelligence
question; therefore, the method
(or methods) selected can pro-
duce profoundly different results
– affecting the analyst’s arrival at
the most accurate answer.
– �� –
a solution to the issue. Differences among various analytic
methodologies, techniques, and tools are not trivial.
In fact, comparing the results of different analysis methods can
be
a valid means of establishing the accuracy of the answer. If
various
means of analysis yield multiple results, a review by different
analysts
makes for the ideal environment for critical thinking to discuss
and
debate those results.
��
Therefore, assessment of the available evidence includes
redirecting thinking, soliciting feedback from other sources,
appraising
the quality of possible answers, and comparing initial goals
with
results. In so doing an analyst employs interpretive and
evaluative

skills to select the best mass of evidence to analyze.
Creating Better Inferences
Critical thinking aims to ensure that inferences are reasonable
and evidence-based. Inference creation begins at the same time
that analysis starts. As each piece, or the whole mass of
evidence, is
considered, inferences are made.
��
Resulting “chains” of inferences
linking the evidence to the hypotheses under consideration are
known
as arguments. Chains of inferences converge – strengthening
the
argument – or diverge – weakening it. The self-reflective nature
of
critical thinking places a check on these inferences. The analyst
asks,
“Do my inferences flow from the evidence?” and, “Are my
inferences
logical given the evidence and other inferences I have made?”
Inferences lead to a search for additional evidence. In other
words, based on inferences drawn from the evidence at hand, an
analyst may infer that there are other sources of evidence to
consider.
For example, such reasoning was cited in the explanation of
why
�� For a more detailed discussion of this and other means
of countering
biases, see Heuer, Psychology, ��0–��; Irving Janis,
Groupthink: Psychological Studies of

Policy Decisions and Fiascoes, �nd Edition (Boston, MA:
Houghton Mifflin Company,
�9��); and Scott Plous, The Psychology of Judgment and
Decision-making (New York,
NY: McGraw Hill, Inc., �99�).
�� Indeed, the analyst’s acceptance – or rejection – of this
evidence is a result
of inferences: either the evidence is valid, credible, and
relevant, or it is not.
– �9 –
the FBI investigated a pond in western Maryland during June
2003, searching for the source of the 2001 anthrax attacks. At
the
time, the FBI believed their primary suspect lived near the pond
in
question. In their search for additional evidence, they reasoned
that
the pond might be something in which evidence had been
hidden.
Thus, inference led the FBI to drain the pond to search for that
evidence.
��
The analyst self-consciously evaluates the thinking process and
the biases that have affected it to reduce unproductive thinking
and
to consciously develop new ways of understanding the evidence
at
hand. This self-regulation also plays a role as the available
evidence

is considered. A means of accomplishing this within a
collaborative
setting is to seek the assistance of colleagues of diverse
backgrounds.
The underlying premise is that their biases differ sufficiently to
enable
productive and thorough analysis to occur.
��
Producing Intelligence
Analysts who produce intelligence assessments and judgments
have the opportunity to employ what Peter Facione considers
the
core cognitive skills of critical thinking: interpretation,
analysis,
evaluation, inference, explanation, and self-regulation.
��
Each
of these competencies contributes to the integration of
available
evidence through a clear line of reasoning to the most probable
answer.
Facione asserts that good critical thinkers can explain “what
they
think and how they arrived at that judgment.”
��9
By documenting the
�� Washington Post, “No Anthrax.”

�� See Robert Callum, “The Case for Cultural Diversity in
the Intelligence
Community,” International Journal of Intelligence and Counter
Intelligence ��, no. �,
Spring �00�: ��–��.
�� Peter A. Facione, Critical Thinking: What It Is and Why
It Counts
(Milbrae, CA: California Academic Press, �99�, updated
�00�), URL<http://
www.insightassessment.com/>, last accessed 22 July 2005, 4.
Cited hereafter as
Facione, Critical Thinking.
��9 Facione, Critical Thinking, �.
– �0 –
reasoning process used to arrive at an answer, analysts move
beyond
merely “stating results [to] justifying procedures, and
presenting
arguments.”
��0
Questioning biases and mindsets encourages
consideration of alternative possibilities that overcome what
Josh
Kerbel refers to as “single–outcome [analytic] trajectories.”
��
Most

importantly, analysts demonstrate a critical spirit – a reflection
of
character.
��
Thus, critical thinking contributes to short-term analysis and
assumes an essential role in longer-term analysis. Indeed,
building a
comprehensive picture of an issue or target requires critical
thinking
to determine which previous reports are included or excluded.
“How
do the parts contribute to the whole?” is one question the
analyst
asks. Another is, “How is the whole greater than the sum of its
parts?” When previously published intelligence reports diverge,
the
critical thinking process helps the analyst ensure that the
divergence
is considered fairly and that the resulting intelligence does not
merely
satisfice.
��0 Facione, Critical Thinking, �.
�� Josh Kerbel, “Thinking Straight: Cognitive Bias in the
US Debate about
China,” Studies in Intelligence �� no. � (�00�), URL:
<http://cia.gov/csi/studies/
vol48no3/index.html>, last accessed 22 February 2006.
�� Facione, Critical Thinking, �. See also Moore and
Krizan, “Intelligence
Analysis,” 9–��.

– �� –
How cAN ANAlysTs be TAugHT
To THiNk criTicAlly?
Many people would rather die than think – in fact, they do.
—Bertrand Russell
Outside the Intelligence Community
Critical thinking offers a framework for structured problem
solving. Yet, despite a corpus of associated literature, critical
thinking remains in its infancy as a discipline. It is still
“largely
misunderstood…existing more in stereotype than in substance,
more in image than in reality.”
��
As Bertrand Russell’s humorous
quip reminds us, critical thinking is not a habit acquired by just
being alive.
Ideally, valuable skills and dispositions should be developed
among prospective analysts before they join intelligence-
producing
corporations. Yet, observations by the author of newly hired
intelligence analysts suggest this happens rarely if at all. This
raises
two questions, “What are the opportunities for prospective
analysts
to become critical thinkers before they are hired?” and often

“Why
do these opportunities not exist?”
Despite its importance, critical thinking is not widely taught
in schools and universities. A mid-�990s California study on
the
role of critical thinking in the curricula of �� public and ��
private
universities concluded that the skill is “clearly an honorific
phrase in
the minds of most educators.”
��
The study concluded that university
�� Richard W. Paul, “A Draft Statement of Principles,” The
National
Council for Excellence in Critical Thinking, URL:
<www.criticalthinking.org/
ncect.html>, last accessed March 18, 2003. The reasons why
critical thinking
remains an undeveloped discipline while important, go beyond
the scope of this
essay and are not addressed.
144 Richard W. Paul, Linda Elder, and Ted Bartell, “Executive
Summary,
– �� –
faculty members “feel obliged to claim both familiarity with it
and
commitment to it in their teaching, despite the fact that…most
have

only a vague understanding of what it is and what is involved
in
bringing it successfully into instruction.”
��
Indeed, the authors of the
study found that while 89% of the faculty they interviewed
“claimed
critical thinking was the primary objective of their instruction,”
only
19% could define the term and only 9% were evidently using it
on
a daily basis in their instruction.
��
If the results of the California
study are representative of the nation at large, they explain why
prospective new hires – themselves college graduates –
generally fail
to exhibit skill in critical thinking at any level of proficiency.
��
Informal conversations with recent hires at NSA support this
premise. Although slightly fewer than half of these
individuals
indicate they have been exposed to critical thinking skills in
college,
most have been exposed only in one class and then only as an
approach to learning the materials covered in that class. While
not
discouraged, respondents apparently were not encouraged to
apply
the skills to other subjects. Thus, a disposition to think

critically is
rarely fostered. Further, when asked to define critical thinking,
most
Study of �� Public Universities and �� Private Universities
To Determine Faculty
Emphasis on Critical Thinking In Instruction,” California
Teacher Preparation for
Instruction in Critical Thinking: Research Findings and Policy
Recommendations, California
Commission on Teacher Credentialing, Sacramento California,
�99� (Dillon, CA:
Foundation for Critical Thinking, �99�), URL:
<criticalthinking.org/schoolstudy.
htm>, last accessed March 18, 2003. Cited hereafter as Paul,
Elder, and Bartell,
Executive Summary.
�� Paul, Elder, and Bartell, Executive Summary.
�� Richard W. Paul, Linda Elder, and Ted Bartell,
California Teacher Preparation
for Instruction in Critical Thinking: Research Findings and
Policy Recommendations California
Commission on Teacher Credentialing, Sacramento California,
�99� (Dillon, CA:
Foundation for Critical Thinking, �99�), ��.
147 A recent but unscientific survey of several major eastern
American
universities’ online catalogs shows the term “critical thinking”
to be used widely
in course descriptions. Further examination of some of those
courses suggests
that East Coast academics may share a similar lack of
understanding about

critical thinking. Although beyond the purview of this paper, it
appears that there
is sufficient evidence to warrant repeating the California survey
on a national
basis.
– �� –
young analysts were unable to do so in any comprehensive
fashion.
��
Finally, few employ any form of critical thinking in their
analytic (or
other) reasoning unless they have been trained to do so.
In fairness, it should be noted that critical thinking awareness,
attitude, and skills varies from one academic discipline to
another. For
example, students of the physical sciences who employ the
scientific
method in a community setting probably have a greater
inclination
toward a basic form of critical thinking if only through
osmosis.
��9
There are also some primary, secondary, and university
education
programs that have adopted a meta-cognitive critical thinking
paradigm. Although the results of the impact of such

programs
are largely anecdotal, at least one study, conducted by Jennifer
Reed
as part of her dissertation found that students’ critical thinking
skills
improved after just one course.
��0
Faculty and administrators of these
programs routinely attend the annual critical thinking
conference
sponsored by the Center for Critical Thinking where their
results
are shared. However, the approximately 500 people who attend
this international event represent a small fraction of the
educators
in the United States.
The elements of scientific method – the formulation of
hypotheses,
collection of relevant data, testing and evaluation of
hypotheses, and
the logical derivation of conclusions – are matched step-by-
step by
�� These conversations occur routinely as part of a training
course the
author teaches to newly hired intelligence and language analysts
at NSA. When
asked to complete the following statement, “In my opinion,
critical thinking
involves…” typical answers center on “thinking outside the
box.”
��9 Francis J. Hughes, conversation with the author,

Washington, DC:
National Defense Intelligence College, � May �00�. Mr.
Hughes is one of the
few proponents and teachers of evidence-based inferential
intelligence analysis,
a means of analysis requiring critical thinking at every stage in
the process.
��0 Jennifer H. Reed, Effect of a Model For Critical Thinking
on Student Achievement
In Primary Source Document Analysis And Interpretation,
Argumentative Reasoning, Critical
Thinking Dispositions, And History Content in a Community
College History Course, PhD
Dissertation, College of Education, University of South
Florida, December �99�,
vii. URL: <http://www.criticalthinking.org/resources/JReed-
Dissertation.pdf>,
last accessed � May �00�.
– �� –
critical thinking.
��
Given that most newly hired intelligence analysts
– at least at NSA – are drawn from fields other than the physical
sciences, one can expect that new hires lack adequate critical
thinking
skills. If NSA’s newly hired intelligence analysts are
representative
of those being hired across the Intelligence Community then it
is
probable that few of the thousands of new hires arrive with

adequate
critical thinking skills.
Inside the Intelligence Community
That analysts need to develop critical thinking skills is
recognized within the Intelligence Community. Heuer wrote in
1999, “[Traditionally], analysts at all levels devote little
attention
to improving how they think.”
��
As a direct response to Heuer’s
criticism, the CIA’s Sherman Kent School includes critical
thinking as
part of the curricula for training new analysts, and recently
initiated
a class in critical thinking.
��
New employees are encouraged to
develop a disposition to think critically as they are taught the
skills
of intelligence analysis.
A similar approach is employed in courses on structured
analysis
methods at the National Defense Intelligence College (NDIC).
Critical
thinking is claimed as a feature of many NDIC courses.
However,
the emphasis in most courses is on topical or issue-related
material,

and only incidentally on the process of thinking. At present,
the skill
itself is largely not taught. Instead, students are expected to
figure
�� Steven D. Schafersman, “An Introduction to Critical
Thinking,” January
1991, URL: <www.freeinquiry.com/critical-thinking.html>, last
accessed 9 March
�00�.
�� Richards J. Heuer, Jr., Psychology of Intelligence
Analysis (Washington, DC:
CIA Center for the Study of Intelligence, �999), ��. Cited
hereafter as Heuer,
Psychology.
�� For a detailed account of how the faculty of the CIA’s
Sherman Kent
School are working to improve their analysts’ knowledge,
skills, and abilities, see
Stephen D. Marrin, “CIA’s Kent School: Improving Training for
New Analysts,”
International Journal of Intelligence and Counter Intelligence
��, no. � (Winter �00�–�00�):
�09–��.
– �� –
it out on their own, encouraged by the thesis writing process.
The
college is aware of the importance of critical thinking and new
core
courses may include formal critical thinking instruction.

New NSA analysts are provided with an introduction to the
solitary part of the skill (as discussed in figure 4) as part of
their
orientation. The agency’s 40-hour follow-on program is the
first
Intelligence Community course to focus primarily on enhancing
analysts’ solitary and communal critical thinking skills. In
addition
to critical thinking skills, participants learn and apply, and then
assess,
the appropriateness of fourteen structured methods of analysis.
The
course has been taught to both U.S. and Allied personnel drawn
from intelligence, counterintelligence, information assurance,
and
law enforcement communities.
��
Training in critical thinking is offered to Defense Intelligence
Agency (DIA) analysts using a variation of the NSA-developed
course. Also, in �00�, DIA tested the “critical thinking skills”
of
a sample of its employees using the Watson-Glaser Critical
Thinking
Appraisal. The instrument claims to “measure abilities involved
in
critical thinking, including the abilities to define problems,
select
important information for the solution to problems, recognize
stated
and unstated assumption, formulate and select relevant and
promising
hypotheses, [and] draw valid conclusions and judge the validity
of

inferences.”
��
The appraisal appears to confuse abilities with skills,
although both do belong within the domain of analysis.
��
Finally, it
appears that the test does not evaluate an individual’s meta-
cognitive
skills in assessing and correcting the process of reasoning.
The increasing opportunities for enhancing critical thinking
skills
and dispositions reflect a recognition of the importance of
critical
154 The syllabus from the course is in the Appendix.
�� Harcourt Assessment, Inc., Local Norms Report, Watson-
Glaser Critical
Thinking Assessment, Report prepared for the Defense
Intelligence Agency, �00�,
�. Cited hereafter as Harcourt, Watson–Glaser.
�� Abilities and skills are easily differentiated. Individuals
are born with
certain abilities (and can improve them through training) but
they must learn a
skill. See for example, Moore and Krizan, “NSA.”
– �� –

thinking in intelligence analysis.
��
Evidence becomes intelligence
through an “ordered thinking process [involving] careful
judgments
or judicious evaluations leading to defensible conclusions” –
through
critical thinking.
��
Former CIA analyst Morgan Jones asserts that
methods for critical thinking and problem solving, if applied,
can
improve the quality of analysis and decisionmaking.
��9
Implications of Teaching Critical Thinking
Although Sherman Kent, Richards J. Heuer, Jr., and others have
over the years addressed intelligence analysis and its
relationship with
critical thinking, recent presidential executive orders and
legislative
mandates are bringing a new emphasis to how the Intelligence
Community can change analytic practices to achieve improved
outcomes. Teaching analysts to be better critical thinkers may
be
seen as an easy way to satisfy these requirements. However,
linkages
between intelligence analysis and critical thinking remain
poorly
understood. Considerable confusion remains about what
constitutes

critical thinking and how it supports intelligence analysis.
A common excuse among analysts to defend their “non-use
of such self-conscious processes is a lack of time.”
��0
Teaching
critical thinking skills is of little value if analysts are not
inclined
to use them.
��
For those who are willing to think critically, various
�� The Director of National Intelligence, John D.
Negroponte, in the
Foreword of the initial National Intelligence Strategy of the
United States of America
(Washington, DC: Office of the DNI, October �00�), wrote
that one of the three
principal tasks for the reformed Intelligence Community is to
“bring more depth
and accuracy to intelligence analysis.” See
http://www.dni.gov/NISOctober�00�.
pdf.
�� Moore and Krizan, “Intelligence Analysis,” ��.
��9 Morgan D. Jones, The Thinker’s Toolkit: 14 Powerful
Techniques for Problem
Solving (New York, NY: Random House, Inc, 1995), xi. Cited
hereafter as Jones,
Thinker’s Toolkit.
��0 Stephen Marrin, email to the author, December �, �00�.

The author
has heard similar complaints from analysts at NSA and DIA.
�� Linda Elder, Alec Fisher, Diane Halpern, Gerald Nosich,
Richard Paul,
and others develop and publish the materials for – as well as
teach – such courses.
– �� –
instructional models offer complementary means of expanding
and
enhancing analysts’ skills. For example, in an area studies class
on
Russia, students might be asked to evaluate and comment in
depth
on what different sources say about the influence of organized
crime
on the national government. Alternatively, critical thinking can
be
injected into the course as part of a problem-solving
curriculum.
Here students focus on how they think as they apply different
strategies
to each assignment, and then, ideally, transfer those enhanced
skills
to their day-to-day analysis. The use of realistic case studies
makes
classroom-acquired knowledge actionable. Self-reflective and
group-
reflective analyses of the process of reasoning keep the classes
focused
toward the working environment of analysts.

Such transformations involve behavior modification and as
such take time. Participants engaged in such instruction cannot
be
expected to become critical thinkers in a one-day or even a
week-long
course. The instructors of the NSA course attempt over a
period
of �0 weeks to transform their students into critical thinkers.
Some
other Community efforts also provide instruction over an
extended
period. Even so, students leaving the course are still novices in
this practice and can slip back to their old methods of
reasoning.
Becoming critical thinkers requires a change in behavior that
extends
long past the end of the formal instruction. There is no
substitute
for continued practice.
Finally, it serves critically thinking analysts poorly if their
management and corporations are indisposed toward the
application
of the skills. It is well known that “engendering the desire to
use
[critical thinking] as a favored means of problem solving and
decision
making prepares the ground for teaching and learning the
[critical
thinking] skills.”
��
Such encouragement can occur in the classroom
but its effectiveness is limited unless the corporation
encourages and

welcomes strategies to employ critical thinking in the
workplace.
However, there are few metrics for determining the
effectiveness of these materials
and courses and so their value remains undetermined. The fact
that the books
developed for, and used in, such courses remain in print (in
successive editions) is
an indicator that at least this approach is popular, if not
effective.
�� Facione, “Disposition.”
– �� –
The best place for such encouragement is from senior and
midlevel
management. In order for this to occur, they too must be
educated
in the methods of, and reasons for, critical thinking.
Evaluating Teaching Models
Removing analysts from their work for education and training
disrupts their primary mission of intelligence production.
Spreading
instruction over time provides a reasonable answer to this
dilemma.
Since improving critical thinking requires a high level of
experiential,
hands-on practice, an extended course offers students time to
practice
and apply what they are learning. This is the model used for the

NSA
course on critical thinking. Feedback from students indicates
that
once-a-week instruction works best. Mission is minimally
disrupted
and there is time to study and practice what is being taught.
However,
the long duration limits students’ work-related travel and leave.
Experiential learning also requires that such classes be
relatively
small and that instructors be proficient. Without a large
teaching
staff, training a large workforce takes years. Since senior
managers
– and their strategic visions – change often, a long-term
corporate
commitment is crucial to success.
��
Such a long-term commitment exists at the Sherman Kent
School
for Intelligence Analysis. The school itself grew out of a
month-long
course on analytic tradecraft developed by former Deputy
Director
for Intelligence Douglas J. MacEachin.
��
Beginning in �99�, the
course was delivered to the entire analytic workforce. The
school
itself was established in �000 and continues to evolve as both

a
training center and a center of best analytic practices.
��
�� Alternately, a large staff can be created, trained, and
assigned to teach
an entire workforce. This was accomplished in a knowledge
domain at NSA in
the �990s. If inducements and cultural change accompanied
such a program, this
approach might present certain advantages even if its costs are
high. An analytic
workforce could be transformed rapidly through such a “boot
camp” approach.
�� Marrin, “Kent School,” ��.
�� Marrin, “Kent School,” �09.
– �9 –
Another means of critical thinking instruction is to create an
interactive, computer-based or web-based course. This has the
benefit of allowing as many analysts to attend as wish to do so.
One
or two instructors working full-time can answer students’
questions.
Exercises allow student self-assessment. However, this is
presently
an unsatisfactory model for critical thinking instruction.
Learning
to think critically requires Socratic interaction and the
impersonal

nature of web-based instruction discourages the practice even
for
the population of dedicated solitary learners. In the future,
multi-
player simulations and other games could be used to reinforce
the
lessons learned. However, these are not yet available.
Although computer- and web-based instruction can be
accomplished at the analyst’s desk, there are other reasons why
this may not be a good idea. Analysts who remain in their
offices
are subject to interruptions. Further, course work is often
relegated
to the end of the workday, when analysts are tired and learning
is impaired. Little learning occurs when taking a class is a
chore.
The current limitations of both classroom and computer- or
web-
based education suggest that experimentation with new means
of
instruction for critical thinking is needed.
Encouraging Analysts to Think Critically
Incorporating critical thinking into both orientation and basic
analyst education and training is one way to help newly hired
analysts
develop their skills. Subsequently placing those analysts in
offices
where critical thinking is practiced is another way to encourage
an analytic culture that fosters thinking critically. Through
direct
exposure to successful, experienced analysts, junior analysts’
doubts
about the employment of critical thinking techniques can be

overcome.
“Skills pay” can be an inducement for analysts to learn and then
employ critical thinking. Rewarding the acquisition,
maintenance,
and use of other special skills is common in government and
industry.
For example, linguists at NSA earn significant bonuses for
achieving
specific levels of foreign-language proficiency – an effective
way
– �0 –
to maintain critical language competencies within one
intelligence
agency’s workforce. Given the high-level concern for the
health of
the intelligence analysis process, direct monetary incentives
should
be available for analysts to master new tools and demonstrate
constructive analytical behaviors.
On the other hand, if an intelligence enterprise fails to
recognize
and reward the acquisition and application of analytic skills
such
as critical thinking, it sends a very clear message: the enterprise
does not value those skills. Faced with such a message and a
lack
of inducements to excel, the best and brightest analysts may
leave,
especially from the more junior ranks – and midlevel employees
with families and mortgages may “retire in place.” Either

outcome
hurts mission-critical functions, and can be avoided. As a
previous
Director of NSA, Air Force Lieutenant General Kenneth A.
Minihan,
noted, “If we don’t win the talent war, it doesn’t matter what
we
invest in the infrastructure.”
��
Recognizing and rewarding critical
thinkers is one way to win that talent war – with the assumption
that really good analysts are more likely to remain active within
their
intelligence agency workforces.
If an analyst adopts a congeries of skills that contributes to
the mastery of critical thinking, and is compensated monetarily,
that mastery needs to be certified. If a curriculum that drives
the
acquisition of those skills is in place, then an assessment of
those skills
can be administered in-house. Specific tests exist for the
assessment of
critical thinking, such as the “Thinking Skills Assessment
Test.”
��
Persuading to Improve Analysis
Teaching critical thinking is but a first step toward improving
analysis. Because analysts and managers have different needs
and

�� LtGen Kenneth A. Minihan, USAF, Director, NSA, in
Robert K.
Ackerman, “Security Agency Transitions from Backer to
Participant,” Signal ��,
no. � (October �99�), ��.
167 For more information, see
http://tsa.ucles.org.uk/index.html, last
accessed �� March �00�.
– �� –
time constraints, multiple versions of a course are needed to
meet the
needs of each group. Analysts and first-line supervisors may
take an
entire course, midlevel managers an abbreviated course, and
senior
managers an overview. This strategy already is employed at the
CIA’s Sherman Kent School where new employees spend ��
weeks
learning analysis skills, techniques, and methods while
supervisors,
depending on their seniority, spend three days or one day
becoming
familiar with these same skills and methods. The analysts also
have
opportunities to apply new skills immediately as they perform
four-
week “internships” in various CIA offices.
Other CIA analysts, already in place, do not have the same
opportunities to receive this training. Thus a two-tiered analyst

population with more skilled junior analysts and more
knowledgeable
senior analysts is being created. Ideally, each will transfer
skills and
knowledge to the others. However, an alternate possibility is
that it
will generate distrust and animosity between the two groups.
Older,
more experienced analysts may resent the opportunities given to
their newer counterparts. This issue is not trivial. Offering the
wrong curriculum to the wrong groups of analysts and
managers
can destroy its effectiveness. Managers in the CIA and
elsewhere can
influence how the new analytic methods being taught will be
adopted,
as agencies respond to new legislative and executive mandates.
Will analysts embrace critical thinking as a means to improve
their
analysis? There are numerous observations of analysts’
reluctance
to adopt new paradigms.
��
This is true even when analysts are
confronted with the fact that their conclusions were wrong. In
referring to her work on the issue of Iraqi WMD, one CIA
analyst
told Senate investigators, “their ‘bottom line’ judgments would
have
remained the same.”
��9
Rob Johnston also found a similar reluctance

to change opinions among the analysts he studied. He noted
that
�� Stephen Marrin, “Homeland Security and the Analysis of
Foreign
Intelligence,” Markle Foundation Task Force on National
Security in the
Information Age, �� July �00�, URL:
<www.markletaskforce.org /documents/
marrin_071502.pdf>, last accessed December 9, 2003, 9. Cited
hereafter as
Marrin, “Homeland Security.”
��9 SSCI, Iraq, �99.
– �� –
although “analysts can change an opinion based on new
information
or by revisiting old information with a new hypothesis,” they
perceive
a loss of trust and respect and a subsequent loss of “social
capital,
or power, within [their] group.”
��0
Yet, a case can be made that analysts, and especially
experienced
analysts, will benefit the most from enhanced critical thinking
skills
training. These are the analysts who are in positions of
technical

leadership, who work the most difficult aspects of complex
targets.
There may be significant consequences if they fail to notice
and
make sense of an issue. On the other hand, the fact that many
of
these senior personnel will soon be eligible for retirement raises
an
important question: Does the corporation get added value from
teaching analysts who will soon retire to think more critically in
their
analysis if they are predisposed not to do so? Maybe so.
Heuer,
while the head of CIA’s Methods and Forecasting Division
found
that analysts, once persuaded to use new analytic methods,
found
them “interesting and well worth doing.”
��
Other intelligence analysts have adopted new analytic methods
that add value to their analyses. In one example, NSA personnel
involved in research and development adopted a means of
matching
target characteristics and their vulnerabilities with exploitation
capabilities and their costs.
��
Such analyses helped ensure that
appropriate resources were dedicated to collection and that such
collection was better tailored to production analysts’ needs.
However,
some research analysts initially refused to employ the model,

claiming
it took too much time even as it reduced the volumes of
information
��0 Johnston, Analytic Culture, ��.
�� Richards J. Heuer, Jr., Adapting Academic Methods and
Models to Governmental
Needs: The CIA Experience (Carlisle, PA: Strategic Studies
Institute, U.S. Army War
College, �� July �9��), �. Referenced in Marrin,
“Homeland Security,” 9.
172 In this context, a “target” refers to an entity –
geographical, individual,
or topical – in which the Intelligence Community has an
interest.
– �� –
they were required to examine.
��
Adoption of the analytic paradigm
required appropriate persuasion.
One interesting means of self-persuasion may be analysts’
frustration with the really hard problem. This frustration may
lead
analysts to try something new.
��
Such frustration occurs more often

among experienced analysts. Years of attempting to make
sense
of overwhelming masses of information with inadequate
analytic
paradigms and technologies leaves some experienced analysts
willing
to grasp at anything that will improve how they work. On the
other hand, newly hired analysts who have not yet experienced
the
frustration of inadequate paradigms for analysis may be
resistant
to adopting rigorous analytic paradigms such as that afforded by
critical thinking.
��
�� Multiple midlevel NSA analysts, interviews with the
author, �99�–
�00�.
�� LT Robert D. Folker, USAF, email to the author, 9
December �00�.
Cited hereafter as Folker, email, 9 December �00�.
�� Folker, email, 9 December �00�.
– �� –
How Does criTicAl THiNkiNg TrANsforM?
Transforming Intelligence Corporations
In order for an institution to change, all affected personnel,
from

the lowest to the highest, must first recognize that change is
needed
and is advantageous – both corporately and personally. As
retired
World Bank executive Stephen Denning argues, this is difficult
to
accomplish. Logically sound arguments do not sway
employees.
Instead, employees remain convinced that what they are doing is
satisfactory.
��
Further, outsiders who attempt to induce change face
opposition because employees presume that external consultants
are
arrogant in suggesting that things are not right, and that change
is
needed.
One way to effect change is through a “springboard story.”
This
approach contrasts with past conventional – and largely
unsuccessful
– transformation efforts that relied on fixing the systems
involved
and were characterized by an overabundance of buzzwords:
Enhance quality. Streamline procedures. Reform and flatten
the organizational structure. Analyze things in terms of
grids and charts. Develop plans in which individuals are
programmed to operate like so many obedient computers.
Hone our interpersonal mechanics and build skill inventories.
Bring to our difficulties a fix-it attitude, as though our
past errors can be easily corrected with straightforward
explanations.

��
�� For a discussion of Denning’s philosophy about change,
see Jeff de
Cagna, “Making Change Happen: Steve Denning Tells the Story
of Storytelling,”
Information Outlook, January �00�, ��–��, and Stephen
Denning, The Springboard:
How Storytelling Ignites Action in Knowledge-Era
Organizations (Boston, MA: Butterworth-
Heinemann, �00�). Cited hereafter as Denning, Springboard.
�� Denning, Springboard, xvii. In 23 years of Intelligence
Community work,
the author heard all of these phrases as slogans for various
programs designed to
“fix” analysis or the corporations involved. These associated
corporate efforts to
bring these goals about often had the opposite effect. In the
process of “flattening
– �� –
As Denning points out, these strategies fail to account for the
messy and chaotic reality in which real organizations live and
work,
especially intelligence agencies which have the charter to
engage
honestly with “the other,” or mission-related targets, outside of
the
bureaucracy in which they serve.
��

The springboard story helps employees at all levels envision
what is needed for the proposed transformation. Denning
asserts
that it “invites them to see analogies from their own
backgrounds,
their own contexts, their own fields of expertise.”
��9
He cautions
that transformational stories are not a panacea – there are
situations
and circumstances in which they are not effective, such as when
the change being proposed is a bad idea.
��0
The key is finding the
appropriate stories within the corporate culture.
One way to find these stories is to examine the best practices
of intelligence analysis successes where critical thinking
played a
role. Just recounting a success is not enough. Listeners need to
be able to identify and empathize with the scenario and its
actors.
The effective story is “about people who have lived [a]
knowledge-
sharing idea and how things happen in a real-life situation.”
��
For
example, conducting a trial or experiment on critical thinking
within

the intelligence analysis process, if it leads to certifiable,
actionable
success, does effectively persuade.
Learning from Early Adopters
Once a sustained number of early adopters openly apply
systematic, critical thinking to hard analytic problems, the stage
is
set for a “tipping point” in the spread of structured methods
across
the organization” at one intelligence agency, more hierarchical
managerial layers
were actually created!
�� Denning, Springboard, xvii.
��9 Denning, Springboard, xix.
��0 Denning, Springboard, xxi.
�� Denning, Springboard, ��.
– �� –
analytic populations.
��
Stanley Feder recounts how a tipping point
began to apply to a political analysis method at the CIA in the
�9�0s
and �9�0s. The method involved two estimative tools known as
Factions and Policon that were used by the “Intelligence

Directorate
and the National Intelligence Council’s Analysis Group to
analyze
scores of policy and instability issues in over �0 countries.”
��
The
reasons for the adoption of these tools remain the same:
“Forecasts
and analyses…have proved to be significantly more precise and
detailed than traditional analyses.”
��
Writing about the method in
1987, Feder predicted that its use would continue to expand.
��
The
method is still in use �9 years after Feder’s article was
published.
However, expanded use failed – perhaps because the tool was
on a
computer platform that ceased to be supported by the Agency.
The
recent transfer of the tool to a new suite of programs
corresponds
with observations that its use is once again expanding.
��
Non-intelligence-related transformational stories can be applied
in the Intelligence Community to facilitate the spread of new
ways

�� Malcolm Gladwell, The Tipping Point: How Little Things
Can Make a Big
Difference (Boston, MA: Little Brown and Company, �000),
��–�9. Gladwell shows
how “social epidemics” can infect a variety of domains.
�� Stanley Feder, “Factions and Policon: New Ways to
Analyze Politics,”
in H. Bradford Westerfield, ed., Inside CIA’s Private World:
Declassified Articles from the
Agency’s Internal Journal, 1955–1992 (New Haven, CT: Yale
University Press, �99�),
��. Cited hereafter as Feder, “Factions and Policon.”
�� Feder, “Factions and Policon,” �9�.
�� Feder, “Factions and Policon,” �9�. Feder was wrong
about its sustained
popular growth.
186 This story is not an isolated instance. In the author’s
experience, initial
implementation and popularization are often followed by a
gradually reduced
user-base. Certain organizations find the tools useful and they
tend to continue
to use them even though any underlying technology may be
obsolete. In the
case of Factions and Policon, the tools were maintained on an
aged Macintosh
computer. They were updated in �00� as part of the work of
an intelligence
research firm in New Mexico. The Factions tool was rewritten
and included in
Landscape Decision®, a suite of modeling and simulation tools
developed under

a research contract with the Department of Defense’s Advanced
Research and
Development Activity (ARDA). The updated technology was
reinserted into the
tasking organization. Other technology transfers are pending.
– �� –
of thinking. For example, two video commercial
advertisements
from the Electronic Data System (EDS) company resonate with
the
challenge of rebuilding intelligence analysis in the face of
skeptics,
and what it takes to be an intelligence analyst.
��
The two related
commercials depict farcical situations based on two well-known
clichés: “You’ve got to build it before you can fly it!” and “It’s
like
herding cats; it can’t be done!” In the former, a group of
mechanics
is building an airplane in mid-flight; while in the other, a group
of cowboys has accomplished the impossible – they are herding
cats across the Great Plains. EDS’ depiction of its
revolutionary
design capabilities bears a message about corporate rebuilding
and
individual transformation. The message encourages managers
and
analysts to compare themselves and their situation to the people
tasked with building an airplane at “�0,000 feet,” or with

herding
cats across a river.
Intelligence analysts and managers who have seen the videos
have had little trouble drawing analogies with the corporate and
personal transformations needed in the Intelligence Community.
That the situations are humorous only adds to a viewer’s buy-
in – they are “hooked” before they realize it. Developing other
stories that specifically apply to corporate transformation
within
the Intelligence Community and its various elements is a logical
next step in encouraging analysts and managers to use and
support
critical thinking in analysis. If these stories can be drawn from
recent
analytic successes, their value can only increase.
The Costs and Benefits of Thinking Critically
Are intelligence failures inevitable if analysts lack critical
thinking skills? Yes. However, critical thinking is not a
panacea.
Intelligence failures still can be expected. If no measure “will
guarantee that accurate conclusions will be drawn from the
incomplete and ambiguous information that intelligence analysts
�� See Building Airplanes in Flight, television
advertisement produced for
EDS by Fallon-McElligott, November �000, videocassette; and
Herding Cats,
television advertisement produced for EDS by Fallon-
McElligott, January �000,
videocassette.

– �� –
typically work with…[and] occasional failures must be
expected,”
why should the Intelligence Community invest in teaching
its analytic workforce to think critically?
��
A brief review of
the causes of failure clarifies the need for such an investment.
Intelligence failures occur in both systematic and functional
domains. Systematic intelligence failures occur when producers
fail to notice phenomena and the consumers of intelligence fail
to
heed warnings or even to notice that they are being warned. It
may
be that both consumers and producers are focused on other
issues
at the time. Alternatively, intelligence-based warning is not
taken
seriously at high official levels. In cases where collaboration
among
intelligence agencies might be critical, key evidence may not be
recognized and shared.
All of these failings are illustrated in the “intelligence failure”
to warn of the attacks of �� September �00� on the World
Trade
Center in New York and the Pentagon near Washington, DC. In
the first place, the policy issue that was at the forefront –
according
to press at the time – was a national missile defense system, not
the
likelihood of a terrorist attack on the United States. Richard

Clarke,
the Clinton and then Bush administrations’ terrorism “czar”
observes
that Bush administration officials were slow to recognize or
consider
the threats posed by Osama bin Laden.
��9
Evidence passed by an
FBI field agent about students learning to fly large aircraft with
little
regard for landing or takeoffs was not taken seriously at higher
levels
within the Bureau.
�90
The Joint Congressional investigation into the
attacks noted in its report that collaboration was lacking
between
agencies, especially between the law enforcement and strategic
�� Heuer, Psychology, ��.
��9 “Clarke: Bush Didn’t see Terrorism as ‘Urgent,’”
CNN.com, �9 May �00�,
URL: <http://www.cnn.com
/�00�/ALLPOLITICS/0�/��/9��.commission/
index.htm>, last accessed 9 March 2006.
�90 Michael Elliott, “How the U.S. Missed the Clues,”
CNN.com, �0 May
�00�, URL: <
http://archives.cnn.com/�00�/ALLPOLITICS/0�/�0/time.
clues/index.html>, last accessed 9 March 2006.

– �9 –
intelligence agencies.
�9�
This was both a cultural and a physical
phenomenon. Airport security breaches during the summer of
�00�
may have been related, but based on assessments of the
catastrophe,
were not connected to the events.
�9�
The same kinds of failings were identified by the Senate
Select Committee on Intelligence in its assessment of the
National
Intelligence Estimate on Iraqi WMD. As has already been
observed,
analysts and their managers were focused on the results and not
the
process.
�9�
Critical thinking can mitigate some common causes of failure
and provide means by which they can be avoided in the future.
Specifically, an intelligence process based on critical thinking
offsets
the following failures:
Analysts are Wrong. It is unrealistic to expect that analysts
can always be correct. Regardless of the processes they

employ, analysts make errors and fail. Anthropologist Rob
Johnston defines errors as “factual inaccuracies in analysis
resulting from poor or missing data.” Conversely, intelligence
failures are “systemic organizational surprise resulting from
incorrect, missing, discarded, or inadequate hypotheses.”
�9�
Critical thinking mitigates these by providing means to assess
errors in reasoning as they occur and before they become
�9� U.S. Congress, Joint Inquiry into Intelligence Community
Activities Before and
After the Terrorist Attacks of September 11, 2001, Report of
the U.S. Senate Select
Committee on Intelligence and U.S. House Permanent Select
Committee On
Intelligence, Together with Additional Views, Senate Report
No. �0�–��/House
Report. No. �0�–�9�, �0�th Congress, �nd Session,
December 2002, xvi.
�9� The author recalls that media reports of persons hopping
over or
otherwise improperly passing airport security checkpoints in
and around Boston,
New York, and Washington seemed to be higher than usual
during the summer
of �00�.
�9� DNI Negroponte, in the National Intelligence Strategy of
the United States
(October �00�, �) highlights the notion that the Community
appropriately
undertakes study of its internal processes, as well as of the

quality of its products,
by addressing both “enterprise” objectives as well as “mission”
objectives.
�9� Johnston, Analytic Culture, �.
– �0 –
systemic failures. Such a meta-cognitive approach to the
analytic process helps keep it under active review at the
highest levels.
Policymakers Ignore Intelligence.
�9�
A critically thinking
analytic population cannot directly affect what a policymaker
can or will do – neither in fact, can a non-critically thinking
analytic population. What critically thinking analysts can
do, however, is present more effective assessments, perhaps
leading policymakers to question their own assumptions on
the issues. Additionally, thinking critically about how analysts
interact with policymakers can identify ways to restructure
the analysis dissemination process to involve policymakers
more effectively. Such a process might also encourage them
to adopt some of the attributes of critical thinking leading
to (it may be presumed) more effective policy.
�9�
Adversary Denies and Deceives. Critical thinking reduces
the effects of adversarial denial and deception by leading
analysts to consider alternative possibilities, to question biases
and assumptions, to examine systematically the validity of

evidence being considered, and to take seriously anomalies
in the evidence.
Adversary is More Capable. In any adversarial system,
there are winners and there are losers. While analysts can do
everything possible to ensure their work is correct, they rarely
work with all the evidence, and indeed may still be deceived.
In such cases, they may come to wrong conclusions. Critical
�9� There is a “classic” argument as to whether this is or is
not an
“intelligence” failure. In summary, the two sides condense as
follows: On the
one hand, intelligence should have been so persuasively
presented as to compel
the policymaker to pay attention. On the other hand,
intelligence should not be
telling the policymaker what to do. The argument goes all the
way to the roots
of the post-World War II strategic intelligence system currently
in place. Former
CIA analyst Jack Davis summarizes the issue in “The Kent-
Kendall Debate of
�9�9,” Studies in Intelligence ��, no. � (Summer �99�):
��–�0.
�9� Cooper, email to the author, �� March �00�. Cited
hereafter as Cooper,
email, �� March �00�.
– �� –
thinking, however, by providing structure and oversight to
their reasoning, provides an audit trail. In this case, the

means by which the analytic conclusions were reached can
be subsequently reviewed, errors and deceptions revealed,
and steps taken to improve the process so that the failure is
not repeated. Indeed, because of its focus on the process,
critical thinking becomes a powerful tool for evaluating and
enhancing analytical reasoning.
Investment in critical thinking as part of the analysis process
minimizes the likelihood of specific failures. With critical
thinking
essential to so many parts of the intelligence production
process,
enhancing it increases the likelihood of intelligence success.
�9�
With
the cost of failure catastrophically high, the Intelligence
Community
is well advised to improve the likelihood of intelligence
success,
including improving critical thinking. As diplomat Richard
Holbrooke
opined in �00�, “intelligence is…indispensable. And its
greatest
successes are preventative.”
�9�
There are other reasons why systemic intelligence failures
occur.
Jeffrey Cooper considers ten pathologies, summarized in table
�,
that impede successful analysis at both the individual and
corporate
levels. Cooper specifically believes that given the

[emphasis] on the systemic nature of the pathologies that
afflict intelligence analysis, structured analytic methods
�9� One metric for intelligence is decreasing intelligence
failures. In this
case, fewer failures could be considered as “improving
intelligence.” But the
reasons for the prevention of the failures need to be considered.
The reduced
number of failures could be due to a critically thinking
workforce. Or it could be
due to coincidences. Although a pointed discussion of metrics
for intelligence is
beyond the scope of this paper, recent articles explore the issue
of intelligence
metrics. See, for example, David T. Moore, Lisa Krizan, and
Elizabeth J. Moore,
“Evaluating Intelligence: A Competency-Based Model,”
International Journal of
Intelligence and Counterintelligence ��, no. � (Summer
�00�): �0�–��0.
�9� Richard Holbrooke, in Judith Miller, “A Battle of Words
Over War
Intelligence,” New York Times, online edition, �� November
�00�, URL: <www.
nytimes.com/2003/11/22/arts/22INTE.html>, last accessed
November 28,
�00�.
– �� –
become a first line of defense in preventing the networks

of errors from developing – they are like “ripstops” that
keep problems from propagating into wider “error-inducing
systems,” in [Charles] Perrow’s terms.
�99
�99 Jeffrey Cooper, email to the author, �� March �00�.
Cooper refers to
Charles Perrow, Normal Accidents: Living with High-Risk
Technologies (Princeton, NJ:
Princeton University Press, �999).
Cooper’s Analytic
Pathology
Description
Inefficient “Account”
Structure
Work is subdivided into accounts with ownership;
provides basis for accountability; ownership inhibits
sharing, cooperation, and collaboration; encourages
“stovepiping” in collection disciplines.
Evidence-Based
Scientism
Descriptive and explanatory intelligence support
current policy and military operations; less support
for anticipatory intelligence; analysis needs to be
“augmented and leavened with intuition, curiosity,
and a thirst for discovery – all essential elements of
good science.”
Tyranny of Current

Intelligence
Response to customers’ current concerns; support
to current, ongoing military operations; little long-
term research.
Over-Emphasis on
Production
Databases need filling; scheduled reports and assess-
ments must be produced on time; metrics for suc-
cess measure data collected and processed; number
of reports issued used to determine and justify re-
sources allocated.
Over-Reliance on
Previous Judgments
Previous reports presumed to be authoritative; pre-
vious reports combined to form new judgments;
agreed-upon positions retained despite newer con-
tradictory evidence.
Table 6: Cooper’s Analytic Pathologies
Source: Summarized from Cooper, Pathologies, �0–�9.
– �� –
Critical thinking combats this by enhancing the processes by
which decisions are made and by which such processes are
reviewed.
This paper focuses primarily on individual experience, and of
course,

intelligence corporations are made up of individuals. If a
sufficient
number of intelligence professionals are thinking critically, it
is likely
they will transform their corporations, if only through the sheer
attrition of non-critically thinking managers and leaders who
are
Cooper’s Analytic
Pathology
Description
Neglect of
Research
Short–term taskings prevent longer–term research;
reward structure favors current reporting, not lon-
ger term work; stunts development of deep target
expertise.
Neglect of
Anticipatory
Intelligence
“Information Revolution” leaves intelligence com-
peting with journalism and Internet for policymak-
ers’ attention; lack of predictive intelligence against
new emerging threats ensures continuing failures to
warn.
Loss of “Keystone
Species” and
“Intellectual
Middleware”

Periodic reductions in force skew analyst demo-
graphics; “Keystone Species” (journeymen analysts)
lacking; journeymen do bulk of professional mainte-
nance; essential for knowledge retention and trans-
fer to apprentices; maintain “intellectual Middle-
ware” or deep understanding of analytic domains
and processes.
Lack of Analytic Tools
and Methods of
Validation
Available and proposed tools not formally validated
for accuracy and usefulness; focus on results, not
processes, to determine success.
Hindrances of
Security Mindset
Security procedures impede multi–source analysis;
protection of sources and methods more impor-
tant than “effective exploitation and cross–fertiliza-
tion;” artificial and outdated mission distinctions
prevent collaboration.
– �� –
retiring from the work force. It should be observed that an
Intelligence
Community enterprise-wide emphasis on developing critical
thinking
skills at all levels would speed the process.

Validation
A critically thinking Intelligence Community remains essential
for effective intelligence reform. Steven Rieber and Neil
Thomason
advance this argument in their recent article. The authors assert
that “the opinions of experts regarding which methods [of
analysis]
work may be misleading or seriously wrong.”
�00
Unfortunately, as
the authors show, past (and present) efforts at intelligence
reform
rely on expert intuitive judgments. However, “[examples] from
a
wide range of fields show that expert opinion about which
methods
work are often [not only] dead wrong…but also are generally
not
self-correcting.”
�0�
Further support of Rieber’s thesis is seen in devil’s
advocacy – not as theoretically applicable but as practiced.
They
cite Irving Janis who, quoting a Stanford political scientist,
notes
that “instead of stirring up much-needed turbulence among the
members of a policymaking group, [devil’s advocacy] may
create
‘the comforting feeling that they have considered all sides of
the
issue.’”

�0�
To mitigate this and similar analysis-improvement fallacies,
Rieber and Thomason argue that improvements in analysis and
any proposed methods of judgment and decision-making
require
validation through scientific study. They note, for example,
that
research reveals “[a] certain cognitive style, marked by an
open-
mindedness and skepticism toward grand theories, is associated
with
�00 Rieber and Thomason, “Creation of a National Institute
for Analytic
Methods,” ��.
�0� Rieber and Thomason, “Creation of a National Institute
for Analytic
Methods,” ��.
�0� Irving L. Janis, Groupthink: Psychological Studies of
Policy Decisions and
Fiascoes, �nd edition. (Boston, MA: Houghton Mifflin and
Company, 1982), 268.
Referenced in Rieber and Thomason, “Creation of a National
Institute for
Analytic Methods,” ��.
– �� –
substantially better judgments about international affairs.”
�0�

As the
present paper has argued, such attitudes are found in critical
thinkers.
Scientific study of this and other methods will determine when
and
how they are appropriate. Critical thinking also supports Rieber
and
Thomason’s call for a “National Institute for Analytic Methods”
by
providing an overarching structure to champion open-
mindedness
and skepticism in the study of which methods are appropriate
for
intelligence analysis. Only then, they argue, will real
intelligence
reform and improvement occur.
Rieber and Thomason’s proposed institute could determine
which strategies are most effective at disposing analysts and
their
corporations to employ critical thinking. Research evidence
indicates
that simply teaching critical thinking (or for that matter,
structured
analytic) skills is insufficient. People will not adopt the
strategies
unless motivated to do so. As noted here, springboard stories
are one
means of implanting a positive disposition toward critical
thinking.
Other means certainly also exist. A desirable objective of
research
in the proposed National Institute for Analytic Methods would
be
to discover and assess what might motivate analysts most

effectively
toward thinking critically.
�0� Rieber and Thomason, “Creation of a National Institute
for Analytic
Methods,” ��.
– �� –
wHAT oTHer PoiNTs of view exisT?
A recent best-selling book advances the comfortable idea that
conscious reasoning may not confer advantage to the reasoner.
�0�
Recent research suggests that whereas simple choices may
benefit
from conscious thought, complex issues are best left to
unconscious
thought, or “deliberation-without-attention.”
�0�
The explanation
of this finding is that in conscious thinking, people face a
severe
limit on the number of factors that they can effectively
consider
simultaneously and, second, that in conscious thought people
“inflate
the importance of some attributes at the expense of others.”

�0�
The
authors base this finding on four experiments with subjects who
were
asked to indicate their preference for various consumer items.
The
experiments involved differing levels of complexity in terms of
factors
to be taken into consideration. They ultimately suggest that:
[there] is no a priori reason to assume that the deliberation-
without-attention effect does not generalize to other types of
choices – political, managerial or otherwise. In such cases,
it should benefit the individual to think consciously about
simple matters and to delegate thinking about more complex
matters to the unconscious.
�0�
It appears true that the human capacity to weigh evidence
consciously is limited to approximately seven factors.
�0�
But this seven-
�0� Malcolm Gladwell, Blink: The Power of Thinking
Without Thinking (New
York, NY: Little Brown and Company, �00�). Gladwell
argues that thinking does
not require detailed assessment of information. Instead, rapid
cognitive responses
are adequate for decisionmaking.
�0� Ap Dijksterhuis, Martin W. Bos, Loran F. Nordgren, and
Rick B. von

Baren, “On Making the Right Choice: The Deliberation-
Without-Attention
Effect,” Science ��, no. �� (�� February �00�),
�00�. Cited hereafter as
Dijksterhuis, “Deliberation-Without-Attention.”
�0� Dijksterhuis, “Deliberation-Without-Attention,” �00�.
�0� Dijksterhuis, “Deliberation-Without-Attention,” �00�.
�0� George A. Miller. “The Magical Number Seven, Plus or
Minus Two,”
The Psychological Review �� (�9��), ��. The paper is
available online: URL: <
http://psychclassics.yorku.ca/Miller/>, last accessed 14 March
2006.
– �� –
item limit is easily extended: people can hierarchically consider
multiple sets of seven items.
�09
Employing structured methods of
reasoning also extends this capacity. In this context, critical
thinking
is at once both a structuring method and a means of assessing
and
monitoring the processes of selecting and using other
structured
methods. Additionally, “deliberation-without-attention” as
applied to
consumer choices in the study noted above was measuring

preferences.
Preferences change from person to person – different people
conclude
differently about oven mitts and cars – as the variety of these
products
on the market, in kitchens, or on the roads makes clear.
��0
Intelligence analysis is not about preferences; it is about best
answers in ambiguous situations with high-stakes implications
and
consequences. As recent intelligence failures make clear,
without
forced consideration of alternatives, results are biased.
Structured
reasoning methods such as Analysis of Competing Hypotheses
(ACH)
develop these alternatives, allow for multiple factors to be
considered
fairly, and extend analyst capacities to assess complex
situations as
accurately as possible.
Intuition is what both Dijksterhuis and Gladwell consider in
their
respective works. However, exactly what is intuition? Rather
than just
appearing from nowhere, intuition is “almost always informed
by
experience and hard knowledge won by reasoning things out.”
��
As

Michael LeGault notes,
good decisions [are] a nuanced and interwoven mental
process involving bits of emotion, observation, intuition,
and critical thinking....The essential background to all this is
a solid base of knowledge...The broader the base, the more
likely all the parts will fit together.
��
�09 William Reynolds, conversation with the author, ��
March �00�.
��0 Oven mitts and cars are two of the categories for which
Dijksterhuis
and his colleagues tested.
�� LeGault, Think, ��.
�� LeGault, Think, ��.
– �� –
Uninformed “intuitive thinking” contributes to intelligence
failures
because it fails to reflect on presuppositions. Such unexamined
biases
and mindsets contributed to the Cuban missile intelligence
failure
described above. Heuer’s outline of the specific biases and
mindsets
that impede effective analysis is reinforced by the findings of
others.

��
For example, Merom, writing about the Cuban crisis, concludes
that intelligence failures occur due to a “lack of commitment to
fundamental principles of investigation and analysis.”
��
Although
intuition or “thin slicing” may be appropriate in some domains,
in intelligence analysis it appears to be associated with
cognitive
impediments that cause costly intelligence failures.
��
However, as Rieber and Thomason note, testing needs to be
conducted to determine which methods do work best in which
situations in intelligence analysis.
��
Their work ought to go further,
developing and modifying methods to overcome the very
limitations
they identify. For example, investigators have found that ACH
suffers
from confirmation bias.
��
The present author notes that this may
be because non-confirmatory reasoning is so difficult for people
to
do.

��
213 See for example, Scott Plous, The Psychology of
Judgment and Decision-making
(New York, NY: McGraw Hill, Inc., 1993); Thomas Gilovich,
Dale Griffin, and
Daniel Kahneman, Eds. Heuristics and Biases: The Psychology
of Intuitive Judgment
(Cambridge, UK: Cambridge University Press, �00�); and
Daniel Kahneman,
Paul Slovic, and Amos Tversky, eds. Judgment under
Uncertainty: Heuristics and Biases
(Cambridge, UK: Cambridge University Press, �9��).
�� Merom, “Estimate,” �9.
�� Gladwell introduces the notion of “thin slicing” as a
reasoning strategy
in Blink.
�� Rieber and Thomason, “Creation of a National Institute
for Analytic
Methods,” ��.
�� Brant A. Cheikes, Mark J. Brown, Paul E. Lehner, and
Leonard Adelman,
Confirmation Bias in Complex Analyses, Mitre Technical
Report MTR 0�B00000��
(Bedford, MA: Center for Integrated Intelligence Systems,
October �00�).
�� The author teaches Analysis of Competing Hypotheses
to both new
and experienced analysts at the National Security Agency. He
notes that holding

a non-confirmatory attitude – which ACH, as developed by
Heuer, requires – is
– �9 –
Finally, it may be a fallacy that conscious and unconscious
reasoning are truly separate. Instead, Morgan Jones (citing
work by
Richard Restak) asserts that “the unconscious has a governing
role in
much that we consciously think and do.”
��9
This means that analysts
commit a variety of “analytic sins,” including focusing on the
initially
favored solution – which is also often the first satisfactory
solution
considered.
��0
Part of the problem seems to arise from confusion
about “‘discussing/thinking hard’ about a problem and
‘analyzing’
it, when the two activities are not at all the same.”
��
Ultimately,
the unconscious or instinctive approach to reasoning seems to
“remain closed to alternatives.”

��
This is an unsatisfactory model
for intelligence analysis where alternatives must be considered.
extremely difficult for people to do. Since the method has
value, a means of
mitigating this difficulty is worth developing.
��9 Jones, Thinker’s Toolkit, �0. Jones bases his assertions
and argument
on the work of Richard Restak, The Brain has a Mind of Its
Own (New York, NY:
Harmony Books, �99�).
��0 Jones, Thinker’s Toolkit, ��.
�� Jones, Thinker’s Toolkit, ��.
�� Jones, Thinker’s Toolkit, ��.
– 90 –
wHAT Does THe fuTure HolD?
Looking ahead, new paradigms for analysis become necessary
in light of the many changes occurring across the globe.
Rapidly
emerging twenty-first century issues and challenges stress both
infrastructures and sensemaking enterprises. The changes
needed
to maintain an edge against adversaries becomes clear as one
considers what it is that analysts do, and how and when, as table
�

illustrates.
Gregory Treverton’s observations about the evolution of
intelligence sensemaking and the organizations that comprise
the
national intelligence enterprise suggest that current “reform”
efforts
are merely the first steps in a much lager – and fundamental –
transformation. A careful, considered examination of what
intelligence
must accomplish and how this can be best achieved may ensure
that
the goals Treverton identifies as necessary for the future are
met – and
met sooner rather than later. Here again critical thinking has
much
to offer. By encouraging discussion of the alternatives and
their
advantages and disadvantages, and by examining analogous
cases,
critical thinkers aid the transformation. Questioning of
assumptions
about the roles of intelligence, its activity, and organization
will reveal
which long-held beliefs need retirement.
Further, technology efforts in support of analysis, such as those
being developed through the efforts of the ARDA – and other –
advanced technology efforts presume that analysts think
critically.
For instance, by focusing “analytic attention on the most critical
information found within massive data,”
��
some research projects

seek to uncover otherwise unknown information that indicates
the potential for strategic surprise.
��
These projects seek to build
�� Critical information is that which is essential for analysts
to make sense
of what their targets are going to do. Often it is buried in
massive amounts of
“noise.”
�� Advanced Research and Development Activity, Broad
Agency Announcement
for the Novel Intelligence from Massive Data (NIMD) R&D
Program, NMA�0�–0�–BAA–
000�, � JUNE �00�, �.
– 9� –
partnerships between analysts and technology applications.
��
In one
case, a research firm has developed technology that enables
analysts
to generate and validate novel hypotheses, distill complex
systems into
digestible elements, create effective narratives, and inspire
creative
thinking.
��
Other examples are illustrated in the adjacent text box.

Follow-on research efforts such as the Collaboration and
Analyst/
System Effectiveness (CASE) effort will develop these efforts
further.
��
The intelligence workforce these ARDA-supported efforts
presuppose
is being hired and trained now.
The future also holds opportunities for further developing
models of reasoning in intelligence. Exactly how analogy and
critical thinking interact, the roles of creative thinking, the
roles
of intuition, and specific strategies for bias mitigation are all
areas
warranting further research. Formal validation of how critical
thinking improves intelligence reasoning is another research
topic.
Rieber and Thomason’s proposed National Institute of Analytic
Methods is an obvious venue for such research. Additionally,
the
ARDA advanced questioning and answering research program,
AQUAINT, as well as other ARDA programs dealing with
predictive
analysis, prior and tacit knowledge, and hypothesis generation
offer
other domains in which such research could occur.
��
Such research
is of both intrinsic value and necessary for developing new

means
225 The five areas on which the NIMD effort focused were
modeling analysts
and how they work; analysts’ prior and tacit knowledge;
creating, representing,
and tracking multiple scenarios, hypotheses, and/or strategies;
massive data
management and analysis; and the means by which analysts
interact with
technology. The author represented his agency in evaluating
and mentoring the
efforts of the researchers.
226 The author is a technical mentor of this firm’s work.
�� For more information, see “Collaboration and
Analyst/System
Effectiveness (CASE),” AFRL Directorate: IFED BAA 0�–0�–
IFKA, �� October
2005, URL: <http://www.rl.af.mil/div/IFK/baa/>, last accessed
11 March
�00�.
�� Advanced Research and Development Activity (ARDA),
“Preliminary
Solicitation Information for Advanced QUestion & Answering
for INTelligence
(AQUAINT),” URL: <http://www.digitalgovernment.org
/library/library/pdf/
preliminary_information.pdf>, last accessed 26 March 2006.
– 9� –

Issue 1970s and 1980s 1990s and 2000s Future
Focus
of Analysis
Continuing large, well-defined issues
and adversaries
Emergence of complex, rapidly
shifting issues and adversaries
Complex, rapidly shifting issues and
adversaries and large, well-defined
issues and adversaries
Space for longer-term thinking Bias toward current intelligence
Both immediate question–answering
and deeper analysis
Organization
and Workflow
Large, centrally organized and managed
Large, centrally organized and
managed
Tailored to rapidly adapt to shifting
foci
Hierarchical
Still hierarchical, though problem-
oriented “centers” added
Flat, problem-centric networks
Institutional and operational memory

mostly in analysts heads
Technology helps to notice what
analysts are watching and asking
Time pressure persistent but low
intensity (mostly)
Time pressure drives toward
premature closure
Technology allows memory even of
hypotheses, data rejected by analysts
Sources
and Methods
Dominated by secret sources
Broader range of sources, but secrets
still primary
Draws on a wide variety of sources,
open and secret
Analysts are separated from collectors Analysts are also their
own collectors Analysts are their own collectors
Analysts mostly passive recipients of
information
Limited searching of available data
Much more aggressive searching and
reaching for data…both classified and

unclassified
Analysis focuses on previous patterns
Same, though growing interest in
new methods and tools for shaping,
remembering, and examining
hypotheses
Formative pattern recognition and
data mining capabilities searches for
out of the ordinary
Analysts operate on basis of own
experience and biases
Limited use of formal method and
technology
Wide use of method and technology
– from aggregating expert views
to searching, data mining, pattern
recognition
Analysts’
Characteristics
Many analysts are deep specialists
Many, perhaps most, analysts are
generalists
Mix of generalists and deep specialists,
both technical and political
Analysts mostly work alone or in small
groups

Analysts mostly work alone or in
small groups
Analysts work in larger virtual
networks
Key analytic choices with analysts Key analytic choices with
analysts Key analytic choices with analysts
Source: Developed by Gregory F. Treverton with input from
the author. Email exchanges 17–18 May 2006.
– 9� –
Issue 1970s and 1980s 1990s and 2000s Future
Focus
of Analysis
Continuing large, well-defined issues
and adversaries
Emergence of complex, rapidly
shifting issues and adversaries
Complex, rapidly shifting issues and
adversaries and large, well-defined
issues and adversaries
Space for longer-term thinking Bias toward current intelligence
Both immediate question–answering

and deeper analysis
Organization
and Workflow
Large, centrally organized and managed
Large, centrally organized and
managed
Tailored to rapidly adapt to shifting
foci
Hierarchical
Still hierarchical, though problem-
oriented “centers” added
Flat, problem-centric networks
Technology helps to notice what
analysts are watching and asking
Time pressure persistent but low
intensity (mostly)
Time pressure drives toward
premature closure
Technology allows memory even of
hypotheses, data rejected by analysts

Sources
and Methods
Dominated by secret sources
Broader range of sources, but secrets
still primary
Draws on a wide variety of sources,
open and secret
Analysts are separated from collectors Analysts are also their
own collectors Analysts are their own collectors
Analysts mostly passive recipients of
information
Limited searching of available data
Much more aggressive searching and
reaching for data…both classified and
unclassified
Analysis focuses on previous patterns
Same, though growing interest in
new methods and tools for shaping,
remembering, and examining
hypotheses
Formative pattern recognition and
data mining capabilities searches for
out of the ordinary
Analysts operate on basis of own
experience and biases
Limited use of formal method and

technology
Wide use of method and technology
– from aggregating expert views
to searching, data mining, pattern
recognition
Analysts’
Characteristics
Many analysts are deep specialists
Many, perhaps most, analysts are
generalists
Mix of generalists and deep specialists,
both technical and political
Analysts mostly work alone or in small
groups
Analysts mostly work alone or in
small groups
Analysts work in larger virtual
networks
Key analytic choices with analysts Key analytic choices with
analysts Key analytic choices with analysts
Source: Developed by Gregory F. Treverton with input from
the author. Email exchanges 17–18 May 2006.

– 9� –
of improving intelligence analysis. Knowing the details of
how
analysts reason and how they might reason more effectively
guides
managers to understand where educational and training efforts
can
be most valuable.
TECHNOLOGICAL ADVANCES IN CRITICAL THINKING
One ARDA research team has created a technological system
employing an
automated critical thinking-based model that provides an
overarching con-
trol structure for analytic reasoning. It includes both model
generation (for
counter–examples) and more traditional argument building. The
system’s
creators assert – based on preliminary testing – that analysts
employing the
system are faster and more accurate problem solvers.
Suppose an analyst develops an argument in the course of using
this system,
that confirms that Saddam Hussein has WMD and an argument
that con-
firms that Saddam does not have WMD. Working both sides of
this issue,
this analyst will make inferences from evidence and link them
with other
evidence-derived inferences. She will not imagine a situation in
which all her
evidence is true, but her hypothesis is not. For example, the
evidence might

include communications from which she infers that those
communicating
– including Saddam’s scientists – believe they have WMD.
Using the system,
she also would be led to consider that the scientists might have
wanted to
deceive Saddam into believing that they were further along than
they were.
The generation of this alternative model can only arise through
a mechanism
that conducts the analyst away from the comfortable, inductive,
confirmation
path associated with the initial hypothesis.
– 9� –
coNclusioN
Never before in our peacetime history have the stakes of
foreign policy been higher.
—Sherman Kent, �9�9
This paper posed the intriguing question: “How can intelligence
analysts be ‘really good’”? Critical thinking, if conceived and
employed by intelligence analysts as suggested here, appears
capable
of leading analysts to adopt personal habits of thought
appropriate
to the resolution of hard intelligence problems. The idea that
intelligence analysts are expected to bring to the table a
capability to
draw reasoned and actionable conclusions from scant and
conflicting

information distinguishes their charge from that of their
academic
brethren. Thus, intelligence analysts in government or other
applied
work environments may deserve the lavish budgets and
technological
capabilities they often enjoy.
To earn their high level of resource support, analysts can take
advantage of a capability to redirect unique intelligence
collection
capabilities. In doing so, they are perfectly positioned to apply
critical
thinking methods to the hard problems. They can, for example,
order up special collection against targets that exhibit even
fleeting
evidence that convincingly disconfirms one or more alternative
hypotheses about an impending threat. This is the nub of Ben-
Israel’s argument for a “logical” approach to intelligence
analysis
for national security, and is an approach no doubt under-used by
those who do not systematically think about threats in the
manner
developed in this paper.
Although most of the specific threats have changed since
Sherman
Kent first wrote in 1949, his epigraph remains as true today as
it
was then: the survival and prosperity of the United States
remain at
stake; they depend on effective, informed foreign policy.
��9
Critical

thinking brings with it an indispensable capability to inform a
rational
��9 Kent, Strategic Intelligence, ix.
– 9� –
foreign policy. “Really good analysts” are those who think
critically
about the issues they work and the problems they solve; they
bring
structured thought to the foreign policy realm.
– 97 –
Appendix
nSA’S CritiCAl thinking And StruCtured
AnAlySiS ClASS SyllAbuS230
Background
Twenty-first Century intelligence issues involve uncertainty,
mysteries, and risk. This differs from the 20th Century
paradigms
of security, secrets, and prevention. Analysis of current
complex
issues requires of its practitioners novel approaches including a
productively imaginative process of inquiry. Questions an
analyst
asks not only serve as devices for attracting existing evidence,
but

also as devices for generating new evidence not presently
considered.
In this way, analysts more thoroughly examine complex issues
and,
aided by technology, are more likely to create novel intelligence
and
prevent strategic surprise.
However, such reasoning is at odds with how people – all
people, including intelligence analysts – naturally think.
Instead,
people seek to confirm the first answer to a problem they
discover,
selectively using evidence to support that position even when
there
is compelling evidence that an alternative hypothesis may
actually be the
correct one. That people routinely fall prey to such poor
thinking is
well documented. Indeed, most commercial advertisers strive to
take advantage of this. So do adversaries. One element of
most
intelligence failures includes poor thinking on the part of
analysts—
poor thinking of which adversaries have taken advantage. So
how
can analysts avoid such thinking?
One solution is to teach intelligence analysts to think critically.
230 This syllabus has been developed and refined by the author
through
several years of teaching critical thinking at the National
Security Agency’s
National Cryptologic School. An early version of the course
bore similarity to

one developed by (then) MSGT Robert D. Folker, while a
student at the (now)
National Defense Intelligence College. Folker’s course focused
on the methods
of analysis, not on the overarching critical thinking. His
course (as written) is
not taught at the college.
– 98 –
Critical thinking provides structure to the reasoning process that
identifies for analysts where they are likely to go astray. It
offers a
means for self-reflective reasoning that leads to improved
thinking.
If such thinking is aided by structured analytic techniques, then
analysts will (and do) improve how they resolve issues with
which
they are confronted. The quality of their work improves.
This critical thinking and analytic problem-solving course
offers
participants a chance to learn a paradigm for critical thinking
and
critically explore 14 different structured methods of analysis.
Texts
by critical thinking experts Richard Paul and Linda Elder, and
structured analysis experts Morgan Jones and Richards Heuer,
as
well as materials developed by the instructor, teach the concepts
and
techniques. Classroom problems as well as operational examples
(introduced and developed by the students from their own work)
reinforce and help transfer what is learned into the operational

environment. A final project developed by student teams
completes
the formal requirements.
Learning to think critically and to solve problems in a
structured manner requires active participation. The class
requires
40 hours of classroom time, consisting of ten sessions of four
hours
each. The method of instruction is Socratic, demanding active
classroom participation. Participants also can expect
homework,
requiring both office and non-office time. Participants will
prepare
reading summaries for each class session, and develop one (or
more)
operational examples of at least one structured analytic
method.
Finally, participants work together on teams to complete
classroom
assignments and a final project dealing with an operational
issue
(employing at least five structured analytic methods).
– 99 –
Administration:
Enrollment: Up to 21 Students.
Class Date/Time/Duration: One 4-hour class per day for 10
weeks.
Class Location: ________________________.
Homework: Yes, but hopefully not too odious. Operational
examples are required. A team project is due at the end of the

course.
Texts:
Elder, Linda, and Richard Paul. The Foundations of Analytic
Thinking:
How to Take Thinking Apart and What to Look for When You
Do (Dillon Beach, CA: Foundation for Critical Thinking,
2003).
Heuer, Richards J., Jr. The Psychology of Intelligence Analysis
(Washington, DC: Center for the Study of Intelligence,
1999), URL: <http://cia.gov/csi/books/19104/index.
html>, last accessed 15 March 2006.
Jones, Morgan D. The Thinker’s Toolkit: Fourteen Skills for
Making
Smarter Decisions in Business and in Life (New York, NY:
Crown
Publishing Group, 1997).
Several handouts (TBD)
Objectives:
The overall objective of the class is to enable you to critically
think about analysis and what analysts are tasked to analyze. A
second objective is to provide you with a set of analytic tools
that are
useful to your analysis. At the end of the class you will be
equipped
with a set of analytic skills and will have honed your critical
thinking
skills, allowing you to better function in the workplace.
Specifically,
the course objectives are as follows:

– 100 –
Upon completion of this course you will be able to:
• Use critical thinking techniques to provide structure to your
analytic reasoning.
• Identify, describe, and employ 14 methods for structured
reasoning.
• Demonstrate critical thinking proficiency through lecture,
classroom participation, and weekly homework assignments.
• Complete a final class assignment using a minimum of five
structured analytic methods presented in this course.
• Apply knowledge of critical thinking by using a set of
analytic
tools designed to hone your skills as an analyst.
In other words, at the end of this 10-week-long class, you will
have
• Learned to critically analyze intelligence-associated data,
information, and evidence.
• Honed your critical thinking skills.
• Built a “toolbox” of analytic and problem-solving methods.
• Become better analysts.
For example, when you approach a problem you will be able to
• Discover the true problem by restating and considering
alternative outcomes.

• Have a variety of methods by which you can organize and
make sense of the relevant evidence.
Formal Requirements:
Written summaries of readings. (No more than one page
per chapter assigned.) The summaries should answer exactly
the questions on page five of the syllabus. The summaries
should
also include answers to the exercises in Morgan Jones’ book.
The
summaries will be typed unless prior arrangements have been
made with the instructor. In-class discussions will draw heavily
on
the readings.
Problems from the work environment (Operational
Exemplars). As we study the elements of reasoning and the
methods of problem solving, we need operational examples
– 101 –
against which to apply/illustrate what we are learning. You will
be
responsible for providing at least one of those examples for the
class
to be presented during the week we discuss the method. The
best
exemplars may be saved for use in subsequent classes.
Final project. Working in teams of three or four, and using any
Problem Restatement and Divergent and Convergent Thinking
plus at least three other methods developed in the course – for a

total of of at least five methods – you will develop an
operational
project to be presented at the last class. You will apply the
elements
of critical thinking to the method chosen as well as the specific
problem, apply the appropriate methods to solve of problem,
report
the results, and evaluate the process. The project will be
presented
during a 15–20 minute briefing. A list of the specific elements
that
must be included and the format by which the project is graded
is
on the last page of this syllabus.
Grading:
• Written Summaries (25%). Due weekly. The first summary
will be graded. Subsequently, a random number generator will
be used to select three (3) additional summaries for grading.
Grading will be based on the “Universal Intellectual Standards”
in The Foundations of Analytic Thinking and on whether
instructions
are followed. For example, if you are asked to identify what
key
questions an author is attempting to answer, it is expected that
you
will provide those questions in the reading summary.
• Class Participation (25%). Since this is a discussion course,
you are expected to engage in the process.
• Operational Exemplars (25%). Assigned the first week. No
longer than 5 minutes each.
• Final Project (25%).

– 102 –
Class Descriptions and Weekly Assignments
(complete prior to each class):
Class 1 – How We Don’t Think and How We Might
Reading Assignment: The Thinker’s Toolkit: Fourteen Skills for
Making Smarter Decisions in Business and in Life, Chapters 1–
2
(Part One).
Psychology of Intelligence Analysis, 1–30.
Critical Thinking: Concepts and Tools, entire work.
Written Assignment: Reading summaries for chapters from
Jones and
Heuer. Exercises in text.
Class Objectives: At the end of Class 1 you will be able to:
• Define a bias and discuss the implications of biases in our
decisions.
• Identify sources of cognitive biases.
• Describe the inherent dangers/benefits of biases and the
difficulty of compensating for perceptual biases.
• Describe the characteristics and three principles of
perception.
• Describe how analysts fall prey to absence-of-evidence
biases.
• Describe how anchoring impacts analytical decision-

making.
• Describe how a target can use assimilation biases to
deceive.
• Acknowledge how analysts unwittingly use confirmation
bias to support early assessments.
• Discuss how hindsight and reliability biases play a part in
intelligence failures.
• Describe how oversensitivity to consistency bias can lead to
undesirable results.
• Discuss how expert biases and the Pollyanna and Cassandra
complexes distort our thinking.
• Describe how cultural, personal, and organizational
– 103 –
mindsets impact analysis.
• Define mindsets, discuss how they are derived, and describe
how they influence predictions.
• Identify how we think.
• Describe critical thinking and the standards used for
evaluating our thinking.
• Identify the elemental structures of thought.
• Describe the differences between inferences and assumptions
in intelligence analysis.

– 104 –
Class 2 – Critical Problem Restatement and
Alternative Thinking
Making Smarter Decisions in Business and in Life, Chapters 3,
5.
Written Assignment: Reading summaries for each chapter.
Exercises
in text.
• Demonstrate knowledge of the critical thinking process by
providing an example that meets the universal intellectual
standards.
• Describe the role of questioning in critical thinking.
• Identify the three types of questions used in critical
thinking.
• Determine the sample domains involved in complex
questions.
• Define problem restatement and apply its use through a
practical example.
• Discuss the role that our biases play in problem
restatement.
• Demonstrate knowledge of the critical thinking process by

providing an example that meets the universal intellectual
standards.
• Define divergent thinking and its benefits when performing
a problem restatement.
• Discuss the pitfall involved in problem definition and how it
relates to problem restatement.
• Describe some effective techniques for problem
restatement.
• Identify the types of problems that benefit from problem
restatement.
• Discuss how points of view influence the critical thinking
process.
• Discuss the logic, benefits, risks, and elements of divergent
thinking.
• Identify the four main ideas of divergent thinking.
– 105 –
• Describe what is needed to move from divergence to
convergence.
• Identify the types of problems that benefit from divergent
thinking.
– 106 –

Class 3 – Pollyanna, Cassandra, and Marshaling
6.
Exercises
in text.
• Discuss the pros-cons-fixes approach to critical thinking
and the six-step method employed by successful analysts.
• Identify the logic behind the pros-cons-fixes approach and
discuss those critical thinking problems best suited to this
method.
• Apply the techniques of sorting, chronologies, and timelines
to critical thinking and identify those critical thinking
problems best suited for this approach.
• Identify the two-step technique used for sorted lists,
chronologies, and timelines.
– 107 –
Class 4 – Causes, Squares, and Shrubs
8,

9.
Exercises
in text.
• Discuss the purpose, logic, underlying question, evidence,
concepts, and types of problems best suited for causal flow
diagramming.
• Identify the major factors and cause-and-effect relationships
influencing causal flow diagramming.
• Characterize the differences between direct and
inverse relationships and their impact on causal flow
diagramming.
• Define a feedback loop and discuss what makes it stable or
unstable.
• Apply your knowledge of causal diagramming by
participating in a classroom exercise.
concepts, and types of problems best suited for scenario
matrices and trees.
• Define and discuss the characteristics and differences
between matrices and scenario trees.
• Apply your knowledge of scenario matrices by participating
in a classroom exercise.
• Apply your knowledge of scenario trees by participating in

a classroom exercise.
– 108 –
Class 5 – Weighing the Likelihood
13.
Exercises
in text.
• Discuss how individuals view issues.
concepts, and types of problems best suited for weighted
ranking.
• List the 9-step process involved in weighted ranking.
• Demonstrate knowledge of weighted ranking through class
and small group exercises.
concepts, and types of problems best suited for probability.
• Define and describe how and why we characterize and
assign events, and how this influences intelligence analysis.
• Discuss and explain the differences between mutually
exclusive and conditionally dependent events relative to

probability.
• Identify the seven steps in creating a probability tree.
• Apply your knowledge of probability by participating in a
classroom exercise.
– 109 –
Class 6 – Testing the Devil’s and Other’s Advocacy
Making Smarter Decisions in Business and in Life, Chapters
11–12.
The Psychology of Intelligence Analysis, Chapter 8.
Exercises
in text.
• Define intelligence.
concepts, and types of problems best suited for devil’s
advocacy.
• Define, describe, and characterize situations appropriate
to employ methods of devil’s advocacy in intelligence
analysis
• Demonstrate prowess in devil’s advocacy through
participation in a practical case study.

concepts, and types of problems best suited for hypothesis
testing.
• Discuss the benefits of hypothesis testing.
• Identify the 8-step process of hypothesis testing.
• Demonstrate prowess in hypothesis testing through
participation in a practical case study.
– 110 –
Class 7 – Analyzing Apples and Toyotas
15, 16.
Exercises
in text.
• Describe how individual viewpoints influence utility
analysis.
• Define utility analysis and discuss the type of problems best
suited for this method.
• Identify the logic of utility analysis.
• Employ utility analysis to arrive at the most cost-effective
solutions.
• List the steps involved in creating a utility tree and apply

this knowledge by participating in classroom exercises.
• List the steps involved in creating a utility matrix and apply
this knowledge by participating in classroom exercises.
• Examine the benefits of structuring as it applies to utility
and probability analysis.
• Describe the differences between utility and probability
analysis.
• Examine multi-perspective utility analysis and its use by
participating in classroom exercises.
• List the 13 steps of multi-perspective utility analysis.
• Apply your knowledge of multi-perspective utility analysis
by participating in a classroom exercise.
– 111 –
Class 8 – And Now for Something Completely Different
Reading Assignment: Review all assigned readings.
Written Assignment: None.
• Describe which structured analysis methods work best in
particular situations.
• Demonstrate knowledge of critical thinking and structured
analysis through a practicum.

• Demonstrate your knowledge of critical thinking and
structured analysis through a small group exercise.
– 112 –
Class 9 – Final Project Preparation
Class Objectives: During Class 9 you will:
• Learn through experience the pros and cons of research,
analysis and reporting in small teams.
– 113 –
Class 10 – Final Project Presentation and Wrap-up
Class Objectives: During Class 10:
• Your small group will make a multi-media presentation on
the analytic problem you chose.
Methods:
• A sample of the problem statement and restatement

process.
• A sample of the convergent and divergent thinking applied
to the problem.
• Concise demonstrations of the three structured analytic
methods used to answer the analytic problem.
Assessment:
• You are evaluated on how clearly the group presents
its project, and on whether the project requirements
listed above are met as shown on the last page of
the syllabus.
– 114 –
Reading Summary Questions:231
For each chapter/work, answer the following questions (in
writing) about
the reading.
1. What is the author’s purpose? If there is more than one
main point, what is the overarching purpose?
2. What are the key questions the author raises/addresses?
3. What evidence does the author provide to support his
argument?
4. What inferences does the author make from the evidence?

5. On what underlying concepts does the author rely?
6. What does the author take for granted? What assumptions
does the author make?
7. What are the implications of the author’s point of view?
What are the implications if we adopt/do not adopt what
the author recommends?
8. What is the author’s point of view? What other points of
view are there?
231 Paul and Elder, Concepts and Tools, 13.
– 115 –
Final Project Evaluation Worksheet232
C
larity
A
ccuracy
Precisio
n
R
elevance
D
epth
B

readth
Lo
gic
Significance
Fairness
Purpose
Key
Questions
Assumptions
Evidence
A: Methods
B: Analysis
Concepts
Inferences
Implications
Other Points
of View
Lessons
Learned
Comments

232 Derived by author from Paul and Elder, Concepts and
Tools, 2–9.
– 116 –
– 117 –
BiBliography
Ackerman, Robert K. “Information Age Poses New Challenges
to
Intelligence.” Signal 53, no. 2 (October 1998): 23–25.
Advanced Research and Development Activity. Broad Agency
Announcement for the Novel Intelligence from Massive Data
(NIMD)
R&D Program. NMA-401-02-BAA-0004, 7 June 2002, 1.
__________. “Preliminary Solicitation Information for
Advanced
QUestion & Answering for INTelligence (AQUAINT).” URL:
<http://www.digitalgovernment.org/library/library/pdf/
preliminary_information.pdf>. Last accessed 26 March
2006.
Air Force Research Laboratory (AFRL). IFED Directorate.
“Collaboration and Analyst/System Effectiveness (CASE).”
BAA 06-02-IFKA, 28 October 2005. URL: <http://www.rl.af.
mil/div/IFK/baa/>. Last accessed 11 March 2006.
Allen, Charles E. “Intelligence: Cult, Craft, or Business?” In
Seminar

on Intelligence, Command, and Control, Guest Presentations,
Spring
2000. Cambridge, MA: Harvard University Program on
Information Resources Policy, I-01-1, April 2000. URL:
<http://www.pirp.harvard.edu/pubs.html>. Last accessed
11 January 2006.
Allison, Graham, and Philip Zelikow. Essence of Decision:
Explaining
the Cuban Missile Crisis, 2nd Edition. New York, NY:
Longman,
1999.
Amuchastegui, Domingo “Cuban Intelligence and the October
Crisis.” In James G. Blight and David A. Welch, Eds.
Intelligence and the Cuban Missile Crisis. London, UK: Frank
Cass, 1998: 88–119.
– 118 –
Anderson, Terence J., and William Twining. Analysis of
Evidence: How
to Do Things with Facts Based on Wigmore’s Science of
Judicial Proof.
Chicago, IL: Northwestern University Press, 1991.
Anderson, Terence J., David A. Schum, and William Twining.
Analysis
of Evidence. Cambridge, UK: Cambridge University Press,
2005.
Bathurst, Robert B. Intelligence and the Mirror: On Creating an
Enemy. Oslo,
Norway: International Peace Research Institute, 1993.

Ben-Israel, Isaac. “Philosophy and Methodology of
Intelligence:
The Logic of the Estimate Process. Intelligence and National
Security 4, no. 4 (October 1989): 660–718.
Blight, James G., and David A. Welch, Eds. Intelligence and
the Cuban
Missile Crisis. London, UK: Frank Cass, 1998.
Boden, Margaret A. The Creative Mind: Myths and
Mechanisms. New
York, NY: Basic Books, 1990.
Brei, William S., CAPT, USAF. Getting Intelligence Right: The
Power of
Logical Procedure. Occasional Paper Number Two.
Washington,
DC: Joint Military Intelligence College, January 1996.
Brent, Joseph. Charles Sanders Peirce: A Life. Bloomington,
IN: Indiana
University Press, 1998.
Brookfield, Stephen D. Developing Critical Thinkers:
Challenging Adults to
Explore Alternative Ways of Thinking and Acting. San
Francisco,
CA: Jossey-Bass Publishers, 1987.
Callum, Robert. “The Case for Cultural Diversity in the
Intelligence Community.” International Journal of Intelligence
and CounterIntelligence 14, no. 1, Spring 2001: 25–48.

– 119 –
Cheikes, Brant A., Mark J. Brown, Paul E. Lehner, and Leonard
Adelman. Confirmation Bias in Complex Analyses, Mitre
Technical
Report MTR 04B0000017. Bedford, MA: Center for Integrated
Intelligence Systems, October 2004.
Christensen, Clayton M. The Innovator’s Dilemma: When New
Technologies
Cause Great Firms to Fail. Boston, MA: Harvard Business
School
Press, 1997.
Christensen, Clayton M., and Michael E. Raynor. The
Innovator’s
Solution
: Creating and Sustaining Successful Growth. Boston, MA:
Harvard Business School Press, 2003.
Christensen, Clayton M., Scott D. Anthony, and Erik A. Roth.
Seeing
What’s Next: Using the Theories of Innovation to Predict
Industry
Change. Boston, MA: Harvard Business School Press, 2004.
“Clarke: Bush Didn’t See Terrorism as ‘Urgent’.” CNN.com,

19 May 2004. URL: <http://www.cnn.com/2004/
ALLPOLITICS/03/24/911.commission/index.htm>. Last
accessed 9 March 2006.
Commission on the Intelligence Capabilities of the United
States
Regarding Weapons of Mass Destruction. Report to the
President
of the United States. Washington DC: Government Printing
Office, March 31, 2005. URL: <http://www.wmd.gov/
report/ index.html>, last accessed 28 July 2005.
Cooper, Jeffrey R. Curing Analytic Pathologies: Pathways to
Improved
Intelligence Analysis. Washington, DC: Center for the Study
of Intelligence, 2005.
__________. Email to the author. 31 March 2006.
Davis, Jack. “The Kent-Kendall Debate of 1949.” Studies in
Intelligence
35, no. 2 (Summer 1991): 37–50.
De Bono, Edward. de Bono’s Thinking Course. New York,
NY: Facts

on File Publications, 1982.
– 120 –
De Cagna, Jeff. “Making Change Happen: Steve Denning Tells
the Story of Storytelling.” Information Outlook, January 2001:
28–35.
De Rivera, Joseph. The Psychological Dimension of Foreign
Policy.
Columbus, OH: Charles E. Merrill Publishing Company,
1968.
Denning, Stephen. The Springboard: How Storytelling Ignites
Action
in Knowledge-Era Organizations. Boston, MA: Butterworth-
Heinemann, 2001.
Devlin, Keith. Goodbye Descartes: The End of Logic and the
Search for
a New Cosmology of the Mind. New York, NY: John Wiley &
Sons, Inc., 1997.

Dewey, John. How We Think: A Restatement of the Relation
of Reflective
Thinking to the Educative Process. New York, NY: D.C. Heath
and Company, 1910.
Dijksterhuis, Ap, Martin W. Bos, Loran F. Nordgren, and Rick
B.
von Baren. “On Making the Right Choice: The Deliberation-
Without-Attention Effect.” Science, 311, no. 5763 (17 February
2006): 1005–1007.
DiSpezio, Michael A. Classic Critical Thinking Puzzles. New
York,
NY: Main Street, 2005.
Durkheim, Emile. The Rules of Sociological Method.
Glencoe, IL: Free
Press, 1938.
Eco, Umberto, and Thomas A. Sebeok. The Sign of Three:
Dupin,
Holmes, Pierce. Bloomington, IN: Indiana University Press,
1988
EDS. Herding Cats. Television advertisement produced for

EDS by
Fallon-McElligott. Videocassette, January 2000.
– 121 –
__________. Building Airplanes in Flight. Television
advertisement
produced for EDS by Fallon-McElligott. Videocassette,
November 2000.
Elder, Linda, and Richard Paul. The Foundations of Analytic
Thinking:
How to Take Thinking Apart and What to Look for When You
Do.
Dillon Beach, CA: The Foundation for Critical Thinking,
2003.
Elliott, Michael. “How the U.S. Missed the Clues.” CNN.com,
20 May 2002. URL: < http://archives.cnn.com/2002/
ALLPOLITICS/05/20/time.clues/index.html>. Last
accessed 9 March 2006.
Facione, Peter A. Critical Thinking: What It Is and Why It

Counts.
Milbrae, CA: California Academic Press, 1998, updated
2004. URL<http://www.insightassessment.com/>. Last
accessed 22 July 2005.
Facione, Peter A., Noreen C. Facione, and Carol A. Giancarlo.
“The Disposition Towards Critical Thinking: Its Character,
Measurement, and Relationship to Critical Thinking Skill,”
Informal Logic 20, no. 1 (2000): 61–84.
__________. Professional Judgment and the Disposition
Toward Critical
Thinking (Milbrae, CA: California Academic Press, 2002).
URL: < http://www.calpress.com/ pdf_files/Prof_jmn.pdf>.
Last accessed 31 March 2006.
Feder, Stanley. “Factions and Policon: New Ways to Analyze
Politics.”
In Westerfield, H. Bradford, ed. Inside CIA’s Private World:
Declassified Articles from the Agency’s Internal Journal, 1955–
1992.
New Haven, CT: Yale University Press, 1995.
Feldman, Daniel. Critical Thinking: Strategies for Decision
Making. Menlo

Park, CA: Crisp Publications, Inc, 2002.
Fisher, Alec. Critical Thinking: An Introduction. Cambridge,
UK:
Cambridge University Press, 2001.
– 122 –
Folker, Robert D., MSGT, USAF. Intelligence Analysis in
Theater Joint
Intelligence Centers: An Experiment in Applying Structured
Methods.
Occasional Paper Number Seven. Washington, DC: Joint
Military Intelligence College, January 2000.
Folker, Robert D., LT, USAF. Email to the author. 9 December
2003.
Fursenko, Aleksandr and Timothy Naftali. “One Hell of a
Gamble:”
Khrushchev, Castro, and Kennedy, 1958–1964. New York, NY:
W. W. Norton & Company, 1997.

Garthoff, Raymond L. “US Intelligence in the Cuban Missile
Crisis.”
In James G. Blight and David A. Welch, Eds. Intelligence
and the Cuban Missile Crisis. London, UK: Frank Cass, 1998:
18–63.
George, Alexander. Presidential Decisionmaking in Foreign
Policy: The
Effective Use of Information and Advice. Boulder, CO:
Westview
Press, 1980.
Gilovich, Thomas. How We Know What Isn’t So: The
Fallibility of
Human Reason in Everyday Life. New York, NY: The Free
Press,
1991.
Gilovich, Thomas, Dale Griffin, and Daniel Kahneman, Eds.
Heuristics
and Biases: The Psychology of Intuitive Judgment. Cambridge,
UK:
Cambridge University Press, 2002.
Gladwell, Malcolm. The Tipping Point: How Little Things Can

Make a Big
Difference. Boston, MA: Little Brown and Company, 2000.
__________. Blink: The Power of Thinking without Thinking.
Boston,
MA: Little Brown and Company, 2005.
Glaser, Edward M. An Experiment in the Development of
Critical Thinking.
New York, NY: AMS Press, 1941.
– 123 –
Grabo, Cynthia M. Anticipating Surprise: Analysis for
Strategic Warning.
Washington DC: Joint Military Intelligence College, 2002.
Gribkov, Anatoli. “The View from Moscow and Havana.” In
Gribkov, Anatoli, and William Y. Smith. Operation ANADYR:
U.S. and Soviet Generals Recount the Cuban Missile Crisis.
Chicago,
IL: Edition q, 1994: 3–76.

Gribkov, Anatoli, and William Y. Smith. Operation ANADYR:
U.S. and
Soviet Generals Recount the Cuban Missile Crisis. Chicago, IL:
Edition q, 1994.
Halpern, Diane. Thought and Knowledge: An Introduction to
Critical
Thinking, 4th Edition. Mahwah, NJ: Lawrence Erlbaum
Associates, Publishers, 2002.
Hampson, Fen Osler. “The Divided Decision-Maker: American
Domestic Politics and the Cuban Crises.” International Security,
9, no 3 (Winter 1984/85): 130-165.
Harcourt Assessment, Inc. Local Norms Report, Watson-Glaser
Critical
Thinking Assessment. Report prepared for the Defense
Intelligence Agency, 2005.
Heuer, Richards J., Jr. “Strategic Deception and
Counterdeception:
A Cognitive Process Approach.” International Studies
Quarterly
25, no. 2 (June 1981): 294–327.

__________. Adapting Academic Methods and Models to
Governmental
Needs: The CIA Experience. Strategic Studies Institute. U.S.
Army War College. July 31, 1978.
__________. Psychology of Intelligence Analysis.
Washington, DC: CIA
Center for the Study of Intelligence, 1999.
Hinsley, F. H., and Alan Stripp, Eds. Codebreakers: The Inside
Story of
Bletchley Park. Oxford, UK: Oxford University Press, 1993.
Holyoak, Keith J., and Paul Thagard. Mental Leaps: Analogy in
Creative
Thought. Cambridge, MA: The MIT Press, 1995.
– 124 –
Hughes, Francis J. Conversation with the author. Washington,
DC:
Joint Military Intelligence College. 8 May 2003.

Hughes, Francis J., and David A. Schum. Credibility
Assessment: A
First Step in Intelligence Analysis. Unpublished tutorial. Joint
Military Intelligence College, April 2003.
Janis, Irving. Groupthink: Psychological Studies of Policy
Decisions and
Fiascoes, 2nd Edition. Boston, MA: Houghton Mifflin
Company,
1982.
Jervis, Robert. System Effects: Complexity in Political and
Social Life.
Princeton, NJ: Princeton University Press, 1997.
Johnson Thomas R., and David A. Hatch. NSA and the Cuban
Missile
Crisis. Fort Meade, MD: National Security Agency Center
for Cryptologic History, 1998. URL: <http://www.nsa.gov/
publications/publi00033.cfm>. Accessed 18 April 2006
Johnston, Rob. Analytic Culture in the U.S. Intelligence
Community: An
Ethnographic Study. Washington, DC: Center for the Study of
Intelligence, 2005.

Jones, Morgan D. The Thinker’s Toolkit: 14 Powerful
Techniques for Problem
Solving. New York, NY: Random House, Inc, 1995.
__________. Conversation with the author. 15 December 2003.
Kahneman, Daniel, Paul Slovic, and Amos Tversky, eds.
Judgment under
Uncertainty: Heuristics and Biases. Cambridge, UK: Cambridge
Kam, Ephraim. Surprise Attack: The Victim’s Perspective.
Cambridge,
MA: Harvard University Press, 1990.
Kent, Sherman. Strategic Intelligence for American World
Policy. Princeton,
NJ: Princeton University Press, 1949.
__________. “A Crucial Estimate Relived.” Studies in
Intelligence 36,
no. 5 (1992): 111–119.

– 125 –
Kerbel, Josh. “Thinking Straight: Cognitive Bias in the US
Debate
about China.” Studies in Intelligence 48, no. 3 (2004), URL:
<http://cia.gov/csi/studies/vol48no3/index.html>. Last
accessed 22 February 2006.
Klein, Gary. Intuition at Work: Why Developing Your Gut
Instincts Will
Make You Better at What You Do. New York, NY: Doubleday,
2003.
Krizan, Lisa. Intelligence Essentials for Everyone. Occasional
Paper
Number Six. Washington, DC: Joint Military Intelligence
College, 1999.
Lebow, Richard Ned, and Janice Gross Stein. “Back to the Past:
Counterfactuals and the Cuban Missile Crisis.” In Philip E.
Tetlock, and Aaron Belkin Counterfactual Thought Experiments
in
World Politics: Logical, Methodological, and Psychological
Perspectives.

Princeton, NJ: Princeton University Press, 1996: 119–148.
LeGault, Michael R. Think: Why Crucial Decisions Can’t Be
Made in the
Blink of an Eye. New York, NY: Threshold Editions, 2006.
Lengel, Allan. “Little Progress in FBI Probe of Anthrax
Attacks:
Internal Report Compiled As Agents Hope for a Break.”
Washington Post. 16 September 2005. A01.
Lengel, Allan, and Guy Gugliotta. “Md. Pond Produces No
Anthrax
Microbes: FBI Sought Clues In Deadly Attacks.” Washington
Post. 1 August 2003. A03.
Lipman, Matthew. Thinking in Education, 2nd Edition.
Cambridge,
UK: Cambridge University Press, 2003.
Lundahl, Arthur C. “Additional Information—Mission 3102.”
Memorandum for Director of Central Intelligence and
Director, Defense Intelligence Agency, 15 October 1962.
In Mary S. McAuliffe, ed., CIA Documents on the Cuban
Missile

Crisis, 1962. Washington DC: Central Intelligence Agency,
1992: 181–182.
– 126 –
__________. “Additional Information—Mission 3107.”
Memorandum for Director of Central Intelligence and
Director, Defense Intelligence Agency, 19 October 1962.
In Mary S. McAuliffe, ed., CIA Documents on the Cuban
Missile
Crisis, 1962. Washington DC: Central Intelligence Agency,
1992: 209.
McAuliffe, Mary S., ed. CIA Documents on the Cuban Missile
Crisis, 1962
Washington DC: Central Intelligence Agency, 1992.
Marrin, Stephen. “Homeland Security and the Analysis of
Foreign
Intelligence,” Markle Foundation Task Force on National
Security in the Information Age, 15 July 2002, URL: <www.
markletaskforce.org/documents/marrin_071502.pdf>. Last
accessed December 9, 2003.

__________. Email to the author, 8 December 2003.
__________. “CIA’s Kent School: Improving Training for New
Analysts.” International Journal of Intelligence and
CounterIntelligence
16, no. 4 (Winter 2003–2004): 609–637.
May, Ernest R. Lessons of the Past: The Use and Misuse of
History in
American Foreign Policy. Oxford, UK: Oxford University
Press,
1973.
Merom, Gil. “The 1962 Cuban Intelligence Estimate: A
Methodological Perspective.” Intelligence and National
Security,14, no. 3 (Autumn 1999): 48–80.
Mid-Level Intelligence Professional. National Security Agency.
Email
to the author, 9 March 2006.
Miller, George A. “The Magical Number Seven, Plus or Minus
Two.” The Psychological Review 63 (1956): 81–97.

– 127 –
Miller, Judith. “A Battle of Words over War Intelligence.”
New York
Times. Online edition, 22 November 2003. URL: <www.
nytimes.com/2003/11/22/arts/22INTE.html>. Last
accessed November 28, 2003.
Millward, William. “Life in and out of Hut 3.” In Hinsley, F.
H., and
Alan Stripp, Eds. Codebreakers: The Inside Story of Bletchley
Park.
Oxford, UK: Oxford University Press, 1993: 17–29.
Moore, David T. Creating Intelligence: Evidence and Inference
in the
Analysis Process. MSSI Thesis. Washington, DC: Joint
Military
Intelligence College, July 2002.
__________. “Species of Competencies for Intelligence
Analysis.”
Defense Intelligence Journal 11, no. 2 (Summer 2002): 97–119.

__________. Personal Notes. Joint Military Intelligence
College,
Denial and Deception Advanced Studies Program, Spring
2005.
Moore David T., and Lisa Krizan. “Intelligence Analysis: Does
NSA
Have What it Takes.” Reprint: NSA Center for Cryptologic
History. Cryptologic Quarterly, 20, nos 1/2 (Summer/Fall
2001): 1–33.
__________. “Core Competencies for Intelligence Analysis at
the National Security Agency.” In Swenson, Russell G., ed.
Bringing Intelligence About: Practitioners Reflect on Best
Practices.
Washington DC: Joint Intelligence Military College, 2003:
95–132.
Moore, David T., Lisa Krizan, and Elizabeth J. Moore.
“Evaluating
Intelligence: A Competency-Based Model.” International
Journal of Intelligence and Counterintelligence 18, no. 2
(Summer
2005): 204–220.

Myers, David G. Intuition: Its Powers and Perils. New Haven,
CT: Yale
– 128 –
Negroponte, John D., Director of National Intelligence.
National
Intelligence Strategy of the United States of America.
Washington,
DC: Office of the Director of National Intelligence, October
2005. URL: <http://www.dni.gov/NISOctober2005.pdf>.
Neustadt, Richard E., and Ernest R. May. Thinking in Time:
The
Uses of History for Decision Makers. New York, NY: The Free
Press, 1986.
Oberdorfer, Don. “Missed Signals in the Middle East.”
Washington
Post Magazine, 17 March 1991.

Paul, Richard W. “A Draft Statement of Principles.” The
National
Council for Excellence in Critical Thinking. URL: <www.
criticalthinking.org/ncect.html>. Last accessed March 18,
2003.
Paul, Richard W., and Linda Elder. The Miniature Guide to
Critical
Thinking Concepts and Tools, 4th Edition. Dillon Beach, CA:
The Foundation for Critical Thinking, 2004.
Paul, Richard W., and Gerald Nosich. A Model for the National
Assessment
of Higher Order Thinking. Dillon Beach, CA: Foundation for
Critical Thinking, n.d.
Paul, Richard W., Linda Elder, and Ted Bartell. “Executive
Summary,
Study of 38 Public Universities and 28 Private Universities
To Determine Faculty Emphasis on Critical Thinking In
Instruction.” California Teacher Preparation for Instruction in
Critical
Thinking: Research Findings and Policy Recommendations.
California

Commission on Teacher Credentialing, Sacramento,
California, 1997. Dillon, CA: Foundation for Critical
Thinking, 1997. URL: <criticalthinking.org/schoolstudy.
htm>. Last accessed March 18, 2003.
– 129 –
Paul, Richard W., Linda Elder, and Ted Bartell. California
Teacher
Preparation for Instruction in Critical Thinking: Research
Findings and
Policy Recommendations. California Commission on Teacher
Credentialing, Sacramento California, 1997. Dillon, CA:
Foundation for Critical Thinking, 1997.
Perrow, Charles. Normal Accidents: Living with High-Risk
Technologies.
Pettee, George S. The Future of American Secret Intelligence.
Washington,
DC: Infantry Journal Press, 1946.

Platt, Washington. Strategic Intelligence Production: Basic
Principles. New
York, NY: Frederick A. Praeger, 1957.
Plous, Scott. The Psychology of Judgment and Decision-
Making. New
York, NY: McGraw Hill, Inc., 1993.
Raasa, Paul J. “The Denial and Deception Challenge to
Intelligence.”
In Godson, Roy, and James J. Wirtz, Strategic Denial and
Deception. New Brunswick, NJ: Transaction Publishers, 2002:
223–228.
Reed, Jennifer H. Effect of a Model For Critical Thinking on
Student
Achievement In Primary Source Document Analysis And
Interpretation,
Argumentative Reasoning, Critical Thinking Dispositions, and
History
Content in a Community College History Course. PhD
Dissertation.
College of Education, University of South Florida, December
1998. URL: <http://www.criticalthinking.org/resources/
JReed-Dissertation.pdf>. Last accessed 6 May 2006.

Restak, Richard, M.D. The Brain Has a Mind of Its Own:
Insights
from a Practicing Neurologist. New York, NY: Harmony
Books,
1991.
Reynolds, William. Conversation with the author, 14 March
2006.
– 130 –
Rieber, Steven, and Neil Thomason. “Toward Improving
Intelligence
Analysis: Creation of a National Institute for Analytic
Methods.” Studies in Intelligence 49, no. 4 (Winter 2006):
71–80.
Schafersman, Steven D. “An Introduction to Critical Thinking,”
January 1991. URL: <www.freeinquiry.com/critical-thinking.
html>. Last accessed 9 March 2006.
Schecter Jerrold L., and Peter S. Deriabin. The Spy Who Saved

the
World. New York, NY: Scribner’s, 1992.
Schum, David A. Evidence and Inference for the Intelligence
Analyst,
2 volumes. Lanham, MD: University Press of America,
1987.
__________. Evidential Foundations of Probabilistic
Reasoning. New
York, NY: John Wiley & Sons, Inc., 1994.
__________. “Species of Abductive Reasoning in Fact
Investigation
in Law” Cardozo Law Review 22, nos. 5/6 (July 2001): 1645–
1681.
__________. “Evidence Marshaling for Imaginative Fact
Investigation.” Artificial Intelligence and Law 9 (2001): 165–
188.
Sebeok, Thomas A. “One, Two, Three Spells UBERTY.” In
Umberto Eco and Thomas A. Sebeok. The Sign of Three:
Dupin, Holmes, Pierce. Bloomington, IN: Indiana University
Press, 1988: 1–10.

Senate Select Committee On Intelligence. Report on the U.S.
Intelligence
Community’s Prewar Intelligence Assessments on Iraq. United
States
Senate. 108th Congress, 7 July 2004.
Simon, Herbert A. “Invariants of Human Behavior.” Annual
Review
of Psychology 41 (1990):1–19.
– 131 –
Stefik, Mark, and Barbara Stefik. Breakthrough: Stories and
Strategies of
Radical Innovation. Cambridge, MA: MIT Press, 2004.
Swenson, Russell G., ed. Research: Design and Methods.
Washington,
DC: Joint Intelligence Military College, 2000.
__________. Bringing Intelligence About: Practitioners Reflect
on Best

Practices. Washington, DC: Joint Intelligence Military College,
2003.
Tetlock, Philip E. Expert Political Judgment: How Good Is It?
How Can
We Know? Princeton, NJ: Princeton University Press, 2005.
Tetlock, Philip E. and Aaron Belkin. Counterfactual Thought
Experiments in
World Politics: Logical, Methodological, and Psychological
Perspectives.
Thagard, Paul. Coherence in Thought and Action. Cambridge,
MA: The
MIT Press, 2000.
Treverton, Gregory F. Conversation with the author, 18 May
2006.
__________. Email exchanges with the author, 17–18 May
2006.
U.S. Congress. Joint Inquiry into Intelligence Community
Activities Before

and After the Terrorist Attacks of September 11, 2001. Report
of the U.S. Senate Select Committee on Intelligence and
U.S. House Permanent Select Committee on Intelligence,
Together with Additional Views. Senate Report No. 107-
351/House Report. No. 107-792. 107th Congress, 2nd Session.
December 2002.
U.S. Department of Defense. Photograph in the John Fitzgerald
Kennedy Library. Boston, MA. PX 66-20:7 14 October
1962.
White, Mark J. Missiles in Cuba: Kennedy, Khrushchev,
Castro, and the
1962 Crisis. Chicago, IL: Ivan R. Dee, 1997.
– 132 –
Wikipedia. “The Free Encyclopedia.” URL:
<www.wkikpedia.org>.
Wirtz, James J. “Organizing for Crisis Intelligence: Lessons
form

the Cuban Missile Crisis.” In James G. Blight and David A.
Welch, Eds. Intelligence and the Cuban Missile Crisis.
London,
UK: Frank Cass, 1998: 120–149.
Wohlstetter, Roberta. “Cuba and Pearl Harbor: Hindsight and
Foresight.” Foreign Affairs 46, no. 3 (July 1965): 691–707.
Woods, Kevin, James Lacey, and Williamson Murray.
“Saddam’s
Delusions: The View from the Inside,” Foreign Affairs, 85,
no. 3 (May/June 2006). URL: <http://www.foreignaffairs.
org/20060501faessay85301/kevin-woods-james-lacey-
williamson-murray/saddam-s-delusions-the-view-from-the-
inside.html>. Accessed 31 March 2006.
Woods, Kevin M., Michael R. Pease, Mark E. Stout, Williamson
Murray, and James G. Lacey, Iraqi Perspectives Project: A
View of Operation Iraqi Freedom from Saddam’s Senior
Leadership.
Norfolk, VA: Joint Center for Operational Analysis, 2006.
URL: <http://www.foreignaffairs.org/special/iraq/ipp.pdf>.
Zlotnick, Jack. “Bayes’ Theorem for Intelligence Analysis.”

Studies
in Intelligence 11, no. 4 (Spring 1972): 1–12. URL: <http://
cia.gov/csi/kent_csi/pdf/v08i4a11p.pdf#page=1>. Last
accessed 7 April 2006.
– 133 –
aBout the author
DaviD t. Moore is a career senior intelligence analyst and
technical director at the National Security Agency where he is
an
advocate for, and mentor of best practices in intelligence
analysis.
He is an adjunct faculty member of the National Cryptologic
School; has taught at the National Defense Intelligence College,
Washington, DC; and at Trinity University, Washington, DC.
Mr. Moore holds a B.A. in sociology from Washington and Lee
University and a Master of Science of Strategic Intelligence
from
the National Defense Intelligence College (formerly the Joint
Military Intelligence College).

Mr. Moore’s other publications include:
• “Species of Competencies for Intelligence Analysis,” Defense
Intelligence Journal, 11, no. 2 (Summer 2002): 97-119.
• “Species of Competencies for Intelligence Analysis,”
American
Intelligence Journal, 23 (2005): 29–43 (an expanded version of
the original article).
• With coauthor Lisa Krizan:
o “Intelligence Analysis, Does NSA Have What it Takes,”
Cryptologic Quarterly, 20, nos. 1/2 (Summer/Fall 2001): 1-32.
o “Core Competencies for Intelligence Analysis at the National
Security Agency,” in Bringing Intelligence About: Practitioners
Reflect on Best Practices, Russell Swenson, ed. (2004): 95-131.
• With coauthors Lisa Krizan and Elizabeth J. Moore
o “Evaluating Intelligence: A Competency-Based Approach,”
in the International Journal of Intelligence and
CounterIntelligence,
18, no. 2 (Summer 2005): 204-220.

• With coauthor William N. Reynolds:
o “So Many Ways To Lie: The Complexity of Denial and
Deception,” Defense Intelligence Journal, 15, no. 2 (Fall 2006):
95-116.
– 134 –
Colophon
A colophon records the details of the layout and design of a
publication. It acknowledges the craftsmanship and art required
in
creating a printed work. Whereas the acknowledgements record
the
textual assistance an author receives, a colophon records the
equally
important technical assistance received.
This Occasional Paper was set in Baskerville, a transitional
typeface
created by Birmingham printer John Baskerville around 1752.

George
Jones redesigned it in 1930 for Linotype-Hell; the International
Typeface Corporation licensed it in 1982. The long, elegant
serifs
and the subtle transfer of stroke weight from thick to very thin
give it
a delicacy and grace that makes it an excellent text typeface.
Titling
and primary headings were set in Castellar, a Monotype
typeface
created by John Peters in 1957. Named after a town in the Alps,
Castellar also was designed for refined display work. Tables
were set
in Gill Sans, designed by artist and type designer Eric Gill in
1931
for Monotype. Unlike many sans serif typefaces it is highly
legible
and readable in text. The cover was set in Californian, a
typeface
originally designed by Frederic Goudy in 1938 and adapted by
various designers for the Font Bureau.
This publication was composed on an Apple iMac and a
Macbook
Pro using Adobe Systems InDesign. Masters were created in

Adobe
Acrobat. Illustrations were prepared in Omni Systems Omni
Graffle
and Adobe Photoshop.
Baskerville, Californian, Castellar, Gill Sans, iMac, Macbook
Pro,
Acrobat, InDesign, Omni Graffle, and Photoshop are registered
trademarks of their respective companies.
Desktop composition by the author.
The cover was designed by Bridgette White.
Published by the National Defense Intelligence College Press,
May
2006; and by the National Defense Intelligence College
Foundation,
March 2007. Printed at the Defense Intelligence Agency,
Washington
DC.

CIA Studies in Intelligence Soviet Deception in the Cuban Missi.docx

CIA Studies in Intelligence Soviet Deception in the Cuban Missi.docx

More Related Content

Similar to CIA Studies in Intelligence Soviet Deception in the Cuban Missi.docx (20)

More from christinemaritza (20)

Recently uploaded (20)

CIA Studies in Intelligence Soviet Deception in the Cuban Missi.docx