The Scientific Method A Guide To Finding Useful Knowledge J Scott Armstrong

The Scientific Method A Guide To Finding Useful
Knowledge J Scott Armstrong download
https://ebookbell.com/product/the-scientific-method-a-guide-to-
finding-useful-knowledge-j-scott-armstrong-50183040
Explore and download more ebooks at ebookbell.com

Here are some recommended products that we believe you will be
interested in. You can click the link to download.
The Scientific Method Reflections From A Practitioner Massimiliano Di
Ventra
https://ebookbell.com/product/the-scientific-method-reflections-from-
a-practitioner-massimiliano-di-ventra-49157412
Solving Everyday Problems With The Scientific Method Thinking Like A
Scientist Mak Don K
https://ebookbell.com/product/solving-everyday-problems-with-the-
scientific-method-thinking-like-a-scientist-mak-don-k-4646184
Solving Everyday Problems With The Scientific Method Thinking Like A
Scientist 2nd Edition Don K Mak
https://ebookbell.com/product/solving-everyday-problems-with-the-
scientific-method-thinking-like-a-scientist-2nd-edition-don-k-
mak-5711460
Archeological Explanation The Scientific Method In Archeology Patty Jo
Watson Steven A Leblanc Charles L Redman
https://ebookbell.com/product/archeological-explanation-the-
scientific-method-in-archeology-patty-jo-watson-steven-a-leblanc-
charles-l-redman-51905094

Our Knowledge Of The External World As A Field For Scientific Method
In Philosophy Bertrand Russell
https://ebookbell.com/product/our-knowledge-of-the-external-world-as-
a-field-for-scientific-method-in-philosophy-bertrand-russell-55063274
The Philosophy Of As If A System Of The Theoretical Practical And
Religious Fictions Of Mankind International Library Of Psychology
Philosophy And Scientific Method Paperback Hans Vaihinger
https://ebookbell.com/product/the-philosophy-of-as-if-a-system-of-the-
theoretical-practical-and-religious-fictions-of-mankind-international-
library-of-psychology-philosophy-and-scientific-method-paperback-hans-
vaihinger-7410120
An Introduction To The Lattice Boltzmann Method A Numerical Method For
Complex Boundary And Moving Boundary Flows Takaji Inamuro
https://ebookbell.com/product/an-introduction-to-the-lattice-
boltzmann-method-a-numerical-method-for-complex-boundary-and-moving-
boundary-flows-takaji-inamuro-48710510
Introduction To The Lattice Boltzmann Method An A Numerical Method For
Complex Boundary And Moving Boundary Flows Inamuro
https://ebookbell.com/product/introduction-to-the-lattice-boltzmann-
method-an-a-numerical-method-for-complex-boundary-and-moving-boundary-
flows-inamuro-232333372
A Pragmatic Introduction To The Finite Element Method For Thermal And
Stress Analysis With The Matlab Toolkit Sofea Draft Petr Krysl
https://ebookbell.com/product/a-pragmatic-introduction-to-the-finite-
element-method-for-thermal-and-stress-analysis-with-the-matlab-
toolkit-sofea-draft-petr-krysl-2629928

The Scientific Method
The scientific method delivers prosperity, yet scientific practice has become
subject to corrupting influences from within and without the scientific
community. This essential reference is intended to help remedy those
threats. The authors identify eight essential criteria for the practice of
science and provide checklists to help avoid costly failures in scientific
practice. Not only for scientists, this book is for all stakeholders of the
broad enterprise of science. Science administrators, research funders, jour-
nal editors, and policymakers alike will find practical guidance on how they
can encourage scientific research that produces useful discoveries.
Journalists, commentators, and lawyers can turn to this text for help with
assessing the validity and usefulness of scientific claims. The book provides
practical guidance and makes important recommendations for reforms in
science policy and science administration. The message of the book is
complemented by Nobel Laureate Vernon L. Smith’s foreword and an
afterword by Terence Kealey.
J. Scott Armstrong has dedicated his life to discovering useful scientific
findings. During his 52 years as a professor at the Wharton School,
University of Pennsylvania, he has delivered over 100 lectures at univer-
sities in 27 countries, won numerous awards, and published four books and
over 400 management science articles.
Kesten C. Green has published nearly 40 scientific works, mostly concerned
with improving the predictive validity of methods and models. Before
becoming an academic to follow his passion for research, he was an
entrepreneur for 30 years. His pursuit of useful knowledge has been a
common thread through his career.

The Scientiﬁc Method
A Guide to Finding Useful Knowledge
J. Scott Armstrong
Wharton School, University of Pennsylvania
Kesten C. Green
University of South Australia

University Printing House, Cambridge CB2 8BS, United Kingdom
One Liberty Plaza, 20th Floor, New York, NY 10006, USA
477 Williamstown Road, Port Melbourne, VIC 3207, Australia
314–321, 3rd Floor, Plot 3, Splendor Forum, Jasola District Centre,
New Delhi – 110025, India
103 Penang Road, #05-06/07, Visioncrest Commercial, Singapore 238467
Cambridge University Press is part of the University of Cambridge.
It furthers the University’s mission by disseminating knowledge in the pursuit of
education, learning, and research at the highest international levels of excellence.
www.cambridge.org
Information on this title: www.cambridge.org/9781316515167
DOI: 10.1017/9781009092265
© J. Scott Armstrong and Kesten C. Green 2022
This publication is in copyright. Subject to statutory exception
and to the provisions of relevant collective licensing agreements,
no reproduction of any part may take place without the written
permission of Cambridge University Press.
First published 2022
A catalogue record for this publication is available from the British Library.
ISBN 978-1-316-51516-7 Hardback
ISBN 978-1-009-09642-3 Paperback
Cambridge University Press has no responsibility for the persistence or accuracy
of URLs for external or third-party internet websites referred to in this publication
and does not guarantee that any content on such websites is, or will remain,
accurate or appropriate.

Michael J. Mahoney
(1946–2006)
Michael J. Mahoney dropped out of high school and worked as a psychiatric aide until
his allergist encouraged him to relocate to Arizona. He received an undergraduate
degree at Arizona State University in 1969 and, later, a PhD in psychology from Stanford.
He joined the faculty at Penn State University in 1972, where he remained until 1985.
Following his time at Penn State, he taught at the University of California, Santa Barbara,
for 5 years then, in 1990, he moved to the University of North Texas where he remained
for 15 years.
Michael was a skeptic and used his skepticism to study current scientific beliefs via
experimentation. He was creative in finding important problems to study. For example,
when he took up weightlifting, he wrote papers on sports psychology. He also became a
national champion weightlifter and served as a resident psychologist to the US
weightlifting team in preparation for the 1980 Moscow Olympic Games.
Michael was, as far as we are aware, the first scientist to undertake a program of
research in which scientists were the subjects. His book, Scientist as Subject: The
Psychological Imperative, published in 1976, led the way to the scientific study of
scientific practices.
Scott became interested in Michael’s research early on. It inspired his first paper on
scientific practice, “Advocacy and Objectivity” (Armstrong, 1979). Although they did
communicate with each other by mail and email, they never met, to Scott’s regret. Scott
regards Michael as the person who had the greatest influence on his work on the
scientific method.

Michael died on May 31, 2006 at the age of 60, in the prime of his career. He had
published more than 250 scholarly articles in psychology, authored and edited 19 books,
was editor of four journals, and served on 23 editorial boards.1
He made many important
scientiﬁc ﬁndings and helped others with their research.
Rest in peace, Michael. You were fearless in your search for truth.
1
Biographical details drawn from Bandura (2008), Marquis et al. (2009), Smucker (2008),
and Warren (2007).

CONTENTS
List of Tables and Checklists x
Foreword by Vernon L. Smith xi
Acknowledgments xvi
Who Is This Book For? xviii
Authors’ Oath for The Scientific Method xx
1 Introduction 1
1.1 Plan of This Book 2
1.2 Scientific Method versus Scientist Opinion 3
1.3 Objective of the Scientific Method 5
1.4 Objectives of Scientific Practice Subverting Science 5
1.5 Operational Guidelines for Scientific Practice 7
2 Defining the Scientific Method 10
2.1 An Aspirational Definition 10
2.2 Criteria for Complying with the Scientific Method 12
3 Checklist for the Scientific Method 24
3.1 Development of the Compliance With Science Checklist 24
3.2 Using the Checklist: For What, How, and by Whom 28
3.3 Not All Checklists Are Useful 29
3.4 Ensuring That Checklists Are Used 30
4 Assessing the Quality of Scientific Practice 31
4.1 How We Reviewed Evidence on Research Practice 32
4.2 Do Journal Papers Help Readers to Make Better Decisions? 33
4.3 More Replications of Important Papers Needed 34
4.4 Are Papers in Scientific Journals Replicable? 35
4.5 How Much of a Concern Is Cheating? 40
4.6 How Efficient Is Research? 42
5 Scientific Practice: Problem of Advocacy 44
5.1 Unnaturalness of Objectivity 44
5.2 Confirmation Bias 47
5.3 How Advocacy Is Practiced 48
5.4 Prevalence and Acceptance of Advocacy 57

6 Scientific Practice: Problem of Journal Reviews 61
6.1 Failure to Find Errors 63
6.2 Advocacy by Reviewers and Editors 63
6.3 Unreliable Reviews 65
6.4 Failure to Improve Papers 65
6.5 Fooled by Fraudulent Reviews and Bafflegab 66
6.6 Distracted by Statistical Significance Tests 68
6.7 Blind to Importance of Authors’
Previous Contributions 70
6.8 Long Delays in Publication 71
7 Scientific Practice: Problem of Government Involvement 72
7.1 Being Useful 72
7.2 Funding 74
7.3 Regulation 83
8 What It Takes To Be a Good Scientist 88
8.1 General Mental Ability 88
8.2 Family Propensity 89
8.3 Early Desire to Do Scientific Research 90
8.4 Personality 91
8.5 Motivated by Intrinsic Rewards 91
8.6 Self-Control 92
8.7 Skepticism 95
8.8 Your Decision 97
8.9 Navigating a Doctoral Program 98
8.10 Acting as a Scientist in all Relevant Roles 98
9 How Scientists Can Discover Useful Knowledge 100
9.1 Identifying Important Problems 100
9.2 Selecting a Problem 105
9.3 Designing a Study 109
9.4 Collecting Data 117
9.5 Analyzing Data 119
10 How Scientists Can Disseminate Useful Findings 121
10.1 Writing a Scientific Paper 121
10.2 Disseminating Findings 128
11 How Stakeholders Can Help Science 139
11.1 Universities 139
viii / Contents

11.2 Scientiﬁc Journals 147
11.3 Governments, Regulators, and Courts 160
11.4 Media and Interested Groups and Individuals 162
12 Rescuing Science from Advocacy 164
12.1 From Where Should Funding Come? 165
12.2 Advocacy 166
12.3 Bureaucracy and Irrelevant Incentives 166
12.4 How Stakeholders Can Contribute 167
Afterword by Terence Kealey 170
References 176
Index 201
ix / Contents

TABLES AND CHECKLISTS
Tables
3.1 Potential users and uses of the Compliance With Science
Checklist page 26
5.1 Prospect theory’s compliance with science 49
5.2 Percentage of psychology studies compliant with science 58
5.3 Advocacy studies’ compliance with science 59
Checklists
3.1 Compliance With Science Checklist page 26
8.1 Self-assessment of self-control 94
9.1 Identifying important problems 101
9.2 Conducting a useful scientific study 106
10.1 Content of a scientific paper 122
10.2 Writing a scientific paper 129
10.3 Disseminating useful scientific findings 130
10.4 Preparing a talk on scientific findings 136
10.5 Making an oral presentation 137
11.1 Elements of a structured abstract 149

FOREWORD
by Vernon L. Smith
I found this book to be a particularly engaging and useful
treatment of scientific method and practice whereby the authors’ target
is to improve practice. There is a lot of subversion out there that they
want to avoid with a content-positive approach. The book’s themes for
improving scientific practice are generated from earthy checklists, all
concerned to improve compliance with the substantive content of sci-
ence. It results in checklists for eleven user categories, from researchers
to courts.
The checklists all derive from key elements of the scientific
method, summarized by eight criteria. I want to apply seven of these
elements to my early work in experimental economics that will help me
to see how this book can help you, which is its purpose.
Study Important Problems
My first experiments were designed to explore questions related
to whether, how, and if buyers and sellers in non-durable goods and
service markets were well-represented by standard supply and demand
(S&D) theory (Smith, 1962). In trying to teach principles I realized that
we economists knew nothing about the relationship between S&D and
what people do in markets. The trading procedure I used was the oral-
outcry two-sided auction, common in securities and commodity
markets, because it appeared that those markets were perceived as
highly competitive and likely to be good beginning models for testing
the theory.

Build on Prior Knowledge
I built upon prior knowledge of the market experiments
reported by Chamberlin (1948), an example of which I had participated
in as a graduate student. I saw limitations in the design and sought to
combine that prior knowledge with independent knowledge of trading
procedures used in open outcry markets. Chamberlin was much influ-
enced by Alfred Marshall in applying the concept of reservation price to
define willingness to pay (or accept) to buyers (sellers) in the experiment
and that carried over into my work. Marshall’s modelling of S&D as
flows into and out of a consumption market led naturally to repeat
trade over time in my design.
What knowledge, how extensive, and when should you acquire
and build on it? My style is to do the experimental designs and work,
based on the motivating ideas, before examining the literature. Either
there is a literature or there is not. If not, it will make little difference. If
there is, review the literature when it comes time to write up your work
and findings. It is then that you know better what to search for and,
more importantly, your work will be independent. Very likely it will be
conceived, designed, and motivated differently, constituting a richer
contribution that can still be extended in ways suggested by
previous results.
Here is a true story on prior knowledge. Roy Radner was
working jointly with a probability theorist on a project. They had a
draft of a paper finished, and Roy’s co-author said he thought it was
ready to submit. Roy said, not quite, as they needed to investigate the
literature. Roy’s co-author said that he would do it, as that would
provide him with a good learning opportunity. After a period, the co-
author returned saying that this was a remarkable literature. Why?
Every paper begins with another paper, not with a problem of the
world.
Use Objective Designs
This criterion was satisfied by assigning private values to
buyers, and costs to sellers that provided well-defined S&D conditions
prior to running each experiment. Eventually, I used other people’s
classes for subjects besides my own classes and prevailed on others to
xii / Foreword

run experiments. To control for economic understanding, all experi-
ments were run on the first day of economics classes before any text
assignments, lectures, or discussion.
I had no experience or background in economics’ experiments,
as it was not yet a field in economics, but I had studied physics and
astronomy, and knew of some of the great experiments in science,
giving me a sense of scientific method. Beyond that, common sense
seemed to be a good guide.
With those foundations, I designed experiments that could
produce findings that would either challenge or support central hypoth-
eses about the operation of markets. For example, competitive equilib-
ria turned out not to require large numbers of sellers and buyers.
Provide Full Disclosure
Although I provided complete narrative descriptions of the
procedures, it is not clear that this was sufficient for a reader to know
how to replicate an experiment. Interestingly, in my second paper the
experimental instructions were included in an appendix, so I corrected
that error (Smith, 1964).
Use Valid and Reliable Data
For experiments, that means replicate. But none of the original
experiments were literally replicated. Motivation for each design and
test was followed by only one experiment. Yet the results were startling.
Overall, across the experiments, there was a strong tendency to con-
verge to the equilibrium specified by the prior S&D. Hence, the diversity
of designs constituted a test of the robustness of equilibrium. The
convergence pattern of results, highly variable in terms of convergence
paths, stood out prominently relative to variability across experiments.
Use Valid Simple Methods
Simplicity of design allowed the convergence pattern across a
diversity of experiments to stand in bold relief. But that outcome was
neither anticipated, nor was it an intentional design feature. The experi-
ments were all simple, and the qualitative results transparent in
xiii / Foreword

demonstrating convergence. One experiment, with perfectly elastic
supply and downward sloping demand failed to converge. It was repli-
cated, using cash payoffs, plus a “commission” to provide a minimum
profit for each trade. The replication with cash incentives converged
(Smith, 1962, chart 4 and n. 9), which provided early proof of the
importance of adequate subject motivation. Subsequently all my experi-
ments paid each subject cash in proportion to profits earned in the
experiment.
Objective Designs: Testing Multiple Hypotheses
The first experiments strongly supported convergence. But is
there a quantitative rule or law of convergence operating across the
experiments? The authors of this book emphasize the importance of
testing multiple reasonable hypotheses as the key to achieving objectiv-
ity in experiments. Two hypotheses were prominent in the microeco-
nomic literature: (1) Walras postulated that prices rise in proportion to
excess demand, fall in proportion to excess supply; (2) Marshall, mod-
eling firm entry, postulated that output from firm entry increased in
proportion to the excess of demand price over supply price – a point
estimate of profit or loss.
The experiment transaction price histories seemed not to sup-
port either (1) or (2); rather a third hypothesis worked much better.
Compute V(pt), the area under the demand, and above the supply
curves, at any price pt). The difference ER = V(pt) – V(p*), where
p* is the equilibrium price, appeared to be a better predictor of price
at pt+1. Buyers (sellers) were cutting prices to avoid loss from failing to
contract. Moreover, V is minimized at p*, so the process was efficient.
New experiment designs, with excess demand constant at all prices pt,
predicted exponential price decay if ER was the rule, constant decay if
Walras was right; the data supported ER (Smith, 1965).
Draw Logical Conclusions
The observed pronounced convergence defied prevailing theory,
widely shared expectations, beliefs, and teaching. The idea that market
participants required either complete information or a Walrasian auc-
tioneer to find prices found no support in the results. Undergraduates,
xiv / Foreword

naive in economics, made excellent subjects. Eventually their results
generalized across a rich variety of groups (Smith, 1991). Finally, the
price paths to equilibrium reﬂected price concessions by buyers or sellers
to avoid failure to make contracts.
Vernon L. Smith
Economic Science Institute
Chapman University
2002 Nobel Laureate in Economics
xv / Foreword

ACKNOWLEDGMENTS
We thank Joel Kupfersmid, Brian Martin, Frank Schmidt, and
Stan Young for reviewing the entire book, and William H. Starbuck for
providing useful guidance during the final stages of our book.
Dennis Ahlburg, Hal Arkes, Peter Ayton, Jeff Cai, Nathan
Cofnas, John Dawes, John Dunn, Lew Goldberg, Anne-Wil Harzing,
Ray Hubbard, Nick Lee, Jay Lehr, Gary Lilien, Byron Sharp, Karl
Teigen, and Malcolm Wright reviewed sections of the book that were
relevant to their expertise.
Harrison Beard, Len Braitman, Heiner Evanschitzky, Bent
Flyvbjerg, Shane Frederick, Gerd Gigerenzer, Andreas Graefe, Jay
Koehler, David Legates, Justin Pearson, Don Peters, Paul Sherman,
and Arch Woodside made useful suggestions.
We also thank the authors that we cited for substantive findings
for checking and improving our summaries of their findings.
Amy Dai and Esther Park – Scott’s research assistants – and
Charles Green did an excellent job of obtaining and analyzing ratings of
papers using the checklists that we have developed for this book, and
helped us to improve the clarity of our writing.
Jonathan Ho volunteered to assist us full time during the
summer of 2019 and continued to provide help during 2020.
We are grateful to our copy editors, Hester Green, Scheherbano
Rafay, Lynn Selhat, and Lisa Zou.
The University of Pennsylvania Library provided much support.
Their Document Delivery group was able to track down papers for us,

suggest relevant papers, and ensure that our citations were
properly formatted.
Scott’s wife, Kay Armstrong, has reviewed nearly all of his
books and writings over his career. No matter how many reviewers
have provided reviews before her, she ﬁnds many ways to make further
improvements. She has done it again for this book.
xvii / Acknowledgments

WHO IS THIS BOOK FOR?
We wrote this book to help researchers make better use of the
scientific method and to write papers and books describing their useful
scientific findings in ways that can be understood by the widest
relevant audience.
Our book is also intended as a resource for all other stakehold-
ers in science. If you are reading this and are not already a scientist, we
expect that you will belong to one or more of the following groups:
People who are considering a career as a scientist, to determine if they
are, in fact, well suited for such a career;
Employers of scientists, to help them make hiring decisions;
PhD students, to demonstrate that they can comply with the scientific
method;
Journal editors, to increase the publication of papers that comply
with the scientific method;
Journal reviewers, to assess the extent to which a paper complies with
the scientific method;
Government regulatory agencies, to assess whether current or pro-
posed regulations are, or will be, effective;
Policy makers in government or private corporations, to assess evi-
dence on alternative policies;
Courts, to assess evidence from expert witnesses;

Consumers, to better understand the evidence for product claims,
such as when evaluating the efﬁcacy of a medical treatment;
Journalists and reporters, to inform readers about the extent to which
a given study complies with science;
Citizens, to analyze evidence for proposed government policies.
xix / Who Is This Book For?

AUTHORS’ OATH FOR THE
SCIENTIFIC METHOD
(1) We, one or both of us, have read each publication that we cite for a
substantive finding.
(2) We attempted to contact the authors we cited for substantive find-
ings to help ensure that we accurately described their findings, and
to determine if we omitted relevant scientific papers, especially those
with scientific evidence that conflicted with our findings.
(3) We used the Criteria for Compliance with the Scientific Method
from our book, and believe the book is compliant with the
scientific method.
(4) Voluntary disclosure: We received no external funding for writing
this book and have no conflicts of interest.

1 INTRODUCTION
The scientific method is largely responsible for improving life
expectancies and the quality of life over the past 2000 years. Individual
scientists, in their efforts to discover how things work and how to make
them better have used the method on their own or in collaboration with
others to make the world a better place.
We believe that there is no way to improve upon the scientific
method. Our aim for this book, then, is to help improve scientific
practice.
The message of our book is a positive one. The scientific method
has worked and does continue to work, to the great benefit of all of us.
While there are considerable problems with the current practice of
science in many fields – which we describe at some length – we provide
a plan for overcoming those problems.
Scientists have long been concerned about problems with scien-
tific practice, and leading scientists have long made recommendations
for the practice of science. Books by philosophers of science – Karl
Popper’s The Logic of Scientific Discovery (1959; see also Thornton,
2018), and Thomas Kuhn’s The Structure of Scientific Revolutions
(1962; see also Bird, 2018) in particular – also spurred interest in
scientific practice.
Then something happened. Michael J. Mahoney published a
book – Scientist as Subject (1976) – describing the findings of his
experimental research on scientific practices. As Mahoney had quickly
learned, scientists were not fond of being experimental subjects.

They claimed that he was behaving unethically by not revealing to them
that they were participating in an experiment.
Despite the backlash from scientist subjects, the number of
papers describing experimental research on scientific practice since
Mahoney (1976) has grown at an increasing rate. We draw on those
papers to describe failings in current scientific practice, and to provide
solutions that are based on scientific research.
Our solutions – recommendations for improving scientific prac-
tice so that it complies more with the scientific method – are for
scientists, and for other stakeholders in the accumulation of scientific
knowledge.
Central to our solutions is the Compliance With Science
Checklist (Chapter 3). All stakeholders can use the checklist. We also
provide nine other checklists, one of which is intended for those con-
sidering a career as a scientist, and the remaining seven are intended to
help practicing scientists with aspects of their role, such as writing a
scientific paper.
We believe that the checklists we provide will be useful to all
scientists, from PhD students and early career researchers to emeritus
professors. We use the term researcher interchangeably with scientist
throughout the book.
Before we continue, here is a caution for scientists and science
stakeholders, delivered in the Nobel Prize lecture by Friedrich von
Hayek:
Yet the confidence in the unlimited power of science is only too
often based on a false belief that the scientific method consists in
the application of a ready-made technique, or in imitating the
form rather than the substance of scientific procedure, as if one
needed only to follow some cooking recipes to solve all social
problems. It sometimes almost seems as if the techniques of
science were more easily learnt than the thinking that
shows us what the problems are and how to approach them.
F. A. Hayek (1974)
1.1 Plan of This Book
We survey the current state of scientific practice in Chapters 4
through 7. Chapter 4 (Assessing the Quality of Scientific Practice)
2 / Introduction

describes how we reviewed the evidence on practice and sets the scene.
Chapter 5 examines the problem of advocacy, Chapter 6 the problem of
mandatory journal peer reviews, and Chapter 7 the problems created by
government funding and regulating research.
In Chapter 8, we describe research findings on what it takes to
be a good and useful scientist.
We then describe solutions in Chapters 9–11. Chapter 9 pro-
vides guidance for scientists on how to discover useful scientific know-
ledge. Chapter 10 provides guidance on how to write a scientific paper
to best communicate useful research findings, and on how to dissemin-
ate those findings to the widest relevant audience. Chapter 11 provides
guidance on how stakeholders can help science in four sections devoted
to university managers, scientific journal editors, governments and
courts, and media and interested individuals respectively.
Chapter 12 summarizes how our checklists and guidance pro-
vide a plan for reforming scientific practice so that the practice of
science matches the scientific method. We expect great benefits to flow
from reducing unscientific practices and increasing the adoption of the
scientific method throughout the research process from the production
to the consumption of useful scientific knowledge.
Finally, an Afterword by Terence Kealey – author of The
Economic Laws of Scientific Research (1996) – provides a fascinating
account of the history of science that revolves around the question
“How do we know this statement is true?” He reinforces and expands
on the conclusion of this book that the increasing involvement of
governments in research from the mid-twentieth century has diverted
science from its true role as an engine for discovering useful truths, and
endorses our recommendations for returning the scientific endeavor to
its true path.
1.2 Scientific Method versus Scientist Opinion
The opinions of scientists – even those of the most eminent –
should not be confused with knowledge obtained from the application
of the scientific method. Scientists’ opinions are examples of the logical
fallacies of appeal to authority and – when the opinion is shared by a
group of scientists – argumentum ad populum.
Not surprisingly, then, scientists’ opinions about the way
things are or will be have not held up well against reality. Cerf and
3 / Scientific Method versus Scientist Opinion

Navasky’s (1998) collection of scientists’ and other widely respected
experts’ opinions, The Experts Speak: A Definitive Compendium of
Authoritative Misinformation, is a thick book (445 pages) with many
examples. Here is one: “Heavier-than-air flying machines are impos-
sible” (Lord Kelvin, British mathematician, physicist, and President of
the British Royal Society, 1895.)
Some of the Cerf and Navasky examples are hilarious, if one
ignores the harm that they caused when they were taken seriously.
There is an abundance of what seem now to have been idiotic prognosti-
cations, supporting George Orwell’s observation that “One has to
belong to the intelligentsia to believe things like that: no ordinary man
could be such a fool” (Orwell, 1945, para 29). The book was suffi-
ciently successful that the authors were encouraged to put out a revised
edition in 1998 with more examples of incorrect, but highly confident,
expert opinions.
An experiment to assess the value of expert judgments was
conducted over a 20-year period (Tetlock, 2005). The 284 experts
who participated were asked to assess the probabilities of various events
occurring for situations in the future. The experts were people whose
professions included “commenting or offering advice on political and
economic trends.” By 2003, Tetlock had accumulated 82,361 forecasts.
He then evaluated the experts’ judgments against the outcomes, and
against predictions from simple statistical procedures, uninformed non-
experts, and well-informed non-experts. The experts barely, if at all,
outperformed the informed non-experts and none of the groups did well
against simple rules and models. (Tschoegl and Armstrong, 2007,
reviewed the book.)
Scott had previously reviewed experimental evidence on experts’
judgmental predictions. The review led him to develop his Seer-Sucker
Theory: “No matter how much evidence exists that seers do not exist,
suckers will pay for the existence of seers” (Armstrong, 1980a).
When knowledge about a situation is at best tentative, scientists
nevertheless can and do use their perceived authority to promote theor-
ies in the hope of persuading voters, government officials, and political
leaders that there is a problem and that government actions that accord
with their opinions are needed. The approach, which is an embodiment
of the “precautionary principle,” has been called “post-normal science”
(Ravetz, 2004). The precautionary principle is an anti-scientific political
principle that, in the absence of objective cost-benefit analyses, is used to
4 / Introduction

call for drastic government actions in response to some scientists’
opinions that bad things will happen otherwise (Green and
Armstrong, 2008).
1.3 Objective of the Scientific Method
Benjamin Franklin believed that universities should be centers
for scientific research. When he founded what is now known as the
University of Pennsylvania, he suggested that faculty be involved in the
“discovery and dissemination of useful knowledge” (Franklin, 1743).
We believe that his suggestions should be the objective of all scientific
research, and the yardstick against which it is judged.
Other pioneers of science professed a similar preference for
usefulness or importance, as we discuss in Chapter 2 (Defining the
Scientific Method). But while the scientific method is efficient for
making useful discoveries – because it is designed to identify the hypoth-
eses that best accord with reality – it is up to scientists to identify the
problems that are most likely to lead to useful discoveries, and that they
can best help with.
1.4 Objectives of Scientific Practice Subverting Science
Outside of the business world, current procedures for the evalu-
ation of researchers’ contributions provide little to encourage them to
achieve the objective of discovering useful knowledge. Instead of assess-
ing the usefulness of scientists’ research findings, their employers use
proxy measures such as the number of papers published in “high-
quality” journals, citation counts, and dollars of grant money received.
The outcome of that approach is consistent with Campbell’s
Law: “The more any quantitative measure is used for social decision-
making, the more subject it will be to corruption pressures and the more
apt it will be to distort and corrupt the social processes it is intended to
monitor” (Campbell, 1979, p. 85). For example, researchers are motiv-
ated to divide a research project into a series of papers, and to include as
co-authors people with little or no involvement in the research project
on the understanding that the favor will be reciprocated. The Economist
(2016) examined more than 34,000 papers listed on Scopus between
1996 and 2015 and found that the average number of authors per paper
grew from 3.2 to 4.4.
5 / Objectives of Scientific Practice Subverting Science

While the proxy measures may have been reliable indicators of
useful scientific findings when they were first adopted, they are no
longer so.
Citation analysis began in 1961 when Eugene Garfield began
publication of the Science Citation Index (SCI) in Philadelphia. The Social
Science Citation Index (SSCI) followed. These provide a valuable service
for scientists who are searching for prior knowledge in their area of study.
Since then, there has been a phenomenal increase in the number
of citations in all fields of science. For example, when Scott joined The
Wharton School in 1968, the author with the most citations in the field
of marketing, Paul Green, received 100 citations in some years. He was
one of the most renowned researchers in marketing of his time. Things
are vastly different now. As at September 2019, one researcher in
marketing had received more than 15,000 citations in one year.
The number of citations that a paper receives might provide an
indicator that it has no scientific value when other scientists fail to cite it.
But even then, a paper that challenges the current orthodoxy in a field
might be ignored regardless of its contribution to scientific knowledge.
The vast majority of papers are not cited in any substantive
manner. For example, Armstrong and Overton (1977) developed a
simple and effective way to estimate non-response bias in mail surveys.
The paper had been cited more than 15,000 times as of early
2020 according to Google Scholar. Yet, in a sample of 50 papers in
the leading journals that cited the Armstrong and Overton procedure,
only one correctly represented the procedure. Most of the citations used
it to support their own incorrect procedures for dealing with non-
response bias, suggesting that the authors of those papers had not even
read the Armstrong and Overton paper. In addition, many of the
citations made mistakes in the references, such as incorrect spellings,
which were often identical to the mistakes made by other authors who
had cited the paper (Wright and Armstrong, 2008).
Do those who cite papers regard them as useful? Apparently
not. An analysis of 12 high-profile scientific papers estimated that about
70–90 percent of cited papers had not been read by those who cited
them (Simkin and Roychowdhury, 2005).
Fire and Guestrin (2019) analyzed more than 120 million papers
in 2,600 fields, with an emphasis on the field of biology. They concluded
that, “citations are not beneficial for comparing researchers . . . even in
the same department” (p. 76).
6 / Introduction

Citations of papers that have been refuted or challenged, or
retracted, contribute to the authors’ citation counts even though they
are detrimental to science. For example, a study of biomedical papers
from 1966 through August 1997 found that 235 papers had been
retracted. These retracted papers were cited 2,034 times, after being
retracted. On all but 19 occasions, the citation was treated as a valid
study (Budd et al., 1998).
Another problem is that many are “mysterious citations.” That
is, the authors do not explain what findings the papers are being cited
for, nor how they were discovered. We provide guidance on avoiding
mysterious and unnecessary citations in Chapter 10 (How Scientists
Can Disseminate Useful Findings) and Chapter 11 (How Stakeholders
Can Help Science).
We discuss the corrupting incentives resulting from the involve-
ment of governments in research via grant funding and regulation in
Chapter 7 (Scientific Practice: Problem of Government Involvement),
and solutions to that problem in Chapter 11 (How Stakeholders Can
Help Science).
1.5 Operational Guidelines for Scientific Practice
In this book, we develop guidelines for implementing the scien-
tific method. To do that, we first derived a list of criteria for complying
with the scientific method from descriptions provided by the founders of
the method. We then translated those criteria into a checklist of oper-
ational guidelines that can be used to determine the extent to which a
research paper complies with the scientific method.
We argue that the criteria should be generally acceptable given
that they are based on those proposed by the scientists who originated
the scientific method, and the success in generating useful knowledge
that following the criteria has had across 21 centuries.
1.5.1 Previous Attempts at Guidelines for Science
In the past, various disciplines developed guidelines for scien-
tific practice based on a consensus of the opinions of scientists working
in the field. An early instance was the Operations Research Society of
America’s “Guidelines by the Ad Hoc Committee on Professional
Standards” (ORSA, 1971). Guidelines have also been developed for
7 / Operational Guidelines for Scientific Practice

medical research and clinical practice, including GRADE (Guyatt et al.,
2008), CONSORT (Moher et al., 2010; Schulz et al., 2010), and
SQUIRE (Davidoff et al., 2008).
For 70 years, the US Supreme Court followed the Frye standard
in assessing scientific evidence. That standard required courts to follow
the “generally accepted opinions of scientists” – another consensus-
based standard.
The court’s consensus standard changed following the 1993
Daubert v. Merrell Dow case, in which the Supreme Court of the
United States unanimously replaced the Frye approach in favor of
assessing evidence on the basis of whether it was the product of “scien-
tific procedures.” To date, about half of US state courts have adopted
the Daubert approach.
To help the courts implement the Daubert standard, descriptions
of scientific procedures have been distributed to all federal judges in the
Reference Manual on Scientific Evidence (Breyer et al., 2011). By 2011, the
third edition contained over one thousand pages. The Daubert standard
has had great effect on the legal system in the United States and, according
to some lawyers, has led to better judgments (e.g., Faigman, 2013).
The Daubert standard does, however, depend upon the authority
of selected experts to choose the proper procedures. In addition, consensus
on what are the appropriate scientific procedures may change over time as
better procedures are discovered. Scientists must, therefore, keep up to date
with the development of procedures and evidence on their validity.
1.5.2 Guidelines Necessary, but Not Sufficient
The development of guidelines, while necessary, is not suffi-
cient, as a number of studies have found. One study examined six
different sets of guidelines for medical research: The authors concluded
that the “implementation of these guidelines has led to only a moderate
improvement in the quality of the reporting of medical research”
(Johansen and Thomsen, 2016).
1.5.3 Mandated Checklists Necessary
The only way we know to ensure compliance with guidelines is
the required and monitored use of an operational checklist of
the guidelines.
8 / Introduction

The effectiveness of monitored checklists has been well docu-
mented. For example, a review of 15 experimental studies in health care
found that validated checklists led to substantial improvements in
patient outcomes. One of the experiments examined the application of
a 19-item checklist for a surgical procedure that was performed on
thousands of patients in eight hospitals around the world. Use of
the checklist reduced mortality rates at those hospitals by half
(Haynes et al., 2009).
Checklists are especially effective when people know little about
the relevant scientific principles. For example, advertising novices were
asked to use a checklist with 195 validated persuasion principles to rate
96 pairs of advertisements. By using the checklist, they made 44 percent
fewer errors than did unaided novices in predicting which advertise-
ments were more effective (Armstrong et al., 2016).
Checklists also help when the users are aware of proper pro-
cedures. For example, an experiment on infection prevention in the
intensive care units of 103 Michigan hospitals required physicians to
follow five well-known guidelines for inserting catheters. Use of the
checklist reduced infection rates from 2.7 per 1,000 patients, to zero
after three months (Hales and Pronovost, 2006).
Users should confirm that they have implemented each item of a
comprehensive checklist, and the use of the checklist should be moni-
tored. In some fields, such as engineering, aeronautics, and medicine,
failure to follow operational, agreed-upon checklists can be used by
courts to assign blame for bad outcomes. In some cases, the failure to
complete a checklist can be grounds for dismissal of an employee.
9 / Operational Guidelines for Scientific Practice

2 DEFINING THE SCIENTIFIC METHOD
The invitation for those nominating candidates for the Nobel
Prize in economics, the “Sveriges Riksbank Prize in Economic Sciences
in Memory of Alfred Nobel,” described the award of the prize as being
“based solely on scientific merit.” No criteria for judging scientific merit
were provided, but nominators were directed to “consider origin and
gender” of the nominees. Without clear criteria for the award, to what
extent can one be confident that the prize was based on the scientific
merit of the findings?
In this chapter we provide an aspirational definition of the
scientific method. The definition is in the form of eight criteria that are
based on the writings of key figures in the development of the scientific
method. We then expand on each of the criteria, describing their
source – where appropriate – and the reasons for their importance for
the scientific method.
2.1 An Aspirational Definition
We sought to define the scientific method in such a way that
most researchers should aspire to the ideal the definition represents. To
do so, we turned to the writings of the developers of the scientific
method. Scientists have been describing elements of the scientific
method since before 400 BC. White (2002) concluded that the modern
scientific method owes its approach to the logical framework of hypoth-
esis testing laid out by Socrates, with later refinements by Plato and
Aristotle. Socrates in effect set out the basis of a valid approach

to seeking knowledge that scientists still use – the use of experiments,
which came to be formally recognized as important much later, excepted.
We concluded that the key elements of the scientific method – as
derived from the words of famous and pioneering scientists – could be
summarized by eight criteria:
1. Study important problems
2. Build on prior knowledge
3. Provide full disclosure
4. Use objective designs
5. Use valid and reliable data
6. Use valid simple methods
7. Use experimental evidence
8. Draw logical conclusions
These criteria are also consistent with the Oxford English Dictionary
(OED), which defines the scientific method as:
commonly represented as ideally comprising some or all of (a)
systematic observation, measurement, and experimentation, (b)
induction and the formulation of hypotheses, (c) the making of
deductions from the hypotheses, (d) the experimental testing of
the deductions, and (if necessary) (e) the modification of the
hypotheses . . . The modern scientific method is often seen as
deriving ultimately from Francis Bacon’s Novum Organum
(1620) and the work of Descartes. In the 20th century, Karl
Popper’s idea of empirical falsification has been important.
OED Online (2018).
In practice, a study can contribute to making a useful scientific discovery
even when it does not on its own comply with all of the criteria. For
example, Einstein drew on the findings of others’ experiments to
develop novel hypotheses about important problems that could in turn
be tested against alternative hypotheses by further experiments.
Papers might also contribute to science by identifying important
problems. Others might contribute by identifying shortcomings in the
papers of other researchers and resolving those issues. Another contri-
bution is to develop objective measures of important variables, and
compile data using those measures, as has been done by scientists at
the University of Alabama at Huntsville in estimating global average
temperatures from satellite readings (Spencer et al., 2017).
11 / An Aspirational Definition

While studies that fall short on some criteria – e.g., by over-
looking prior knowledge – might nevertheless turn out to provide a
useful contribution to research on a problem, studies that failed to use
an objective design (criterion 4) are unlikely to do so. In order to claim
that a principle or method is scientific, studies of the problem would,
when taken together, need to satisfy all eight criteria.
We consider that the support of meta-analyses of objective
studies that collectively comply with all eight criteria for science are
necessary for rational policy making. The requirement is particularly
important for government laws and regulations, because they involve
duress rather than voluntary transactions.
2.2 Criteria for Complying with the Scientific Method
We now expand on the eight criteria for complying with the
scientific method that we described above.
2.2.1 Study Important Problems
According to the general spirit of this book, which values everything in
its relation to Life, knowledge which is altogether inapplicable to the
future is nugatory.
Charles Sanders Peirce
(1958, para 56)
Scientists in the past sought to address important problems. Robert
Boyle, a founder of the English Royal Society, wrote in 1646 that
the founders valued “no knowledge but that it has a tendency to use”
(as quoted by O’Connor and Robertson, 2004).
Some scientists argue that research that does not obviously lead to
useful findings is nevertheless important because of potential future useful-
ness. While that may turn out to be true in some cases, identifying problems
that are currently in need of solutions to research is more likely to produce
useful findings than is research based on curiosity about a non-problem.
Addressing currently pressing problems can lead, and has led,
to advances in scientific knowledge that go well beyond finding solu-
tions to those problems, as the following quotation illustrates.
[T]he practical sciences incessantly egg on researches into
theory. For considerable parts of chemical discovery we have
12 / Defining the Scientific Method

to thank the desire to find a substitute for quinine or to make
quinine itself synthetically, to obtain novel and brilliant dye-
stuffs, and the like. The mechanical theory of heat grew out of
the difficulties of steam navigation. For it was first broached by
Rankine while he was studying how best to design marine
engines. Then again, one group of scientists sometimes urges
some overlooked phenomenon upon the attention of another
group. It was a botanist who called van’t Hoff’s attention to the
dependence of the pressure of sap in plants upon the strength of
the solution, and thus almost instantaneously gave a tremen-
dous impulse to physical chemistry. In 1820, Kästner, a manu-
facturer of cream of tartar in Mulhouse, called the attention of
chemists to the occasional, though rare, occurrence in the wine
casks of a modification of tartaric acid, since named racemic
acid; and from the impulse so given has resulted a most import-
ant doctrine of chemistry, that of the unsymmetric carbon
atom, as well as the chief discoveries of Pasteur, with their far-
reaching blessings to the human species. Charles Sanders Peirce
(1958, para 52)
If research on relatively narrow current problems can lead the curious
scientist to such widely important discoveries as are described in the
quotation from Peirce (1958) above, the case for studying non-problems
at someone else’s expense seems weak when researcher time is a limited
resource. Of course, if there is a willing well-informed funder for such
activity, including self-funding, then that is the business of the parties
concerned, and good luck to them.
2.2.2 Build on Prior Knowledge
Progress in science requires that scientists become familiar with
prior knowledge and methods for the given problem. Newton (1675)
referred to the process as “standing on the shoulders of giants.”
Despite the logical necessity of doing so, researchers often fail to
comprehensively review the existing evidence, perhaps because doing so
greatly increases the time needed to complete a publication. Because the
reviewers used by journal editors are often unaware of relevant prior scien-
tific findings, an author’s failure to identify relevant prior research can go
undetected. As a consequence, researchers are prone to making rediscoveries.
13 / Criteria for Complying with the Scientific Method

In one example, Kahneman (2011) concluded that people pro-
cess information differently depending on the nature of the decision. He
referred to the phenomenon as “slow versus fast,” or “System 1” and
“System 2” decision-making. His was at least the third discovery of the
concept. In 1913 it was called “short circuit versus long-circuit” think-
ing as described by Hollingworth (1913). Half-a-century later, the
concept was referred to as “low involvement versus high-involvement”
by Krugman (1965). Whatever name the concept is given, it has been an
important condition to consider for persuasion for over a century now.
For more on this, see Armstrong (2010, pp. 21–22).
2.2.3 Provide Full Disclosure
The scientific method depends heavily on replication, and repli-
cation requires full disclosure of methods. Replications are needed to
help determine whether potentially useful scientific findings should be
accepted and acted upon.
A paper that does not provide all necessary information for
replication may, nevertheless, contribute to science if it at least
addresses an important problem. Other researchers can conduct exten-
sions that test the same issue. The extensions can help to allay concerns
about findings that arise when disclosure is incomplete.
2.2.4 Use Objective Designs
The founders whose writings we used to develop the definition of
the scientific method recognized early on that objectivity is hard to achieve.
They also recommended a solution. Sir Isaac Newton, for example,
described four “Rules of Reasoning in Philosophy” in the third edition of
his Philosophiae Naturalis Principia Mathematica (1726, pp. 387–389).
His fourth rule, in Motte’s translation from Latin, states, “In experimental
philosophy we are to look upon propositions collected by general induc-
tion from phenomena as accurately or very nearly true, notwithstanding
any contrary hypotheses that may be imagined, till such time as other
phenomena occur, by which they may either be made more accurate, or
liable to exceptions” (Newton, 1729, vol. 2, p. 205, emphasis added).
We refer to this solution as Multiple Reasonable Hypotheses
Testing, or MRHT. One should include all reasonable hypotheses or

describe why that was not feasible. MRHT stands in contrast to the
approach that has become accepted practice in psychology and the
social sciences: Null Hypothesis Statistical Testing, or NHST.
The increase in productivity that arose from the English
Agricultural Revolution illustrates the importance of MRHT.
Agricultural productivity saw little improvement until landowners in
the 1700s began to conduct experiments comparing the effects of alter-
native ways of growing crops. The Industrial Revolution progressed in
the same manner (Kealey, 1996, pp. 47–89).
Chamberlin (1890) claimed that disciplines that conduct
experiments to test multiple reasonable hypotheses progress greatly,
while those that do not, progress little. Nearly three-quarters of a
century later, Platt (1964) reiterated Chamberlin’s conclusion because
researchers in many ﬁelds of science were still ignoring the original
advice.
MHRT has also led to advances in medical knowledge. For
example, one study examined all papers that used MRHT that were
published in the New England Journal of Medicine from 2001 to 2010
(Prasad et al., 2013). The study found that 146 medical treatment
recommendations were reversed as a consequence of experiments using
MRHT. The reversals amounted to 40 percent of all procedures tested.
MRHT has also led to the growth of useful knowledge in engineering,
forecasting, persuasion, and technology.
2.2.5 Use Valid and Reliable Data
Validity is the extent to which the data measure the concept that
they purport to measure. Validity is not a trivial matter. Many disputes
arise due to differences in how concepts are measured. For example,
what is the best way to measure inequality among people? Is it best
assessed only in terms of money income, or should it also include the
effects of taxes, wealth, transfer payments, home production, etc.?
These measures produce different ﬁndings and policies. More funda-
mentally, should inequality be assessed in terms of life satisfaction
instead of income? Money income is, after all, only one of several means
to achieve the desired end of happiness. People routinely trade off
money income to do work that provides greater intrinsic satisfaction
or to live somewhere that they prefer.

Reliability is established when other researchers, using the same
procedures, can reproduce findings. Reliability can be improved by
using all relevant data that are available such as when using a time-
series. As Sir Winston Churchill said, “The longer you can look back,
the farther you can look forward.”
Data that has been subject to unexplained revisions should not
be used. Enough said.
2.2.6 Use Valid Simple Methods
There is, perhaps, no beguilement more insidious and dangerous than
an elaborate and elegant mathematical process built upon
unfortified premises.
Chamberlin (1899, p. 890)
Validity requires that the method used has been tested and found to be
useful for the problem at hand. Simple methods are those that can be
understood by those who might have an interest in reading or replicat-
ing the paper. Complex methods make it difficult for others to under-
stand the research, spot errors, and replicate the study.
The call for simplicity in science started with Aristotle but is
usually attributed to Occam as “Occam’s Razor.” Yet, academics and
consultants love complex methods. So do their clients. After all, if the
process were simple they would ask, “Why are we paying all that
money?” For a further discussion of why complexity proliferates, see
Hogarth (2012).
The 1976 Nobel Laureate in Economics, Milton Friedman,
stressed the importance of testing the predictive validity of hypotheses
against new, or out-of-sample, observations (1953). Is there a conflict
between predictive validity and simplicity? Apparently not.
Comparative studies have shown the superior predictive validity of
simple methods in out-of-sample tests across diverse problems. The
experiments on the predictive validity of simple alternatives to multiple
regression analysis by Czerlinski et al. (1999), and by Gigerenzer et al.
(1999) are elegant examples.
In our review of the evidence on the predictive validity of
Occam’s Razor, we defined a “simple method” as one for which an
intelligent person could understand: (a) procedures; (b) representation
of prior knowledge; (c) relationships among the elements; and (d)

relationships among models, predictions, and the decisions that might
be made (Green and Armstrong, 2015). We found 32 published studies
that compared forecasts from simple methods with forecasts from more
complex methods that had been proposed by their authors as a way to
improve accuracy. We hired university students to rate complexity
against the simplicity criteria listed above. Simplicity improved out-of-
sample predictive validity in all 32 studies involving 97 experimental
comparisons. On average, complex methods had errors for out-of-
sample predictions that were 27 percent larger for the 25 papers that
provided quantitative comparisons. The strength and consistency of the
findings astonished us and are a caution to researchers who assume that
complex data modelling methods have predictive validity.
2.2.7 Use Experimental Evidence
The testing of the hypothesis proceeds by deducing from it
experimental consequences almost incredible, and finding that
they really happen, or that some modification of the theory is
required, or else that it must be entirely abandoned.
These experiments need not be experiments in the narrow and
technical sense, involving considerable preparation. That prep-
aration may be as simple as it may. The essential thing is that it
shall not be known beforehand . . . how these experiments will
turn out. Charles Sanders Peirce (1958, paras 83, 90)
Experiments emerged as a key element of the scientific method
in the practice of the natural sciences in the sixteenth century.
The importance of experiments was generally not recognized in
medical research and the social sciences until the nineteenth century
(DiNardo, 2018).
Robert Boyle and other scientists established the forerunner of
the modern-day Royal Society around 1645 to acquire knowledge
through experiments. The value the society placed on experiments was
highlighted by the appointment of Robert Hooke as a Curator of
Experiments who was tasked with “furnish[ing] them every day on
which they met with three or four considerable experiments”
(O’Connor and Robertson, 2004). The society translates its Latin
motto, nullius in verba, as “take nobody’s word for it.” It expresses
the Royal Society Fellows’ determination “to withstand domination of

authority and to verify all statements by an appeal to facts determined
by experiment” (Royal Society, 2019).
Experiments can be controlled, quasi-controlled – include some,
but not all, important causal variables – or natural. Laboratory experi-
ments allow for more control over conditions, while field experiments
are more realistic. Interestingly, a comparison of findings from labora-
tory versus field experiments in 14 areas of organizational behavior
concluded that they produced similar findings (Locke, 1986). Vernon
Smith demonstrated that “laboratory” (controlled) experiments can be
used to test competing hypotheses in economics. He found that very
simple experiments could be devised that would replicate the relevant
behaviors of participants in real markets (Smith, 2002).
Experiments have been conducted in fields of science as diverse
as astronomy (e.g. Ostro, 1993, described the use of radar to conduct
experiments on the scale of the solar system and gravitation, among
other things), evolutionary biology (e.g., Schluter, 1994, conducted
experiments to test theories about the effect of resource competition
among species on evolution), geology (Kuenen, 1958, described the use
of experiments in geology starting with those of Sir James Hall, who
began conducting his experiments in 1790), paleontology (e.g., Oehler,
1976, described experiments that simulated fossilization in synthetic
chert), and zoology (e.g., Erlingsson, 2009, described the rise of experi-
mental zoology in Britain during the 1920s).
Darwin is most famous for his theory of evolution, but he also
devoted much time to testing hypotheses with experiments. For
example, he hypothesized, contrary to then current belief, that plants
move, and designed experiments that tracked plant movement
(Hangarter, 2000). But not all research problems are amenable to
testing by way of experiments that are controlled by the researcher, as
Mayr (1997) described in his book on the science of biology: “Much
progress in the observational sciences is due to the genius of those who
have discovered, critically evaluated, and compared . . . natural experi-
ments in fields where a laboratory experiment is impractical, if not
impossible” (p. 29).
Natural experiments have been used to test competing theories
in the physical sciences; for example, Maupertuis’s expedition to
Lapland over the winter of 1736–1737 to undertake observations
that would test the Cartesian theory that the earth is taller than it is
broad against Newton’s theory that the opposite is the case. More

famously, Eddington’s 1919 expeditions were mounted to determine
whether Einstein’s or Newton’s gravitation theories provided the better
prediction of phenomena by taking advantage of the natural experiment
provided by a solar eclipse (Sponsel, 2002).
Hypotheses on the distribution of plants from Darwin’s specu-
lations and ﬁndings from experiments on the survival and dispersal of
plant seeds (Carlquist, 2009) were tested by the natural experiment of
the 1883 eruption of the island of Krakatoa (Krakatau). The eruption
sterilized what was left of the island such that most plant life – with the
possible exception of some grasses – would have to have arrived on or
over open sea. Nine months after the eruption, there was no sign of
plant life, but by 1930 the whole island was covered with dense forest
(Went, 1949).
Gould (1970) advocated greater use of experiments in paleon-
tology – “we must include the experimental approach . . . and not
remain tied to the observational mode of traditional natural history”
(p. 88) – and described prior studies that used natural experiments. He
quoted Seilacher on the topic: “One cannot make experiments with
organisms that became extinct hundreds of million years ago. Still, isn’t
it an experimental approach if the belemnites’ habits were tested
through the reactions of its commensals? The fact that the actual test
was made long before man’s existence does not alter the principles of its
evaluation” (Gould, 1970, p. 89).
Variations between the societies of different countries, regions,
states, and communities, and changes over time provide natural experi-
ments against which researchers can test hypotheses from alternative
theories. Diamond and Robinson’s (2010) edited book Natural
Experiments of History includes seven analyses of political and social
arrangements and their economic outcomes or causes using natural
experiments from history. Alternative arrangements for managing
common pool resources provide natural experiments that allowed
testing of hypotheses on whether sustainable management arrange-
ments can arise by trial and error, or whether they must be imposed
by a political authority (Ostrom, 1990). Variations in regulations
between US counties and states, and over time, allowed Lott (2010) to
test hypotheses on the relationship between gun control and crime.
Note that some scientists consider the term “natural experi-
ments” to be only a metaphor for studies that literally test hypotheses
by making observations, or “observational studies,” and not true

experiments. We prefer to use the term “natural experiments” in order
to distinguish studies that are properly designed to test alternative
hypotheses by identifying situations in which observations might turn
out to falsify them, and reserve the term “observational studies” for
studies that do not test hypotheses or that develop hypotheses to fit
observations.
For ideas and guidance on designing experiments see Shadish,
Cook, and Campbell’s (2001) book Experimental and Quasi-
Experimental Designs for Generalized Causal Inference. They describe
diverse and creative ways to conduct experiments. Another resource is
Dunning’s (2012) book Natural Experiments in the Social Sciences:
A Design-Based Approach, the first part of which is devoted to “dis-
covering natural experiments.”
Experiments guided by sound theoretical reasoning provide the
only valid and reliable way to establish causal relationships. Causality
cannot be identified by “machine learning” methods, known by names
such as artificial intelligence, data mining, factor analysis, and stepwise
regression. We described the lack of evidence that the models that are
the product of machine learning methods have any predicted validity in
our 2018 and 2019 co-authored papers.
Machine learning models violate the scientific method because
they fail to incorporate prior knowledge from experimental studies and
coherent theory. The models are also vulnerable to including variables
that have no known causal relationship to the variable of interest.
As economist Friedrich Hayek warned in his Nobel Prize lecture,
“in economics and other disciplines that deal with essentially complex
phenomena, the aspects of the events to be accounted for about which
we can get quantitative data are necessarily limited and may not include
the important ones” (Hayek, 1974).
Meta-analyses of experimental data are the gold standard of
evidence. Meta-analyses combine the results of all experimental studies
on the issue being studied, no matter the type of experiment. For
example, a meta-analysis of 40 experiments on how communication
affects persuasion found the conclusions from field and laboratory
studies were similar (Wilson and Sherrell, 1993).
Findings from experimental studies do, however, often
differ from those based on non-experimental data. For example, expert
judgments and non-experimental research typically conclude that con-
sumer satisfaction surveys improve consumer satisfaction. However,

well-designed experiments showed that they harm satisfaction because
customers look for bad things to report. They also create dissatisfaction
among those providing the services. The problems went away when
people were asked what they liked about the product or service (Ofir
and Simonson, 2001).
Non-experimental data from hundreds of thousands of users
showed that female hormone-replacement therapy helped to preserve
youth and ward off a variety of diseases in older women. The findings
were replicated. However, subsequent experimental studies found that
the treatment could actually be harmful. The favorable findings from
the non-experimental data occurred because the women who used the
new medicine were generally more concerned about their health and
sought out ways to stay healthy (Avorn, 2004).
Kabat’s (2008) book on environmental hazards – examining
such topics as DDT, electromagnetic fields from power lines, radon, and
second-hand smoke – concluded that analysis of non-experimental data
in studies on health had often misled researchers, doctors, patients, and
the public.
Non-experimental data analyses lend themselves to advocacy stud-
ies. They allow researchers to produce “evidence” for almost any hypoth-
esis by attributing causal relationships to correlations in survey data.
Vernon Smith, a pioneer of experimental economics and a
2002 Nobel Laureate in Economics, suggested that what can be learned
from well-designed laboratory experiments is only limited by the
ingenuity and creativity of the researcher.
What are the limits of laboratory investigation? I think any
attempt to define such limits is very likely to be bridged by the
subsequent ingenuity and creativity ... of some experimentalist.
Twenty-five years ago I could not have imagined being able to do
the kinds of experiments that today have become routine in our
laboratories. Experimentalists also include many of us who see no
clear border separating the lab and the field. Vernon Smith (2003,
p. 474, n. 27).
There may be problems or situations for which experiments are not
possible. In such cases, analyses of non-experimental data may be useful
for helping to identify whether hypothesized causal relationships are
plausible. For situations in which causal relationships have been estab-
lished, analyses of non-experimental data can help to assess effect sizes.

Some philosophers of science have theorized that experiments
cannot do what scientists expect them to: contribute to knowledge by
rejecting or supporting hypotheses. As we hope is clear from this book,
we disagree, strongly. Philosopher of science Deborah Mayo and prac-
titioner of science Vernon Smith have also disagreed, as follows.
In principle the D-Q problem1
is a barrier to any defensible notion
of a rational science that selects theories by a logical process of
confrontation with scientific evidence. This is cause for joy not
despair. Think how dull would be a life of science if, once we were
trained, all we had to do was to turn on the threshing machine of
science, feed it the facts and send its output to the printer. In
practice the D-Q problem is not a barrier to resolving ambiguity
in interpreting test results. The action is always in imaginative new
tests and the conversation it stimulates. My personal experience as
an experimental economist since 1956, resonates well with Mayo’s
critique of Lakatos:
Lakatos, recall, gives up on justifying control; at best we decide –
by appeal to convention – that the experiment is controlled ...
I reject Lakatos and others’ apprehension about experimental
control. Happily, the image of experimental testing that gives these
philosophers cold feet bears little resemblance to actual experi-
mental learning. Literal control is not needed to correctly attribute
experimental results (whether to affirm or deny a hypothesis).
Enough experimental knowledge will do. Nor need it be assured
that the various factors in the experimental context have no influ-
ence on the result in question – far from it. A more typical strategy
is to learn enough about the type and extent of their influences and
then estimate their likely effects in the given experiment. Vernon
Smith (2002, p. 106, quoting Mayo, 1996, p. 240)
2.2.8 Draw Logical Conclusions
Francis Bacon (1620 [1863]) reinforced Aristotle’s assertion
that the scientific method involves logical induction from systematic
1
The Duhem-Quine problem is the assertion that designing an experiment to test a hypothesis
is not possible without making assumptions or involving additional hypotheses that may
themselves be the cause of the experiment’s support for or rejection of the hypothesis (the
authors).

observation. Conclusions should follow logically from the evidence
provided in a paper.
How might logic be used to compare competing hypotheses?
Here is an example: compare the hypothesis that people in a given
community in a rich country will be happier if the government redistrib-
utes money income from higher income people to those with lower
incomes (Hypothesis #1), with the hypothesis that people in a commu-
nity who are happier are more productive and earn more money
(Hypothesis #2), and with the hypothesis that the happiness of people
within a community is more affected by their relative status than by
their absolute money income (Hypothesis #3). The latter hypotheses
lead to policy conclusions that are opposite to the those from the ﬁrst.
Frey’s (2018) summary of evidence from happiness research provides
support for Hypothesis #2 and #3, and cautions against Hypothesis #1.
If the research addresses a problem that involves strong emo-
tions, consider writing the conclusions using symbols in order to check
the logic. For example, the argument “if P, then Q. Not P, therefore not
Q” is easily recognized as a logical fallacy – known as “denying the
antecedent” – but recognition is not easy for contentious issues, such as
the relationship between guns and crime.
Violations of logic are common in the social sciences. We
suggest asking researchers who have different views on the problem
you are studying to check your logic. Logic does not change over time,
nor does it differ by ﬁeld. Thus, Beardsley’s (1950) Practical Logic
continues to be useful. For an additional discussion of logical fallacies,
see the website www.logicalfallacies.org.

3 CHECKLIST FOR THE SCIENTIFIC METHOD
We intend that our checklist provides a common understanding
among all stakeholders in science of what the scientiﬁc method entails.
To that end, we describe it in terms that are simple and
commonly understood.
In this chapter, we outline how we developed the Compliance
with Science Checklist. We then present the checklist of eight criteria for
complying with the scientiﬁc method and 26 items to help check
whether the criteria are met. The checklist is intended for all stakehold-
ers of science. We describe how the checklist can be used, and list
stakeholders and what they can use the checklist for in Table 3.1. We
caution that checklists are only useful if they are logical and based on
evidence, and if they are used.
3.1 Development of the Compliance With Science Checklist
Checklists draw upon the decomposition principle, which
reduces a complex problem into simpler parts. One solves or makes
estimates for, or rates, each part, and then calculates an aggregate
solution, or overall rating.
Our review of experimental evidence showed that decompos-
ition typically provides substantial improvements in predictive validity.
For example, in three experiments on subjects’ decisions for job and
college selection, judgmental decomposition resulted in more accurate
judgments than holistic ratings (Arkes et al., 2010). Similarly, an experi-
ment in which members of the Society for Medical Decision Making

evaluated presentations at their annual convention found that decom-
posed ratings were more reliable than holistic ratings (Arkes et al.,
2006). For additional experimental evidence on the value of decom-
position, see MacGregor (2001).
To develop a checklist of criteria for compliance with the scien-
tific method, we reviewed experimental research on scientific practice
(described in Chapter 4). Based on the research findings, we designed
operational guidelines for each of the eight criteria. For example, to
gauge a paper’s objectivity, the checklist asks raters to determine
whether a paper compares multiple reasonable hypotheses.
As we will show in this book, the Compliance With Science
Checklist provides a valid and reliable way to rate the extent to which
Table 3.1. Potential users and uses of the Compliance With Science Checklist
Researchers ▪ determining which findings to cite
▪ ensuring that their own papers comply
▪ informing clients, editors, users, and readers on the extent to
which their paper complies
Journals ▪ setting expectations of authors
▪ identifying which criteria were met
▪ selecting which papers to publish
Universities ▪ training, hiring, promoting, and dismissing scientists
▪ setting expectations of researchers
▪ disseminating useful scientific findings
Think Tanks ▪ assessing papers to identify the scientific criteria that were met
Funders ▪ requiring research to meet scientific criteria
Awards
Committees
▪ choosing recipients who made useful scientific discoveries
Certifiers ▪ independently assessing the extent to which papers provide
useful scientific findings
Managers ▪ assessing the value of published findings
Journalists ▪ reporting the extent to which studies address important
problems and comply with science
Regulators ▪ developing, revising, and rescinding regulations based on
compliance with science
Law Courts ▪ assessing the value of evidence
25 / The Compliance With Science Checklist

papers – or methods or policies – comply with the scientific method.
This checklist, along with other checklists in this book, is also provided
at GuidelinesForScience.com. To ensure that raters understood the
guidelines, we pretested the checklist many times by examining the
inter-rater reliability of the ratings for each of the criteria.
Checklist 3.1 is the result of our efforts: it provides 26 oper-
ational items to rate compliance with the eight criteria of the
scientific method.
Checklist 3.1 Compliance With Science Checklist
Paper title:
Reviewer: Date: Time spent (minutes):
Instructions for Raters
1. Skim the paper while you complete the checklist as a skeptical reviewer.
2. Rate each lettered item, below, marking the relevant checkbox to indicate
True if the research complies,
F/? (False/Unclear) if the research does not comply, or if you are unsure.
IMPORTANT: If you are not convinced that the paper complied, rate the item F/?
3. If you rate an item True, give reasons for your rating in your own words.
4. Rate criteria 1–8 as True by marking the checkbox only if all lettered items for
the criterion are rated T.
First assess whether the paper complies with the
lettered items under each criterion below. Then assess
whether it complies with each of the eight criteria
based on compliance with the lettered items. Avoid
speculation.
1. Problem is important for decision-making, policy,
or method development
□ True
T F/?
a. Importance of the problem clear from the title,
abstract, result tables, or conclusions
□ □
b. Findings add to cumulative scientific knowledge □ □
c. Uses of the findings are clear to you □ □
d. The findings can be used to improve people’s
lives without resorting to duress or deceit
□ □
2. Prior knowledge was comprehensively reviewed
and summarized
□ True
T F/?
a. The paper describes objective and
comprehensive procedures used to search for
prior useful scientific knowledge
□ □
26 / Checklist for the Scientific Method

Checklist 3.1 cont’d
b. The paper describes how prior substantive
findings were used to develop hypotheses (e.g.
direction and magnitude of effects of causal
variables) and research procedures
□ □
3. Disclosure is sufficiently comprehensive for
understanding and replication
□ True
T F/?
a. Methods are fully and clearly described so as to
be understood by all relevant stakeholders,
including potential users
□ □
b. Data are easily accessible using information
provided in the paper
□ □
c. Sources of funding are described, or absence of
external funding noted
□ □
4. Design is objective (unbiased by advocacy) □ True
T F/?
a. Prior hypotheses are clearly described (e.g.,
regarding directions and magnitudes of
relationships, and effects of conditions)
□ □
b. All reasonable hypotheses are included in the
design, including plausible naive, no-meaningful-
difference, and current-practice hypotheses
□ □
c. Revisions to hypotheses are described, or
absence of revisions noted
□ □
5. Data are valid (true measures) and reliable
(repeatable measures)
□ True
T F/?
a. Data were shown to be relevant to the problem □ □
b. All relevant data were used, including the
longest relevant time-series
□ □
c. Reliability of data was assessed □ □
d. Other information needed for assessing the validity
of the data is provided, such as adjustments, known
shortcomings and potential biases
□ □
6. Methods were validated (proven fit for purpose)
and simple
□ True
T F/?
a. Methods were explained clearly and shown
valid – unless well known to intended readers,
users, and reviewers, and validity is obvious
□ □
27 / The Compliance With Science Checklist

3.2 Using the Checklist: For What, How, and by Whom
The Compliance With Science Checklist is intended to help
researchers discover useful scientific knowledge and stakeholders to
evaluate research.
As far as we are aware, the Compliance With Science Checklist
is the only checklist designed for assessing the extent to which a paper
complies with the scientific method. For example, a major US research
funding body, the National Science Foundation, states that the agency
was created by Congress in 1950 with a mission to “promote the
Checklist 3.1 cont’d
b. Methods were sufficiently simple for potential
users to understand
□ □
c. Multiple validated methods were used □ □
d. Methods used cumulative scientific knowledge
explicitly
□ □
7. Experimental evidence was used to compare
alternative hypotheses
□ True
T F/?
a. Experimental evidence was used to compare
hypotheses under explicit conditions
□ □
b. Predictive validity of hypotheses was tested
using out-of-sample data
□ □
8. Conclusions follow logically from the evidence
presented
□ True
T F/?
a. Conclusions do not go beyond the evidence in
the paper
□ □
b. Conclusions are not the product of confirmation bias □ □
c. Conclusions do not reject a hypothesis by
denying the antecedent
□ □
d. Conclusions do not support a hypothesis by
affirming the consequent
□ □
Describe the most important scientific finding in your own words.
Sum the criteria (1–8) rated True for compliance: [ ] of 8
An electronic version of this checklist is available at guidelinesforscience.com.
28 / Checklist for the Scientific Method

Random documents with unrelated
content Scribd suggests to you:

The Project Gutenberg eBook of The God Next
Door

This ebook is for the use of anyone anywhere in the United States
and most other parts of the world at no cost and with almost no
restrictions whatsoever. You may copy it, give it away or re-use it
under the terms of the Project Gutenberg License included with this
ebook or online at www.gutenberg.org. If you are not located in the
United States, you will have to check the laws of the country where
you are located before using this eBook.
Title: The God Next Door
Author: William R. Doede
Illustrator: Larry Ivie
Release date: April 8, 2016 [eBook #51699]
Most recently updated: October 23, 2024
Language: English
Credits: Produced by Greg Weeks, Mary Meehan and the Online
Distributed Proofreading Team at http://www.pgdp.net
*** START OF THE PROJECT GUTENBERG EBOOK THE GOD NEXT
DOOR ***

THE GOD NEXT DOOR
By BILL DOEDE
Illustrated by IVIE
[Transcriber's Note: This etext was produced from
Galaxy Magazine August 1961.
Extensive research did not uncover any evidence that
the U.S. copyright on this publication was renewed.]

The sand-thing was powerful, lonely and
strange. No doubt it was a god—but who
wasn't?
Stinson lay still in the sand where he fell, gloating over the success
of his arrival.
He touched the pencil-line scar behind his ear where the cylinder
was buried, marveling at the power stored there, power to fling him
from earth to this fourth planet of the Centaurian system in an
instant. It had happened so fast that he could almost feel the warm,
humid Missouri air, though he was light years from Missouri.

He got up. A gray, funnel-shaped cloud of dust stood off to his left.
This became disturbing, since there was scarcely enough wind to
move his hair. He watched it, trying to recall what he might know
about cyclones. But he knew little. Weather control made cyclones
and other climatic phenomena on earth practically non-existent. The
cloud did not move, though, except to spin on its axis rapidly,
emitting a high-pitched, scarcely audible whine, like a high speed
motor. He judged it harmless.
He stood on a wide valley floor between two mountain ranges. Dark
clouds capped one peak of the mountains on his left. The sky was
deep blue.
He tested the gravity by jumping up and down. Same as Earth
gravity. The sun—no, not the sun. Not Sol. What should he call it,
Alpha or Centaurus? Well, perhaps neither. He was here and Earth
was somewhere up there. This was the sun of this particular solar
system. He was right the first time.
The sun burned fiercely, although he would have said it was about
four o'clock in the afternoon, if this had been Earth. Not a tree, nor a
bush, nor even a wisp of dry grass was in sight. Everywhere was
desert.
The funnel of sand had moved closer and while he watched it, it
seemed to drift in the wind—although there was no wind. Stinson
backed away. It stopped. It was about ten feet tall by three feet in
diameter at the base. Then Stinson backed away again. It was
changing. Now it became a blue rectangle, then a red cube, a violet
sphere.
He wanted to run. He wished Benjamin were here. Ben might have
an explanation. What am I afraid of? he said aloud, a few grains
of sand blowing in the wind? A wind devil?
He turned his back and walked away. When he looked up the wind
devil was there before him. He looked back. Only one. It had moved.
The sun shone obliquely, throwing Stinson's shadow upon the sand.
The wind devil also had a shadow, although the sun shone through

it and the shadow was faint. But it moved when the funnel moved.
This was no illusion.
Again Stinson felt the urge to run, or to use the cylinder to project
himself somewhere else, but he said, No! very firmly to himself. He
was here to investigate, to determine if this planet was capable of
supporting life.
Life? Intelligence? He examined the wind devil as closely as he
dared, but it was composed only of grains of sand. There was no
core, no central place you could point to and say, here is the brain,
or the nervous system. But then, how could a group of loosely
spaced grains of sand possibly have a nervous system?
It was again going through its paces. Triangle, cube, rectangle,
sphere. He watched, and when it became a triangle again, he
smoothed a place in the sand and drew a triangle with his forefinger.
When it changed to a cube he drew a square, a circle for a sphere,
and so on. When the symbols were repeated he pointed to each in
turn, excitement mounting. He became so absorbed in doing this
that he failed to notice how the wind devil drew closer and closer,
but when he inhaled the first grains of sand, the realization of what
was happening dawned with a flash of fear. Instantly he projected
himself a thousand miles away.
Now he was in an area of profuse vegetation. It was twilight. As he
stood beside a small creek, a chill wind blew from the northwest. He
wanted to cover himself with the long leaves he found, but they
were dry and brittle, for here autumn had turned the leaves. Night
would be cold.
He was not a woodsman. He doubted if he could build a fire without
matches. So he followed the creek to where it flowed between two
great hills. Steam vapors rose from a crevice. A cave was nearby and
warm air flowed from its mouth. He went inside.

At first he thought the cave was small, but found instead that he
was in a long narrow passageway. The current of warm air flowed
toward him and he followed it, cautiously, stepping carefully and
slowly. Then it was not quite so dark. Soon he stepped out of the
narrow passageway into a great cavern with a high-vaulted ceiling.
The light source was a mystery. He left no shadow on the floor. A
great crystal sphere hung from the ceiling, and he was curious about
its purpose, but a great pool of steaming water in the center of the
cavern drew his attention. He went close, to warm himself. A stone
wall surrounding the pool was inscribed with intricate art work and
indecipherable symbols.
Life. Intelligence. The planet was inhabited.
Should he give up and return to earth? Or was there room here for
his people? Warming his hands there over the great steaming pool
he thought of Benjamin, and Straus, and Jamieson—all those to
whom he had given cylinders, and who were now struggling for life
against those who desired them.
He decided it would not be just, to give up so easily.
The wide plaza between the pool and cavern wall was smooth as
polished glass. Statues lined the wall. He examined them.
The unknown artist had been clever. From one angle they were
animals, from another birds, from a third they were vaguely
humanoid creatures, glowering at him with primitive ferocity. The
fourth view was so shocking he had to turn away quickly. No
definable form or sculptured line was visible, yet he felt, or saw—he
did not know which senses told him—the immeasurable gulf of a
million years of painful evolution. Then nothing. It was not a curtain
drawn to prevent him from seeing more.
There was no more.

He stumbled toward the pool's wall and clutched for support, but his
knees buckled. His hand slid down the wall, over the ancient
inscriptions. He sank to the floor. Before he lost consciousness he
wondered, fleetingly, if a lethal instrument was in the statue.
He woke with a ringing in his ears, feeling drugged and sluggish.
Sounds came to him. He opened his eyes.
The cavern was crowded. These creatures were not only humanoid,
but definitely human, although more slight of build than earth
people. The only difference he could see at first sight was that they
had webbed feet. All were dressed from the waist down only, in a
shimmering skirt that sparkled as they moved. They walked with the
grace of ballet dancers, moving about the plaza, conversing in a
musical language with no meaning for Stinson. The men were dark-
skinned, the women somewhat lighter, with long flowing hair, wide
lips and a beauty that was utterly sensual.
He was in chains! They were small chains, light weight, of a metal
that looked like aluminum. But all his strength could not break them.
They saw him struggling. Two of the men came over and spoke to
him in the musical language.
My name is Stinson, he said, pointing to himself. I'm from the
planet Earth.
They looked at each other and jabbered some more.
Look, he said, Earth. E-A-R-T-H, Earth. He pointed upward,
described a large circle, then another smaller, and showed how Earth
revolved around the sun.
One of the men poked him with a stick, or tube of some kind. It did
not hurt, but angered him. He left the chains by his own method of
travel, and reappeared behind the two men. They stared at the place
where he had been. The chains tinkled musically. He grasped the
shoulder of the offender, spun him around and slapped his face.
A cry of consternation rose from the group, echoing in the high
ceilinged cavern. SBTL! it said, ZBTL ... XBTL ... zbtl.

The men instantly prostrated themselves before him. The one who
had poked Stinson with the stick rose, and handed it to him. Still
angered, Stinson grasped it firmly, with half a notion to break it over
his head. As he did so, a flash of blue fire sprang from it. The man
disappeared. A small cloud of dust settled slowly to the floor.
Disintegrated!
Stinson's face drained pale, and suddenly, unaccountably, he was
ashamed because he had no clothes.
I didn't mean to kill him! he cried. I was angry, and....
Useless. They could not understand. For all he knew, they might
think he was threatening them. The object he had thought of as a
stick was in reality a long metal tube, precisely machined, with a
small button near one end.
This weapon was completely out of place in a culture such as this.
Or was it? What did he know of these people? Very little. They were
humanoid. They had exhibited human emotions of anger, fear and,
that most human of all characteristics, curiosity. But up to now the
tube and the chain was the only evidence of an advanced
technology, unless the ancient inscriptions in the stone wall of the
pool, and the statues lining the wall were evidences.
There was a stirring among the crowd. An object like a pallet was
brought, carried by four of the women. They laid it at his feet, and
gestured for him to sit. He touched it cautiously, then sat.
Instantly he sprang to his feet. There, at the cavern entrance, the
wind devil writhed and undulated in a brilliant harmony of colors. It
remained in one spot, though, and he relaxed somewhat.
One of the women came toward him, long golden hair flowing, firm
breasts dipping slightly at each step. Her eyes held a language all

their own, universal. She pressed her body against him and bore him
to the pallet, her kisses fire on his face.
Incongruously, he thought of Benjamin back on earth, and all the
others with cylinders, who might be fighting for their lives at this
moment. He pushed her roughly aside.
She spoke, and he understood! Her words were still the same
gibberish, but now he knew their meaning. Somehow he knew also

that the wind devil was responsible for his understanding.
You do not want me? she said sadly. Then kill me.
Why should I kill you?
She shrugged her beautiful shoulders. It is the way of the Gods,
she said. If you do not, then the others will.
He took the tube-weapon in his hands, careful not to touch the
button. Don't be afraid. I didn't mean to kill the man. It was an
accident. I will protect you.
She shook her head. One day they will find me alone, and they'll kill
me.
Why?
She shrugged. I have not pleased you.
On the contrary, you have. There is a time and place for everything,
though.
Suddenly a great voice sounded in the cavern, a voice with no
direction. It came from the ceiling, the floor, the walls, the steaming
pool. It was in the language of the web-footed people; it was in his
own tongue. No harm must come to this woman. The God with
fingers on his feet has decreed this.
Those in the cavern looked at the woman with fear and respect. She
kissed Stinson's feet. Two of the men came and gave her a brilliant
new skirt. She smiled at him, and he thought he had never seen a
more beautiful face.
The great, bodiless voice sounded again, but those in the cavern
went about their activities. They did not hear.
Who are you?

Stinson looked at the wind devil, since it could be no one else
speaking, and pointed to himself. Me?
Yes.
I am Stinson, of the planet Earth.
Yes, I see it in your mind, now. You want to live here, on this
planet.
Then you must know where I came from, and how.
I do not understand how. You have a body, a physical body
composed of atoms. It is impossible to move a physical body from
one place to another by a mere thought and a tiny instrument, yet
you have done so. You deserted me out in the desert.
I deserted you? Stinson cried angrily, You tried to kill me!
I was attempting communication. Why should I kill you?
He was silent a moment, looking at the people in the cavern.
Perhaps because you feared I would become the God of these
people in your place.
Stinson felt a mental shrug. It is of no importance. When they
arrived on this planet I attempted to explain that I was not a God,
but the primitive is not deeply buried in them. They soon resorted to
emotion rather than reason. It is of no importance.
I'd hardly call them primitive, with such weapons.
The tube is not of their technology. That is, they did not make it
directly. These are the undesirables, the incorrigibles, the
nonconformists from the sixth planet. I permit them here because it
occupies my time, to watch them evolve.
You should live so long.
Live? the wind devil said. Oh, I see your meaning. I'd almost
forgotten. You are a strange entity. You travel by a means even I
cannot fully understand, yet you speak of time as if some event
were about to take place. I believe you think of death. I see your

physical body has deteriorated since yesterday. Your body will cease
to exist, almost as soon as those of the sixth planet peoples. I am
most interested in you. You will bring your people, and live here.
I haven't decided. There are these web-footed people, who were
hostile until they thought I was a God. They have destructive
weapons. Also, I don't understand you. I see you as a cone of sand
which keeps changing color and configuration. Is it your body?
Where do you come from? Is this planet populated with your kind?
The wind devil hesitated.
Where do I originate? It seems I have always been. You see this
cavern, the heated pool, the statues, the inscriptions. Half a million
years ago my people were as you. That is, they lived in physical
bodies. Our technology surpassed any you have seen. The tube
these webfoots use is a toy by comparison. Our scientists found the
ultimate nature of physical law. They learned to separate the mind
from the body. Then my people set a date. Our entire race was
determined to free itself from the confines of the body. The date
came.
What happened?
I do not know. I alone exist. I have searched all the levels of time
and matter from the very beginning. My people are gone.
Sometimes it almost comes to me, why they are gone. And this is
contrary to the greatest law of all—that an entity, once in existence,
can never cease to exist.
Stinson was silent, thinking of the endless years of searching
through the great gulf of time. His eyes caught sight of the woman,
reclining now on the pallet. The men had left her and stood in
groups, talking, glancing at him, apparently free of their awe and
fear already.

The woman looked at him, and she was not smiling. Please ask the
Sand God, she said, to speak to my people again. Their fear of him
does not last. When He is gone they will probably kill us.
As for the webfoots, the wind devil, or Sand God, said, I will
destroy them. You and your people will have the entire planet.
Destroy them? Stinson asked, incredulously, all these people?
They have a right to live like any one else.
Right? What is it—'right?' They are entities. They exist, therefore
they always will. My people are the only entities who ever died. To
kill the body is unimportant.
No. You misunderstand. Listen, you spoke of the greatest law. Your
law is a scientific hypothesis. It has to do with what comes after
physical existence, not with existence itself. The greatest law is this,
that an entity, once existing, must not be harmed in any way. To do
so changes the most basic structure of nature.
The Sand God did not reply. The great bodiless, directionless voice
was silent, and Stinson felt as if he had been taken from some high
place and set down in a dark canyon. The cone of sand was the
color of wood ashes. It pulsed erratically, like a great heart missing a
beat now and then. The web-footed people milled about restlessly.
The woman's eyes pleaded.
When he looked back, the Sand God was gone.
Instantly a new note rose in the cavern. The murmur of
unmistakable mob fury ran over the webfoots. Several of the men
approached the woman with hatred in their voices. He could not
understand the words now.
But he understood her. They'll kill me! she cried.
Stinson pointed the disintegrating weapon at them and yelled. They
dropped back. We'll have to get outside, he told her. This mob will
soon get out of hand. Then the tube won't stop them. They will rush
in. I can't kill them all at once, even if I wanted to. And I don't.

Together they edged toward the cavern entrance, ran quickly up the
inclined passageway, and came out into crisp, cold air. The morning
sun was reflected from a million tiny mirrors on the rocks, the trees
and grass. A silver thaw during the night had covered the whole area
with a coating of ice. Stinson shivered. The woman handed him a
skirt she had thoughtfully brought along from the cavern. He took it,
and they ran down the slippery path leading away from the
entrance. From the hiding place behind a large rock they watched,
as several web-footed men emerged into the sunlight. They blinked,
covered their eyes, and jabbered musically among themselves. One
slipped and fell on the ice. They re-entered the cave.
Stinson donned the shimmering skirt, smiling as he did so. The
others should see him now. Benjamin and Straus and Jamieson.
They would laugh. And Ben's wife, Lisa, she would give her little-girl
laugh, and probably help him fasten the skirt. It had a string, like a
tobacco pouch, which was tied around the waist. It helped keep him
warm.
He turned to the woman. I don't know what I'll do with you, but
now that we're in trouble together, we may as well introduce
ourselves. My name is Stinson.
I am Sybtl, she said.
Syb-tl. He tried to imitate her musical pronunciation. A very nice
name.
She smiled, then pointed to the cavern. When the ice is gone, they
will come out and follow us.
We'd better make tracks.
No, she said, we must run, and make no tracks.
Okay, Sis, he said.
Sis?

That means, sister.
I am not your sister. I am your wife.
What?
Yes. When a man protects a woman from harm, it is a sign to all
that she is his chosen. Otherwise, why not let her die? You are a
strange God.
Listen, Sybtl, he said desperately, I am not a God and you are not
my wife. Let's get that straight.
But....
No buts. Right now we'd better get out of here.
He took her hand and they ran, slid, fell, picked themselves up
again, and ran. He doubted the wisdom of keeping her with him.
Alone, the webfoots were no match for him. He could travel instantly
to any spot he chose. But with Sybtl it was another matter; he was
no better than any other man, perhaps not so good as some
because he was forty, and never had been an athlete.
How was he to decide if this planet was suitable for his people,
hampered by a woman, slinking through a frozen wilderness like an
Indian? But the woman's hand was soft. He felt strong knowing she
depended on him.
Anyway, he decided, pursuit was impossible. They left no tracks on
the ice. They were safe, unless the webfoots possessed talents
unknown to him.
So they followed the path leading down from the rocks, along the
creek with its tumbling water. Frozen, leafless willows clawed at their
bodies. The sun shone fiercely in a cloudless sky. Already water ran
in tiny rivulets over the ice. The woman steered him to the right,
away from the creek.

Stinson's bare feet were numb from walking on ice. Christ, he
thought, what am I doing here, anyway? He glanced down at Sybtl
and remembered the webfoots. He stopped, tempted to use his
cylinder and move to a warmer, less dangerous spot.
The woman pulled on his arm. We must hurry!
He clutched the tube-weapon. How many shots in this thing?
Shots?
How often can I use it?
As often as you like. It is good for fifty years. Kaatr—he is the one
you destroyed—brought it from the ship when we came. Many times
he has used it unwisely.
When did you come?
Ten years ago. I was a child.
I thought only criminals were brought here.
She nodded. Criminals, and their children.
When will your people come again?
She shook her head. Never. They are no longer my people. They
have disowned us.
And because of me even those in the cavern have disowned you.
Suddenly she stiffened beside him. There, directly in their path,
stood the Sand God. It was blood red now. It pulsed violently. The
great voice burst forth.
Leave the woman! it demanded angrily. The webfoots are nearing
your position.
I cannot leave her. She is helpless against them.
What form of primitive stupidity are you practicing now? Leave, or
they will kill you.
Stinson shook his head.

The Sand God pulsed more violently than before. Ice melted in a
wide area around it. Brown, frozen grass burned to ashes.
You will allow them to kill you, just to defend her life? What
business is it of yours if she lives or dies? My race discarded such
primitive logic long before it reached your level of development.
Yes, Stinson said, and your race no longer exists.
The Sand God became a sphere of blue flame. A wave of intense
heat drove them backward. Earthman, the great voice said, go
back to your Earth. Take your inconsistencies with you. Do not come
here again to infect my planet with your primitive ideas. The
webfoots are not as intelligent as you, but they are sane. If you
bring your people here, I shall destroy you all.
The sphere of blue fire screamed away across the frozen wilderness,
and the thunder of its passing shook the ground and echoed among
the lonely hills.
Sybtl shivered against his arm. The Sand God is angry, she said.
My people tell how he was angry once before, when we first came
here. He killed half of us and burned the ship that brought us. That
is how Kaatr got the tube-weapon. It was the only thing the Sand
God didn't burn, that and the skirts. Then, when he had burned the
ship, the Sand God went to the sixth planet and burned two of the
largest cities, as a warning that no more of us must come here.
Well, Stinson said to himself, that does it. We are better off on Earth.
We can't fight a monster like him.
Sybtl touched his arm. Why did the Sand God come? He did not
speak.
He spoke to me.
I did not hear.

Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.
More than just a book-buying platform, we strive to be a bridge
connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.
Join us on a journey of knowledge exploration, passion nurturing, and
personal growth every day!
ebookbell.com

The Scientific Method A Guide To Finding Useful Knowledge J Scott Armstrong

More Related Content

Similar to The Scientific Method A Guide To Finding Useful Knowledge J Scott Armstrong (20)

Recently uploaded (20)

The Scientific Method A Guide To Finding Useful Knowledge J Scott Armstrong