Talking Theory

Talking Theory Five Perspectives on Theory Construction in Psychology

Voodoo Correlations are Everywhere—Not Only in Neuroscience Klaus Fiedler, University of Heidelberg

Voodoo Correlations in Neuroscience Or “Puzzlingly High Correlations,” at least (Vul, Harris, Winkielman, & Pashler, 2009) Selecting voxels based on thresholds leads to distorted estimates of correlations

Other Research Biases—Study design Sampling stimuli Generally an intuitive process; researcher implicitly simulates what is expected to be observed Random sampling of stimuli can eliminate some effects, ex. Overconfidence & trivia questions (e.g., Gigerenzer, Hoffrage, & Kleinbölting, 1991) Pilot testing of stimuli/tasks X values are auditioned to correlate with Y

Other Research Biases—Selecting Variables Dependent variables What is chosen to be studied affects theoretical perceptions Ex. TV & aggression Sampling levels of IVs Magnitude of ΔY is easily manipulated by ΔX Effect sizes are useful, but can also be misleading

Other Research Biases—Analysis Mediator variables Trifling mediators (Y’) can be found if they are correlated enough with Y Moderator variables Strong moderators can be overlooked or assumed in certain paradigms, overestimating actual effects Publishing Weak findings will be “optimized” Analyses will be re-run until they are “successful” File-drawer problem

Fiedler: Conclusions Aim for representative designs (Brunswick, 1955) When that fails, treat stimuli as random factors, as appropriate, to enhance external validity Preplanned comparisons >2 levels of IV Convergent validation Multiple methods to converge on the existence of an effect Usually underestimates the true effect size, but can provide a lower limit Study boundary conditions Can reveal important moderators, enabling conditions Or rather, make implicit moderators explicit

The Problem of Circularity in Evidence, Argument, and Explanation Ulrike Hahn, Cardiff University

Question How are we to effectively argue in psychology?

Circular Argumentation Reassertion: ex. “God exists, because God exists.” A because A Clearly convertible: ex. “Opium induces sleep because it has a soporific quality.” A because A’ Self-dependent justification: ex. “God exists because the bible says so, and the Bible is the word of God.” A because B; B because A

Circular Argumentation Is it invalid? No; on the contrary, it’s always valid Is it persuasive? Sometimes yes, actually Reassertion and convertible cases are not very persuasive; miss the point of argumentation: belief change Self-dependent case is frequently used in hypothesis testing: ex. “Electrons exist because their signature effects can be seen in cloud chambers” Modus ponens argument: “if P, then Q”; falsifiable, but not entirely verifiable Persuasiveness of argument rests on conditional probabilities [What’s your P(e|~h)? Eliminating alternative hypotheses]

3 Domains of Circularity 1: Circular Explanations Instead of explaining, merely re-describes the observed phenomenon Gigerenzer (2009) makes grand accusations against much of psychology Matching bias in Wason selection task is a result of greater “transparency” – but what does that mean?

Levels of Explanation Explanation / Mechanism Theory Data

3 Domains of Circularity 2: Circular Analysis Voodoo correlations—practice of pre-selecting data Is it circular? No: not technically an argument No claim nor evidence, but bias introduced to data Presents a limit on the “interestingness” of findings Is it wrong? Not really… but it’s not right, either; why bother? Inferences based on this data are statistically invalid Consider model fitting Model is constructed to accommodate data Evaluating freedom of model (i.e., minimizing parameters) helps to guard against false positives

3 Domains of Circularity 3: Circular Methods Refers to Fiedler (2011) Circular? Not technically, but significant concerns about “interestingness”

Hahn: Conclusions Q: Is psychology a circular science? A: No; at least, not more than any other science. Q: Why write a paper about circularity in psychology when there is none? A: To be pedantic. Be aware at what level you are speaking/thinking about a phenomenon

Much Pain, Little Gain? Paradigm-Specific Models and Methods in Experimental Psychology Thorstein Meiser, University of Mannheim

Using Paradigms Paradigms can be useful, but they they can also hinder scientific progress Wason card selection task was developed to study deductive reasoning, but turns out people don’t reason deductively with it Inappropriate for studying deductive reasoning, but fecund paradigm for other ideas

Variables & Constructs Manifest X & Y variables in a given study map onto latent constructs ξ & η Considering a given paradigm as being sui generis can divorce the real constructs Example: DRM paradigm purports to be an instance of false memory, but is also an instance of category learning (e.g., Hintman, 1986) Need to bridge gaps in literatures, connect disparate literatures Like Bobbie Spellman did with category induction & persuasion (Ranganath, Spellman, & Joy-Gaba, 2010)

Meiser: Conclusions We should aspire to whittle down the universe of plausible explanations and constructs, not add to it Relying on traditional paradigms (and accounts of paradigms) can obfuscate further knowledge to be learned from them Paradigm-independent methodologies, such as signal detection theory, are advocated

Friends and Foes of Theory Construction in Psychological Science: Vague Dichotomies, Unified Theories of Cognition, and the New Experimentalism Leonel Garcia-Marques & Mario B. Ferreira, University of Lisbon

Dichotomous Theories Competing choices ex. Modal vs. amodal conceptualization, analogical vs. propositional representation In reality, two out of an infinite possibility of theories (e.g., Goodman’s (1946) concept of “grue”) Textbook experiments—illustration of a particular theory, leave aside complications, exceptions, etc. Problem of theoretical mimcry—same result can be accounted for by modification of alternate theory

Unified Theories of Cognition Attempts to explain cognitive architecture at a grand scope John Anderson’s ACT & ACT-R, SOAR, etc. Avoid problems of more specific theories, but suffer their own problems Task theories—how is the specific instantiation represented? Parameter overfitting—what can explain everything explains nothing

The New Experimentalism Dichotomies present the possibility of unproductive theoretical mimics UTCs are limited by parameter fitting and task theories Good experiments: Lead to productive errors (effetive error-correcting mechanisms) Results are only weakly theory-dependent Tested hypotheses are easily falsifiable Severe tests of specific hypotheses (low false positive rate) The results of good experiments can effectively serve to limit the search space of possible theories More novel results must be accommodated by new theories, reducing theory glut

Garcia-Marques & Ferreira: Conclusions Generate more useful hypotheses Have >2 levels of IV Describe data better; ex. Exploratory analyses, robust descriptive statistics Use effect sizes Test alternate accounts

Why the Cogntive Approach in Psychology Would Profit From a Functional Approach and Vice Versa Jan De Houwer, Ghent University

Extrapolation from Behavior The cognitive perspective aims to uncover the mental operations that are responsible for behavior Behavioral effects are often taken as proxies for mental events Ex. Classical conditioning paradigm is taken to mean formation of association between stimuli in memory Mental construct is not a necessary component; alternate causes could explain effects at least part of the time Could take a tentative view; what’s the harm? Researchers could take construct for granted, become conditioned and shut out alternate explanations Theories that are invested in particular behavioral effects could be massively disrupted if they are found to be poor proxies

Questions Does this entail a difference in how we talk about experiments, or about how we think about them? (Both) Is this an argument for a stricter separation between experimentation and theorizing? (Yes)

De Houwer: Conclusions Don’t skip steps in your explanations Before you propose a mental construct as the explanans of a behavioral effect, be sure you know what the environmental explanans is This goes double for imaging studies Five suggestions: Use separate functional and mental terms Define behavioral effects by causal impact of specific environmental stimuli on specific aspects of behavior Don’t treat behavioral effects as proxies of mental constructs First describe empirical findings in functional terms; should be abstract and general Research should be directed to advance both mental and functional explanations

Talking Theory

More Related Content

What's hot (18)

Viewers also liked (6)

Similar to Talking Theory (20)

More from Colin Smith-Clark (7)

Recently uploaded (20)

Talking Theory

Editor's Notes