![c
2007 Massachusetts Institute of Technology
All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means
(including photocopying, recording, or information storage and retrieval) without permission in writing from the
publisher.
For information about special quantity discounts, please email special sales@mitpress.mit.edu.
This book was set in LaTex by the authors.
Printed and bound in the United States of America.
Library of Congress Cataloging-in-Publication Data
Toward brain-computer interfacing / edited by Guido Dornhege ... [et al.] ; foreword by Terrence J. Sejnowski.
p.; cm. – (Neural information processing series)
”A Bradford book.”
Includes bibliographical references and index.
ISBN 978-0-262-04244-4 (hardcover : alk. paper)
1. Brain-computer interfaces. I. Dornhege, Guido. II. Series.
[DNLM: 1. Brain Mapping. 2. User-Computer Interface. 3. Brain–physiology. 4. Psychomotor Performance.
5. Rehabilitation–instrumentation. WL 335 T737 2007]
QP360.7.T69 2007
612.8’2–dc22
2007000517
10 9 8 7 6 5 4 3 2 1](https://image.slidesharecdn.com/2074768-250610114936-b7395cd4/85/Toward-Braincomputer-Interfacing-1st-Edition-Guido-Dornhege-8-320.jpg)
![1.6 Recording Methods, Paradigms, and Systems for Brain-Computer Interfacing 9
Figure 1.2 Generic noninvasive BCI setup: signals are recorded, e.g., with EEG, meaningful
features are extracted and subsequently classified. Finally, a signal is extracted from the classifier
that provides the control signal for some device or machine.
1.6.1.1 Brain Signals Recorded from the Scalp (EEG-BCIs)
In a typical BCI setting, participants are presented with stimuli or are required to perform
specific mental tasks while the electrical activity of their brains is being recorded by EEG
(see figure 1.2 for a general setup of a noninvasive BCI). Extracted and relevant EEG
features can then be fed back to the user by so-called closed-loop BCIs. Specific features of
the EEG are either regulated by the BCI user (slow cortical potentials (SCP), sensorimotor
rhythms (SMR)) or are elicited by visual, tactile, or auditory stimulation (event-related
potentials, namely the P300 or steady-state [visually-]evoked potentials (SS[V]EP)). In the
following paragraphs we provide a short description of the physiology of these features
and their use for brain-computer interfacing.
Slow Cortical Potentials (SCP)
Research over the past thirty years on SCPs and their regulation led to the excitation-
threshold-regulation theory (Birbaumer et al. (1990, 2003); Strehl et al. (2006)). The
vertical arrangement of pyramidal cells in the cortex is essential for the generation of SCP
(see figure 1.3). Most apical dendrites of pyramidal cells are located in cortical layers I and
II. Depolarization of the apical dendrites giving rise to SCP is dependent on sustained
afferent intracortical or thalamocortical input to layers I and II, and on simultaneous
depolarization of large pools of pyramidal neurons. The SCP amplitude recorded from
the scalp depends upon the synchronicity and intensity of the afferent input to layers I and
II (Speckmann et al. (1984)). The depolarization of cortical cell assemblies reduces their
excitation threshold such that firing of neurons in regions responsible for specified motor or
cognitive tasks is facilitated. Negative amplitude shifts grow with increasing attentional or
cognitive resource allocation. Cortical positivity may result from active inhibition of apical
dendritic neural activity or simply from a reduction of afferent inflow and subsequent
reduced postsynaptic activity. In any case, positive SCPs are considered to increase the
excitation threshold of upper cortical layers via a negative feedback loop involving the
basal ganglia and the reticular nucleus of the thalamus. Increasing cortical negativity is
accompanied by increased activation of inhibitory striatal nuclei that leads to an increase](https://image.slidesharecdn.com/2074768-250610114936-b7395cd4/85/Toward-Braincomputer-Interfacing-1st-Edition-Guido-Dornhege-24-320.jpg)
Toward Braincomputer Interfacing 1st Edition Guido Dornhege
- 1. Toward Braincomputer Interfacing 1st Edition Guido Dornhege download https://ebookbell.com/product/toward-braincomputer- interfacing-1st-edition-guido-dornhege-4149536 Explore and download more ebooks at ebookbell.com
- 2. Here are some recommended products that we believe you will be interested in. You can click the link to download. Toward Braincomputer Interaction In Paralysis A New Approach Based On Visual Evoked Potentials And Depthoffield 1st Edition Anibal Cotrina Auth https://ebookbell.com/product/toward-braincomputer-interaction-in- paralysis-a-new-approach-based-on-visual-evoked-potentials-and- depthoffield-1st-edition-anibal-cotrina-auth-5882504 Towards Practical Braincomputer Interfaces Bridging The Gap From Research To Realworld Applications 1st Edition Brendan Z Allison https://ebookbell.com/product/towards-practical-braincomputer- interfaces-bridging-the-gap-from-research-to-realworld- applications-1st-edition-brendan-z-allison-4404600 Raveling The Brain Toward A Transdisciplinary Neurorhetoric 1st Edition Jordynn Jack https://ebookbell.com/product/raveling-the-brain-toward-a- transdisciplinary-neurorhetoric-1st-edition-jordynn-jack-23894914 Zen And The Brain Toward An Understand Of Meditation And Consciousness Austin https://ebookbell.com/product/zen-and-the-brain-toward-an-understand- of-meditation-and-consciousness-austin-11640830
- 3. Zenbrain Horizons Toward A Living Zen James H Austin Md https://ebookbell.com/product/zenbrain-horizons-toward-a-living-zen- james-h-austin-md-56401730 Mindbraingene Toward Psychotherapy Integration Hardcover John Arden https://ebookbell.com/product/mindbraingene-toward-psychotherapy- integration-hardcover-john-arden-7347546 Theodore W Berger Dennis L Glanzman Toward Replacement Parts For The Brain Implantable Biomimetic Electronics As Neural Prostheses https://ebookbell.com/product/theodore-w-berger-dennis-l-glanzman- toward-replacement-parts-for-the-brain-implantable-biomimetic- electronics-as-neural-prostheses-56389312 Neurobiology Of Social Behavior Toward An Understanding Of The Prosocial And Antisocial Brain 1st Edition Michael Numan https://ebookbell.com/product/neurobiology-of-social-behavior-toward- an-understanding-of-the-prosocial-and-antisocial-brain-1st-edition- michael-numan-5138316 Chronic Medical Disease And Cognitive Aging Toward A Healthy Body And Brain Kristine Yaffe https://ebookbell.com/product/chronic-medical-disease-and-cognitive- aging-toward-a-healthy-body-and-brain-kristine-yaffe-51268884
- 6. Neural Information Processing Series Michael I. Jordan and Thomas Dietterich, editors Advances in Large Margin Classifiers, Alexander J. Smola, Peter L. Bartlett, Bernhard Schölkopf, and Dale Schuurmans, eds., 2000 Advanced Mean Field Methods: Theory and Practice, Manfred Opper and David Saad, eds., 2001 Probabilistic Models of the Brain: Perception and Neural Function, Rajesh P. N. Rao, Bruno A. Olshausen, and Michael S. Lewicki, eds., 2002 Exploratory Analysis and Data Modeling in Functional Neuroimaging, Friedrich T. Som- mer and Andrzej Wichert, eds., 2003 Advances in Minimum Description Length: Theory and Applications, Peter D. Grunwald, In Jae Myung, and Mark A. Pitt, eds., 2005 Nearest-Neighbor Methods in Learning and Vision: Theory and Practice, Gregory Shakhnarovich, Piotr Indyk, and Trevor Darrell, eds., 2006 New Directions in Statistical Signal Processing: From Systems to Brains, Simon Haykin, José C. Prı́ncipe, Terrence J. Sejnowski, and John McWhirter, eds., 2007 Predicting Structured Data, Gökhan Bakir, Thomas Hofmann, Bernard Schölkopf, Alexan- der J. Smola, Ben Taskar, S. V. N. Vishwanathan, eds., 2007 Toward Brain-Computer Interfacing, Guido Dornhege, José del R. Millán, Thilo Hinter- berger, Dennis J. McFarland, Klaus-Robert Müller, eds., 2007 Large Scale Kernel Machines, Léon Bottou, Olivier Chapelle, Denis DeCoste, Jason West- man, eds., 2007
- 7. Toward Brain-Computer Interfacing edited by Guido Dornhege José del R. Millán Thilo Hinterberger Dennis J. McFarland Klaus-Robert Müller foreword by Terrence J. Sejnowski A Bradford Book The MIT Press Cambridge, Massachusetts London, England
- 8. c 2007 Massachusetts Institute of Technology All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the publisher. For information about special quantity discounts, please email special sales@mitpress.mit.edu. This book was set in LaTex by the authors. Printed and bound in the United States of America. Library of Congress Cataloging-in-Publication Data Toward brain-computer interfacing / edited by Guido Dornhege ... [et al.] ; foreword by Terrence J. Sejnowski. p.; cm. – (Neural information processing series) ”A Bradford book.” Includes bibliographical references and index. ISBN 978-0-262-04244-4 (hardcover : alk. paper) 1. Brain-computer interfaces. I. Dornhege, Guido. II. Series. [DNLM: 1. Brain Mapping. 2. User-Computer Interface. 3. Brain–physiology. 4. Psychomotor Performance. 5. Rehabilitation–instrumentation. WL 335 T737 2007] QP360.7.T69 2007 612.8’2–dc22 2007000517 10 9 8 7 6 5 4 3 2 1
- 9. Contents Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Terrence J. Sejnowski Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi 1 An Introduction to Brain-Computer Interfacing . . . . . . . . . . . . . . 1 Andrea Kübler and Klaus-Robert Müller I BCI Systems and Approaches 27 2 Noninvasive Brain-Computer Interface Research at the Wadsworth Center 31 Eric W. Sellers, Dean J. Krusienski, Dennis J. McFarland, and Jonathan R. Wolpaw 3 Brain-Computer Interfaces for Communication in Paralysis: A Clinical Experimental Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Thilo Hinterberger, Femke Nijboer, Andrea Kübler, Tamara Matuz, Adrian Furdea, Ursula Mochty, Miguel Jordan, Thomas Navin Lal, N. Jeremy Hill, Jürgen Mellinger, Michael Bensch, Michael Tangermann, Guido Widman, Christian E. Elger, Wolfgang Rosenstiel, Bernhard Schölkopf, and Niels Birbaumer 4 Graz-Brain-Computer Interface: State of Research . . . . . . . . . . . . 65 Gert Pfurtscheller, Gernot R. Müller-Putz, Alois Schlögl, Bernhard Graimann, Reinhold Scherer, Robert Leeb, Clemens Brunner, Claudia Keinrath, George Townsend, Carmen Vidaurre, Muhammad Naeem, Felix Y. Lee, Selina Wriessnegger, Doris Zimmermann, Eva Höfler, and Christa Neuper
- 10. vi Contents 5 The Berlin Brain-Computer Interface: Machine Learning-Based Detection of User Specific Brain States . . . . . . . . . . . . . . . . . . . . . . . . . 85 Benjamin Blankertz, Guido Dornhege, Matthias Krauledat, Volker Kunzmann, Florian Losch, Gabriel Curio, and Klaus-Robert Müller 6 The IDIAP Brain-Computer Interface: An Asynchronous Multiclass Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 José del R. Millán, Pierre W. Ferrez, and Anna Buttfield 7 Brain Interface Design for Asynchronous Control . . . . . . . . . . . . . 111 Jaimie F. Borisoff, Steve G. Mason, and Gary E. Birch II Invasive BCI Approaches 123 8 Electrocorticogram as a Brain-Computer Interface Signal Source . . . . 129 Jane E. Huggins, Bernhard Graimann, Se Young Chun, Jeffrey A. Fessler, and Simon P. Levine 9 Probabilistically Modeling and Decoding Neural Population Activity in Motor Cortex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 Michael J. Black and John P. Donoghue 10 The Importance of Online Error Correction and Feed-Forward Adjustments in Brain-Machine Interfaces for Restoration of Movement . . . . . . . . . 161 Dawn M. Taylor 11 Advances in Cognitive Neural Prosthesis: Recognition of Neural Data with an Information-Theoretic Objective . . . . . . . . . . . . . . . . . . . . . 175 Zoran Nenadic, Daniel S. Rizzuto, Richard A. Andersen, and Joel W. Burdick 12 A Temporal Kernel-Based Model for Tracking Hand Movements from Neural Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 Lavi Shpigelman, Koby Crammer, Rony Paz, Eilon Vaadia, and Yoram Singer III BCI Techniques 203 13 General Signal Processing and Machine Learning Tools for BCI Analysis 207 Guido Dornhege, Matthias Krauledat, Klaus-Robert Müller, and Benjamin Blankertz
- 11. Contents vii 14 Classifying Event-Related Desynchronization in EEG, ECoG, and MEG Signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 N. Jeremy Hill, Thomas Navin Lal, Michael Tangermann, Thilo Hinterberger, Guido Widman, Christian E. Elger, Bernhard Schölkopf, and Niels Birbaumer 15 Classification of Time-Embedded EEG Using Short-Time Principal Component Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 Charles W. Anderson, Michael J. Kirby, Douglas R. Hundley, and James N. Knight 16 Noninvasive Estimates of Local Field Potentials for Brain-Computer Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 Rolando Grave de Peralta Menendez, Sara Gonzalez Andino, Pierre W. Ferrez, and José del R. Millán 17 Error-Related EEG Potentials in Brain-Computer Interfaces . . . . . . . 291 Pierre W. Ferrez and José del R. Millán 18 Adaptation in Brain-Computer Interfaces . . . . . . . . . . . . . . . . . . 303 José del R. Millán, Anna Buttfield, Carmen Vidaurre, Matthias Krauledat, Alois Schlögl, Pradeep Shenoy, Benjamin Blankertz, Rajesh P. N. Rao, Rafael Cabeza, Gert Pfurtscheller, and Klaus-Robert Müller 19 Evaluation Criteria for BCI Research . . . . . . . . . . . . . . . . . . . . 327 Alois Schlögl, Julien Kronegg, Jane E. Huggins, and Steve G. Mason IV BCI Software 343 20 BioSig: An Open-Source Software Library for BCI Research . . . . . . . 347 Alois Schlögl, Clemens Brunner, Reinhold Scherer, and Andreas Glatz 21 BCI2000: A General-Purpose Software Platform for BCI . . . . . . . . . 359 Jürgen Mellinger and Gerwin Schalk V Applications 369 22 Brain-Computer Interfaces for Communication and Motor Control—Perspectives on Clinical Applications . . . . . . . . . . . . . . . 373 Andrea Kübler, Femke Nijboer, and Niels Birbaumer
- 12. viii Contents 23 Combining BCI and Virtual Reality: Scouting Virtual Worlds . . . . . . 393 Robert Leeb, Reinhold Scherer, Doron Friedman, Felix Y. Lee, Claudia Keinrath, Horst Bischof, Mel Slater, and Gert Pfurtscheller 24 Improving Human Performance in a Real Operating Environment through Real-Time Mental Workload Detection . . . . . . . . . . . . . . . . . . . 409 Jens Kohlmorgen, Guido Dornhege, Mikio L. Braun, Benjamin Blankertz, Klaus-Robert Müller, Gabriel Curio, Konrad Hagemann, Andreas Bruns, Michael Schrauf, and Wilhelm E. Kincses 25 Single-Trial Analysis of EEG during Rapid Visual Discrimination: Enabling Cortically Coupled Computer Vision . . . . . . . . . . . . . . . . . . . . . 423 Paul Sajda, Adam D. Gerson, Marios G. Philiastides, and Lucas C. Parra References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441 Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 491 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503
- 13. Foreword The advances in brain-computer interfaces in this book could have far-reaching conse- quences for how we interact with the world around us. A communications channel that bypasses the normal motor outflow from the brain will have an immediate benefit for paraplegic patients. Someday the same technology will allow humans to remotely con- trol agents in exotic environments, which will open new frontiers that we can only dimly imagine today. The earliest systems to be developed were based on noninvasive electroencephalo- graphic (EEG) recordings. Because these systems do not require invasive surgical implants, they can be used for a wide range of applications. The disadvantage is the relatively low rate of signaling that can be achieved. Nonetheless, advances in signal processing tech- niques and the development of dry electrodes make this an attractive approach. Three separate research areas have contributed to major advances in invasive brain- computer interfaces. First, the neural code for motor control was uncovered based on recordings from single neurons in different cortical areas of alert primates. The second was the development of mathematical algorithms for converting the train of spikes recorded from populations of these neurons to an intended action, called the decoding problem. Third, it was necessary to achieve stable, long-term recordings from small, cortical neurons in a harsh aqueous environment. For both invasive and noninvasive BCIs interdisciplinary teams of scientists and engi- neers needed to work closely together to create successful systems. Success in brain-computer interfaces has also depended on the remarkable ability of the brain to adapt to unusual tasks, none more challenging than “mind control” of extracorpo- real space. We are still at an early stage of development, but the field is moving forward rapidly and we can confidently expect further advances in the near future. Terrence J. Sejnowski La Jolla, CA
- 14. Preface The past decade has seen a fast growing interest to develop an effective communication interface connecting the human brain to a computer, the “brain-computer interface” (BCI). BCI research follows three major goals: (1) it aims to provide a new communication channel for patients with severe neuromuscular disabilities bypassing the normal output pathways, (2) it provides a powerful working tool in computational neuroscience to con- tribute to a better understanding of the brain, and finally (3)—often overseen—it provides a generic novel independent communication channel for man-machine interaction, a di- rection that is at only the very begining of scientific and practical exploration. During a workshop at the annual Neural Information Processing Systems (NIPS) conference, held in Whistler, Canada, in December 2004, a snapshot of the state of the art in BCI research was recorded. A variety of people helped in this, especially all the workshop speakers and attendees who contributed to lively discussions. After the workshop, we decided that it would be worthwhile to invest some time to have an overview about current BCI research printed. We invited all the speakers as well as other researchers to submit papers, which were integrated into the present collection. Since BCI research has previously not been covered in an entire book, this call has been widely followed. Thus, the present collection gathers contributions and expertise from many important research groups in this field, whom we wholeheartedly thank for all the work they have put into our joint effort. Note, of course, that since this book is the outcome of a workshop, it cannot cover all groups and it may— clearly unintentionally—contain some bias. However, we are confident that this book covers a broad range of present BCI research: In the first part we are able to present overviews about many important noninvasive (that is, without implanting electrodes) BCI groups in the world. We have been also able to win contributions from a few of the most important invasive BCI groups giving an overview of the current state of the invasive BCI research. These contributions are presented in the second part. The book is completed by three further parts, namely an overview of state-of-the-art techniques from machine learning and signal processing to process brain signals, an overview about existing software packages in BCI research, and some ideas about applications of BCI research for the real world.
- 15. xii Preface It is our hope that this outweighs the shortcomings of the book, most notably the fact that a collection of chapters can never be as homogeneous as a book conceived by a single author. We have tried to compensate for this by writing an introductory chapter (see chapter 1) and prefaces for all five parts of the book. In addition, the contributions were carefully refereed. Guido Dornhege, José del R. Millán, Thilo Hinterberger, Dennis J. McFarland, and Klaus-Robert Müller Berlin, Martigny, Tübingen, Albany, August 2006 Acknowledgments Guido Dornhege and Klaus-Robert Müller were funded by BMBF (FKZ 01IBE01A, 16SV2231, and 01GQ0415), EU PASCAL. Guido Dornhege furthermore acknowledges the support by the Fraunhofer Society for the BCI project. José del R. Millán acknowl- edges support from the European IST Programme FET Project FP6-003758, the European Network of Excellence “PASCAL,” and the Swiss National Science Foundation NCCR “IM2.” Thilo Hinterberger was supported by the German Research Society (DFG, SFB 550, and HI 1254/2-1) and the Samueli Institute, California. Dennis McFarland was sup- ported by NIH (NICHD (HD30146) and NIBIB/NINDS (EB00856)) and by the James S. McDonnell Foundation. Klaus-Robert Müller also wishes to thank, for warm hospitality during his stay in Tübingen at the Max-Planck Institute for Biological Cybernetics, the Friedrich Miescher Laboratory and the Department of Psychology at University of Tbin- gen, where part of this book was written. Klaus-Robert Müller furthermore acknowledges generous support by the Fraunhofer Society for the BCI endeavor and in particular his sabbatical project. Finally, we would like to thank everybody who contributed toward the success of this book project, in particular to Mel Goldsipe, Suzanne Stradley, Robert Prior, to all chapter authors, and to the chapter reviewers.
- 16. 1 An Introduction to Brain-Computer Interfacing Andrea Kübler Institute of Medical Psychology and Behavioural Neurobiology Eberhard-Karls-University Tübingen, Gartenstr. 29 72074 Tübingen, Germany Klaus-Robert Müller Fraunhofer–Institute FIRST Intelligent Data Analysis Group (IDA) Kekuléstr. 7, 12489 Berlin, Germany Technical University Berlin Str. des 17. Juni 135 10 623 Berlin, Germany 1.1 Abstract We provide a compact overview of invasive and noninvasive brain-computer interfaces (BCI). This serves as a high-level introduction to an exciting and active field and sets the scene for the following sections of this book. In particular, the chapter briefly assembles information on recording methods and introduces the physiological signals that are being used in BCI paradigms. Furthermore, we review the spectrum from subject training to machine learning approaches. We expand on clinical and human-machine interface (HMI) applications for BCI and discuss future directions and open challenges in the BCI field. 1.2 Overview Translating thoughts into actions without acting physically has always been material of which dreams and fairytales were made. Recent developments in brain-computer interface (BCI) technology, however, open the door to making these dreams come true. Brain- machine interfaces (BMI1) are devices that allow interaction between humans and artificial devices (for reviews see e.g. Kübler et al. (2001a); Kübler and Neumann (2005); Lebedev and Nicolelis (2006); Wolpaw et al. (2002)). They rely on continuous, real-time interaction between living neuronal tissue and artificial effectors. Computer-brain interfaces2 are designed to restore sensory function, transmit sensory information to the brain, or stimulate the brain through artificially generated electrical sig-
- 17. 2 An Introduction to Brain-Computer Interfacing nals. Examples of sensory neuroprostheses are the retina implant (e.g. Eckmiller (1997); Zrenner (2002)) and the cochlear implant, which circumvents the nonfunctioning audi- tory hair cells of the inner ear by transmitting electrically processed acoustic signals via implanted stimulation electrodes directly to the acoustic nerve (e.g., Zenner et al. (2000); Merzenich et al. (1974); Pfingst (2000)). Further, with an implanted stimulating neuropros- thesis, hyperactivity of the subthalamic nuclei can be inhibited to improve Parkinsonian symptoms (e.g., Mazzone et al. (2005); Benabid et al. (1991)). Brain-computer interfaces provide an additional output channel and thus can use the neuronal activity of the brain to control artificial devices, for example, for restoring motor function. Neuronal activity of few neurons or large cell assemblies is sampled and processed in real-time and converted into commands to control an application, such as a robot arm or a communication program (e.g., Birbaumer et al. (1999); Müller-Putz et al. (2005b); Taylor et al. (2002); Hochberg et al. (2006); Santhanam et al. (2006); Lebedev and Nicolelis (2006); Haynes and Rees (2006); Blankertz et al. (2006a); Müller and Blankertz (2006)). Brain activity is either recorded intracortically with multielectrode arrays or single electrodes, epi- or subdurally from the cortex or from the scalp. From the broad band of neuronal electrical activity, signal detection algorithms filter and denoise the signal of interest and decoded information is commuted into device commands. Over the past twenty years, increased BCI research for communication and control has been driven by a better understanding of brain function, powerful computer equipment, and by a growing awareness of the needs, problems, and potentials of people with disabilities (Wolpaw et al. (2002); Kübler et al. (2001a)). In addition to addressing clinical and quality of life issues, such interfaces constitute powerful tools for basic research on how the brain coordinates and instantiates human behavior and how new behavior is acquired and maintained. This is because a BCI offers the unique opportunity to investigate brain activity as an independent variable. In traditional psychophysiological experiments subjects are presented with a task or stimuli (independent variables), and the related brain activity is measured (dependent variable). Conversely, with neurofeedback by means of a BCI, subjects can learn to deliberately increase or decrease brain activity (independent variable) and changes in behavior can be measured accordingly (dependent variable). Studies on regulation of slow cortical potentials, sensorimotor rhythms, and the BOLD response (see below) yield various specific effects on behavior, such as decreased reaction time in a motor task after activation of contralateral motor cortex (Rockstroh et al. (1982)), faster lexical decisions (Pulvermüller et al. (2000)), or improved memory performance as a function of deactivation of the parahippocampal place area (Weiskopf et al. (2004b)). In these examples, the link between activation and deactivation of a specific cortical area and changes in behavior is quite evident. More general effects on learning such as better musical performance in music students (techniques and subjective interpretation) and better dancing performance in dance students (technicality, flair, overall execution) were observed after regularization of alpha and theta activity (Gruzelier and Egner (2005); Raymond et al. (2005)) An often overlooked direction of BCI applications beyond clinical and basic research aspects is the yet unexplored use of BCI as an additional independent channel of man-
- 18. 1.3 Approaches to BCI Control 3 machine interaction (see chapters 23, 24 and 25 for first examples in this direction of research). In particular, brain signals can provide direct access to aspects of human brain state such as cognitive workload, alertness, task involvement, emotion, or concentration. The monitoring of these will allow for a novel technology that directly adapts a man- machine interface design to the inferred brain state in real-time. Furthermore, BCI technology can in the near future serve as an add-on when devel- oping new computer games, for example, fantasy games that require the brain-controlled mastering of a task for advancing to the next game level. A variety of technologies for monitoring brain activity may serve as a BCI. In addi- tion to electroencephalography (EEG) and invasive electrophysiological methods, these in- clude magnetoencephalography (MEG), positron emission tomography (PET), functional magnetic resonance imaging (fMRI), and optical imaging (functional near infrared spec- troscopy, fNIRS). As MEG, PET, and fMRI are demanding, tied to the laboratory, and expensive, these technologies are more suitable to address basic research questions and short-term intervention for location of sources of brain activity and alteration of brain ac- tivity in diseases with known neurobiological dysfunction. In contrast, EEG, NIRS, and invasive devices are portable, and thus may offer practical BCIs for communication and control in daily life. Current BCIs for human users have been mainly used for cursor control and communi- cation by means of selection of letters or items on a computer screen (e.g., Birbaumer et al. (1999); Blankertz et al. (2006a); Hochberg et al. (2006); Obermaier et al. (2003); Wolpaw and McFarland (2004)). An overview of BCI applications in clinical populations is given in chapter 22. Interfaces between machine and the animal brain have been used to control robotic arms (e.g., Taylor et al. (2002); Wessberg et al. (2000), and for a review, see Lebedev and Nicolelis (2006)). However, before BCIs can be utilized across a wide range of clinical or daily life settings, many open technological issues must be resolved. Sensors are the bottleneck of todays invasive and noninvasive BCIs: invasive sensors can last only a limited time before they lose signal (Hochberg et al. (2006); Nicolelis et al. (2003)), and noninvasive sensors need long preparation time due to the use of conductive gel. More neurobiological and psychological research is necessary to understand the interaction between neurons and behavior related to the use of BCIs. Already machine learning and advanced signal processing methods play a key role in BCI research as they allow the decoding of different brain states within the noise of the spontaneous neural activity in real-time (see chapters 9, 11, 12, 13, 14, 15, 16, and 18). There is, however, a need for continuous improvement; in particular, higher robustness, online adaptation to compensate for nonstationarities, sensor fusion strategies, and techniques for transferring classifier or filter parameters from session to session are among the most burning topics. 1.3 Approaches to BCI Control Two separate approaches to BCI control exist, while almost all BCIs realize a mix- ture of both approaches: (1) Learning to voluntarily regulate brain activity by means of
- 19. 4 An Introduction to Brain-Computer Interfacing neurofeedback and operant learning principles. Following subject training, in which the subject learns to regulate a specific brain activity by means of feedback, different brain states can be produced on command and, thus, become suitable as control commands. (2) Machine learning procedures that enable the interference of the statistical signature of spe- cific brain states or intentions within a calibration session (see chapter 5). 1.3.1 The Biofeedback Approach—Voluntary Control of the Brain Response Biofeedback is a procedure that, by means of feedback of a (seemingly) autonomous parameter, aims at acquiring voluntary control over this parameter. Participants receive visual, auditory, or tactile information about their cardiovascular activity (heartrate, blood pressure), temperature, skin conductance, muscular activity, electrical brain activity (EEG, MEG), or the blood oxygen level dependent (BOLD) response (with fMRI). In discrete or continuous trials, the participants are presented with the task to either increase or decrease the activity of interest. By means of the feedback signal, participants receive continuous information about the alteration of the activity. At the end of the trial, participants are informed about their performance (e.g., by highlighting a correctly hit target) and correct trials may be positively reinforced by a smiling face (Kübler et al. (1999); see also chapter 3) or by earning tokens that can be exchanged later for toys (e.g., training of children with ADHD in first author’s affiliation). If participants are repeatedly trained, they learn to manipulate the activity of interest, which is then—at least to a certain extent— under voluntary or conscious (cortical) control. 1.3.2 The Machine Learning Approach—Detection of the Relevant Brain Signal A somewhat opposite approach3 is the machine learning approach to BCI, where the train- ing is relocated from the subject to the learning algorithm. Thus, decoding algorithms are individually adapted to the users that perform the task. For obtaining a qualitative impres- sion about the variability that is to be compensated, see chapter 13, figures 13.1 and 13.2, where different individuals perform finger tapping or motor imagery. Note that even the intraindividual variance between sessions is high. Learning algorithms require examples from which they can infer the underlying statistical structure of the respective brain state. Therefore, subjects are first required to repeatedly produce a certain brain state during a calibration session (e.g., for the BBCI, this calibration session takes approximately twenty minutes, see chapter 5). Even from such a small amount of data, current learning machines can extract spatiotemporal blueprints of these brain states, which are readily usable in the subsequent feedback session. The tackling of the enormous trial-to-trial variability is a ma- jor challenge in BCI research. We believe that advanced techniques for machine learning are an essential tool in this endeavor. The use of state-of-the-art learning machines enables not only the achievement of high decision accuracies for BCI (e.g., chapters 5, 6, 9, 12, 13, and 14), but also, as a by-product of the classification, the few most salient features for classification are found, which can then be matched with neurophysiological knowledge. In this sense, machine learning approaches are useful beyond the pure classification or adaptive spatiotemporal filtering step, as they can contribute to a better interpretation and
- 20. 1.4 Clinical Target Groups—Individuals in Need of a BCI for Motor Control and Communication 5 understanding of a novel paradigm per se (see Blankertz et al. (2006b)). Thus, machine learning can be usefully employed in an exploratory scenario, where (1) a new paradigm is tested that also could generate unexpected neurophysiological signals, (2) a hypothesis about underlying task relevant brain processes is generated automatically by the learning machine through feature extraction, and (3) the paradigm can be refined, and thus a better understanding of the brain processes could be achieved (see figure 13.8). In this sense, a machine learning method offering explanation can be of great use in the semiautomatic ex- ploration loop for testing new paradigms. Note that this holds also for data analysis beyond decoding of brain signals. 1.3.3 Integration of the Two Approaches The two last paragraphs reflect opposite positions. In practice, BCIs will neither rely solely on feedback learning of the users nor only on machine learning. For example, in the BBCI (see chapter 5) that has no explicit user biofeedback training, a user’s brain activity will adapt to the settings of the decoding algorithm when using the BCI in feedback mode, such that the most successful EEG activity pattern will be repeatedly produced. Thus, a coadaptation of the learning user and algorithm occurs inevitably. However, it remains unclear how to optimally bring these two interacting learning systems into synchrony; a thorough study is still missing. Experimentally, the two learning systems can be coupled using online learning (see chapter 18 for discussion). It is furthermore important to note that in a proportion of the subject population, typically in 20 percent of the users, one is unable to successfully classify the brain activation patterns. We refer to this group as the BCI illiterates. This finding holds no matter whether machine learning or biofeedback is used to train the subjects. Further research is needed to fully understand and overcome the BCI illiteracy phenomenon. 1.4 Clinical Target Groups—Individuals in Need of a BCI for Motor Control and Communication A variety of neurological diseases such as motor neuron diseases, spinal cord injury, stroke, encephalitis, or traumatic brain injury may lead to severe motor paralysis, which may also include speech. Patients may have only a few muscles to control artificial devices for communicating their needs and wishes and interacting with their environment. We refer to the locked-in state if some residual voluntary muscular movement, such as eye or lip movement, is still possible. People who lost all voluntary muscular movement are referred to as being in the complete locked-in state (see also chapter 22 and Birbaumer (2006a)). In the realm of BCI use, it is of particular importance how and how much the brain is affected by disease. To provide a detailed discussion of all diseases that may lead to the locked-in syndrome would go beyond the scope of this introduction. Thus, we will refer to only those diseases that have been repeatedly reported in the BCI literature, that is amyotrophic lateral sclerosis, high spinal cord injury, and stroke; all three diseases have quite different effects on the brain.
- 21. 6 An Introduction to Brain-Computer Interfacing 1.4.1 Amyotrophic Lateral Sclerosis Amyotrophic lateral sclerosis (ALS) is a progressive neurodegenerative disease involving the first and second motoneurons and the central nervous system (see also chapter 22). Patients with ALS show global brain atrophy with regional decreases of grey matter density being highest in right-hemispheric primary motor cortex and left-hemispheric medial frontal gyrus (Kassubek et al. (2005)). White matter reduction is found along the corticospinal tracts, in the corpus callosum, and in frontal and parietal cortices. Clinical symptoms are atrophic paresis with fasciculations mostly starting in hands and lower arms. With progressive neuronal degeneration, patients become severely physically impaired. In later stages of the disease, speech, swallowing, and breathing are also affected. Patients succumb to respiratory failure unless they choose artificial ventilation via tracheotomy. Patients with tracheotomy may render the locked-in state with only residual muscular movement or even the completely locked-in state. Cognitive impairment has been reported repeatedly (Hanagasi et al. (2002); Ringholz et al. (2005)), but improved learning has also been shown (Lakerfeld et al. (submitted); Rottig et al. (2006)). Emotional processing seems to be altered such that positive and negative extremes are attenuated (Lulé et al. (2005)). Quality of life in ALS patients is surprisingly high and within the range of patients with nonfatal diseases such as diabetes or irritable bowl syndrome (Kübler et al. (in preparation)). One important component of individual quality of life repeatedly mentioned by patients, specifically as the disease progresses, is the ability to communicate. 1.4.2 Cervical Spinal Cord Injury Most often spinal cord injury follows trauma. It may also occur due to acute ischaemia in the arteria spinalis-anterior or acute compression. Acute symptoms are spinal shock with atonic paresis below the lesion, atonic bladder, paralysis of the rectum, disturbed sensitivity in all qualities (pain, pressure, temperature) and vegetative dysfunction. These symptoms continue into the post-traumatic phase and are endorsed by painful, involuntary stretching and bending of extremities (so-called spinal automatisms). Cervical spinal cord injury has been shown to be accompanied by local cortical grey matter reduction in somatosensory areas (S1) bilaterally located posterior to the hand region in M1. Atrophy also occurred in the right leg area and extended to parietal BA5 in the left hemisphere (Jurkiewicz et al. (2006)). Several years post trauma, patients may be well adapted to a life with impairment, experience a balanced emotional life, and lead an intact social life. Pain contributes to poorer quality of life, and gainful employment is related to high quality of life (Lundqvist et al. (1991)). Clinically relevant symptoms of depression occur specifically in the first year post injury (Hancock et al. (1993)). 1.4.3 Brain Stem Stroke The classic locked-in syndrome as defined by Bauer and colleagues is characterized by total immobility except for vertical eye movement and blinking (Bauer et al. (1979); Smith and Delargy (2005)). Most often the locked-in syndrome is of cerebrovascular origin such
- 22. 1.5 Brain-Computer Interfaces for Healthy Subjects 7 that thrombotic occlusion of the arteria basilaris leads to infarction in the ventral pons (Katz et al. (1992); Patterson and Grabois (1986)). As a result, corticobulbar and cortical spinal tracts are interrupted as are both the supranuclear and postnuclear oculomotor fibers. If movements other than vertical eye movement is preserved, the locked-in syndrome is referred to as incomplete, and if no movement, and thus no communication, is possible as total (Bauer et al. (1979)). Higher cortical areas or subcortical areas besides the brain stem are not affected. Consequently, consciousness and cognition are usually unimpaired in such patients. A survey on quality of life in chronic locked-in patients (more than one year after diagnosis) with no major motor recovery, revealed no differences to healthy controls in the perception of mental and general health (Laureys et al. (2005)). In a survey (N = 44) by Leon-Carrion et al., less than 20 percent of the patients described their mood as bad (5 percent) or reported to be depressed (12.5 percent) and 81 percent met with friends more than twice a month (Leon-Carrion et al. (2002)). Many locked-in patients return home from hospital and start a different but meaningful life (Laureys et al. (2005)). 1.5 Brain-Computer Interfaces for Healthy Subjects Applications of BCI technology go beyond rehabilitation. Although BCI for healthy sub- jects is pursued much less, it is of high industrial relevance. It is less the desire to com- municate for the healthy: this is much more easily done via keyboard, computer mouse, speech, or gesture recognition devices. It is this additional independent channel “BCI” for man-machine interaction (see chapters 23, 24 and 25 for first examples in this direction of research) that has remained unexplored. Brain signals read in real-time on a single trial basis could provide direct access to human brain states, which can then be used to adapt the man-machine interface on the fly. One application field could be monitoring tasks such as alertness monitoring, where the brain holds the key to access information that can oth- erwise not be easily acquired. Signals of interest to be inferred from brain activity are cognitive workload, alertness, task involvement, emotion, or concentration. For instance, workload could be assessed in behavioral experiments by measuring reaction times. How- ever, this would give very indirect and therefore imprecise measures with respect to tem- poral resolution, quality, and context. The online monitoring of cognitive workload could contribute to construct better systems in safety critical applications (see chapter 24). A fur- ther direction is the direct use of brain states in computer applications or as novel features for computer gaming (see figure 1.1). The latter is an interesting challenge since the game interfaces should be able to compensate for the imperfect signal of a BCI. In other words, if the classification rate of a BCI is 95 percent, then the respective computer game interface will have to be robust with respect to the 5 percent errors that will inevitably occur. Tetris, although already successfully played with the BBCI system, is a good example of a game where small errors can seriously spoil the course of the game. The current state of EEG sensor technology and the price of EEGs are major obstacles for a broad use of BCI technology for healthy users. However, once fashionable, cheap, contactless EEG caps are available—for example, in the shape of baseball caps—a wide market and application perspective will immediately open.
- 23. 8 An Introduction to Brain-Computer Interfacing Figure 1.1 The simple game of Pong is revived in a new technological context: imagination of the right hand moves the cursor to the right, imagination of the left hand pushes the cursor to the left. In this manner, the ball that is reflected from the sides of the game field can be hit by the brain-racket. Thus, the user can use his intentions to play “Brain Pong.” Dornhege (2006); Krepki (2004); Krepki et al. (2007). 1.6 Recording Methods, Paradigms, and Systems for Brain-Computer Interfacing Current BCIs differ in how the neural activity of the brain is recorded, how subjects (humans and animals) are trained, how the signals are translated into device commands, and which application is provided to the user. An overview of current noninvasive BCIs is provided in chapters 2–7, while invasive BCIs are discussed in chapters 8–12. An overview of existing software packages is found in chapters 20 and 21. 1.6.1 Noninvasive Recording Methods for BCI The electrical activity of the brain can be recorded noninvasively with electroencephalogra- phy (EEG) (e.g., Birbaumer et al. (1999); Pfurtscheller et al. (2000b); Wolpaw et al. (2003); Blankertz et al. (2006a)). The current produced by neural activity induces a magnetic field that can be recorded with magnetoencephalography (MEG) (Birbaumer and Cohen (2005)). Increased neural activity is accompanied by locally increased glucose metabolism, resulting in increased glucose and oxygene consumption. As a consequence of glucose con- sumption, cranial arteries dilate, allowing for increased blood flow, that results in hyper- oxygenation of the active tissue. Imaging techniques make use of the different magnetic and optical properties of oxygenated and deoxygenated hemoglobin. The different mag- netic properties of the ferrous on the heme of oxy- and deoxyhemoglobin are the basis of the blood oxygen level dependent (BOLD) response measured with functional magnetic resonance imaging (fMRI) (Chen and Ogawa (2000)). Oxy- and deoxyhemoglobin have different optical properties in the visible and near infrared range. The changes in the ratio of oxygenated hemoglobin to blood volume due to neural activity is measured with near infrared spectroscopy (NIRS) (Bunce et al. (2006)). In the following sections, we briefly review noninvasive BCIs categorized according to the recording techniques.
- 24. 1.6 Recording Methods, Paradigms, and Systems for Brain-Computer Interfacing 9 Figure 1.2 Generic noninvasive BCI setup: signals are recorded, e.g., with EEG, meaningful features are extracted and subsequently classified. Finally, a signal is extracted from the classifier that provides the control signal for some device or machine. 1.6.1.1 Brain Signals Recorded from the Scalp (EEG-BCIs) In a typical BCI setting, participants are presented with stimuli or are required to perform specific mental tasks while the electrical activity of their brains is being recorded by EEG (see figure 1.2 for a general setup of a noninvasive BCI). Extracted and relevant EEG features can then be fed back to the user by so-called closed-loop BCIs. Specific features of the EEG are either regulated by the BCI user (slow cortical potentials (SCP), sensorimotor rhythms (SMR)) or are elicited by visual, tactile, or auditory stimulation (event-related potentials, namely the P300 or steady-state [visually-]evoked potentials (SS[V]EP)). In the following paragraphs we provide a short description of the physiology of these features and their use for brain-computer interfacing. Slow Cortical Potentials (SCP) Research over the past thirty years on SCPs and their regulation led to the excitation- threshold-regulation theory (Birbaumer et al. (1990, 2003); Strehl et al. (2006)). The vertical arrangement of pyramidal cells in the cortex is essential for the generation of SCP (see figure 1.3). Most apical dendrites of pyramidal cells are located in cortical layers I and II. Depolarization of the apical dendrites giving rise to SCP is dependent on sustained afferent intracortical or thalamocortical input to layers I and II, and on simultaneous depolarization of large pools of pyramidal neurons. The SCP amplitude recorded from the scalp depends upon the synchronicity and intensity of the afferent input to layers I and II (Speckmann et al. (1984)). The depolarization of cortical cell assemblies reduces their excitation threshold such that firing of neurons in regions responsible for specified motor or cognitive tasks is facilitated. Negative amplitude shifts grow with increasing attentional or cognitive resource allocation. Cortical positivity may result from active inhibition of apical dendritic neural activity or simply from a reduction of afferent inflow and subsequent reduced postsynaptic activity. In any case, positive SCPs are considered to increase the excitation threshold of upper cortical layers via a negative feedback loop involving the basal ganglia and the reticular nucleus of the thalamus. Increasing cortical negativity is accompanied by increased activation of inhibitory striatal nuclei that leads to an increase
- 25. 10 An Introduction to Brain-Computer Interfacing Negative surface potential underneath electrode Cortical layers Nonspecific thalamic afferents Apical dendrite Sink Current flow Source Pyramidal cell Figure 1.3 Negative slow cortical potentials at the surface of the cortex originate from afferent thalamic or cortical input to the apical dendrites in layers I and II. The extracellular surrounding of the dendrites is electrically negative, leading to current flow into the cell mediated by positive sodium ions (sink). Intracellularly, the current flows toward the soma (source). This fluctuation of ions generates field potentials that can be recorded by electrodes on the scalp (from Kübler et al. (2001a), figure 4, with permission). of the excitation threshold of upper cortical layers, thereby preventing overexcitation (Birbaumer et al. (2003); Hinterberger et al. (2003c); Strehl et al. (2006)). A strong relationship among self-induced cortical negativity, reaction time, signal detec- tion, and short-term memory performance has been reported in several studies in humans and monkeys (Lutzenberger et al. (1979, 1982); Rockstroh et al. (1982)). Tasks requir- ing attention are performed significantly better when presented after spontaneous or self- induced cortical negativity. Slow Cortical Potentials as Input for a BCI (SCP-BCI) The SCP-BCI requires users to achieve voluntary regulation of brain activity. Typically, the SCP-BCI presents users with the traditional S1-S2 paradigm, which in the sixties led Walter and colleagues to the detection of the contingent negative variation (CNV) (Walter et al. (1964)): a negative SCP shift seen after a warning stimulus (S1) two to ten seconds before an imperative stimulus (S2) that requires participants to perform a task (e.g., a
- 26. 1.6 Recording Methods, Paradigms, and Systems for Brain-Computer Interfacing 11 Figure 1.4 Course of slow cortical potentials (SCP) averaged across 600 trials (amplitude as a function of time). The grey line shows the course of SCP when cortical negativity has to be produced to move the cursor toward the target at the top of the screen, the black line when cortical positivity is required to move the cursor toward the bottom target. Negative and positive SCP amplitudes clearly differ between the two tasks providing a binary response. At the beginning of a trial the task is presented, accompanied by a high-pitched tone (S1—warning stimulus) indicating that two seconds later the active phase will start providing SCP feedback to the user. The active phase is introduced by a low-pitched tone (S2—imperative stimulus). Between S1 and S2 a contingent negative variation (CNV) develops, which indicates that the user is preparing to perform the task. button press or cursor movement). The CNV (see figure 1.4) indicates depolarization and, thus, resource allocation for task performance as described above. Similarly, the SCP-BCI presents users with a high-pitched tone (S1) that indicates to the user that two seconds later, simultaneously with a low-pitched tone (S2), feedback of SCPs will start either visually as cursor movement on a monitor or by auditory means with instrumental sounds (Hinterberger et al. (2004a); Kotchoubey et al. (1997); Kübler et al. (1999)). Users are presented with two tasks, for example, cursor movement into targets either at the top or bottom of the screen, or an increase or decrease in the pitch of tones. To perform the task, BCI users have to produce positive and negative SCP amplitude shifts as compared to a baseline (see figure 1.4). SCP amplitude shifts must be above or below a predefined threshold to be classified as negative or positive. Severely paralyzed patients communicated extended messages with the SCP-BCI (Birbaumer et al. (1999); Neumann et al. (2003)) (see chapter 3). Sensorimotor Rhythms (SMR) Sensorimotor rhythms include an arch-shaped μ-rhythm (see figure 1.5), usually with a frequency of 10 Hz (range 8–11 Hz), often mixed with a β (around 20 Hz) and a γ component (around 40 Hz) recorded over somatosensory cortices, most preferably over C3 and C4. Spreading to parietal leads is frequent and also is seen in patients with
- 27. 12 An Introduction to Brain-Computer Interfacing Figure 1.5 Upper trace: μ-rhythm over sensory motor areas. Lower trace: desynchronization of μ-rhythm through movement imagery. amyotrophic lateral sclerosis (Kübler et al. (2005a)). Recently, an ALS patient showed left-hand movement-imagery-related SMR modulation at P4, which is in accordance with increased parietal activation during hand movement imagery in ALS patients as measured with functional magnetic resonance imaging (Kew et al. (1993); Lulé et al. (in press)). The SMR is related to the motor cortex with contributions of somatosensory areas such that the beta component arises from the motor and the alphoid μ-component from sensory cortex. SMR is blocked by movements, movement imagery, and movement preparation; thus, it is seen as an “idling” rhythm of the cortical sensory region. In cats, μ-rhythm-like activity has been shown to originate from the nucleus ventralis posterior of the thalamus. Usually μ- rhythm activity is not readily seen in scalp-recorded spontaneous EEG activity and it thus historically has been believed to occur in only a small number of adult persons. However, with better signal processing it has been shown to be ubiquitous in adults. Immediately, scalp-detectable μ-rhythm may, however, be an indicator of pathology. It was reported to accompany autonomic and emotional dysfunction such as migrane, bronchial asthma, tinnitus, anxiety, aggressiveness, and emotional instability. It is also often seen in patients with epilepsy. Three theories for the neurophysiological basis of the μ-rhythm exist: (1) it could be the correlate of neuronal hyperexcitability as specifically expressed in pathology, (2) it could be a sign of cortical inhibition, which would explain the blocking of μ-rhythm by movement or movement imagery, or (3) it may be interpreted as somatosensory “cortical idling,” adding the component of afferent input (summarized according to Niedermeyer (2005b)). Sensorimotor Rhythms as Input for a BCI (SMR-BCI) Sensorimotor rhythms (SMR) decrease or desynchronize with movement or preparation for movement and increase or synchronize in the postmovement period or during relaxation (Pfurtscheller et al. (1999)). Furthermore, and most relevant for BCI use by locked- in patients, they also desynchronize with motor imagery. Thus, to modulate the SMR amplitude no actual movement is required. Many BCI groups choose SMR as input signal because—at least in healthy participants—they are easy to regulate by means of motor imagery (see figure 1.6 and chapters 2–5). Modulation of SMR can be achieved within the first training session where the subjects are instructed to imagine left and right hand and foot movement (e.g., Blankertz et al. (2006a), see also chapter 5). After
- 28. 1.6 Recording Methods, Paradigms, and Systems for Brain-Computer Interfacing 13 Figure 1.6 EEG frequency spectrum of an ALS patient at electrode position Cp4 as a function of amplitude averaged over about 160 trials. The grey line shows the averaged EEG when the cursor has to be moved into the top target at the right-hand-side margin of the screen, the black line when the cursor has to be moved into the bottom target. Downward cursor movement is achieved by motor imagery (in this case left-hand movement leading to SMR modulation at Cp4), upward cursor movement by “thinking of nothing.” During downward cursor movement, the SMR amplitudes decrease (desynchronize) in the α and β band leading to a binary signal. subsequent machine learning (about two minutes on a standard PC) and visual inspection, individualized spatiotemporal filters and classifiers are ready to be used for feedback. Good subjects are then able to achieve information transfer rates of fifty bits per minute in asynchronous BCI mode (for a comparison of different evaluation criteria, see chapter 19). Even under the extremely challenging conditions of life demonstrations at CeBit 2006 in Hanover, Germany, subjects were able to achieve on average a selection rate of five to eight letters per minute in a spelling task (see chapter 5 and Müller and Blankertz (2006)). To achieve similar EEG patterns of imagined movements as compared to actual move- ments, it is important to instruct participants to imagine movement kinesthetically, meaning to “feel and experience” the movement instead of simply visualizing a movement (Neuper et al. (2005)). As with the SCP-BCI, to operate the SMR-BCI subjects are required to reg- ulate the SMR amplitude and are thus provided with visual (Pfurtscheller et al. (2006c); Wolpaw et al. (2003)) or auditory feedback (Hinterberger et al. (2004a); Nijboer et al. (in press)) (see also chapter 3). Typically, subjects are shown two or more targets on a monitor in which the cursor has to be moved by means of SMR amplitude modulation (see also chapters 2, 4, and 5). In a recent study with four ALS patients, it was shown that SMR regulation is possible despite considerable degeneration of cortical and spinal motor neu- rons (Kübler et al. (2005a)). However, the SMR amplitude is much lower in patients as compared to healthy individuals.
- 29. 14 An Introduction to Brain-Computer Interfacing Event-Related Potentials Event-related potentials (ERPs) are electrocortical potentials that can be measured in the EEG before, during, or after a sensory, motor, or psychological event. They have a fixed time delay to the stimulus and their amplitude is usually much smaller than the ongoing spontaneous EEG activity. The amplitudes are smaller because ERPs are more localized in the corresponding cortical areas. They are less frequent than the spontaneous EEG waves with similar shape and amplitude (Birbaumer and Schmid (2006)). To detect ERPs, averaging techniques are used. An averaged ERP is composed of a series of large, biphasic waves, lasting a total of five hundred to thousand milliseconds. Error monitoring of the brain is also accompanied by evoked potentials referred to as error related potentials. These deflections in the EEG may be used for error detection in a BCI (see chapter 17). In the following two paragraphs, a short overview is provided of the P300 component of the event-related potential and the visually (and sensorily) evoked potential for BCI use. BCIs on the basis of visually evoked potentials and visual P300 require intact gaze. P300 The P300 is a positive deflection in the electroencephalogram (EEG) time-locked to auditory or visual stimuli (see figure 1.7). It is typically seen when participants are required to attend to rare target stimuli presented within a stream of frequent standard stimuli (Squires et al. (1977)), an experimental design referred to as the oddball paradigm (Fabiani et al. (1987)). The P300 amplitude varies as a function of task characteristics such as discriminability of standard and target stimuli (Johnson and Donchin (1978)), loudness of tones (Squires et al. (1977)), overall probability of the target stimuli, the preceding stimulus sequence (Squires et al. (1976)), and the electrode position (Squires et al. (1977)). Mostly observed in central and parietal regions, it is seen as a correlate of an extinction process in short-term memory when new stimuli require an update of representations. P300 as Input Signal for a BCI (P300-BCI) As early as the late eighties, Farwell and Donchin had shown that the P300 component of the event-related potential can be used to select items displayed on a computer monitor (Farwell and Donchin (1988)). The authors presented their participants with a 6 x 6 matrix where each of the 36 cells contained a character or a symbol. This design becomes an oddball paradigm by first intensifying resp. flashing each row and column for 100 ms in random order and second by instructing participants to focus attention to only one of the 36 cells. Thus, in one sequence of 12 flashes (6 rows and 6 columns are highlighted), the target cell will flash only twice, constituting a rare event compared to the 10 flashes of all other rows and columns and therefore eliciting a P300 (see figure 1.7). Selection occurs by detecting the row and column that elicit the largest P300 (see also chapter 2). The P300- BCI does not require self-regulation of the EEG. All that is required from users is that they are able to focus attention and gaze on the target letter albeit for a considerable amount of time. Over the past five years, the P300 has received increasing amounts of attention as a BCI control signal. For example, a number of offline studies have been conducted to improve
- 30. 1.6 Recording Methods, Paradigms, and Systems for Brain-Computer Interfacing 15 Figure 1.7 Averaged EEG at the vertex electrode (Cz) of an ALS patient using a 7 x 7 P300 spelling matrix. The black line indicates the EEG response to 2448 standard stimuli and the grey line to 51 target letters (oddball) that have to be selected from the spelling matrix. A positive deflection as a response to targets can be seen in the time window between 200 and 500 ms. the classification rate of the P300 Speller (Kaper et al. (2004); Serby et al. (2005); Xu et al. (2004); He et al. (2001); Thulasidas et al. (2006)). Using a support vector machine classifier, Thulasidas et al. report online selection of three characters per minute with 95 percent accuracy (Thulasidas et al. (2006)). Bayliss showed that the P300 also can be used to select items in a virtual apartment, provided presentation of targets constitute an oddball paradigm (Bayliss et al. (2004)). In 2003, Sellers, Schalk, and Donchin published the first results of an ALS patient using the P300 Speller (Sellers et al. (2003)). In recent studies, Sellers et al. and Nijboer et al. presented results of the P300 speller used by ALS patients indicating that ALS patients are able to use the P300-BCI with accuracies up to 100 percent (Nijboer et al. (submitted); Sellers et al. (2006b)). It was also shown that the P300 response remains stable over periods of twelve to more than fifty daily sessions in healthy volunteers as well as in ALS-patients (Nijboer et al. (submitted); Sellers and Donchin (2006)). Piccione et al. tested the P300 as a control signal for a BCI in seven healthy and five paralyzed patients (Piccione et al. (2006)). As in the other studies, task completion and good performance was achieved after little time, thus there was no need for time- consuming training. However, the patients’ performance (68.6 percent) was worse than that of healthy participants (76.2 percent). In particular, those patients who were more impaired performed worse than did healthy participants, whereas there was no difference between less impaired patients and healthy participants (Piccione et al. (2006)). Recently, Vaughan et al. introduced a P300-BCI for daily use in a patient’s home environment (Vaughan et al. (2006)). Auditorily presented oddball paradigms may be used for patients with restricted or lost eye movement and are currently being investigated (Sellers and Donchin (2006); Hill et al. (2005)).
- 31. 16 An Introduction to Brain-Computer Interfacing SSVEP After visual stimulation (e.g., an alternating checkerboard), evoked potentials can be recorded from the visual cortex in the occipital lobe (O1, O2, Oz—according to the international 10-20 system). A visually evoked potential becomes steady if the presentation rate of stimuli is above 6 Hz (Gao et al. (2003b)). When participants focus their gaze on a flickering target, the amplitude of the steady-state visually evoked potential (SSVEP) increases at the fundamental frequency of the target and second and third harmonics (Wang et al. (2006); Müller-Putz et al. (2005b)). Amplitude and phase of the SSVEP depend on stimulus parameters such as repetition rate and contrast. The frequency resolution of the SSVEP is about 0.2 Hz and the bandwidth in which the SSVEP can be detected reliably is between 6 and 24 Hz (Gao et al. (2003b)). SSVEPs as Input Signal for a BCI (SSVEP-BCI) Like the P300-BCI, the SSVEP-BCI requires attention and intact gaze but no user train- ing as the cortical response is elicited via external stimulation (see chapter 4). To elicit SSVEPs, targets with different flickering frequencies are presented on a monitor (Wang et al. (2006)) or on a board with light emitting diodes (LED) (Müller-Putz et al. (2005b); Gao et al. (2003b)). The number of targets realized in a BCI varies from 4 (Müller-Putz et al. (2005b)) up to 48 (Gao et al. (2003b)). Classification accuracies of more than 90 percent correct are often reported (Kelly et al. (2005); Nielsen et al. (2006); Trejo et al. (2006)). In a 9-target SSVEP-BCI, healthy participants spelled out their phone number and birth date with a spelling rate of 7.2–11.5 selections per minute (information transfer rate of 18.37–27.29 bits/min) (Nielsen et al. (2006)), and in an 11-target SSVEP-BCI with an average accuracy of 83.3 percent (23.06 bits/min) (Lee et al. (2006)). A caveat of all SSVEP approaches to BCI control is their dependence on intact gaze, which renders them unsuitable for patients with restricted eye movement. Two studies address this issue. Kelly et al. investigated classification accuracies when users were not required to focus gaze on the flickering targets but on a fixation cross between two targets— a condition the authors refer to as covert attention (Kelly et al. (2005)). A decrease in accuracy was observed from about 95 percent when targets were fixated directly to about 70 percent in the covert attention condition. Thus, at least a rather simple two-target SSVEP paradigm might be used by locked-in patients albeit with reduced accuracy. A BCI based on steady-state evoked potentials completely independent of vision was introduced by Müller- Putz and colleagues (Müller-Putz et al. (2006)). The authors used vibratory stimulation of left- and right-hand fingertips to elicit somatosensory steady-state evoked potentials (SSSEP, see figure 1.8). The EEG was recorded from central electrodes (C3, Cz, and C4—according to the international 10-20 system). In each trial, both index fingers were stimulated simultaneously at different frequencies and participants were instructed via arrows on a computer screen to which finger they should pay attention. Online accuracies of four participants varied between 53 (chance level) and 83 percent correct, but offline classification was between 65 and 88 percent correct. Albeit not yet as reliable as the SSVEP-BCI the SSSEP-BCI may become an option for patients with impaired vision.
- 32. 1.6 Recording Methods, Paradigms, and Systems for Brain-Computer Interfacing 17 Figure 1.8 Peak at 31 Hz recorded at C3 when focusing attention on stimulation of right index finger (left panel) and 26 Hz at C4 when focusing on stimulation of left index finger. Both peaks reflect the stimulation frequency correctly (We thank Dr. Gernot Müller-Putz from the Laboratory of Brain-Computer Interfaces Institute for Knowledge Discovery at the Technical University of Graz, Austria, for this picture and the permission of reproduction). 1.6.1.2 Combinations of Signals It is a well-known fact that different physiological phenomena, for example, slow corti- cal potential shifts such as the premovement Bereitschaftspotenzial or differences in spa- tiospectral distributions of brain activity (i.e., focal event-related desynchronizations), code for different aspects of a subject’s intention to move. While papers noted the potential of combining these multiple modalities, it was first explored systematically by Dornhege et al. (2004a). Their work showed that BCI information transfer rates can be boosted signif- icantly when combining different EEG features. From a theoretical point of view, feature combination is most beneficial if the features of the single modalities have maximal statis- tical independence. High mutual independence can be measured in EEG features and thus subject dependent improvements of up to 50 percent relative classification performance gain are observed when using combined features in an offline evaluation (Dornhege et al. (2004a); Dornhege (2006)). The use of robust well-regularized classifiers is mandatory in this “sensor-fusion” process because otherwise the model complexity is hard to control in such high dimensional feature spaces (see chapter 13). We conjecture that not only combi- nations between different EEG modalities but also between different recording technolo- gies will be useful in the future, for example, between fMRI and EEG, or between local field potentials and spike data. Machine learning will be challenged by fusing different time-scales and their underlying statistical processes. 1.6.1.3 The Magnetic Activity of the Brain The magnetic field generated by electrical brain activity can be measured by means of magnetoencephalography (MEG). To date, this method is used only in laboratory settings and is consequently not suitable for a BCI for communication and control in the patient’s home environment. However, the advantages of MEG as compared to EEG, namely better spatial resolution leading to a precise localization of cortical activation related to a specific
- 33. 18 An Introduction to Brain-Computer Interfacing task or sensory stimulation and higher signal-to-noise ratio, especially for higher frequency activity like gamma band activity, render it a viable tool for short-term intervention and rehabilitation (see chapter 14). In a study with three tetraplegic patients after cervical spinal cord injury, Kauhanen et al. achieved the same classification accuracies in MEG data as compared to EEG data (Kauhanen et al. (2006)). The patient’s task was to attempt finger movement, and data were analyzed offline. Lal et al. showed that regulation of the magnetic activity of the brain by means of motor imagery can be used to select letters on a computer screen, but participants were not yet provided with online feedback of MEG activity; instead they were provided with feedback of results, that is, a smiling face after correct classification or selection of the correct letter (Lal et al. (2005b)). Mellinger et al. even provide online MEG feedback for healthy participants during motor imagery (Mellinger et al. (under revision)). Three of five participants achieved cursor control of 90 percent accuracy or more within the first training session. Thus, learning to regulate brain activity by means of MEG feedback and achieved accuracies were comparable to EEG (Blankertz et al. (2006a)). MEG may be used to localize the focus of activity during motor imagery if EEG provides no clear results (see chapter 22). Currently, MEG feedback during motor imagery is used to train chronic stroke patients to reactivate the paralyzed limb provided that not the entire motor cortex or pyramidal tracts are lesioned. Chronic stroke patients undergo an MEG feedback training such that their paralyzed limb is provided with an orthosis that opens and closes the paralyzed hand (Birbaumer and Cohen (2005)). Motor imagery opens the orthosis whereas relaxation (thinking of nothing) closes it. This training provides the patients with self-induced sensory feedback of the paralyzed limb. The idea behind this being that activation of a sensorimotor network enables patients to relearn motor functions (Braun et al. (submitted)) (see also chapter 22). 1.6.1.4 The Blood Oxygen Level Dependent Response (BOLD) For the past approximately five years it has been possible to use the blood oxygen level de- pendent (BOLD) response as input signal for a BCI. Local concentration of deoxygenated hemoglobin in brain tissue depends on neuronal activity and metabolism and changes can be measured with functional magnetic resonance imaging (fMRI). Compared to EEG, fMRI allows spatial resolution in the range of millimeters and a more precise allocation of neuronal activity. Additionally, activation in subcortical areas can be recorded. Due to recent advances in acquisition techniques, computational power, and algorithms, the func- tional sensitivity and speed of fMRI was increased considerably (Weiskopf et al. (2004b)) and the delay of feedback could be reduced to below two seconds (Weiskopf et al. (2003)), which allows the use of this technique as real-time fMRI. Target areas for feedback were sensory (S1, e.g., Yoo et al. (2004)) and motor areas (M1, e.g., DeCharms et al. (2004); SMA Weiskopf et al. (2004b)), the parahippocampal place area (Weiskopf et al. (2004b)), the affective and cognitive subdivision of the anterior cingulate cortex (ACC) (Weiskopf et al. (2003)), and rostral ACC (DeCharms et al. (2005)). Learning of self-regulating the BOLD response was reported in all studies that included subject training to regulate the BOLD response, and some reported behavioral effects in relation to activation or deacti- vation of target areas: Increase of activation in the affective subdivision of the ACC led to
- 34. 1.6 Recording Methods, Paradigms, and Systems for Brain-Computer Interfacing 19 higher valence and arousal ratings of the subjective affective state (Weiskopf et al. (2003)). Better encoding of words after down regulation of the parahippocampal place area (as compared to the supplementary motor area) and decreased reaction time in a motor task after upregulation of the supplementary motor area (as compared to the parahippocam- pal place area) was demonstrated (Weiskopf et al. (2004b)). Regulation of the insula, an area involved in emotional processing, also proved possible and was shown to increase the negative valence of participants when confronted with negative stimuli such as pictures of violence or mutilated bodies (Sitaram et al. (2006)). Recently, specific effects on pain perception as a function of self-regulation of the rostral part of the ACC was reported in the first clinical study including patients with chronic pain. In healthy subjects, the au- thors controlled for effects of repeated practice, brain region, feedback, and intervention. In chronic pain patients, only feedback was controlled such that one group received feed- back of the BOLD response in the rostral ACC and another of skin conductance, heart rate, and respiration. Healthy participants were presented with nociceptive heat stimuli. Only in those healthy participants and pain patients who received real-time feedback of the BOLD response in the rostral ACC, an area known to be involved in pain perception, were changes in pain ratings found (DeCharms et al. (2005)). This study already demonstates the possible power of the fMRI-BCI for treating clinical groups if the neurobiological basis of the disorder is known. For example, hypoactivation in orbitofrontal and limbic areas in- volved in emotional processing were found in psychopaths (Birbaumer et al. (2005)), and hypofunction in dorsolateral and dorsomedial prefrontal cortex and the pregenual part of the ACC is consistently found in depressed patients (Davidson et al. (2002)). Even more complex cognitive functions as needed for the game of paper, rock, and scissors could be decoded successfully with fMRI by Kamitani and Tong (2005). Most recently, Owen et al. successfully distinguished activation patterns to motor imagery (playing tennis) and spatial navigation (through one’s own house starting at the front door) in a patient diagnosed with persistent vegetative state, and could thus show that she was consciously aware (Owen et al. (2006)). For further reference, see also the review by Haynes and Rees (2006). 1.6.1.5 Near Infrared Spectroscopy (NIRS) as a Recording Method for BCI The advantage of functional MRI as compared to EEG is its 3D spatial resolution. How- ever, fMRI is expensive and bound to the laboratory. Near infrared spectroscopy offers a comparable spatial resolution albeit restricted to cortical areas (depth 1–3 cm) with much less technical effort and costs. Moreover the NIRS-BCI is portable and could thus be used in a patient’s home environment. The NIRS-BCI system presented by Sitaram and colleagues incorporates the so-called continuous wave technique. Regional brain activation is accompanied by increases in regional cerebral blood flow (rCBF) and the regional cerebral oxygen metabolic rate (rCMRO2). The increase of rCBF exceeds that of rCMRO2 resulting in a decrease of deoxygenated hemoglobin in venous blood. Thus, the ratio of oxygenated to deoxygenated hemoglobin is expected to increase in active brain areas and is measured with NIRS. The continuous wave approach uses multiple pairs or channels of light sources and light detectors operating at two or more discrete wavelengths. The light source may be a
- 35. 20 An Introduction to Brain-Computer Interfacing Figure 1.9 Exemplary data from a healthy participant performing motor imagery (right hand movement). The solid line indicates course of oxygenated (Oxy HB; HB = hemoglobin) and the dashed line of deoxygenated hemoglobin (Deoxy HB) averaged across a full session (80 trials) from channel 7 on the contralateral (left) hemisphere (close to the C3 electrode position as per the 10-20 system) for the duration 0–140 time points after stimulus presentation. 140 time points are equal to 10 s of execution of the motor imagery task at a sampling rate of 14 Hz. (We thank Ranganatha Sitaram from the Institute of Medical Psychology and Behavioural Neurobiology, University of Tübingen, for this picture and the permission of reproduction). laser or a light emitting diode (LED). The optical parameter measured is attenuation of light intensity due to absorption by the intermediate tissue. The concentration changes of oxygenated and deoxygenated hemoglobin are computed from the changes in the light intensity at different wavelengths (Sitaram et al. (2007)). It has been shown already that brain activation in response to motor movement and imagery can be readily detected with NIRS (see figure 1.9, Coyle et al. (2004); Sitaram et al. (2005, 2007)). 1.6.2 Invasive Recording Methods for BCI Invasive recording methods either measure the neural activity of the brain on the cortical surface (electrocorticography, ECoG) or intracortically from within the (motor) cortex (see figure 1.10 for general setup). These methods have strong advantages in terms of signal quality and dimensionality. However, they require surgery and the issues of long- term stability of implants and protection from infection arise (Hochberg et al. (2006)). The decision of a BCI user for the one method over the other will strongly depend on
- 36. 1.6 Recording Methods, Paradigms, and Systems for Brain-Computer Interfacing 21 a Visual, tactile, or proprioceptive feedback Telemetry (receiver) Telemetry (sender) Control logic Robotic arm Signal preprocessing unit Recording system (intracortical) Translation to control signal b Figure 1.10 Generic setup of an invasive BCI (left) and picture of an array electrode placed into the monkey cortex (right). Figure (b) from Nicolelis (2001) by permission. the purpose of BCI use; for example, multidirectional neuroprosthesis control may only be possible with intracortical recordings, whereas communication at a speed of approximately ten selections per minute can be achieved with noninvasive methods. We might speculate that invasive methods have to proof substantially better than noninvasive methods to become attractive for possible users. 1.6.2.1 Brain Signals Recorded from the Surface of the Cortex (ECoG) The electrocorticogram (ECoG) uses epidural or subdural electrode grids or strips to record the electrical activity of the cortex. It is an invasive procedure that requires craniotomy for implantation of electrodes (see Leuthardt et al. (2006b)). However, the procedure becomes less invasive when less electrodes are required (strips instead of grids) because strips may be inserted via a small hole in the scalp. The main advantages of ECoG are a higher spatial resolution than the EEG (tenths of millimeters versus centimeters), broader bandwidth (0– 200 Hz versus 0–40 Hz) that allows also recording of γ band activity, and higher amplitude (50–100 μV versus 5–20 μV) and less vulnerability to artifacts such as electromyogram (Leuthardt et al. (2004)). Commonly, ECoG is used to localize seizure activity in patients with epilepsy before they undergo surgery. Studies on the feasibility of ECoG for BCI were thus largely conducted with epilepsy patients and are reviewed in detail in chapter 8. To our knowledge, only one ALS patient consented to grid implantation for the purpose of controlling a BCI for communication, but communication was not achieved (Birbaumer (2006a,b), see chapter 22). Most of these studies performed offline open-loop analysis of ECoG data (Huggins et al. (1999); Levine et al. (1999)). Using Distinction Sensitive Learning Vector Quantization (DSLVQ) for offline classification of data recorded during self-paced middle finger extension, Scherer et al. reported accuracies between 85 and 91
- 37. 22 An Introduction to Brain-Computer Interfacing percent (Scherer et al. (2003)). Hill et al. applied autoregressive models and support vector machine classification to data obtained during motor imagery and achieved accuracies around 75 percent (Hill et al. (2006)). Few studies closed the loop and provided feedback of ECoG to the participants (Felton et al. (2005); Leuthardt et al. (2004, 2006a); Wilson et al. (2006)). In each study by Leuthardt et al., the ECoG of four patients was recorded with electrode grids or strips over prefrontal, temporal, sensorimotor, and speech areas. Patients were required to perform and imagine motor and speech tasks such as opening and closing the right or left hand, protruding the tongue, shrugging shoulders, or saying the word move. Each task was associated with a decrease in μ- and β-rhythm and an increase of gamma-rhythm amplitudes over prefrontal, premotor, sensorimotor, or speech areas. The spatial and spectral foci of task-related ECoG activity were similar for action and imagery. Frequency bands in the gamma range were most often chosen for online control, and during movement imagery accuracies achieved within a brief training of 3– 24 minutes were between 73 and 98 percent. Wilson et al. proposed to use multimodal imagery for cursor control and showed that cursor control can be achieved with nonmotor imagery such as auditory imagery (a favorite song, voices, phone) (Wilson et al. (2006)). In a completely paralyzed ALS patient implanted with a 32-electrode grid, classification of signals related to motor imagery was at the chance level (Birbaumer (2006a); Hill et al. (2006); see also chapter 22). More than one year after implantation approximately 50 percent of the electrodes provide stable and clear signal recording (unpublished data from first author’s affiliation). 1.6.2.2 Brain Signals Recorded from Within the Cortex Intracortical signal acquisition can be realized with single, few, or multiple electrodes (arrays) that capture the action potentials of individual neurons. Electrode tips have to be in close proximity to the signal source and the arrays have to be stable over a long period of time. With two exemplary ALS patients, Kennedy and Bakay showed that humans are able to modulate the action potential firing rate when provided with feedback (Kennedy and Bakay (1998)). The authors implanted into the motor cortex a single electrode with a glass tip containing neurotrophic factors. Adjacent neurons grew into the tip and after a few weeks, action potentials were recorded. One patient was able to move a cursor on a computer screen to select presented items by modulating his action-potential firing rate (Kennedy et al. (2000, 2004)). In Mehring et al. (2003), it was demonstrated that hand movements could be estimated from local field potentials. Multielectrode arrays for intracortical recording are still to be improved for clinical application (Nicolelis (2003); Nicolelis et al. (2003), see figure 1.10). They have been used in animals with stable recordings for up to two years (Nicolelis et al. (2003); Donoghue (2002)). Recent results by Hochberg et al. (2006) for human patients show that stable long-term recordings are possible but at the expense of losing signal at a large number of electrodes. Several groups use multielectrode recording to detect activation patterns related to movement execution in animals (Carmena et al. (2003); Paninski et al. (2004); Taylor et al. (2003); Chapin et al. (1999)). The action-potential firing rate in motor areas contains sensory, motor, perceptual, and cognitive information that allows the estimation
- 38. 1.6 Recording Methods, Paradigms, and Systems for Brain-Computer Interfacing 23 of a subject’s intention for movement execution, and it was shown that 3D hand trajectories can be derived from the activity pattern of neuronal cell assemblies in the motor cortex by appropriate decoding (Serruya et al. (2002)). For example, Taylor et al. realized brain- controlled cursor and robot arm movement using recordings from a few neurons (18 cells) in the motor cortex only (Taylor et al. (2002)). Rhesus macaques learned first to move a cursor into eight targets located at the corners of an imaginary cube with real hand movements. Accompanying neural activity patterns were recorded and used to train an adaptive movement prediction algorithm. After sufficient training of subjects and algorithm, subjects’ arms were restricted and cursor movement was performed by brain control. Similarly, rhesus monkeys were trained to move a brain-controlled robot arm in virtual reality (Taylor et al. (2003)) and then to feed themselves with a real robot arm (Schwartz (2004b)). Recently, Musallam et al. presented data from three monkeys that were implanted with electrode arrays in the parietal reach area, area 5, and the dorsal premotor cortex (Musallam et al. (2004)). Subjects were first trained to reach for targets at different positions on a screen after a delay of 1.2 to 1.8 seconds following cue presentation. Neural activity during the memory period was correctly decoded with an accuracy of about 64 percent. The authors then trained subjects to associate visual cues with the amount, probability, or type of reward (orange juice versus water). Neural activity was then found to alter as a function of expected reward and thus represented additional information for classification. Accordingly, classification results could be improved by 12 percent. Santhanam et al. (2006) used a 96-electrode array implanted in the monkey dorsal premotor cortex and report selection rates of 6.5 bits per second. This astonishing high information transfer rate was achieved in an instructed delay reach task with ultra-short trial lengths around 250 ms. Integration over spike activity in very short time windows was enough for these excellent decoding results. Hochberg et al. (2006) report on a study where an array of 96 electrodes was implanted in a human subject diagnosed with tetraplegia three years after high spinal cord injury. With the array position being the primary motor cortex, it could be demonstrated that spike patterns were modulated by hand movement intention. A decoding algorithm based on a linear filter provided a “neural cursor” to the subject, who was then able to operate the TV, to open or close a prosthetic hand even while in a conversation, or to accomplish other tasks. The authors furthermore report a considerable loss of recorded units after 6.5 months, which again underlines the necessity to advance sensor technology. It is important to note that this was the first pilot clinical trial with an intracortical array implantation in humans. These and other experimental works reveal that it is possible to derive limb or cursor movement directly from the neural activity patterns of the cortex with appropriate decoding algorithms (see also Lebedev and Nicolelis (2006)). Finally, a simultaneous stimulation of the reward area and the sensory area in the rat allowed the control over movement patterns of the rat (Talwar et al. (2002)). A more recent work by Chapin studies how to perform a stimulation of the sensory areas to ultimately supply artificial sensory feedback for neuroprosthetics (Chapin (2006)).
- 39. 24 An Introduction to Brain-Computer Interfacing 1.7 Concluding Discussion Brain-Machine Interfacing—be it invasive or noninvasive—has witnessed a recent explo- sion of research. The reason for this increased activity is the wide application potential that the field is bearing. (1) Clinical applications of BCI (such as those outlined in chap- ter 22) become evident, and work in the invasive BMI community shows the potential for future use in neuroprosthetics (Leuthardt et al. (2006a)). This is in particular underlined by the first successful human clinical trial reported by Hochberg et al. or a recent monkey study by Santhanam et al. that explores the “speed-limit” of invasive brain-computer inter- facing (Hochberg et al. (2006); Santhanam et al. (2006)). Similar success can be seen in noninvasive BCIs, where two-dimensional cursor control allows a richer repertoire of com- munication and higher information transfer rates (ITR) (Wolpaw and McFarland (2004)). The Berlin BCI system is now able to almost completely dispense with subject training, an important progress that nevertheless has yet to be verified for disabled users. Event-related potentials such as the P300 and the steady-state visually evoked potential provide to date the highest ITR for noninvasive BCIs (Nijboer et al. (submitted); Nielsen et al. (2006); Lee et al. (2006)). The Tübingen, Albany, and Graz BCIs are used for rehabilitation in exem- plary patients. Overall, there is about a factor of ten in information transfer rate between invasive and noninvasive BCIs and it will depend on each individual patient whether the risk of surgery and potential inflammation incurred in the invasive methods will justify this gain. (2) Although clinical application in rehabilitation will always serve as a main motiva- tion and driving force for BCI research, it is the fascination for the brain itself and the urge to better understand its function that also drives BCI researchers. In fact, BCIs are a unique new tool that have emerged over the past years to analyze seminal questions in brain re- search such as plasticity, dynamics, representation, neural coding, intention, planing, and learning in a very direct manner. Invasive BMIs can now record from several hundreds of electrodes and can thus directly study, for example, the change of neural code during learning. Noninvasive BCIs allow researchers to watch how the brain alters and instantiates behavior and cognition in real-time. (3) Finally, there is a variety of applications that in- corporate advanced signal processing, such that single trial data can be classified robustly. This step forward allows BCI researchers to contribute to general topics in the domain of human-machine interaction. The exploration of the novel independent communication and control channel BCI to assess the users state in a direct manner opens a broad field and it remains to be seen how far the BCI channel will prove to be useful when considering typical HMI applications like assessment of cognitive workload (see chapter 24), alert- ness, task involvement, emotion, or concentration. Clearly new systems that can use BCI for navigating or gaming in virtual worlds (see chapter 23) and for enhancing and improv- ing man-machine interaction are on the way (see chapter 24). It is important to note that the above applications will be limited to the noninvasive EEG based systems due to their comparatively low risk and cost. Many open problems remain on the path toward better brain-computer interfacing and broader applicability of the BCI technology. As very extensively outlined by, for exam- ple, Lebedev and Nicolelis (2006); Nicolelis (2001) and Nicolelis et al. (2003), it will be
- 40. 1.7 Concluding Discussion 25 important to advance recording and transmission technology such that chronical implants become possible that can persist for a long time with very low risk and telemetrically trans- mit signals to BCI. A better understanding of basic neuroscience issues like representation, plasticity, and learning will allow the construction of better BMIs. Similar reasoning also holds for noninvasive BCIs where contactless wearable sensors are a necessary condition for a wide applicability of BCIs even outside medical domains, for example, for com- puter gaming and general man-machine interfacing applications such as usability studies. Overall, it will be essential to advance signal processing and machine learning technology to build faster, better, more adaptive, and most important more robust systems. What we defined as the phenomenon of BCI-illiteracy has to be investigated in more depth to un- derstand whether there will always be a part of the population that is unable to operate a BCI and for what reasons. Knowing the amazing possibility of humans to learn a task and observing the considerable inter- and intraindividual signal variances, it seems reasonable to make BCIs fully adaptive. Unsolved, however, is how we can get these two complex learning systems—the machines and the human brains—in synchronicity such that stable BCI control becomes the rule and not the exception. To investigate long-term stability of BCI systems clearly, more long-term clinical trials are necessary. Gradual improvement in all these directions will be indispensible for the future success of this lively and vigorous field. Acknowledgments We thank Boris Kotchoubey, Michael Schröder, and Guido Dornhege for valuable com- ments on the manuscript. This work is funded by the Deutsche Forschungsgemeinschaft (DFG), the National Institutes of Health (NIH), the Bundesministerium für Bildung und Forschung (BMBF), and the IST Programme of the European Community. This publica- tion reflects only the authors’ views. Notes (1) BCI and BMI are used as synonyms. (2) We distinguish here between brain-computer interfaces that listen to the neural code and computer-brain interfaces that are also able to transmit information from the computer toward the brain. (3) Popularized under the slogan “let the machines learn” by the Berlin Brain-Computer Interface group (BBCI).
- 41. I BCI Systems and Approaches
- 42. Introduction This part provides an insight into a representative variety of BCI systems that are currently being pursued in research labs. A distinctive feature in BCI studies is the paradigm used for the interaction between user and computer. On one hand there are systems that require an active and voluntary strategy for generating a specific regulation of an EEG parameter such as the motor-related μ-rhythm or the self-regulation of slow cortical potentials (SCP). On the other hand there are passive paradigms, where participants only have to passively view an item for selection. Those systems detect the evoked responses such as P300 as presented in chapter 2 or make use of steady-state evoked potentials (SSVEP) as presented in chapter 4. Finally, one distinction between BCI labs is based on the realization of the system. Most groups, as introduced in chapters 2, 3, and 7, use extensive subject training. So, users have to adapt their brain signals to a fixed decoding algorithm, that is, the learning is on the subject side. Over the past five years, the Berlin group has established a paradigm change, where learning is now done by the computer, following the motto “let the machines learn.” Now several groups have adopted this principle. Examples for this approach are discussed in chapters 4, 5, and 6. Note that even if a pure machine learning approach was intended, the subject will inevitably learn once feedback has started, so in principle BCI systems will always have both aspects: subject and machine training. In this section six major BCI labs introduce their systems. For further ideas we refer to Babiloni et al. (2004), Gao et al. (2003b), Sykacek et al. (2003), Thulasidas et al. (2006), Kauhanen et al. (2006), and Kaper et al. (2005). Note that this list can never be complete. Chapter 2 outlines the Albany BCI, where a user is trained to manipulate his μ and β rhythms to control a cursor in 1- or 2D. Furthermore, BCI control based on the P300 paradigm is shown. Similar to Albany, the Tübingen BCI, outlined in chapter 3, train their subjects to adapt to the system using slow cortical potentials. The group uses BCI as a means for communication of ALS patients with the outside world and as the design of this interaction. Further BCI systems discussed in the chapter are P300 and μ-rhythm-based BCIs, an interesting new BCI paradigm based on auditory stimulation and the use of invasive techniques like ECoG for BCI. In chapter 4 the main research directions of the Graz BCI are depicted. The group is broadly exploring the whole BCI field from sensors, feedback strategies, and cognitive aspects to novel signal processing methods, with excellent results. The Graz BCI is shown to be not only of use for patients but also it contributes to general man-machine interaction as demonstrated for a moving in a VR environment. Typically, only a few electrodes and
- 43. 30 BCI Systems and Approaches machine learning techniques combined with user adaptation are employed to achieve BCI control. Chapter 5 introduces the Berlin BCI. Compared to training times of weeks or even months in other BCIs, the BBCI allows for subject control after 30 minutes. This drastic decrease in training time became possible by virtue of advanced machine learning and signal processing technology. The chapter presents online feedback studies based on the physiological signals’ preparatory potential and μ-rhythm modulation. The study shows that after less than one hour, five of six untrained subjects were able to achieve high performances when operating a variety of different feedbacks. Similar to the Berlin approach, the Martigny BCI introduced in chapter 6 tries to relocate the effort from the subject training to the machine by using machine learning techniques and online adaptation to realize a BCI. In particular, online adaptation is an important direction to compensate for the intrinsic nonstationarities found in EEG signals. Finally, the ideas of the Vancouver BCI are introduced in chapter 7. The main focus here is to establish an asynchronous BCI for patients, that is, a system that detects whether a user is intending something or not. To achieve this goal, the authors also use machine learning techniques that adapt the machine to the user. The cheapest, most popular, and thus most commonly used measuring device for non- invasive BCI is certainly EEG, but recently also BCI experiments using fMRI (cf., e.g., Weiskopf et al. (2004a); Kamitani and Tong (2005)) and MEG were conducted success- fully (cf. Mellinger et al. (2005); Kauhanen et al. (2006)). So far fMRI and MEG are too expensive for a broad use in BCI, but they have been very important for a better under- standing of the physiological phenomena in the context of BCI control (cf. Haynes and Rees (2006)). Thilo Hinterberger, Guido Dornhege, and Klaus-Robert Müller
- 44. 2 Noninvasive Brain-Computer Interface Research at the Wadsworth Center Eric W. Sellers, Dean J. Krusienski, Dennis J. McFarland, and Jonathan R. Wolpaw Laboratory of Nervous System Disorders Wadsworth Center New York State Department of Health Albany, NY 12201-0509 2.1 Abstract The primary goal of the Wadsworth Center brain-computer interface (BCI) program is to develop electroencephalographic (EEG) BCI systems that can provide severely disabled individuals with an alternative means of communication and/or control. We have shown that people with or without motor disabilities can learn to control sensorimotor rhythms recorded from the scalp to move a computer cursor in one or two dimensions and we have also used the P300 event-related potential as a control signal to make discrete selections. Overall, our research indicates there are several approaches that may provide alternatives for individuals with severe motor disabilities. We are now evaluating the practicality and effectiveness of a BCI communication system for daily use by such individuals in their homes. 2.2 Introduction Many people with severe motor disabilities require alternative methods for communica- tion and control because they are unable to use conventional means that require voluntary muscular control. Numerous studies over the past two decades indicate that scalp-recorded EEG activity can be the basis for nonmuscular communication and control systems, com- monly called brain-computer interfaces (BCIs) (Wolpaw et al. (2002)). EEG-based BCI systems measure specific features of EEG activity and translate these features into de- vice commands. The most commonly used features have been sensorimotor rhythms (Wol- paw et al. (1991, 2002); Wolpaw and McFarland (2004); Pfurtscheller et al. (1993)), slow cortical potentials (Birbaumer et al. (1999, 2000); Kübler et al. (1998)), and the P300 event-related potential (Farwell and Donchin (1988); Donchin et al. (2000); Sellers and
- 45. 32 Noninvasive Brain-Computer Interface Research at the Wadsworth Center Donchin (2006)). Systems based on sensorimotor rhythms or slow cortical potentials use components in the frequency or time domain that are spontaneous in the sense that they are not dependent on specific sensory events. Systems based on the P300 response use time-domain EEG components that are elicited by specific stimuli. At the Wadsworth Center, our goal is to develop a BCI that is suitable for everyday use by severely disabled people at home or elsewhere. Over the past 15 years, we have developed a BCI that allows people, including those who are severely disabled, to move a computer cursor in one or two dimensions using μ and/or β rhythms recorded over sensorimotor cortex. More recently, we have expanded our BCI to include use of the P300 response that was originally described by Farwell and Donchin (1988). Fundamental to the efficacy of our system has been BCI2000 (Schalk et al. (2004)), the general-purpose software system that we developed and that is now used by more than one hundred BCI laboratories around the world (see chapter 21 for a complete description of the BCI2000 system). 2.3 Sensorimotor Rhythm-Based Cursor Control Users learn during a series of training sessions to use sensorimotor rhythm (SMR) am- plitudes in the μ (8–12 Hz) and/or β (18–26 Hz) frequency bands over left and/or right sensorimotor cortex to move a cursor on a video screen in one or two dimensions (Wolpaw and McFarland (1994, 2004); McFarland et al. (2003)). This is not a normal function of this brain signal, but rather the result of training. The SMR-based system uses spectral features extracted from the EEG that are spontaneous in the sense that the stimuli presented to the subject provide only the possible choices and the contingencies are arbitrary. The SMR-based system relies on improvement of user performance as a result of prac- tice (McFarland et al. (2003)). This approach views the user and system as the interaction of two dynamic processes (Taylor et al. (2002); Wolpaw et al. (2000a)), and can be best conceptualized as coadaptive. By this view, the goal of the BCI system is to vest control in those signal features that the user can most accurately modulate and optimize the transla- tion of these signals into device control. This optimization is presumed to facilitate further learning by the user. Our first reports of SMR use to control a BCI used a single feature to control cursor movement in one dimension to hit a target located at the top or bottom edge of a video monitor (Wolpaw et al. (1991)). In 1993 we demonstrated that users could learn to control the same type of cursor movement to intercept targets starting at a variable height and moving from left to right across the screen (McFarland et al. (1993)). Subsequently, we used two channels of EEG to control cursor movement independently in two dimensions so users could hit targets located at one of the four corners of the monitor (Wolpaw and McFarland (1994)). We also evaluated using one-dimensional cursor control with two to five targets arranged along the right edge of the monitor (McFarland et al. (2003)). This task is illustrated in figure 2.1a. Cursor control in these examples was based on a weighted sum of one or two spectral features for each control dimension. For example, an increase in the amplitude of the 10Hz μ rhythm, located over the sensorimotor cortex (electrode C3), could move the target up and a decrease in the amplitude of this μ-rhythm could serve to
- 46. 2.3 Sensorimotor Rhythm-Based Cursor Control 33 a b Figure 2.1 (a) One-dimensional four-target SMR control task (McFarland et al. (2003)). (b) Two- dimensional eight target SMR control task (Wolpaw and McFarland (2004)). (1) The target and cursor are present on the screen for 1 s. (2a) The cursor moves steadily across the screen for 2 s with its vertical movement controlled by the user. (2b) The cursor moves in two dimensions with direction and velocity controlled by the user until the user hits the target or 10 s have elapsed. (3) The target flashes for 1.5 s when it is hit by the cursor. If the cursor misses the target, the screen is blank for 1.5 s. (4) The screen is blank for a 1-s interval. (5) The next trial begins. move the target down. In this case, feature selection was based on inspection of univariate statistics. We found that a regression approach is well suited to SMR cursor movement since it provides continuous control in one or more dimensions and generalizes well to novel target configurations. The utility of a regression model is illustrated in the recent study of SMR control of cursor movement in two dimensions described in Wolpaw and McFarland (2004). An example trial is shown in figure 2.1b. A trial began when a target appeared at one of eight locations on the periphery of the screen. Target location was block-randomized (i.e., each occurred once every eight trials). One second later, the cursor appeared in the middle of the screen and began to move in two dimensions with its movement controlled by the user’s EEG activity. If the cursor reached the target within 10 s, the target flashed as a reward. If it failed to reach the target within 10 s, the cursor and the target simply disappeared. In either case, the screen was blank for one second, and then the next trial began. Users initially learned cursor control in one dimension (i.e., horizontal) based on a regression function. Next they were trained on a second dimension (i.e., vertical) using a different regression function. Finally the two functions were used simultaneously for full two-dimensional control. Topographies of Pearson’s r correlation values for one user are shown in figure 2.2, where it can be seen that two distinct patterns of activity controlled cursor movement. Horizontal movement was controlled by a weighted difference of 12-Hz μ-rhythm activity between the left and right sensorimotor cortex (see figure 2.2, left topography). Vertical movement was controlled by a weighted sum of activity located
- 47. 34 Noninvasive Brain-Computer Interface Research at the Wadsworth Center Figure 2.2 Scalp topographies (nose at top) of Pearson’s r values for horizontal (x) and vertical (y) target positions. In this user, horizontal movement was controlled by a 12-Hz μ-rhythm and vertical movement by a 24-Hz β-rhythm. Horizontal correlation is greater on the right side of the scalp, whereas vertical correlation is greater on the left side of the scalp. The topographies are for R rather than R2 to show the opposite (i.e., positive and negative, respectively) correlations of right and left sides with horizontal target level (Wolpaw and McFarland (2004)). over left and right sensorimotor cortex in the 24-Hz β-rhythm (see figure 2.2, right topography). This study illustrated the generalizability of regression functions to varying target configurations. This 2004 study also determined how well users could move the cursor to novel loca- tions. Targets were presented at sixteen possible locations consisting of the original eight targets and eight additional targets that were on the periphery in the spaces between the original eight and not overlapping with them. Target location was block-randomized (i.e., each occurred once in sixteen trials). The average movement times to the original locations was compared with the average movement times to the novel locations. In the first of these sessions, movement time was slightly but not significantly longer for the novel targets, and this small difference decreased with practice. These results illustrated that ordinary least-squares regression procedures provide efficient models that generalize to novel tar- get configurations. Regression provides an efficient means to parameterize the translation algorithm in an adaptive manner that smoothly transfers to different target configurations during the course of multistep training protocols. This study clearly demonstrated strong simultaneous independent control of horizontal and vertical movement. This control was comparable in accuracy and speed to that reported in studies using implanted intracortical electrodes in monkeys (Wolpaw and McFarland (2004)). We have also evaluated various regression models for controlling cursor movement acquired from a four-choice, one-dimensional cursor movement task (McFarland and Wolpaw (2005)). We found that using more than one EEG feature improved performance (e.g., C4 at 12Hz and C3 at 24Hz). In addition, we evaluated nonlinear models with linear regression by including cross-product (i.e., interaction) terms in the regression function. While the translation algorithm could be based on either a classifier or a regression function, we concluded that a regression approach was more appropriate for the cursor
- 48. 2.3 Sensorimotor Rhythm-Based Cursor Control 35 Figure 2.3 Comparison of regression and classification for feature translation. For the two-target case, both methods require only one function. For the five-target case, the regression approach still requires only a single function, while the classification approach requires four functions (see text for full discussion). movement task. Figure 2.3 compares the classification and regression approaches. For the two-target case, both the regression approach and the classification approach require that the parameters of a single function be determined. For the five-target case, the regression approach still requires only a single function when the targets are distributed along a single dimension (e.g., vertical position on the screen). In contrast, for the five-target case the classification approach requires that four functions be parameterized. With even more and variable targets, the advantage of the regression approach becomes increasingly apparent. For example, the positioning of icons in a typical mouse-based graphical user interface would require a bewildering array of classifying functions, while with the regression approach, two dimensions of cursor movement and a button selection serve all cases. We have conducted preliminary studies that suggest users are also able to accurately control a robotic arm in two dimensions by applying the same techniques used for cursor control. A more recent study shows that after encountering a target with the cursor, users are able to select or reject the target by performing or withholding hand-grasp imagery (McFarland et al. (2005)). This imagery evokes a transient response that can be detected and used to improve the overall accuracy by reducing unintended target selections. As these results illustrate, training of SMRs has the potential to be extended to a variety of applications, and the control obtained for one task can transfer directly to another task. Our current efforts toward improving the SMR paradigm are refining the one- and two- dimensional control procedures with the intention of progressing to more choices and to higher dimensional control. This includes the identification or transformation of EEG features so that the resulting control signals are as independent, trainable, stable, and
- 49. 36 Noninvasive Brain-Computer Interface Research at the Wadsworth Center a b Figure 2.4 (a) A 6 × 6 P300 matrix display. The rows and columns are randomly highlighted as indicated by column 3. (b) Average waveforms for each of the 36 cells contained in the matrix from electrode Pz. The target letter “O” (thick waveform) elicited the largest P300 response, and a smaller P300 response is evident for the other characters in column 3 or row 3 (medium waveforms) because these stimuli are highlighted simultaneously with the target. All other cells indicate nontarget stimuli (thin waveforms). Each response is the average of 30 stimulus presentations. predictable as possible. With control signals possessing these traits, the user and system adaptations should be superior, and thus the required training time should be reduced and overall performance improved. 2.4 P300-Based Communication We have also begun to use and further develop the potential of the P300 class of BCI systems. In the original P300 matrix paradigm introduced by Farwell and Donchin (1988), the user is presented with a 6 × 6 matrix containing 36 symbols. The user focuses attention on the desired symbol in the matrix while the rows and columns of the matrix are highlighted in a random sequence of flashes. A P300 response occurs when the desired symbol is highlighted. To identify the desired symbol, the classifier determines the row and the column that the user is attending to (i.e., the symbol that elicited a P300) by weighting specific spatiotemporal features that are time-locked to the stimulus. The intersection of this row and column defines the selected symbol. Figure 2.4 shows a typical P300 matrix display and the averaged event-related potential responses to the intensification of each cell. The cell containing the letter “O” was the target cell and elicited the largest P300 response when highlighted. To a lesser extent the other characters in the row or the column containing the O also elicited a P300 because these cells are simultaneously highlighted with the target cell. Our focus has been on improving matrix speller classification. These studies examined variables related to stimulus properties, presentation rate, classification parameters, and classification methods. Sellers et al. (2006a) examined the effects of matrix size and interstimulus interval (ISI) on classification accuracy using two matrix sizes (3 × 3 and
- 50. 2.4 P300-Based Communication 37 Figure 2.5 Montages used to derive SWDA classification coefficients. Data were collected from all 64 electrodes; only the indicated electrodes were used to derive coefficients (see text). 6 × 6), and two ISIs (175 and 350 ms). The results showed that the amplitude of the P300 response for the target items was larger in the 6 × 6 matrix condition than in the 3 × 3 matrix condition. These results are consistent with a large number of studies that show increased P300 amplitude with reduced target probability (e.g., Duncan-Johnson and Donchin (1977)). Our lab has tested several variables related to classification accuracy using the stepwise discriminant analysis (SWDA) method (Krusienski et al. (2005)). We examined the effects of channel set, channel reference, decimation factor, and the number of model features on classification accuracy (Krusienski et al. (2005)). The factor of channel set was the only factor to have a statistically significant effect on classification accuracy. Figure 2.5 shows examples of each electrode set. Set 1 (Fz, Cz, and Pz) and set 2 (PO7, PO8, and Oz) performed equally, and significantly worse than set 3 (set 1 and set 2 combined). In addition, set 4 (which contained 19 electrodes) was no better than set 3 (which contained 6 electrodes). These results demonstrate at least two important points: First, 19 electrode locations appear to provide no more useful information beyond that provided by the 6 electrodes contained in set 3. Second, electrode locations other than those traditionally associated with the P300 response provide unique information for classification of matrix data. Occipital
- 51. 38 Noninvasive Brain-Computer Interface Research at the Wadsworth Center a b Time (ms) Figure 2.6 (a) Example waveforms for target (black) and nontarget (grey) stimuli for electrodes PO7, Pz, and PO8. The target waveform represents the average of 480 stimuli and the nontarget waveform represents the average of 2400 stimuli. The P300 response is evident at Pz and a negative deflection preceding the P300 is evident at PO7 and PO8. (b) r2 values that correspond to the waveforms shown in panel a. electrodes (e.g., Oz, PO7, and PO8) have previously been included in matrix speller data classification (Kaper et al. (2004); Meinicke et al. (2002)). In addition, Vaughan et al. (2003a) showed that these electrode locations discriminate target from nontarget stimuli, as measured by r2 , but the nature of the information provided by the occipital electrodes has not been rigorously investigated. Examination of the waveforms suggests that a negative deflection preceding the P300 response provides this additional unique information (see figure 2.6a). While a relationship to gaze cannot be ruled out at this time, it is likely that the essential classification-specific information recorded from the occipital electrodes is not produced because the user fixates the target item. An exogenous response to a stimulus occurs within the first 100 ms of stimulus presentation and appears as a positive deflection in the waveform (Skrandies (2005)). In contrast, the response observed at PO7 and PO8 is a negative deflection that occurs after 200 ms. The r2 values remain near zero until approximately 200 ms, also suggesting a negligible exogenous contribution. Moreover, whether or not this negativity is specific to the matrix style display or also present in standard P300 tasks is yet to be determined. While it is reasonable to assume that the user must be able to fixate for the response to be elicited, Posner (1980) has shown that nonfixated locations can be attended to. To our knowledge, P300-BCI studies that examine the consequences of attending to a location other than the fixated location have not been conducted. Furthermore, one may also assume that fixating a nontarget location may have a deleterious effect on performance because it is harder to ignore distractor items located at fixation than it is to ignore distractor items located in the periphery (Beck and Lavie (2005)). At the same time, fixation alone is not sufficient to elicit a P300 response. Evidence for this is provided by studies that present target and nontarget items at fixation in a Bernoulli series (e.g., Fabiani et al. (1987)). If fixation alone were responsible for the P300, both the target and nontarget items would
- 52. 2.5 A Portable BCI System 39 produce equivalent responses because all stimuli are presented at fixation. Hence, we argue that a visual P300-BCI is not classifying gaze in a fashion analogous to the Sutter (1992) steady-state visually evoked potential system. To be useful a BCI must be accurate. Accurate classification depends on feature extrac- tion and the translation algorithm being used for classification (Krusienski et al. (2005)). Currently, we are testing several alternative classification methods in addition to SWDA. To date, we have tested classifiers derived from linear support vector machines, Gaussian support vector machines, Pearson’s correlation method, Fisher’s linear discriminant, and SWDA. The preliminary results reveal minimal differences among several different classi- fication algorithms. The SWDA method we have been using for our online studies perform as well as, or better than, any of the other solutions we have tested offline (Krusienski et al. (2006)). 2.5 A Portable BCI System In addition to refining and improving SMR- and P300-BCI performance we are also focused on developing clinically practical BCI systems. We are beginning to provide severely disabled individuals with BCI systems to use in their daily lives. Our goals are to demonstrate that the BCI systems can be used for everyday communication and that using a BCI has a positive impact on the user’s quality of life (Vaughan et al. (2006)). In collaboration with researchers at the University of Tübingen and the University of South Florida, we have conducted many experimental sessions at the homes of disabled individuals (e.g., Kübler et al. (2005a); Sellers and Donchin (2006); Sellers et al. (2006c)). This pilot work has identified critical factors essential for moving out of the lab and into a home setting where people can use a BCI in an autonomous fashion. The most pressing needs for a successful home BCI system are developing a more compact system, making the system easy to operate for a caregiver, and providing the user with effective and reliable communication applications. The current home system includes a laptop computer, a flat panel display, an eight- channel electrode cap, and an amplifier with a built in A/D board. The amplifier has been reduced to 15 × 4 × 9 cm, and we anticipate a smaller amplifier in the future. We have addressed making the system more user-friendly by automating some of the processes in the BCI2000 software and employing a novice user level that allows the caregiver to start the program with a short series of mouse clicks. Thus, the caregiver’s major task is placing and injecting gel into the electrode cap, which takes about five minutes. We have also modified the BCI2000 software to include a menu-driven item selection structure that allows the user to navigate various hierarchical menus to perform specific tasks (e.g., basic communication, basic needs, word processing, and environmental controls) in a more expedient manner than earlier versions of the SMR (Vaughan et al. (2001)) and P300 (Sellers et al. (2006c)) software. In addition, we incorporated a speech output option for users who desire this ability. A more complete description of the system is provided in Vaughan et al. (2006).
- 53. 40 Noninvasive Brain-Computer Interface Research at the Wadsworth Center Finally, we have provided one severely disabled user with an in-home P300 system that he uses for daily work and communication tasks. He is a 48-year-old man with amyotrophic lateral sclerosis (ALS) who is totally paralyzed except for some eye movement. Since installation, the BCI has been used at least five times per week for up to eight hours per day. The format is a 9 × 8 matrix of letters, numbers, and function calls that operates as a keyboard and makes the computer and Windows-based programs (e.g., Eudora, Word, Excel, PowerPoint, Acrobat) completely accessible via EEG control. The system uses an ISI of 125 ms with a stimulus duration of 62.5 ms, and each series of intensifications lasts for 12.75 s. On a weekly basis the data is uploaded to an ftp site and analyzed in the lab, and classification coefficients are updated via our previously described SWDA procedure (Krusienski et al. (2005); Sellers and Donchin (2006); Sellers et al. (2006a)). The user’s average classification accuracy for all experimental sessions has been 88 percent. These results have demonstrated that a P300-BCI can be of practical value for individuals with severe motor disabilities, and that caregivers who are unfamiliar with BCI devices and EEG signals can be trained to operate and maintain a BCI (Sellers et al. (2006c)). We plan to enroll additional users in the coming months. 2.6 Discussion The primary goal of the Wadsworth BCI is to provide a new communication channel for severely disabled people. As demonstrated here, the SMR and P300 systems employ very different approaches to achieve this goal. The SMR system relies on EEG features that are spontaneous in the sense that the stimuli presented to the user provide information regarding SMR modulation. In contrast, the P300 response is elicited by a stimulus con- tained within a predefined set of stimuli and depends on the oddball paradigm (Fabiani et al. (1987)). The SMR system uses features extracted by spectral analysis while the P300 system uses time-domain features. While the P300 can be characterized in the frequency domain (e.g., Cacace and McFarland (2003)), to our knowledge, this has not been done for P300-BCI use. We use regression analysis with the SMR system and classification for the P300 system. The regression approach is well suited to the SMR cursor movement application since it provides continuous control in one or more dimensions and generalizes well to novel target configurations (McFarland and Wolpaw (2005)). In contrast, the classification approach is well suited to the P300 system where the target is treated as one class and all other alternatives are treated as the other class. Done in this way, a single discriminant function generalizes well to matrices of differing sizes. Finally, these two BCI systems differ in terms of the importance of user training. BCI users can learn to control SMRs to move a computer cursor to hit targets located on a computer screen. This is not a normal function of this brain signal, but, rather, is the result of training. In contrast, the P300 can be used for communication purposes without extensive training. The SMR system relies on improvement of user performance as a result of practice (McFarland et al. (2003)), while the P300 system uses a response that appears to remain relatively constant across trials in terms of waveform morphology (Cohen
- 54. 2.6 Discussion 41 User User User BCI system BCI system BCI system Machine learning conditioning Operant coadaptation Optimized Figure 2.7 Three concepts of BCI operation. The arrows through the user and/or the BCI system indicate which elements adapt in each concept. and Polich (1997); Fabiani et al. (1987); Polich (1989)) and classification coefficient performance (Sellers and Donchin (2006); Sellers et al. (2006a)). An SMR-BCI system is more suitable for continuous control tasks such as moving a cursor on a screen; although Piccione et al. (2006) have shown that a P300 system can be used to move a cursor in discrete steps, albeit more slowly than with an SMR system. While most BCI researchers agree that coadaptation between user and system is a central concept, BCI systems have been conceptualized in at least three ways. Blankertz et al. (e.g., Blankertz et al. (2003)) view BCI to be mainly a problem of machine learning; this view implicitly sees the user as producing a predictable signal that needs to be discovered. Birbaumer et al. (e.g., Birbaumer et al. (2003)) view BCI to be mainly an operant conditioning paradigm, in which the experimenter, or trainer, guides or leads the user to encourage the desired output by means of reinforcement. Wolpaw et al. (2000a) and Taylor et al. (2002) view the user and BCI system as the coadaptive interaction of two dynamic processes. Figure 2.7 illustrates these three views of BCI. The Wadsworth Center SMR system falls most readily into the coadaptive class, while the Wadsworth Center P300 system is most analogous to the machine learning model. Ultimately, determining which of these views (or other conceptualizations of BCI systems) is most appropriate must be empirically evaluated for each BCI paradigm. We feel that one should allow the characteristics of the EEG feature(s) to dictate the BCI system design and this will determine the most effective system for a given user. We currently test users on the SMR- and P300-based BCI systems and then select the most appropriate system based on analyses of speed, accuracy, bit rate, usefulness, and likelihood of use (Nijboer et al. (2005)). This may prove to be the most efficient model as we move BCI systems into people’s homes.
- 55. Another Random Scribd Document with Unrelated Content
- 56. filled with stagnant water which would have done no discredit to that of the lower Thames. Here we outspanned, and here we breakfasted. These breakfasts under the dome of heaven are not to be looked back upon with rapture. Picnicking is an excellent relaxation in England, but a picnic without shade, without cooling drinks, without pasties and salads and jellies and pies, without white tablecloths and bright knives, without even shelter from incessant dust, lacks much. Tinned provisions are, no doubt, excellent and nourishing, but oh, the weariness of them! And oh, the squalor of the single tin mug, which never loses the taste of what it last had in it! And oh, the meanness of the one tin plate which does duty for every meal, and every phase of it! Perhaps of all unappetising adjuncts to a breakfast the tin of preserved milk, which has been opened two days and is already becoming disgustingly familiar, is the most aggressive. The hot climate and the indefatigable ant and the fly do little to make the items of a meal attractive. What does not rapidly decompose promptly dries up. On one occasion a roasted fowl was brought up reverently to Frere in a tin box, but when it came to be eaten it had dried into a sort of papier mâchê roast fowl, and was like the viands which are thrown at the police at pantomimes. We brought many varieties of preserved food with us, and of much of it the question could not fail to arise as to whether it had ever been worth preserving. Many had experience, too, of the inventive art of the shopkeeper, as shown in the evolution of canteens and pocket table knives. The canteen, when unstrapped, tends to fall into a hundred parts, and can never be put together again. It is a prominent or
- 57. generic feature of most canteens that the kettle should look as little like a kettle as possible, and that everything should pack into a frying-pan. The pocket picnicking knife contains a knife, a fork, a spoon, and a corkscrew. The fork runs into everything and prevents the knife from being carried in the pocket. The spoon and fork are jointed for more convenient stowing, and at crises in a meal they are apt to bend weakly in the middle and then to incontinently shut up. The outspanning and the inspanning at Pretorius' Farm occupied over two hours, and then the march was resumed. A better country was reached as we neared the river, and it was a pleasant sight to see the tumbling stream of the Lesser Tugela, and to find in one valley the pretence of a garden and a house among trees. This was at Springfield, which place we reached at 2.30 P.M. The march of some eighteen miles had therefore been effected in two treks. At Springfield the camping ground was the least dreary of any the hospital had experience of, and the proximity of the Lesser Tugela made bathing possible. After a few days at Springfield we moved on to Spearman's Farm, where we camped by the hill called Mount Alice. The return from Spearman's after Spion Kop and Vaal Krantz was even more monotonous than the going forth. The first journey was to Springfield, which was reached at sundown, and where we bivouacked for the night. Springfield was left at dawn, and the next night was spent in a bivouac at Frere. On the following day, before the sun was well up, we took the last stage of the march and reached Chieveley. Here all enjoyed once more the luxury of having tents overhead, for during the crawling journey over the veldt we slept in wagons, on wagons, or under wagons.
- 58. XV SPEARMAN'S FARM In a lonely valley under the mimosa-covered heights which dominate the Great Tugela is the lonely homestead of Spearman's Farm. Those who built it and made a home in it could have had little thought that it would one day figure in the annals of history. The farmhouse and the farm buildings and the garden were enclosed by a rough stone wall, and upon this solitary homestead the hand of the Boer had fallen heavily. The house had been looted, and what was breakable in it had been broken. The garden had been trampled out of recognition, the gates were gone, the agricultural implements had been wantonly destroyed, and the unpretending road which led to the farm was marked by the wheels of heavy guns. The house was small and of one story, and was possessed of the unblushing ugliness which corrugated iron alone can provide. The door swung open, and any could enter who would, and through the broken windows there was nothing to be seen but indiscriminate wreckage. There was about the little house and its cluster of outbuildings a suggestion of the Old Country, and it wanted but a rick or so, and a pond with white ducks to complete a picture of a small English farm. The garden had evidently been the subject of solicitous care, and was on that account all the more desolate, and what delight it ever had had been trampled out of it by countless hoofs or obliterated by the rattling passage over it of a battery or so of artillery.
- 59. At the back of the farm, and at the foot of a green kopje, was a quaint little burial ground--little because it held but two graves, and quaint because these were surmounted by unexpected stone memorials of a type to be associated with a suburban English cemetery. These monuments were fitly carved, and were distinctly the product of no mean town, and they were to the memory respectively of George Spearman and of Susan Spearman. For some undefinable reason these finished memorials, so formal and so hackneyed in their design, appeared inappropriate and even unworthy of the dignity of the lonely graves at the foot of the kopje. Some more rugged emblem, free from artificiality and from any suggestion of the crowded haunts of men, would have covered more fittingly the last resting-place of these two pioneers. A few trees, almost the only trees within sight, shaded the little graveyard, and the trees and the monuments were enclosed by a very solid iron railing. It was in the shadow of this oasis that the dead from our hospital were buried. XVI THE HOSPITAL AT SPEARMAN'S The hospital reached Spearman's on January 16th, and was pitched at the foot of the hill, upon the summit of which the naval gun was firing. We were, therefore, close to those scenes of fighting which were to occupy the next few weeks, and too close for comfort to the
- 60. great 4.7 gun, the repeated booming of which often became a trouble to those who were lying ill in the hospital. The heights that dominated the southern bank of the Tugela were very steep on the side that faced the river, but on the side that looked towards Spearman's the ground sloped gradually down into a wide plain which, like other stretches of veldt, was dotted with kopjes and slashed with dongas. Anyone who mounted the hill at the back of the hospital would come by easy steps to an abrupt ridge, beyond which opened a boundless panorama. In the valley below this crest was the winding Tugela, and just across the dip rose the solemn ridge of Spion Kop. Far away in the distance were the purple hills which overshadowed Ladysmith. If the crest were followed to the right the ground rose until at last the summit of the naval hill was reached, and here were the handy men and their big gun. From this high eminence a splendid view was obtained of the country we desired once more to possess. The Tugela glistened in the sun like a band of silver, and over the plain and in and out among the kopjes and round the dongas the brown road wound to Ladysmith. The road was deserted, and the few homesteads which came into view showed no signs of life. At the foot of the hill was Potgieter's Drift, while above the ford was a splashing rapid, and below was the pont which our men had seized with such daring. The face of the hill towards the river was covered with mimosa trees and with cactus bushes and aloes, and this unexpected wealth of green almost hid the red and grey boulders which clung to the hill-side. Among the rocks were many strange flowers, many unfamiliar plants, and creeping things innumerable. This was a
- 61. favourite haunt of the chameleon, and I believe it was here that the hospital chameleon was captured. The quiet of the place, when the guns had ceased, was absolute, and was only broken by the murmur of the numerous doves which occupied the mimosa woods. The whole place seemed a paradise of peace, and there was nothing to suggest that there were some thousands of grimy men beyond the river who were busy with the implements of death. On looking closely one could see brown lines along many of the hillsides, and these said lines were trenches, and before the hubbub began men in their shirt-sleeves could be seen working about them with pickaxes and shovels. I should imagine that few modern battles have been viewed by the casual onlooker at such near proximity and with such completeness in detail as were the engagements of Spion Kop and Vaal Krantz, when viewed from the high ground above our hospital. The hospital, though now more than twenty-five miles from the railway, was very well supplied with almost every necessity and with the amplest stores of food. Bread was not to be obtained, or only on occasion, when it would be brought up by an ambulance on its return from Frere. We had with us, however, our flocks and herds, and were thus able to supply the sick and wounded with fresh milk, and the whole hospital with occasional fresh meat. We were a little short of water, and fuel was not over abundant. As a result the washing of clothes, towels and sheets presented the same type of problem as is furnished by the making of bricks without straw. The aspect of a flannel shirt that has been washed by a Kaffir on the remote veldt leaves on the mind the impression that the labour of the man has been in vain.
- 62. Our stay at Spearman's was extended to three weeks, and we dealt with over a thousand wounded during that period, and I am sure that all those who came within our lines would acknowledge that at No. 4 they found an unexpected degree of comfort and were in every way well done for. On the Sunday after our arrival the wounded began to come in. Thirteen only came from the division posted at Potgieter's Drift, the rest came from Sir Charles Warren's column. Increasing numbers of wounded came in every day in batches of from fifty to one hundred and fifty. They were all attended to, and were sent on to Frere as soon as possible. All the serious cases, however, were kept in the hospital. XVII THE TWO WHITE LIGHTS Many of the wounded who were brought in between the 18th and the 24th of January came in after sundown. The largest number arrived on the night of Monday, the 22nd. It was a very dark night. The outline of the tents and marquees was shadowy and faint. The camp was but the ghost of a camp. Here and there a feeble light would be shining through the fly of a marquee, and here and there an orderly, picking his way among the tent ropes by the aid of a lantern, would light up a row or two in the little canvas town. In the front of the camp was the flagstaff, high up upon which were suspended the two white lights which marked the situation of the
- 63. hospital. These lamps only sufficed to illumine a few of the tents in the first line. The flaps of these tents were probably secured and the occupants asleep. It was a weary journey to the hospital, and one can imagine with what eagerness the tired, hungry, aching wounded would look ahead for the two white lights. Rocking in pain on a crawling ox wagon, or jolted in the rigid fabric of an ambulance, the way must have seemed unending. Tumbling along in the dark, with no sound but the creaking of the wagon and the incessant moans of the shapeless, huddled figures who were lying in the cart, the journey might well have been one never to be forgotten. How many a time a tired head must have been lifted up from the straw to see if there were yet any sign of the two white lights. Would the journey never end, and the pain never cease? and was the broken limb to be wrenched every time the blundering wagon pitched and rolled? And why had the man who had talked so much ceased to speak--and indeed to breathe? Would they drive through the dark for eternity? and would they never come in view of the two white lights? It was a miserable sight to see these belated wagons come in, and they would often rumble in all night. They emerged one by one out of the darkness and drew up in the open space between the two central lines of tents, and between the few uplifted lanterns held by the sergeants and the men on duty. After they had deposited their load they moved away and vanished again into the night. Some of the wounded in the wagons were sitting up, but the majority were lying on the straw with which the wagon would be littered. Some were asleep and some were dead; and by the light of the lanterns the wagon seemed full of khaki-coloured bundles, vague
- 64. in outline and much stained with blood, with here and there an upraised bandage, and here and there a wandering hand, or a leg in crude splints, or a bare knee. And round about all a medley of rifles, boots, haversacks, helmets, cartridge pouches and tin canteens. What the journey must have been to many I could gather from an incident of one of these dreary nights. A wagon had reached the hospital lines and was waiting to be unloaded. A man with a shattered arm in a sling was sitting up, and at his feet a comrade was lying who had been very hard hit, and who had evidently become weaker and less conscious as the wagon had rolled along. The apparently sleeping man moved, and, lifting his head to look at his pal, who was sitting above him, asked wearily, for probably the fiftieth time, Don't you see nothing yet, Bill, of the two white lights? XVIII AFTER SPION KOP On Wednesday, January 24th, came the terrible affair of Spion Kop. On the previous day some hint of what was expected was foreshadowed in the order that an additional hundred bell tents were to be erected in No. 4 Field Hospital. These tents were obtained from a brigade who were bivouacking, and were all pitched by Wednesday afternoon. They represented accommodation for an additional number of five hundred wounded, and it was, therefore, evident that an important engagement was at hand.
- 65. On Thursday the wounded came pouring in, and they came in the whole day and until late at night, until the hospital was full. The number admitted on that day was nearly six hundred. Those who were deposited in the bell tents had to lie on stretchers. All were provided with blankets. In spite of the immense number of the wounded, they were all got under shelter by Thursday night, and had had their more serious injuries attended to, and were made as comfortable as circumstances would admit. Some of the staff went round with water and food, and others with morphia, while a third party made it their business to see that every man was bestowed as comfortably as extemporised pillows or change of posture could make him. The pillows were represented by helmets, or by the happy combination of helmet and boot, or by haversacks or rolled- up tunics. The volunteer ambulance corps and the coolie bearers did excellent service. The larger number of the wounded were on the top of Spion Kop. The path down was about two miles, was steep, and in places very difficult. The carriage of the wounded down the hill had all to be by hand. From the foot of the hill to the hospital the carriage was by ambulance wagons and in some cases by bearers. All the stretchers had hoods. There was no doubt that the wounded suffered much on account of the tedious transport, but it was rendered as little distressing as possible. The surgeons who went after the wounded on the top of the hill told us that the sight of the dead and injured was terrible in the extreme, the wounds having been mostly from shell and shrapnel; some men had been blown almost to pieces. The weather on Wednesday was warm, but was not to be compared with the intense
- 66. heat on the day of the battle of Colenso. The temperature was that of a hot summer's day in England. Thursday was fortunately cloudy and much cooler. As to the wounded, there was the usual proportion of minor injuries, but on the whole the wounds were much more severe than those received at Colenso. This is explained by the large number of wounds from shell and shrapnel. The men, however, were much exhausted by the hardships they had undergone. In many instances they had not had their clothes off for a week or ten days. They had slept in the open without great-coats, and had been reduced to the minimum in the matter of rations. The nights were cold, and there was on nearly every night a heavy dew. Fortunately there was little or no rain. The want of sleep and the long waiting upon the hill had told upon them severely. There is no doubt also that the incessant shell fire must have proved a terrible strain. Some of the men, although wounded, were found asleep upon their stretchers when brought in. Many were absolutely exhausted and worn out independently of their wounds. In spite of all their hardships the wounded men behaved splendidly, as they always have done. They never complained. They were quite touching in their unselfishness and in their anxiety not to give trouble; but it was evident enough that they were much depressed at the reverse. The shell wounds were the most terrible and the most difficult to treat. One man had most of his face shot away, including both eyes. Another had the forearm shot off and two fearful wounds of each thigh dividing the anterior muscles to the bone. In one case a shrapnel had opened a main artery in the forearm, and the man
- 67. came down safely with a tourniquet on his brachial artery composed of a plug of cake tobacco and the tape of a puttie. I cannot help thinking that this ingenious tourniquet was the work of one of the handy men. XIX THE STORY OF THE RESTLESS MAN The following incident may serve to illustrate the often-expressed unselfishness of the soldier, and his anxiety to do what he can for a comrade in trouble. Among the wounded who came down from Spion Kop was a private, a native of Lancashire, who had been shot in the thigh. The thigh-bone was broken, and the fracture had been much disturbed by the journey to the hospital. The man was given a bedstead in one of the marquees; the limb was adjusted temporarily, and he was told to keep very quiet and not to move off his back. Next morning, however, he was found lying upon his face, with his limb out of position and his splints, as he himself confessed, all anyhow. He was remonstrated with, but excused himself by saying, But you see, doctor, I am such a restless man. The limb was more elaborately adjusted, and everything was left in excellent position. Next morning, however, the restless man was found lying on the floor of the marquee, and in his bed was a man who had been shot through the chest. The marquee was crowded and the number of beds were few, and those who could not
- 68. be accommodated on beds had to lie on stretchers on the ground. The man who was shot in the chest had come in in the night, and had been placed on the only available stretcher. The restless man proceeded to explain that the newcomer seemed worse off than he was, and that he thought the man would be easier on the bed, so he had induced the orderlies to effect the change. The man who was shot in the chest died suddenly, and in due course the restless man was back in his own bed once more. It was not, however, for long, for on another morning visit the Lancashire lad was found on the floor again, and again beamed forth an explanation that one of the wounded on the ground, who had come in late, seemed to be very bad, and so he had changed over. The present occupant of the bed was in a few days moved down to the base, and the restless man was in his own bed again. But not many days elapsed before he discovered among the fresh arrivals an old chum, who longed to lie on a bed, and thus the good-hearted North-countryman found himself once more on the floor. The moving of a man with a broken thigh from a bed to the ground and back again means not only such disordering of splints and bandages, but much pain to the patient and no little danger to the damaged limb. So this generous lad was talked to seriously, and with a faintly veiled sternness was forbidden to give up his bed again on any pretence. In the little attempt he made to excuse himself he returned once more to his original joke and said, with a broad grin: But you see, doctor, I am such a restless man. XX
- 69. DID WE WIN? One instance of the indomitable pluck of the British soldier deserves special notice. A private in the King's Royal Rifles, of the name of Goodman, was brought from Spion Kop to No. 4 Field Hospital in an ambulance with many others. He was in a lamentable plight when he arrived. He had been lying on the hill all night. He had not had his clothes off for six days. Rations had been scanty, and he had been sleeping in the open since he left the camp. He had been struck in the face by a fragment of shell, which had carried away his right eye, the right upper jaw, the corresponding part of the cheek and mouth, and had left a hideous cavity, at the bottom of which his tongue was exposed. The rest of his face was streaked with blood, which was now dried and black--so black that it looked as if tar had been poured on his head and had streamed down his cheek and neck. Eight hours had been occupied on the journey to the hospital, and eight hours is considered to be long even for a railway journey in a Pullman car. He was unable to speak, and as soon as he was settled in a tent he made signs that he wanted to write. A little memorandum book and a pencil were handed to him, and it was supposed that his inquiry would be as to whether he would die--what chance he had? Could he have something to drink? Could anything be done for his pain? After going through the form of wetting his pencil at what had once been a mouth, he simply wrote: Did we win? No one had the heart to tell him the truth. His memorandum-book--which is in my possession--was used by him while he remained speechless in the hospital, and certain of the
- 70. notes he made in it, and which are here appended, speak for themselves: Water. I haven't done bleeding yet. I've got it this time. I think my right eye is gone, and I can hardly swallow. There are no teeth in front. It aches a lot. I'm lying the wrong way for my wound. I found the trenches. I've had all the officers over to see me. He is pleased, the doctor. Did my haversack come with me? If it did, there is some tobacco in it. You can give it to them that smoke. Poor Goodman, he had no mouth to smoke with himself. I am glad to say he reached England, is in good health, and is as cheery as ever. XXI THE FIGHTING SPIRIT The circumstances under which men enlist in the Army are, no doubt, varied enough. But not a few find their place under the colours in obedience to that fighting spirit which has for centuries been strong in the hearts of the islanders from Great Britain and Ireland. That spirit has anyhow carried the colours over the world.
- 71. Among the wounded there are many who, to use an expression common on the soldiers' lips, were fed up with the war: they had had enough of it. There were others who were eager to be at it again, who felt that they had a score to wipe off; and even among the desperately hurt there would be here and there a man keen for revenge, and full of a passionate desire to have another go at 'em. These men, ill as they often were, would describe with a savage delight, and in savage language, the part they had played in the battle out of which they had been finally dragged on a stretcher. A little success, a victory however small, did much to lessen the torment of a wound and to gild the contemplation of a life henceforth to be spent as a cripple. One gallant lad had been paralysed by a Mauser at short range, and had little prospect of other than permanent lameness. He had been in the assault on Vaal Krantz, had escaped without hurt until just towards the end, and was shot as his victorious company were rushing the last trench. After he had been examined, and while he was still lying on his stretcher, I could not avoid the remark, This is a bad business. To which he replied, Yes, but we took the bally trench. To many and many of the dying the last sound of which they were conscious must have belonged to the clamour of war, and it was well for those who heard, or fancied they heard, above the roar of guns the shout of victory. One officer, dying in the hospital at Spearman's, had his last moments made happy by the sound of battle. He had sunk into a state of drowsiness, and was becoming gradually unconscious. Every now and then the boom of the 4.7 gun, firing from the hill above us, would rattle through the tents, and with each shot a smile would come over his face, and he would mutter
- 72. with great satisfaction, They are getting it now. He repeated these words many times, and they were, indeed, the last he uttered. Things were evidently going better with the army in his dream than they were at that moment with the real regiments by the river. Some most vivid suggestions of what may pass through the soldier's mind during the actual circumstances of war were afforded by the utterances of more or less unconscious men when passing under the influence of chloroform in the operation-tent. Before they fell into the state of sleep, it was evident that the drug, with its subtle intoxicating power, brought back to the fading sense some flash of a scene which may have been real, but which was rendered lurid, spectral, and terrifying by the action of the poison. Under this condition incoherent words of command would be uttered in rapid tones, full of an agony of eagerness and haste; and cries for help would be yelled forth in what seemed to be a maniacal frenzy. Many of the actual utterances that escaped these unconscious lips, and gave glimpses of a phantom war as seen through the vapor of chloroform, were too fragmentary to be remembered, but two at least were muttered with such an emphasis of horror that I took note of them. One of the wounded from Spion Kop had evidently engraved upon his mind the hideous scene of slaughter which the trenches on that hill presented. As he was being anæsthetised it was apparent that in his dream he was back again in the trenches, and was once more among his dead and mangled comrades. The vision of one wounded man especially haunted him and fascinated him, and at last he screamed out: There goes that bloke again whose leg was shot away; blimy, if he ain't crawling now!
- 73. Another poor fellow had before his eye the spectre of an awful kopje. His fragmentary utterances made vivid the unearthly land he was traversing. All who stood by could picture the ghostly kopje, and could almost share in his anguish when he yelled: There they are on the hill! For God's sake, shoot! Why don't we shoot? XXII THE BODY-SNATCHERS Early in the campaign Colonel Gallwey, the P.M.O., organised a volunteer ambulance corps. Two thousand bearers were wanted, and in a few days two thousand were enrolled. Their duties were to carry the wounded off the field, to transport serious cases from the advanced hospitals or dressing stations to the stationary field hospital, and thence, if need be, to the railway. There were to be twelve on a stretcher. This corps contained examples of all sorts and conditions of men--labourers, mechanics, gentlemen, dock loafers, seamen, dentists, a chemist or two, a lawyer or two, tram drivers, clerks, miners, and shop assistants. Many were refugees from the Transvaal, and the majority had been thrown out of work of some kind or another by the war. A chance of getting employment had, no doubt, induced many to enlist, while probably the greater number were attracted by a spirit of adventure, by a desire to get to the front and to see something of the pomp and circumstance of war.
- 74. They formed a strange company when they mustered at Pietermaritzburg--a section of a street crowd in their everyday clothes, or in such clothes as were selected for roughing it. There was immense variety in the matter of hats. Belts were a feature. The flannel shirt, which was practically de rigueur, was replaced in an instance or two by a jersey. Collars were not worn; neckties were optional. There was no fixed fashion in the matter of boots; they varied from canvas shoes, worthy of a dandy at the seaside, to top boots fit for a buccaneer. As to the men themselves, they were of all ages, heights, shapes, and sizes--the men of a crowd. Some were sunburned, and some were pale. Some were indifferent, but most were eager. Some were disposed to assume a serious military bearing, while others appeared to regard the venture as a silly joke of which they were beginning to be a little ashamed. There is no doubt that the corps was in appearance not impressive. They were wild and shabby looking, disordered, unsymmetrical, and bizarre. They were scoffed at; and acquired the not unkindly meant title of the body-snatchers. Later on the exuberant invention of the soldier dignified them by the titles of the catch-'em-alive-oh's or the pick-me-ups. It is needless to say that a good number of unsuitable and undesirable men had found their way into the ranks. These were gradually weeded out, and under the discreet command of Major Wright the corps improved day by day, until the time Spearman's was reached they formed a very efficient, reliable, and handy body of men. They did splendid service, and one which was keenly appreciated. They were the means of saving many lives and an
- 75. infinite amount of pain. Their longest tramp, of which I had knowledge, was from Spearman's to Frere, a distance of twenty-five miles. They showed the usual British indifference under fire, and went without hesitancy wherever they were led. Unfortunately it happened that many of the worthy body-snatchers were wounded, and not a few of them were killed. In the early days of their career the catch-'em-alive-oh's fell upon bad times. They knew little of camp life, and less of the art of getting the most out of it. They had no organisation among themselves, and many were incompetent to shift alone. They began as a mob, and they tried to live as a mob, and the result was that about the time of Colenso they had little comfort but that which is said by the moralist to be derived from labour. In their camp after the battle they had time to settle down. They entered the camp a thriftless crowd, and came out of it a company of handy men. They were popular with the soldiers. They had the gift of tongues of a kind, and could compete with most in the matter of lurid language. Their incessant hunger and indiscriminate thirst were a matter for admiration. They were good-hearted, and, although they looked wild, they meant well. Many a wounded man has been rocked to sleep on their stretchers, and on more than one dying ear the last sound that fell was the tramp of their untidy feet. XXIII SEEING THEM OFF
- 76. On the afternoon of Thursday, February 8th, the news came to the hospital at Spearman's that the army was once more to retire, and signs were already abroad to show that the retreat had commenced. At the same time an order arrived to the effect that all the wounded were to be moved at sunrise on the following day to Frere. Our stay at Spearman's--extended now to three weeks--had therefore come to an end. Among those left in the hospital were 150 patients whose condition was more or less serious. They had been kept under care as long as possible in order to avoid or postpone the danger of the long journey to the base. It was determined that these 150 men should be carried down to Frere on stretchers and by hand. And this was done, and well done, by the much-ridiculed corps of body- snatchers. It was no light undertaking, for the distance was twenty-five miles, and the road was dusty and not of the best. Every step had to be tramped under a glaring sun, and the heat of that day was great. Allowing twelve men to a stretcher, 1,800 men would be required. This number was forthcoming at sunrise, and they accomplished the march in the day, reaching Frere at sundown. This was a splendid piece of work. It is not hard to surmise what would have happened to many of those who were the most ill if their journey to Frere had been by the ox-wagon, or by the still less easy ambulance. As it was, the whole convoy went down with comfort, and only one man died on the way, and he had indeed just reached his journey's end when his life ebbed away.
- 77. Long before sunrise on the morning of the departure from Spearman's the hospital was astir; and while it was yet dark lights could be seen in most of the tents, and lanterns carried by orderlies or coolies were moving here and there among the grey lines. The two white lights which hung from the flag-pole in front of the hospital were still shining. By the time the shadows had vanished and the light of the dawn fell upon No. 4, it was in a state of untidy turmoil. Everyone was on the alert to see them off. In the marquees the last dressings were being carried out by candle-light. Clothes were being got together; helpless men were being dressed; blankets were being rolled up, and such comforts as the hospital could provide were being packed for the wounded to take with them on their journey. Cherished possessions were being dragged out from under pillows, to be safely disposed in a haversack or a boot. The grey light fell upon orderlies in their shirt sleeves bustling from tent to tent; upon piles of provision cases and of forage which were being turned out; upon heaps of stretchers; upon the rolled-up kit of the Army Medical Corps men; upon melancholy coolies who had been up all night, and were still crawling about, and were still in their night attire. This night outfit would consist, probably, of a turban, a mealie sack round the neck, and a decayed army mackintosh on the body; or of a turban, a frock-coat, which might at one time have graced Bond Street, and bare legs. Here and there in the indistinct light would be seen the white apron and trim dress of a nurse, who still carried the lantern she had had with her since the small hours of the morning. All were anxious to be up in time to see them off.
- 78. In due course, and even yet before the sun could be seen, the Volunteer Ambulance Corps began to form up outside the camp. They were nearly two thousand strong, and they were a wild-looking company. There was, however, more uniformity in their clothing now, because they had been supplied with khaki tunics, and with occasional khaki trousers. Some wore putties, some gaiters, and some had tucked their trousers inside their socks. A few had cut their trousers off about the knee and were distinguished by bare legs. A gaiter on one leg and a puttie on the other was not considered to be in any way démodé. Their hats were still very varied, but many had possessed themselves of helmets which had been picked up on the field. Uniformity and smartness could, however, not be expected if one man wore a helmet and the next a tam-o'-shanter, the third a bowler hat, and the fourth a squasher or a headpiece of his own designing. They had red-cross brassards on their left arms, but these had become merely fluttering bits of colouring. This weird corps carried their possessions with them, and it was evident that in transporting their impedimenta they had appreciated the value of the division of labour. Many had military water-bottles, which they had probably picked up. Others carried their water in glass bottles, which dangled from their waists. Hanging about their bodies by strings or straps would be various useful domestic articles. Attached to one man would be a bundle of firewood, to another a saucepan, to a third a kettle and a lantern. Here a man would have in the place of a sabre-tache a biscuit tin suspended by a cord, or a hatchet and a tin-opener, or a spare pair of boots, which swung bravely as he marched. A popular vade mecum was an empty jam
- 79. tin (much blackened by the smoke of the camp fire) with a wire handle, and evidence that it represented a cooking-pot. Belts, knives, sticks, overcoats, rolled-up mackintoshes, and a general tint of sunburn and dirt completed the uniform of this strange company. Before they entered the camp the wounded had been brought out on stretchers. The stretchers were placed on the grass, side by side, in long rows which extended across the breadth of the hospital. The men lying on them were not pleasant to look at. They formed a melancholy array of bad cases. Each man was covered by a brown blanket, and within the hood of the stretcher were his special belongings, his boots and his haversack, and, with them, such delicacies for the journey as a pot of jam, a chunk of bread, some biscuits, a lump of tinned meat in a newspaper, and bottles (mostly with paper corks) containing water or milk or tea. Those on the stretchers presented bandaged legs and bandaged arms, splints of all kinds, covered-up eyes and bound-up heads, and the general paraphernalia of an accident ward. Some of the faces were very pinched and pale, for pain and loss of blood and exhaustion had caused the sunburn to fade away. The light of the dawn fell upon this woe-begone line, and dazzled the eyes of many with the unaccustomed glare. Those who were not too ill were in excellent spirits, for this was the first step on the journey homewards. Such were excited, garrulous and jocular, and busy with pipes and tobacco. A few were already weary, and had on their lips the oft-repeated expression that they were fed up with the war. Many a head was lifted out of the hood to see if any old chum could be recognised along the line, and from those would come such exclamations as: Why are you here, Tom? Where have
- 80. you been hit? Ain't this a real beanfeast? Thought you were stiff. We're on the blooming move at last. Many of the men on the stretchers were delirious, and some were almost unmanageable. One poor fellow was babbling about the harvest and the time they were having. He was evidently in his dream once more among the cornfields of England, and among plenteous beer. Another shook the canvas hood of his stretcher and declared with vehemence that he would not go in any bally sailing boat, he was going in a steamer, and the colonel would never let his men go in a rotten sailing ship. Whereupon he affirmed that he was going to chuck it, and proceeded to effect his purpose by rolling off his stretcher. When the Volunteer Ambulance Corps marched along the line of stretchers they were the subject of much chaff, and many comments such as these burst forth: You're being paraded before the General. So buck up! Pull up yer socks. You with the kettle! Do you take yourself for a gipsy van? We ain't buying no hardware to-day--go home. You know there's a Government handicap on this job, and half a crown to the man who gets in first, so you had better hurry my stretcher along. And so on; in the dialect of London, of Dublin, of Lancashire, and of Devon, with infinite variety and with apparent good spirits. There were many anxious cases among this crowd on the stretchers. One, for example, was an Irishman named Kelly, a private in the Lancashire Fusiliers. He was as plucky a soldier as the plucky soil of Ireland has ever produced. His right arm had been smashed on Spion Kop. He had been on the hill two nights; and when the darkness fell had spent his time in crawling about on the
- 81. ground, holding the sleeve of his shattered arm between his teeth, dragging his rifle with his left hand, and searching the bodies of the dead for any water that may have been left in their water-bottles. He had lost an incredible amount of blood, and when he reached the hospital it was necessary to amputate the whole upper limb, including the shoulder-blade and the collar-bone. He went through this ordeal with infinite courage and with irrepressible good humour. He had been the strong man of his regiment and a great boxer, and, as he casually said, He should miss his arm. Kelly's spirits were never damped, and he joked on all topics whenever he had the strength to joke. He was a little difficult to manage, but was as docile as a lamb in the hands of the Sister who looked after him, and for whom he had a deep veneration. Nothing in the ordinary way upset this gallant Irishman, but just before the convoy started he did for once break down. Two bottles of English beer had found their way into the camp as a precious gift. Kelly was promised these bottles to take with him on his journey. In due course they were deposited in the hood of his stretcher. When his eyes fell upon the delectable vision of English beer he could stand no more, and Kelly wept. I little thought when I saw Kelly off at Spearman's that the next time I should say good-bye to him would be in a hansom cab in Pall Mall; but so it was. When all was ready the stretchers were lifted off the ground in order, and the bearers filed out of the camp and on to the dusty track. The morning was like that of a summer's day in England, and we watched the long convoy creep along the road until it was nearly
- 82. Welcome to our website – the perfect destination for book lovers and knowledge seekers. We believe that every book holds a new world, offering opportunities for learning, discovery, and personal growth. That’s why we are dedicated to bringing you a diverse collection of books, ranging from classic literature and specialized publications to self-development guides and children's books. More than just a book-buying platform, we strive to be a bridge connecting you with timeless cultural and intellectual values. With an elegant, user-friendly interface and a smart search system, you can quickly find the books that best suit your interests. Additionally, our special promotions and home delivery services help you save time and fully enjoy the joy of reading. Join us on a journey of knowledge exploration, passion nurturing, and personal growth every day! ebookbell.com