Eben Myers & Santosh Mathan - Building Robot Factory: A Game Design and Brain Science Collaboration

Editor's Notes

• #6: Executive functions (EF) are a set of processes involved in complex cognition – processes that play a crucial role in moderating complex tasks that require solving novel problems, modifying behaviour in the light of new information, generating strategies, and sequencing complex actions (Elliott, 2003). ARP is known to depend critically on EF, evidenced by substantial overlap in their core computational properties (i.e., as general-purpose cognitive processes that underlie intelligent human behavior; Diamond, 2013) and their underlying neural correlates (i.e., dependence on lateral and anterior prefrontal cortex; Duncan et al., 2000; Duncan, 2012). Broad consensus suggest that executive function relies on three central cognitive processes Switching: shifting between tasks or mental sets Updating: updating and monitoring working memory representations Inhibition: suppression of stimuli and responses that are unrelated to the task
• #8: Two key findings from the research literature on cognitive skill acquisition highlight the promise and pitfalls of attempts to improve ARP. The first is that practice and training have robust and highly predictable impact on the acquisition of cognitive skills. Thus, performance of skills ranging from simple motor responses to complex cognitive operations improves with a regular power law function (Logan, 1988; VanLehn, 1996). Evidence suggests that reasoning and problem solving skills are no exception to this rule (Carlson et al., 1990; Mackey et al., 2011), indicating that ARP can be strengthened through training. However, a second key finding is that the acquired skills are typically highly specific: extended training at a given task often leads to little or no general improvement, even for tasks that on the surface appear closely related (‘near transfer’; VanLehn, 1996; Diamond, 2013; Enriquez-Geppert et al., 2013). For example, working memory capacity for numbers can be increased 15-fold with several hundred hours of focused training, without having any impact on working memory for letters (Ericcson & Staszewski, 1989). Evidence of far transfer—where training on a task improves performance on superficially very different tasks—is more scarce still (Diamond, 2013; VanLehn, 1996). Yet the essence of human fluid intelligence is its transferability—by its very nature, ARP must be domain general and widely applicable.
• #9: According to this analysis, core executive functions can be characterized as cognitive skills and as such can be trained, but any single training task (or any limited number of tasks) will be too high-level to generalize effectively to novel task contexts, which is the very essence of ARP. By contrast, our training regime, FAST, mitigates against over-training of hyper-specific skills by isolating component executive processes (update, switch, inhibit) through the requirement to combine them repeatedly and in novel ways with a range of mental operations (logical, relational) across a range of cognitive domains (symbolic, visual) to create a rich but systematically mapped training space. Additionally, to promote flexible and generalizable use of knowledge, we vary the contexts in which practice occurs by changing online task instructions every few trials (cf., Cole et al., 2010; Dumontheil et al., 2011). These instructions add a second task that must be carried out with the ongoing primary task. The ability to reliably generate new task models flexibly in response to changing instructions is central to the relationship between working memory (WM) and fluid intelligence. Moreover, WM tasks that involve instructed performance of this kind very strongly activate the network of brain regions implicated in high-level reasoning and problem solving (Dumontheil et al., 2011) and correlate more strongly with fluid intelligence than standard WM paradigms (Duncan et al., 2012). All components of FAST require subjects to rapidly encode novel sets of instructions and use these to guide their action. This encode-perform cycle is repeated with entirely new task requirements every 3-4 minutes, such that training emphasizes the general capacity to flexibly configure cognitive resources according to current requirements, rather than the specific ability to adopt any one configuration in particular. In addition, specific components of the training regime emphasize hypothesis testing and inference under conditions of uncertainty, akin to the classical Wisconsin Card Sort Test of executive function.
• #10: According to this analysis, core executive functions can be characterized as cognitive skills and as such can be trained, but any single training task (or any limited number of tasks) will be too high-level to generalize effectively to novel task contexts, which is the very essence of ARP. By contrast, our training regime, FAST, mitigates against over-training of hyper-specific skills by isolating component executive processes (update, switch, inhibit) through the requirement to combine them repeatedly and in novel ways with a range of mental operations (logical, relational) across a range of cognitive domains (symbolic, visual) to create a rich but systematically mapped training space. Additionally, to promote flexible and generalizable use of knowledge, we vary the contexts in which practice occurs by changing online task instructions every few trials (cf., Cole et al., 2010; Dumontheil et al., 2011). These instructions add a second task that must be carried out with the ongoing primary task. The ability to reliably generate new task models flexibly in response to changing instructions is central to the relationship between working memory (WM) and fluid intelligence. Moreover, WM tasks that involve instructed performance of this kind very strongly activate the network of brain regions implicated in high-level reasoning and problem solving (Dumontheil et al., 2011) and correlate more strongly with fluid intelligence than standard WM paradigms (Duncan et al., 2012). All components of FAST require subjects to rapidly encode novel sets of instructions and use these to guide their action. This encode-perform cycle is repeated with entirely new task requirements every 3-4 minutes, such that training emphasizes the general capacity to flexibly configure cognitive resources according to current requirements, rather than the specific ability to adopt any one configuration in particular. In addition, specific components of the training regime emphasize hypothesis testing and inference under conditions of uncertainty, akin to the classical Wisconsin Card Sort Test of executive function.
• #19: The proposed research will seek to augment the activity of the cortical substrate supporting executive functions through tES. Neuroimaging studies have revealed that executive control functions stem from the activity of a distributed fronto-parietal network (Barbey et al., 2012; Cole and Schneider, 2007). As Duncan (2012) and Niendam et al. (2012) have reviewed, the dorsolateral prefrontal cortex (DLPFC), anterior cingulate cortex (ACC), and lateral parietal cortex (LPC) play important roles in the executive control network across many different task domains. DLPFC has considerable connectivity with sensory and motor regions, and plays a central role in executive control by maintaining rules for action and guiding response selection and inhibition. ACC contributes to this network by det ecting processing conflicts and situations that demand executive control—serving to engage DLPFC. Extensive connectivity with the parietal cortex provides DLPFC with information on the salience of stimuli, and stimulus-response pairings, while DLPFC supports parietal cortex in guiding shifts attentional focus depending on task demands. Our objective is to enhance the activity of this crucial network during cognitive task performance using tES, with the aim of enhancing skill-acquisition. As the model in Figure 1 illustrates, we expect tES to act synergistically with cognitive training (Krause & Kadosh, in press) to boost the efficacy of the core processes associated with ARP.
• #20: The proposed research will seek to augment the activity of the cortical substrate supporting executive functions through tES. Neuroimaging studies have revealed that executive control functions stem from the activity of a distributed fronto-parietal network (Barbey et al., 2012; Cole and Schneider, 2007). As Duncan (2012) and Niendam et al. (2012) have reviewed, the dorsolateral prefrontal cortex (DLPFC), anterior cingulate cortex (ACC), and lateral parietal cortex (LPC) play important roles in the executive control network across many different task domains. DLPFC has considerable connectivity with sensory and motor regions, and plays a central role in executive control by maintaining rules for action and guiding response selection and inhibition. ACC contributes to this network by det ecting processing conflicts and situations that demand executive control—serving to engage DLPFC. Extensive connectivity with the parietal cortex provides DLPFC with information on the salience of stimuli, and stimulus-response pairings, while DLPFC supports parietal cortex in guiding shifts attentional focus depending on task demands. Our objective is to enhance the activity of this crucial network during cognitive task performance using tES, with the aim of enhancing skill-acquisition. As the model in Figure 1 illustrates, we expect tES to act synergistically with cognitive training (Krause & Kadosh, in press) to boost the efficacy of the core processes associated with ARP.
• #21: The proposed research will seek to augment the activity of the cortical substrate supporting executive functions through tES. Neuroimaging studies have revealed that executive control functions stem from the activity of a distributed fronto-parietal network (Barbey et al., 2012; Cole and Schneider, 2007). As Duncan (2012) and Niendam et al. (2012) have reviewed, the dorsolateral prefrontal cortex (DLPFC), anterior cingulate cortex (ACC), and lateral parietal cortex (LPC) play important roles in the executive control network across many different task domains. DLPFC has considerable connectivity with sensory and motor regions, and plays a central role in executive control by maintaining rules for action and guiding response selection and inhibition. ACC contributes to this network by det ecting processing conflicts and situations that demand executive control—serving to engage DLPFC. Extensive connectivity with the parietal cortex provides DLPFC with information on the salience of stimuli, and stimulus-response pairings, while DLPFC supports parietal cortex in guiding shifts attentional focus depending on task demands. Our objective is to enhance the activity of this crucial network during cognitive task performance using tES, with the aim of enhancing skill-acquisition. As the model in Figure 1 illustrates, we expect tES to act synergistically with cognitive training (Krause & Kadosh, in press) to boost the efficacy of the core processes associated with ARP.
• #26: Given the challenges we faced and the success we found, we thought it would be worth sharing some reflections on what worked and what didn’t.
• #27: There were three key factors in our success.
• #28: Iteration was huge. We iterated everything, including the design. These were three ideas that were culled from a much larger brainstorm process. These were presented to the team in very rough form. Robot Factory was the consensus – both for it’s flexibility and style.
• #29: We spend about 3 weeks in design and then had a playable proof of concept 2 weeks later.
• #30: Here’s the POC with rudimentary configuration and two tasks.
• #31: From there we released the game weekly (for the most part) for the larger team to review and critique. In just over a year of development, we had 34 builds (there were breaks in development during the experiments and holidays). This rapid pace of development led to quicker turnaround on feedback and better course correction.
• #32: Here’s a shot from the more mature game.
• #33: In order to maintain this pace, we had to have stellar communication between the many entities that needed to have meaningful input on the game. Not only did the game have to have scientific validity within each task, but the dynamic progression of difficulty had to conform to the best understanding of current science, too. Additionally, the game had to be instrumented to collect a lot of very specific data. And we had our sponsors at IARPA to keep in the loop as well.
• #34: For much of the project, our game dev team participated in 2 weekly meetings: one specifically for the game and one centered around experimental design. This allowed us frequent touch points to continuously clarify understanding, but it also gave the dev team insight into the greater scientific process to which the game was central.
• #35: One key tool was a spreadsheet mapping the space of potential game types. This was generated by Honeywell staff with input from all partners. If gave Simcoach a framework from which to build automata representing generic games. These were, in turn, used to power specific tasks.
• #36: Image: More complicated page of the spreadsheet. Note that the spreadsheet evolved over time (iterated along with game)
• #37: Simple example of generic task description
• #38: Example of specific spatial stimulus
• #39: Example of specific verbal stimulus
• #40: Another useful tool/process was GitHub, which we used to release each build to the larger team. We then used the integrated issue tracking feature to maintain conversations about each issue.
• #41: Probably the most important driver of success in this fast-paced project was the attitude of all collaborators involved. We all started from a place of mutual respect (which isn’t necessarily a given in all projects) and proceeded in a fully collaborative manner as described previously. Perhaps most importantly, we were honest and critical (which was enabled by our foundation of respect), constantly striving to make the best possible product.
• #42: Adding Mike Wagner to our team for this project was critical in terms of delivering the level of data collection, accuracy, software engineering, etc.
• #43: Even with all the things that went right, there were challenges and things that could have been better. Most of our issues stemmed from attempting to “build the airplane while flying”…
• #44: …or maybe it was more like this. We had a year to not only build the game but also collect enough data to show that it worked (or at least had high potential). This compressed our schedule and put enormous pressure on the development team to deliver something that was not only fun and functional but scientifically valid too.
• #45: As always with a tight timeline, you have to make sacrifices. We ended up not testing our product as rigorously as it probably needed to be. This meant that each experiment served as an additional round of QA, with bugs and feature requests cycling back into development for the next round. Clearly, that’s not ideal. Allotting more time for testing would have meant less time to develop features (given the fixed timeline), but at least the decision could have been made with more intention.
• #46: With each iteration of the product and the attendant feedback from the larger team, the dev team learned more and more about the scientific needs of the project. Our understanding increased at an incredible speed, but still, there were times where key features weren’t fully understood until the last minute (or at worst, when it was too late). This led to a lot of scope creep and last minute heroics (or less-than-heroics). This, again, is a situation where some additional time to do some intentional planning may have been beneficial to setting scope/goals earlier.