AugCog Overview
- 1. AugCog System Architecture Semi-Processed Sensor Data Gauge Data Tasks Cognitive State Assessment Task SystemSensor Data Sensor Data Processing
- 2. Polly Tremoulet, PhD. Lockheed Martin Advanced Technology Laboratories Performance Augmentation through Cognitive Enhancement (PACE) HCI International / AugCog International July 25, 2005
- 3. Overview • Background – Augmented Cognition – Sensors and Cognitive States – Mitigation Strategies • System – Design Goals – Component descriptions – Task selection – Modality selection • Ongoing and future work
- 4. Sensors Background: Augmented Cognition • Goal: Maximize operator cognitive performance in dynamic, complex operational environments • Approach: Biophysical sensor technology assesses operator cognitive state – Detects, predicts, avoids overload to reduce operator error and maximize effectiveness • Benefit: Improve operator performance – Increase situation understanding – Reduce errors – Improve accuracy Domain Simulation Cognitive State Assessor User
- 5. Current Suite of Sensors • EEG – Placement: • Monopolar placement of sensors along middle of head • Bipolar placement of sensors on both sides of the head – Sensors: Electrodes – Preprocessing: None • EKG – Placement: Traditional placement on left and right shoulders and abdomen – Sensors: Electrodes – Preprocessing: None • GSR – Placement: Non-traditional placement on toes rather than fingers – Sensors: Electrodes – Preprocessing: None • Down-selection criteria: – Correlation with performance – Interoperability with other sensors – Physical discomfort for users/subjects – Portability and robustness in operational environments
- 6. Sensor Data Processing Path Gauges MitigationsCSASensors Pre- Processing EEG fNIR GSR EKG Pupil Sequencing Pacing Reinforce- ment Modality Switch Spatial WM Verbal WM Workload Arousal Neural Network
- 7. Mitigation Strategies • Pacing – Delegation – Defer – Decomposition • Intelligent Sequencing – Ordering based on modality and priority • Modality switching – Changing presentation modality based upon capacity • Multi-modality reinforcement
- 8. Initial Gauge and Mitigation Options Gauge Trigger Mitigation logic Workload Above threshold Pacing = change timing of Secondary tasks Decomposition = break down Primary and/or Secondary tasks Arousal Below Range Above threshold Request attention / alert Offload/delegate work Offload/delegate work Decomposition Spatial WM High compared to Verbal WM Sequencing Verbal Modality Shift Chunking Verbal WM High compared to Spatial WM Sequencing Spatial Modality Shift Chunking
- 9. PACE High-Level Software Architecture External Application User Environment Director (ED) Active Task Manager (ATM) Task Information Manager (TIM) Cognitive State Assessor (CSA) System Interface Director (SID) Adaptive Workload Director (AWD) Delegation Manager (DM) Delegated Tasks Proposed Tasks Cognitive State Sensor Data Task Interactions Presented Tasks Proposed Tasks New Tasks User Actions User Performance Configuration Files Task Attributes
- 10. Overarching Architectural Concepts• Domain Neutrality – In order to provide the most generally useful and reusable system, as many components as possible are written without reference to domain. – Configuration files allow tasks, priorities, and application information to be specified per-domain – Certain components include domain-specific extensions to manage domain-specific logic • Component Separability – CommsProvider interface allows easy exchange of underlying communications layer – All components operate independently, subscribing for and publishing particular types of messages through CommsProvider – Allows reconfiguration of system to separate machines and eases integration with other applications
- 11. Configuration Files • Purpose: Allow per-domain and run-time configuration of tasks • Used primarily by TIM but also used by Environment Director and potentially others • XML-based formats for each configuration file: – Augmentation – configuration and selection of mitigation strategies – Modalities – specification of modalities in which tasks may be presented – Presentation – specification of modalities supported by external applications – Priorities – assignment of priorities and urgency of different types of tasks App User ED ATM TIM CSA SID AWD DM Conf
- 12. Task Information Manager • Purpose: Manage the creation, evaluation, and decomposition of individual tasks • Creates new tasks in response to external stimulus • Implementation for TTWCS experiments creates tasks based on a scenario script • Monitor performance of the user to provide feedback and potentially influence mitigations • Perform task decomposition and combination (not currently being used as a mitigation) App User ED ATM TIM CSA SID AWD DM Conf
- 13. Adaptive Workload Director • Purpose: Manage the set of tasks awaiting user attention • Maintains priority-based queue of pending tasks • Maintains dependency graph indicating tasks which are dependent upon the completion of other tasks before they may be presented to the user • Proposes tasks to present to the System Interface Director • Tasks are proposed upon completion of a task, rejection of a proposed task, and on a periodic update (10 sec.) • Tasks to propose are selected based on their priority and how long they’ve been waiting in the queue • Tasks which are rejected can be replaced on the queue, sent to the TIM for decomposition into smaller tasks, or sent to the Delegation Manager for the task to be handled elsewhere App User ED ATM TIM CSA SID AWD DM Conf
- 14. Delegation Manager • Purpose: Reassign tasks to a peer, either another human user or an intelligent agent • The functionality of the DM is not being used for TTWCS, as only a single operator is responsible for handling all tasks App User ED ATM TIM CSA SID AWD DM Conf
- 15. System Interface Director • Receives periodic updates of cognitive state from Cognitive State Assessor • Receives task proposals from the Adaptive Workload Director • Using cognitive state and currently active mitigation strategies decides whether to accept the proposed task or to reject the task, sending it back to the Adaptive Workload Director • Accepted tasks are passed on to the Environment Director App User ED ATM TIM CSA SID AWD DM Conf •Purpose: Perform mitigations based on the current cognitive state of the user
- 16. • Purpose: Evaluate the current cognitive state of the user • Currently implemented as Proxy to Labview implementation • Labview performs data exchange with sensor systems via established protocol and executes neural network function • Gauge values are sent out of CSA to the System Interface Director • Also includes capability to provide current performance as inputs to neural network, but this is not currently used in TTWCS domain App User ED ATM TIM CSA SID AWD DM Conf Cognitive State Assessor
- 17. Environment Director • Purpose: Manage the presentation of tasks through the external application • Monitors the modalities currently being used on all external applications by tasks which currently have user attention • Receives proposed tasks from the System Interface Director • Examines tasks and attempts to select a presentation modality based on the task’s preferred modality as well as the application’s modality capabilities • If no available modality can be found to successfully present the task, it will be rejected and sent back to the Adaptive Workload Director App User ED ATM TIM CSA SID AWD DM Conf
- 18. Active Task Manager • Purpose: Manage the progression of actions associated with individual tasks • Receives newly presented tasks and user actions associated with tasks from Environment Director • Determines the appropriate next step in the task whenever a user takes an action, sending out system actions to the Environment Director • For TTWCS, interacts with Expert Model to generate a score of the user’s performance on completed tasks App User ED ATM TIM CSA SID AWD DM Conf
- 19. jTTWCS Application • Purpose: Provide to the operator an interface through which experimental tasks may be performed • Recognize and forward user-initiated actions – Alert Responses – Retargetting Solutions – Coverage Indications • React to system-initiated actions – Begin New Scenario – Add Emergent Target – Display Alert Question • Provide Expert Model to score user responses for each type of task App User ED ATM TIM CSA SID AWD DM Conf
- 20. Launch Area Preplanned Health and Status points Primary (Default) d- Target Guidance Waypoint Loiter Pattern Alternate (Flex) f-Target Branch Point Time-critical (emergent) e- Target The Tactical Tomahawk cruise missile represents the next generation of cruise missiles with: –On-board mission planning –Inflight retargeting –Battle damage assessment This weapon will now be able to service high-priority, time-critical targets, more quickly and effectively. 3. Emergent (e-target) Missions 1. Default (d-target) Missions 2. Flex (f-target) Missions Tactical Tomahawk Application Domain
- 21. TTWCS Problem Space: increasing cognitive demands • Launch Area Coordinator (LAC) acting as strike controller will need to: – Review Exception Reports – Re-allocate missions to shooters on ships – Review Waiver Reports – De-conflict and re-allocate missiles & air tracks – Review shooter casualty reports – Re-allocate and order backup – Monitor missiles – Re-target and Re-strike • Apply learned heuristics: • Who’s in range? Who’s been on station longer? Who will be off- station earliest? What is my resource availability?
- 22. Task Selection in PACE • Tasks are inserted by application or TIM’s task generator • Tasks are decomposed into forest of subtasks, as needed – E.g. two button clicks two trees • (Sub)tasks assigned priorities and inserted into a queue – Prioritization function of insertion time, urgency, etc • Proposed tasks are examined by SID and compared to CSA’s most recent assessment of cognitive workload • Appropriate modality for next task in queue is selected App User ED TIM CSA SID AWD DM Conf ATM
- 23. Modality Selection in PACE • Each task is defined with a preferred modality – E.g. alerts prefer text-window panels, but may be delivered via speech • Application interface specifies all possible modalities for each task and quality rankings for each modality • SID examines available modalities and proposed task. – Task rejected if no slots available in any possible modality, o/w • SID accepts task and designates it for modality of greatest utility – Utility = combination of task preference and application’s modality quality and user’s cognitive capacity for task
- 24. Additional LM ATL Components Developed • Log Analyzer – Data combined from multiple ACES XML log files into one, easy to read spreadsheet – ACES logs quickly distributed to Sub-Contractors • Scenario Generator – Enables realistic, rapid creation of scenarios by all groups – Playback enables review of scenarios at different speeds • ACES (AugCog Experimental System) Launcher and Distribution Tool – Every component of the ACES system can be started up or shutdown by pressing a button – Simple install script • All required libraries are included • Runs “out of the box” with no compilation or compatibility issues
- 25. Future Directions • Mitigation Strategy research – Appropriate application of delegation – Multi-modal reinforcement strategies • Using task context to control application of mitigations • Transitioning PACE to the field: – HCI evaluation: work in progress – Training operators to use complex applications – Improving command and control operator performance in operational environments
- 27. Why this is NOT just Advanced HCI Cognitive Model Measured Verbal Task Performance is Optimal Task System inhibits Mitigation CSA Hysteresis and Smoothing Neural Network Verbal Gauge Spatial Gauge Sensors Verbal Only Task 1.Anticipates when gauge will reach threshold 2.Threshold is set to avoid becoming seriously overloaded. Task System turns on Mitigation
- 28. PACE Architecture Cognitive Workload Assessor External Actuators/ Sensors Tasks delegated to other operators or software agents New Tasks Maintains a virtual work environment that optimizes communication between operator and machine Maintains definition and state of all operator tasks both current and historic Measures the operator’s ability to handle the current and projected workload Optimizes presentation of current tasks within the operator’s virtual work environment Maintains a plan that optimizes the operator’s ability to handle the current workload Human Work Space Task Space External Task Information Manager System Interface Director Environment Director Operator Actions Domain & Application Independent Domain & Application Dependent Adaptive Workload Director • Manages Tasks, Alerts and Contexts • Monitors User Performance • Listens to Cognitive Workload Level • Directs Cognitive Augmentations – Sequencing – Pacing – Modality Shifts – Chunking – Delegation
- 29. LM ATL AugCog Environment: Augmented Cognition Experimental System – ACES • Experiment environment – Controlled – Repeatable – Scorable – Portable • Provides realistic and discrete events • Isolates memory-intensive tasks • Separable spatial and verbal activities • Modular: able to gradually increase complexity
- 30. CLIP System Configuration Semi- Processed Sensor Data CSA System Gauge Data Task System Tasks Sensor Data Sensor Data Processing System
- 31. Sensor Data Processing Systems • Sensor Data Processing Systems connect directly to a set of sensors • Minimal processing is performed on that data, producing a periodic report on all pertinent sensor values • Sensor data is passed through the network to the CSA System Sensors Sensor Data Processing System Semi- Processed Sensor Data CSA System
- 32. CSA System • The CSA System receives sensor data from the various Sensor Systems. • Using an ANN, these sensor values are processed into a set of Gauge values. • Current gauge values are periodically sent to the Task System to affect its mitigation strategy. CSA System Semi- Processed Sensor Data Task System Gauge DataSensor Data Processing System
- 33. Task System • Task System receives Gauge Data from CSA System. • Based on current Cognitive State, additional tasks are proposed to the user or rescheduled if Cognitive State indicates a potential overload • Tasks which are deferred due to Cognitive State are retained and re- proposed at a later time when the user’s state is more conducive to completing that task. Task System Tasks CSA System Gauge Data User
- 34. Neural Network Vital Statistics • Inputs: 234 excluding fNIR, 252 including fNIR – For each feed, 3 inputs: now, 0.5 sec ago, 1.0 sec ago – GSR: 1 x 3 – IBI: 1 x 3 – fNIR: 6 x 3 – EEG: 74 x 3 • Combination of direct measurements and calculated values such as vigilance – Pupillometry: 2 x 3 • Outputs: 2 – Spatial Working Memory – Verbal Working Memory • Hidden/Internal Nodes: 200 (single hidden layer) • Type: Feed-forward • Training Method: Standard Back-propagation
- 35. Building the Neural Network • Data Collection – Collected data during several scenario runs for 3 subjects – Subjects performed same types of tasks to be used during CVE • Training – Untrained network created in Stuttgart Neural Network Simulator (SNNS) – SNNS provided with data files from scenarios – 1000 training epochs executed • Standard back-propagation, no momentum factor, learning rate = 0.2 – Resulting network converted to C-function to be embedded within Labview sensor pre-processing system • Other experimentation – Other networks and training methods were attempted and this was the best combination found
- 36. Using the Neural Network • CSA System – Reads sensor values – Passes them to them to the Neural Network every 0.5 seconds • Neural Network – Processes sensor data and returns gauge value estimations • PACE System Interface Director – Examines current cognitive state – Perform hysteresis and smoothing on cognitive state values • If user has been in high verbal memory workload for at least 5 seconds, postpone low-priority verbal tasks • If user has been in high spatial memory workload for at least 5 seconds, postpone low-priority spatial tasks
- 37. Task Description and Stimuli Retarget task – Reassign missiles to service higher priority emergent targets instead of their default target destinations. • Goal is to service as many emergent targets as possible, while maintaining coverage on as many high and medium default targets as possible. – Tactical Targeting Alert task – Respond to questions from a commanding officer about an ongoing strike – Commander and Team Online Interruptions Location task – Upon Inquiry, determine what targets can/cannot be covered based on missile coverage zones. – Situation Awareness
- 38. Benefits of AugCog in TTWCS domain • Augmented Cognition system in TTWCS environment will increase operator performance – Number of missiles simultaneously monitored – Number of alerts handled successfully – Overall number of emergent targets handled correctly – Enable operators to employ new capabilities effectively: • Redirection and flex missions • Multiple engagements • Overlapping strike packages • Augmented Cognition system in TTWCS environment will reduce manning
Editor's Notes
- #22: The simultaneous occurrence of tasks between these major groups of tasks will really produce cognitive bottlenecks. The TSC must have the “battlegroup” and joint force persepectives.