![1. INTRODUCTION
During Carnegie Mellon University's Robotics
Institute 25th anniversary, Raj Reddy said, "the
biggest barrier is (developing) computers that learn
with experience and exhibit goal-directed
behaviour. If you can't build a system that can
learn with experience, you might as well forget
everything else". This involves developing agents
characterized by the ability to:
• learn from their own experience and the
experience of human commanders,
Figure 2. Example of a Microworld.
• accumulate learning over long time periods on
various microworld models and use what is Creating a microworld for strategic decisions is
learned to cope with new situations, primarily a cognitive engineering task. A
microworld (e.g. like a map with overlaid
• decompose problems and formulate their own information) is a form of symbolic language that
representations, recognising relevant events should represent the necessary objects and
among the huge amounts of data in their dynamics, but not fall into the trap of becoming as
"experience", complex as the environment it seeks to express
(Friman and Brehmer 1999). It should also avoid
• exhibit adaptive, goal directed behaviour and the potential pitfalls of interaction, including the
prioritise multiple, conflicting and time human desire for “more”, when more is not
varying goals, necessarily better. Omedei et al (2004) have shown
that humans under certain conditions display
• interact with humans and other agents using weakness in self-regulation of their own cognitive
language and context to decipher and respond load. Believing for example, that “more detailed
to complex actions, events and language. information is better”, individuals can become
information overloaded, fail to realise that they are
Extant agent-based modelling and simulation overloaded and mission effectiveness suffers.
environments for military appreciation appear to Similarly, a belief that “more reliable information
occupy extreme ends of the modelling spectrum. is better”, individuals can be informed or observe
At one end, Agent-Based Distillations (ABD) that some sources are only partially reliable, fail to
provide weakly expressive though highly pay sufficient attention to the reliable information
computable (many games per second execution) from those sources and mission effectiveness
models using attraction-repulsion rules to represent suffers. Cognisant of these issues, we seek a
intent in simple automata (Horne and Johnson formalism which will allow the embodiment of an
2003). At the other end of the spectrum, large scale austere, adaptable and agreed representation.
systems such as the Joint Synthetic Armed Forces Though there is a formal definition of agreement in
(JSAF) and Joint Tactical Level Simulation (JTLS) a mathematical sense based on homomorphism as
require days to months and many staff to set up for we describe below, in practice, humans are the
a single simulation run (Matthews and Davies purveyors of authority for agreed models.
2001). We propose a microworld-based
representation, which falls between these extremes 2.1. Microworld Formalism
in that it may be used to achieve statistically-
significant results yet also allows for decision We use the formalisation of multi-agent stochastic
interaction. games (Fundenberg and Tirole 1995) as our
generalised scientific model for microworlds. A
2. INTERACTIVE MICROWORLDS multi-agent stochastic game may be formalised as
a tuple N , S , A, T , R where N is the number of
Microworlds have been used throughout military
agents, S the states of the microworld,
history to represent terrain, force disposition and
movements. Typically such models were built as A = A1 × A2 × ... × AN concurrent actions, where
miniatures on the ground as depicted in figure 2 or Ai is the set of actions available to agent i , T is
on a board grid, but these days are more usually the stochastic state transition function for each
computer-based. action, T : S × A × S → [0,1] , and
2196](https://image.slidesharecdn.com/machine-learning-for-adversarial-agent-microworlds908/85/Machine-Learning-for-Adversarial-Agent-Microworlds-2-320.jpg)
Machine Learning for Adversarial Agent Microworlds
- 1. Machine Learning for Adversarial Agent Microworlds J. Scholz1, B. Hengst2, G. Calbert1, A. Antoniades2, P. Smet1, L. Marsh1, H-W. Kwok1, D. Gossink1 1 DSTO Command and Control Division, 2NICTA, E-Mail: Jason.Scholz@defence.gov.au Keywords: Wargames, Reinforcement learning, Hierarchy; Representation. EXTENDED ABSTRACT We describe the results of RL approaches to microworld applications of progressively We describe models and algorithms to support increasing complexity, as determined by the state- interactive decision making for operational action space dimension. These include discrete national and military strategic courses of action. space-time (grid-world) models of modified Chess and Checkers, and a military campaign-level game Abstract representations or ‘microworlds’ have with concurrent moves, called “TD-Island”. We been used throughout military history to aid in also detail complexity reduction through conceptualization and reasoning of terrain, force hierarchical decomposition in a continuous space disposition and movements. With the introduction model of an operational level air-combat model as of digitized systems into military headquarters the is shown in figure 1. capacity to degrade decision-making has become a concern with these representations. Maps with Effects (root) overlays are a centerpiece of most military Attacking force Defending force headquarters and may attain an authority which exceeds their competency, as humans forget their limitations both as a model of the physical Target(t) Defend(d) environment, and as a means to convey command Strike mission intent (Miller 2005). The design of microworlds Strike Diversionary Mission (t) may help to alleviate human-system integration CAP(d) Intercept(j) problems, by providing detail where and when it is Release appropriate and recognizable abstractions where it weapon CAP is not. Ultimately, Lambert and Scholz (2005 p.29) Max intercept Target dist Out intercept view “the division of labor between people and mission In Avoid mission machines should be developed to leave human threats Shoot decision making unfettered by machine Tarmac interference when it is likely to prove heroic, and Loiter Take off Straight Land wait Nav(p) enhance human decision making with automated decision aids, or possibly override it with Figure 1: Hierarchical decomposition of a automated decision making, when it is likely to be complex operational air-game. hazardous.” Cognisant of these issues, we state a formalism for microworlds initially to manage Given this decomposition, current hierarchical RL representation complexity for machine learning. approaches are not sufficient in an adversarial This formalism is based on multi-agent stochastic context as game-theoretic notions such as mixed or games, abstractions and homomorphism. randomised strategies, to keep the opponent “second-guessing” are absent. We briefly describe According to Kaelbling et al (1996), a new approach to hierarchical learning. This “Reinforcement learning (RL) is the problem faced combines both the “win-or-learn-fast” algorithm by an agent that must learn behaviour through trial which learns mixed strategies and the MAXQ and error interactions with a dynamic algorithm developed to learn through the structure environment.” As RL is frustrated by the curse of of the hierarchy. dimensionality, we have pursued methods to exploit temporal abstraction and hierarchical Finally, we compare the performance of human organisation. As noted by Barto and Mahadevan guessed parameter importance with that of 2003 (p. 23), “Although the proposed ideas for machine learned importance in a modified game of hierarchical RL appear promising, to date there has Checkers with hidden-pieces. Our results show been insufficient experience in experimentally that in new or novel domains, such as the hidden testing the effectiveness of these ideas on large Checkers game, machine-learned parameters fare applications.” better in play, as the algorithms can gain experience faster than humans. 2195
- 2. 1. INTRODUCTION During Carnegie Mellon University's Robotics Institute 25th anniversary, Raj Reddy said, "the biggest barrier is (developing) computers that learn with experience and exhibit goal-directed behaviour. If you can't build a system that can learn with experience, you might as well forget everything else". This involves developing agents characterized by the ability to: • learn from their own experience and the experience of human commanders, Figure 2. Example of a Microworld. • accumulate learning over long time periods on various microworld models and use what is Creating a microworld for strategic decisions is learned to cope with new situations, primarily a cognitive engineering task. A microworld (e.g. like a map with overlaid • decompose problems and formulate their own information) is a form of symbolic language that representations, recognising relevant events should represent the necessary objects and among the huge amounts of data in their dynamics, but not fall into the trap of becoming as "experience", complex as the environment it seeks to express (Friman and Brehmer 1999). It should also avoid • exhibit adaptive, goal directed behaviour and the potential pitfalls of interaction, including the prioritise multiple, conflicting and time human desire for “more”, when more is not varying goals, necessarily better. Omedei et al (2004) have shown that humans under certain conditions display • interact with humans and other agents using weakness in self-regulation of their own cognitive language and context to decipher and respond load. Believing for example, that “more detailed to complex actions, events and language. information is better”, individuals can become information overloaded, fail to realise that they are Extant agent-based modelling and simulation overloaded and mission effectiveness suffers. environments for military appreciation appear to Similarly, a belief that “more reliable information occupy extreme ends of the modelling spectrum. is better”, individuals can be informed or observe At one end, Agent-Based Distillations (ABD) that some sources are only partially reliable, fail to provide weakly expressive though highly pay sufficient attention to the reliable information computable (many games per second execution) from those sources and mission effectiveness models using attraction-repulsion rules to represent suffers. Cognisant of these issues, we seek a intent in simple automata (Horne and Johnson formalism which will allow the embodiment of an 2003). At the other end of the spectrum, large scale austere, adaptable and agreed representation. systems such as the Joint Synthetic Armed Forces Though there is a formal definition of agreement in (JSAF) and Joint Tactical Level Simulation (JTLS) a mathematical sense based on homomorphism as require days to months and many staff to set up for we describe below, in practice, humans are the a single simulation run (Matthews and Davies purveyors of authority for agreed models. 2001). We propose a microworld-based representation, which falls between these extremes 2.1. Microworld Formalism in that it may be used to achieve statistically- significant results yet also allows for decision We use the formalisation of multi-agent stochastic interaction. games (Fundenberg and Tirole 1995) as our generalised scientific model for microworlds. A 2. INTERACTIVE MICROWORLDS multi-agent stochastic game may be formalised as a tuple N , S , A, T , R where N is the number of Microworlds have been used throughout military agents, S the states of the microworld, history to represent terrain, force disposition and movements. Typically such models were built as A = A1 × A2 × ... × AN concurrent actions, where miniatures on the ground as depicted in figure 2 or Ai is the set of actions available to agent i , T is on a board grid, but these days are more usually the stochastic state transition function for each computer-based. action, T : S × A × S → [0,1] , and 2196
- 3. R = (R1 , R2 ,..., RN ) , where Ri is the immediate 2.2. Grid World Examples reward for agent i, Ri : S × A → ℜ .The objective Chess and Checkers Variant Microworlds is to find an action policy π i (S ) for each agent i to maximise the sum of future discounted rewards We initially focused on variants of abstract games Ri . This is effectively, a multi-agent Markov in discrete time and space such as Chess and Decision Process (MDP). Checkers. We deliberately introduced asymmetries into the games and conducted sensitivity analysis In multi-agent stochastic games each agent must through Monte-Carlo simulation at various ply choose actions in accordance with the actions of depths up to seven. The six asymmetries were other agents. This makes the environment either materiel, where one side commences with a essentially non-stationary. In attempting to apply materiel deficit, planning, with differing ply search learning methods to complex games, researchers depths, tempo, in which one side was allowed to have developed a number of algorithms that make a double move at some frequency, stochastic combine game-theoretic concepts with dynamics where pieces are taken probabilistically reinforcement learning algorithms, such as the and finally hidden pieces, where one side Nash-Q algorithm (Hu and Wellman 2003). possessed pieces that could be moved, though the opponent could only indirectly infer the piece Abstractions, Homomorphism and Models location (Smet et al 2005). To make learning feasible we need to reduce the The TDLeaf algorithm is suitable for learning complexity of problems to manageable value functions in these games (Baxter et al, 1998). proportions. A challenge is to abstract the huge TDLeaf takes the principal value of a minimax tree state-space generated by multi-agent MDPs. To do (at some depth) as the sample used to update a this, we plan to exploit any structure and parameterised value function through on-line constraints in the problem to decompose the MDP. temporal differences. Independently discovered by Beal and Smith in 1997, TDLeaf has been applied Models can be thought of as abstractions that to classical games such as Backgammon, Chess, generally reduce the complexity and scope of an Checkers and Othello with impressive results environment to allow us to focus on specific (Schaeffer, 2000). problems. A good or homomorphic model may be thought of as a many-to-one mapping that For each asymmetric variant, we learned a simple preserves operations of interest as shown in figure evaluation function, based on materiel and 3. If the environment transitions from mobility balance. The combined cycle of environmental states w1 to w2 under environmental evaluation function learning and Monte Carlo simulation allowed us to draw general conclusions dynamics, then we have an accurate model if the about the relative importance of materiel, planning, model state m1 transitions to m2 under a model tempo, stochastic dynamics and partial dynamic and states w1 map onto m1 , w2 map observability of pieces (Smet et al 2005) and will onto m2 and environmental dynamics map onto be described further in section 4. the model dynamics. Our approach to playing alternate-move partially- observable games combined some aspects of information theory and value-function based reinforcement learning. During the game, we stored not only a belief of the distribution of our opponent’s pieces, but conditioned on one of these states, a distribution of the possible positions of our pieces (a belief of our opponent’s belief). This enabled the use of ‘entropy-balance,’ the balance Figure 3: Homomorphic models between relative uncertainty of opposition states as one of the basis functions in the evaluation The purpose of a microworld is to represent an function (Calbert and Kwok 2004). Simulation agreed model of a situation. The following models results from this are described in section 4. are constructed on the basis of our observations of military planning and general strategic thought. TD-Island Microworld TD-Island stands for ‘Temporal Difference Island’, indicating the use of learning to control elements 2197
- 4. in an island-based game. Space and time are number of different basis functions, which form discrete in our model. Each of 12 discrete spatial the heart of the value function approximation. states is either of type land or sea (figure 4). 2.3. Continuous Time-Space Example Tempest Seer This is a microworld for evaluating strategic decisions for air operations. Capabilities modelled include various fighters, air to air refuelling, surface to air missiles, ground based radars, as well as logistical aspects. Airports and storage depots are also included. The model looks at a maximum 1000 km square of continuous space, with time Figure 4: TD-Island microworld. advancing in discrete units. It is illustrated in figure 5. The Tempest Seer microworld will enable Elements in the game include jetfighters, army users to look at the potential outcomes of strategic brigades and logistics supplies totaling twenty per decisions. Such decisions involve the assignment side. Unlike Chess, two or more pieces may of aircraft and weapons to missions, selection of occupy the same spatial state. Further from Chess, targets, defence of assets and patrolling of the air in TD-Island, there is concurrency of action in that space. While these are high-level decisions, all elements may move at each time step. This modelling the repercussions of such decisions induces an action-space explosion with around one accurately requires attention to detail. This million move options per turn (compared with microworld is therefore at a much lower level of Chess with thirty six moves per turn). This makes abstraction than TD-Island. centralized control difficult, particularly as the number of elements may be large. The TD-Island Radar agent uses domain symmetries and heuristics to prune the action-space, inducing the homomorphic Fuel Storage model. Some symmetries in the TD-Island game are Fuel Transporter simple to recognise. For example, if considering the set of all possible concurrent actions of Roads jetfighters, striking targets is the set of all possible permutations of jetfighters assigned to these Jetfighter targets. One can eliminate this complexity by considering only a single permutation. By Airport considering this single permutation, one “breaks” the game symmetry, reducing the overall Figure 5: Tempest Seer microworld. concurrent action space dimension which induces a homomorphic model (Calbert 2004). Given the complexity of this microworld, it is natural to attempt to systematically decompose its The approach to state-space reduction in the TD- structure into a hierarchically organized set of sub- Island game is similar to that found in Chess based tasks. The hierarchy would be decomposed into reinforcement learning studies, such as the higher-level strategic tasks such as attack and KnightCap game (Baxter et al 1998). A number of defend at the top, with lower level tactical level features or basis functions are chosen and the tasks near the bottom of the hierarchy, with importance of these basis functions forms the set primitive or atomic moves at the base. Such a of parameters to be learned. The history of Chess hierarchy has been composed for the Tempest Seer has developed insights into the appropriate choice game as shown in figure 1. of basis functions (Furnkranz and Kubat 2001). Given this decomposition, one can apply the Fortunately there are considerable insights into MAXQ-Q algorithm to learn game strategies abstractions important in warfare, generating what (Dieterrich, 2002). One can think of the game is termed ‘operational art’. Such abstractions may being played through the execution of differing include the level of simultaneity, protection of subroutines, these being nodes of the hierarchy as forces, force balance and maintenance of supply differing states are encountered. Each node can be lines amongst others (US Joint Chiefs of Staff considered as a stochastic game, focused on a 2001). Each of these factors may be encoded into a 2198
- 5. particular aspect of play such as targeting, or at an 3. SUMMARY ISSUES elemental level taking-off or landing of aircraft. We have found the use of action and state Though the idea of learning with hierarchical symmetry, extended actions, time-sharing, deictic decomposition is appealing, in order to make the representations, and hierarchical task application practical, several conditions must decompositions useful for improving the efficiency apply. First, our human-designed hierarchy must of learning in our agents. Action symmetries arise, be valid, in the sense that at the nodes of the for example, when actions have identical effects in hierarchy capture the richness of strategic the value or Q-function (Ravindran 2004, Calbert development used by human players. Second, 2004). Another form of action abstraction is to crucially, one must be able to abstract out define actions extended in time. They have been irrelevant parts of the state representation, at variously referred to as options (Sutton et al 1999), differing parts of the hierarchy, in a process called sub-tasks, macro-operators, or extended actions. In structural abstraction. As an example, consider the multi-agent games the action space “take-off” node. Provided all aircraft, including the enemies, are sufficiently far from the departing A = A1 × A2 × ... × AN describes concurrent actions, aircraft, we can ignore their states. The states of where Ai is the set of actions available to agent i. other aircraft can effectively be safely abstracted out, without altering the quality of strategic The number of concurrent actions is A = ∏Ai i . learning. Without such abstraction, hierarchical We can reduce the problem by constraining the learning is in fact far more expensive than general agents so that only one agent can act at each time- “flat” learning (Dietterich, 2002). Finally, the step. State symmetry arises when behaviour over current theory of hierarchical learning must be state sub-spaces is repetitive throughout the state modified to include explicit adversarial reasoning, space (Hengst 2002), or when there is some for example the randomisation of strategies to keep geometric symmetry that solves the problem in one the opponent “second-guessing” so to speak. domain and translates to others. Deictic We have completed research into incorporating representations only consider state regions adversarial reasoning into hierarchical learning important to the agent, for example, the agent’s through a vignette in which taxi’s compete for direct field of view. Multi-level task passengers, and are able to “steal” passengers. The decompositions use a task hierarchy to decompose hierarchical decomposition for this game is shown the problem. Whether the task hierarchy is user in figure 6. specified as in MAXQ (Dietterich 2000) or machine discovered as in HEXQ (Hengst 2002), Root they have the potential to simultaneously abstract both actions and states. Get Deliver Part of our approach to address achieving human agreement and trust in the representation has been a move from simple variants of Chess and PickUp Steal Navigate (t) Evade PutDown Checkers to more general microworlds like TD Island and Tempest Seer. We advocate an open- interface component approach to software development, allowing the representations to be North South East West Wait dynamically adaptable as conditions and hence models need to change. 4. RESULTS OF A MICROWORLD GAME Figure 6: Hierarchical decomposition for the competitive taxi problem. We illustrate learning performance for a Checkers- variant microworld. In this variant, each side is In an approach to combining hierarchy with game- given one hidden piece. If this piece reaches the theory, one of the authors has combined MAXQ opposition baseline, it becomes a hidden king. learning with the “win-or-learn fast” or WOLF After some experience in playing this game, we algorithm (Bowling and Veloso, 2003) which estimated the relative importance of standard adjusts the probabilities of executing a particular pieces, hidden unkinged and hidden kinged pieces. strategy according to whether you are doing well We then machine learned our parameter values in the strategic play or losing. using the belief based TDLeaf algorithm described earlier. 2199
- 6. Research Council. We wish to acknowledge the inspiration of Dr Jan Kuylenstierna and his team from the Swedish National Defence College. 7. REFERENCES Alberts, D., Gartska, J., Hayes, R., (2001) Understanding information age warfare, CCRP Publ., Washington, DC. Barto, A.G., Mahadevan, S., (2003) Recent advances in hierarchical reinforcement learning. Discrete-Event Systems, 13:41-77. Figure 7. Results for Checkers with hidden pieces. Special Issue on Reinforcement Learning. Figure 7 details the probability of wins, losses and Baxter, J.; Tridgell, A.; and Weaver, L, (1998) draws for three competitions each of 5000 games. TDLeaf (λ ) : combining temporal In the first competition, an agent with machine- difference learning with game tree search. learned evaluation function weights was played Proceedings of the Ninth Australian against an agent with an evaluation function where Conference on Neural Networks: 168-172. all weights were set as uniformly important. In the second competition, an agent with machine- Beal, D.F. and Smith, M.C., (1997) Learning piece learned evaluation function weights was played values using temporal differences, ICCA against an agent with a human-estimated Journal 20(3): 147-151. evaluation function. Finally, an agent using human estimated weights played against an agent using Bowling, M and Veloso, M., (2002) Multiagent uniform values. The agent with machine-estimated learning using a variable learning rate. evaluation function weights wins most games. This Artificial Intelligence, Vol. 136, pp. 215- result points towards our aim to construct adequate 250. machine learned representations for new unique strategic situations. Calbert, G., (2004), Exploiting action-space symmetry for reinforcement learning in a 5. FUTURE WORK AND CONCLUSIONS concurrent dynamic game, Proceedings of the International Conference on We will continue to explore the theme of Optimisation Techniques and Applications. microworld and agent design and fuse our experience in games, partially-observable planning Calbert, G and Kwok, H-W, (2004), Combining and extended actions. Though we have identified entropy based heuristics, minimax search good models as homomorphic, there is little work and temporal differences to play hidden describing realistic approaches to constructing state games. Proceedings of the Florida these homomorphisms for microworlds or Artificial Intelligence Symposium, AAAI adversarial domains. Press. We are also interested in capturing human intent Dietterich, T.G, (2000), Hierarchical reinforcement and rewards as perceived by military players in learning with the MAXQ function microworld games. This may be viewed as an decomposition, Journal of Artificial inverse MDP, where one determines rewards and Intelligence Research, Vol. 13, pp. 227- value function given a policy, as opposed to 303. determining a policy through predetermined rewards (Ng and Russell 2000). We believe that Friman, H. and Brehmer, B., (1999), Using augmentation of human cognitive ability will microworlds to study intuitive battle provide an improvement in the ability for military dynamics: a concept for the future, commanders to achieve their intent. ICCRTS, Rhode Island, June 29 - July 1. 6. ACKNOWLEDGEMENTS Fundenberg, D. and Tirole, J., (1995), Game Theory. MIT Press. National ICT Australia is funded through the Australian Government's Backing Australia's Ability initiative, in part through the Australian 2200
- 7. Furnkranz, J., Kubat, M. (eds), (2001), Machines More is better? Problems of self regulation that learning to play games. Advances in in naturalistic decision making settings, in Computation Theory and Practice, Vol.8. Montgomery, H., Lipshitz, R., Brehmer, B. Nova Science Publishers, Inc. (Eds.), Proceedings of the 5th Naturalistic Decision Making Conference. Hengst, B., (2002), Discovering hierarchy in reinforcement learning with HEXQ, Ravindran, B., (2004), An algebraic approach to Proceedings of the Nineteenth International abstraction in reinforcement learning". Conference on Machine Learning, pp. 243- Doctoral Dissertation, Department of 250, Morgan-Kaufman Publ. Computer Science, University of Massachusetts, Amherst MA. Horne, G., Johnson, S., Eds., (2003), Maneuver Warfare Science, USMC Project Albert Schaeffer, J., (2000), The games computers (and Publ. people) play. Advances in Computers 50 , Marvin Zelkowitz editor, Academic Press, Hu, J., Wellman, M.P., (2003), Nash-Q learning pp. 189-266. for general sum stochastic games, Journal of Machine Learning Research, Vol. 4, pp. Smet, P., Calbert, G., Kwok, H-W, Scholz, J., 1039-1069. Gossink, D., (2005), The effects of materiel, tempo and search depth in win Kaelbling L.P., Littman M.L., Moore A.W., loss ratios in chess, Proceedings of (1996), Reinforcement learning: a survey, Australian Artificial Intelligence Journal of Artificial Intelligence Research, Conference. Vol. 4, pp. 237-285. Sutton, R., Precup, D. and Singh, S., (1999), Kuylenstierna, J., Rydmark, J. and Fahrus, T. Between MDPs and semi-MDPs: A (2000), The value of information in war: framework for temporal abstraction in some experimental findings”, Proc. of the reinforcement learning. Artificial 5th International Command and Control Intelligence, Vol. 112. pp. 181-211. Research and Technology Symposium (ICCRTS), Canberra, Australia. US Joint Chiefs of Staff, (2001), Joint Doctrine Keystone and Capstone Primer, Appendix Lambert, D., Scholz, J.B., (2005), A dialectic for A, 10 Sept. network centric warfare, Proceedings of the http://www.dtic.mil/doctrine/jel/new_pubs/ 10th International Command and Control primer.pdf Research and Technology Symposium (ICCRTS), MacLean, VA, June 13–16. http://www.dodccrp.org/events/2005/10th/ CD/papers/016.pdf Matthews, K, Davies, M., (2001), Simulations for joint military operations planning, Proceedings of the SIMTEC Modelling and Simulation Conference. Miller, C.S., (2005), A new perspective for the military: looking at maps within centralized command and control systems, Air and Space Power Chronicles, 30 March. http://www.airpower.maxwell.af.mil/airchr onicles/cc/miller.html Ng, A.Y. and Russell, S., (2000), Algorithms for inverse reinforcement learning, Proceedings of the Seventeenth International Conference on Machine Learning. Omodei, M. M., Wearing, A. J., McLennan, J., Elliott, G. C., & Clancy, J. M., (2004), 2201