MODELS 2022 Journal-First presentation: ETeMoX - explaining reinforcement learning
0 likes28 views
The document discusses the necessity for explainability in reinforcement learning systems as they become more complex and potentially opaque 'black boxes.' It outlines various types of explanations, their applications, and how model-driven engineering can assist in creating these explanations through a reusable trace metamodel. Additionally, it presents case studies and experiments focusing on optimizing storage and queries for better understanding user interactions and system behaviors.
MODELS 2022 Journal-First presentation: ETeMoX - explaining reinforcement learning
- 1. ETeMoX: explaining reinforcement learning J. M. Parra-Ullauri1 , A. García-Domínguez2 , N. Bencomo3 , C. Zheng4 , C. Zhen5 , J. Boubeta-Puig6 , G. Ortiz6 , S. Yang7 1: Aston University, 2: University of York, 3: Durham University 4: University of Oxford, 5: University of Science and Technology of China 6: University of Cadiz, 7: Edinburgh Napier University MODELS 2022 - Thursday October 27th , 2022
- 2. October 27th , 2022 Event-driven temporal models for explanations - ETeMoX: explaining reinforcement learning Rising need for explanations in self-adaptation / AI ● Software is being written to deal with more and more complex environments, where they need to reconfigure themselves and learn from experience ● If not careful, these systems can be “black boxes” where we can only take their decisions at face value - it will be hard to calibrate our trust on them ● We want to be able to ask things like: ○ Why did they take that action? ○ Why did they not take that *other* action? ○ How do you (roughly) work? ● The “right to explanation” is being enshrined in the GDPR, or the IEEE P7001 standard for transparency of autonomous systems ● There is an entire field on eXplainable AI (XAI) 2
- 3. October 27th , 2022 Event-driven temporal models for explanations - ETeMoX: explaining reinforcement learning Types of explanations and stages involved ● Explanations can be broadly classified by scope into: ○ Local - for a specific decision ○ Global - for the overall behaviour of the system (usually, a simplified behavioural model) ● Adadi et al. identified four uses for explanations: ○ To justify decisions impacting people ○ To control systems into an envelope of good behaviour ⬅ ○ To discover knowledge from the system behaviour ○ To improve the system by highlighting flaws ⬅ ● Neerincx considered three stages for producing these explanations: ○ Generation - obtain necessary data and reason about it ⬅ ○ Communication - show it to consumer (human / system) ⬅ ○ Reception - was it effective and efficient? 3
- 4. October 27th , 2022 Event-driven temporal models for explanations - ETeMoX: explaining reinforcement learning How can MDE help XAI? ● In Model-Driven Engineering (MDE), we already have significant experience abstracting away unnecessary complexity ● At design time, we raise the level of abstraction so developers of a system can think in terms of their domain concepts ● We can also do this while the system is running - we can build a model of what the system is perceiving, thinking, and doing (a runtime model) ● If we decide on a common trace metamodel for this, we can reuse efforts on introducing explainability across systems 4
- 5. October 27th , 2022 Event-driven temporal models for explanations - ETeMoX: explaining reinforcement learning MDE: Reusable Trace Metamodel - common half 5
- 6. October 27th , 2022 Event-driven temporal models for explanations - ETeMoX: explaining reinforcement learning MDE: Reusable Trace Metamodel - specific half ● First half of the metamodel is reusable across systems making their own decisions ● Second half of the metamodel is specific - this one is for systems using Q-Learning (a type of Reinf. Learning) ● A decision takes into account the Q-values of each Action ● Observations have rewards associated to them, and map to a state in the Q-table 6
- 7. October 27th , 2022 Event-driven temporal models for explanations - ETeMoX: explaining reinforcement learning MDE: Indexing Models into Temporal Graph DBs ● At each system timeslice, the runtime model is indexed into a temporal graph ● Efficient representation of a graph’s full history, using copy-on-write state chunks ● Implemented by Greycat (from Hartmann et al.), and used by Eclipse Hawk for automated model indexing ● More details here: https://www.eclipse.org/hawk/ad vanced-use/temporal-queries/ 7
- 8. October 27th , 2022 Event-driven temporal models for explanations - ETeMoX: explaining reinforcement learning MDE: History-Aware Model Querying ● For explanation generation, we can query the temporal graph ● We created a Hawk-specific dialect of EOL with time-aware predicates and properties ● More details in our MODELS 2019 paper AGD, NB, JMPU and LGP, ‘Querying and annotating model histories with time-aware patterns’, http://dx.doi.org/10.1109/MODELS. 2019.000-2 8 Version traversal x.versions, x.next, x.prev, x.time, x.earliest, x.latest… Temporal assertions x.always(version | p), x.never(v | p), x.eventually(v | p)... Predicate-based scoping x.since(v | p), x.until(v | p)... Context-based scoping x.sinceThen, x.untilThen… Unscoping x.unscoped
- 9. October 27th , 2022 Event-driven temporal models for explanations - ETeMoX: explaining reinforcement learning Scaling up temporal graphs to large event volumes ● We first applied history-based expls. to Bayesian Learning-based systems ○ Partially Observable Markov Decision Processes (POMDP) ○ Had a case study on data mirroring over the network (Remote Data Mirroring) ○ Wasn’t too resource-intensive (we could simply record all versions) ● Then we tried applying it to a Reinforcement Learning system ○ Tens of training epochs, each with thousands of episodes ○ Original RL system had per-timeslice MongoDB with GBs of records to be indexed ○ RL system changed to send updates directly to Hawk - CoW reduced storage needs ○ Still a lot of history to go through - queries could take a long time! ● Do we really need all this history? ○ Answer: No. ○ How do we select the “right” moments, without imposing too much load? 9
- 10. October 27th , 2022 Event-driven temporal models for explanations - ETeMoX: explaining reinforcement learning Event-Driven Monitoring: Complex Event Processing 10
- 11. October 27th , 2022 Event-driven temporal models for explanations - ETeMoX: explaining reinforcement learning Event-driven Temporal Models for eXplanations (ETeMoX) 11
- 12. October 27th , 2022 Event-driven temporal models for explanations - ETeMoX: explaining reinforcement learning Case study: airborne base stations 12
- 13. October 27th , 2022 Event-driven temporal models for explanations - ETeMoX: explaining reinforcement learning Experiment 1: Evolution of metrics (optionally sampled) ● We evaluated the impact of sampling at different rates on the accuracy of a query providing the historic reward values during the RL training ● We set up the CEP engine with Esper EPL rules as in the top right ● We observed linear decreases in storage required depending on sampling rate ● 10% sampling is safe, more than that depended on the RL algorithm (DQN is sensitive!) 13 Q-Learning DQN
- 14. October 27th , 2022 Event-driven temporal models for explanations - ETeMoX: explaining reinforcement learning Experiment 2: Exploration vs Exploitation (1/2) 14 ● RL systems don’t always pick the best option (exploitation): they try things sometimes to learn more (exploration) ● How often does this happen? ● We compared two approaches to track this: a. CEP pattern to detect exploration/exploitation and only index episodes with exploration b. EOL query on full history, to check CEP pattern correctness ● Q-Learning explored 1.41% of the time, SARSA 7.99%, DQN 7.82%
- 15. October 27th , 2022 Event-driven temporal models for explanations - ETeMoX: explaining reinforcement learning Experiment 2: Exploration vs Exploitation (2/2) 15 ● We tried using the exploration CEP rule as a filter for metric evolution, too
- 16. October 27th , 2022 Event-driven temporal models for explanations - ETeMoX: explaining reinforcement learning Experiment 3: user handovers between stations ● To provide continuous service, a user may be handed over to another station ● We wrote an EOL query to detect handovers in the system history ○ Handover: signal-to-noise ratio changes significantly across stations between timepoints ○ Found 1784 handovers in Q-Learning, 590 in SARSA, 82176 in DQN ● These queries required many checks: ○ 10 episodes, 2000 time steps ○ 2 stations, 1050 users ○ All together: 42M combinations to check! ● Required times: ○ 917s for Q-Learning ○ 1,132s for SARSA ○ 7,914s for DQN 16
- 17. October 27th , 2022 Event-driven temporal models for explanations - ETeMoX: explaining reinforcement learning What’s next? ● Optimise queries via Hawk timeline annotations and CEP time windows ● Explanations for other uses: ○ Human-in-the-loop (SAM 2022 presentation on Monday showed early work) ○ Hyper-parameter optimisation (external system requests explanations and drives change) ○ Global explanations of the system behaviour (event graphs) ● Studying explanation reception: ○ Effectiveness of explanation formats: plots, results, generated text, diagrams… ○ Focused on system developers so far: look into less technical audiences ○ Consider existing models for evaluation ■ Technology Acceptance Model (Davis) ■ XAI metrics (Rosenfeld) 17
- 18. Thank you! Antonio García-Domínguez @antoniogado - a.garcia-dominguez@york.ac.uk Juan Marcelo Parra-Ullauri j.parra-ullauri@aston.ac.uk 18