BayesiaLab Knowledge Elicitation Environment
- 1. Plan Modeling by Brainstorming BayesiaLab’s Knowledge Elicitation BAYESIALAB 5.0 Knowledge Environment Elicitation Environment An innovative Brainstorming Tool Dr. Lionel JOUFFE May 2010 ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 1
- 2. Plan Modeling by Brainstorming MODELING BY BRAINSTORMING BAYESIALAB 5.0 Knowledge MODELING BY BRAINSTORMING Elicitation Environment All models are wrong; the practical question is how wrong do they have to be to not be useful (Box&Draper 87) ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 2
- 3. Designing a Model for Decision Support Every Company is faced to complex decisions that need to be rationally supported Plan Sometime, there are too few data available, or no data at all, to allow using data mining and data analysis technics to automatically build a Decision Support System Modeling by Brainstorming Experts have gathered invaluable Tacit Knowledge through BAYESIALAB 5.0 their experience Knowledge Elicitation We need to Convert this Tacit Knowledge into Explicit Environment Knowledge and use it to build a model We want actionable models to allow What-if scenarios (simulation and/or diagnosis), drivers analysis, ... Bayesian Belief Networks (BBNs) are ideal models for such problematics: their graphical representation allows a manual design by using expert knowledge, and their probabilistic engines offer powerful simulation capabilities ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 3
- 4. BBNs are made of Two Distinct Parts Structure Directed Acyclic Graph (DAG), i.e. no directed loop Plan Nodes represent the variables Each node has a set of exclusive states (e.g.: Young, Adult, Aged) Modeling by Brainstorming Arcs represent the direct probabilistic influences between the BAYESIALAB 5.0 variables (possibly causal) Knowledge Elicitation Environment Parameters Probability distributions are associated to each node, usually by using tables CONDITIONAL PROBABILITY DISTRIBUTION MARGINAL A smoker has a 60% of risk of suffering PROBABILITY DISTRIBUTION from a Bronchitis, whereas the risk of We consider a population made a non smoker is 30% only of 40% of Adults ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 4
- 5. BBNs are Powerful Inference Engines We get some evidence on the states of a subset of variables: Hard positive and negative evidence, Likelihoods, Probability distributions, Mean values Plan We take these findings into account in a rigorous way to update our belief on the states of all the other variables Modeling by Brainstorming Probability distributions on their values BAYESIALAB 5.0 Knowledge Multi-Directional Inference (Simulation and/or Diagnosis) Elicitation Environment Prior Distribution Posterior Distribution The evidence on Smoker (a new probability distribution) allows to update the probability distribution of Age (Diagnosis) and Bronchitis (Simulation) ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 5
- 6. BBN Modeling by Brainstorming Plan Clear definition of the BBN’s objective(s) (e.g.: Improvement of the Product/Service Quality, improvement of the Purchase Intent, improvement of the Company’s performance, ...) Modeling by Brainstorming Identification of the conceptual dimensions that are linked to those objectives (e.g.: Human resources, Management, BAYESIALAB 5.0 Production, Marketing, ...) Knowledge Elicitation Environment Definition of the group of experts that will fully cover all the dimensions (and the different geographical zones), with a small redundancy to allow fruitful debates Brain Storming Sessions with this group of Experts to manually build the BBN ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 6
- 7. The Structure The Directed Acyclic Graph The structure elicitation is probably the simplest task of the Brainstorming session For each identified conceptual dimension Plan Definition of the main variables Modeling by Definition of the exclusive states of those variables Brainstorming Creation of one node per identified variable BAYESIALAB 5.0 Knowledge Elicitation Brainstorming to define the direct relationships between the Environment variables, and addition of the corresponding arcs between those dependent variables ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 7
- 8. The Parameters Probability Distributions Probabilities do not have to be exact to be useful For each root node, i.e. without incoming arc, definition of the marginal probability distribution over the defined states Plan For each node with incoming arc(s), definition of the conditional probability distribution over the defined states, for each combination of the states of Modeling by its connected nodes Brainstorming Each expert gives his/her belief on the distributions BAYESIALAB 5.0 Knowledge Elicitation Environment There are various kinds of biases to be aware of Cognitive (Plausibility, Control, Availability, Anchoring) Emotional (Mood, Motivation) Group (Anchoring, Herding) Facilitator (can be biased toward charismatic experts or toward the last expressed opinion) ☛ Use the new BayesiaLab’s Knowledge Elicitation environment ©2010 BAYESIA SAS to reduce these biases, to improve traceability, to gather all the All rights reserved. Forbidden reproduction in whole or part useful knowledge, .... without the Bayesia’s express written permission 8
- 9. Plan Modeling by Brainstorming BAYESIALAB 5.0 BAYESIALAB 5.0 Knowledge Elicitation Environment Knowledge Elicitation Environment ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 9
- 10. The Experts Definition of the group of Experts Plan Modeling by Brainstorming BAYESIALAB 5.0 Knowledge Elicitation Environment ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 10
- 11. The Experts This Expert Editor allows defining: The Expert’s name, its Credibility (that will be use globally during the consensus computation), her/his Picture, a Comment to describe her/his area of expertise. The last field contains the number of assessments realized by the expert on the Plan current network Modeling by Brainstorming BAYESIALAB 5.0 Knowledge Elicitation Environment - Group of experts can be Imported/Exported - The Open Session button allows opening an Online Brainstorming Session* - The Generate Tables button allows generating a Bayesian network by using the assessments of the selected experts only ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 11 * Available on subscription only
- 12. The Experts’ Assessments Plan Modeling by Brainstorming BAYESIALAB 5.0 Knowledge Elicitation Environment Selecting a cell in the probability table activates the Assessment button for assessing the question corresponding to the selected line, i.e. what is the marginal probability distribution of Age over the 3 defined states? ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 12
- 13. The Experts’ Assessments Pressing the Assessment button opens the Assessment Editor that allows the Facilitator to manually add, delete and modify Experts’ Assessments. Plan The Post Assessment button can be used by the Facilitator to Post the question to the BayesiaLab’s secured website for an online assessment Modeling by Brainstorming BAYESIALAB 5.0 Knowledge Elicitation Environment ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 13
- 14. The Expert Online Assessment Tool The secured website Plan Modeling by The Expert’s name, The session name Brainstorming case sensitive! BAYESIALAB 5.0 Knowledge Elicitation Environment Once logged in, the Expert is waiting for a question ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 14
- 15. The Expert Online Assessment Tool: Example with the 3 states variable Age Once the Facilitator has posted the question with the Assessment Editor, the question is displayed on the Expert’s webpage. Plan The question is relative to the node “Age”, that has 3 states: Young, Adult and Aged. There are then 3 sliders for the probability distribution assessment, and another one Modeling by for the confidence Brainstorming There is no context (root node). This is then a marginal probability BAYESIALAB 5.0 Knowledge Elicitation Check Environment box for fixing the probability of the state Pie Chart representing the probability distribution specified with the sliders The label corresponds to the The Confidence level the expert comment field can be has specified with the used for explaining the ©2010 BAYESIA SAS Confidence Slider (ranging assessment All rights reserved. Forbidden reproduction in whole or part from “I Do not Know” to “I without the Bayesia’s express written permission am Certain”) 15
- 16. The Expert Online Assessment Tool: Example with the binary variable Cancer The context variables in the BBN Plan Hovering over the context variables returns the comment Modeling by associated to the corresponding node, if any Brainstorming BAYESIALAB 5.0 Knowledge Elicitation Environment This question is relative to node Cancer, and the specific Context is “Age = Adult” and “Smoker = Yes” ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 16
- 17. The Facilitator’s tool This listener allows following the status of the Experts’ assessments Plan Modeling by Brainstorming BAYESIALAB 5.0 Knowledge Elicitation Once the Expert validates her/his assessment, this assessment is sent to the BayesiaLab’s server and the Facilitator’s listener is automatically updated Environment Clicking on OK makes BayesiaLab harvesting the assessments. Closing the window cleared the question from the webpage of the Experts that do not have ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 17
- 18. The Facilitator’s tool Plan This gray part corresponds to the Experts’ probability distribution assessments Modeling by Brainstorming BAYESIALAB 5.0 Knowledge Elicitation Environment This second part contains the Expert’s name, the Assessment’s Confidence, the associated Comment and the Time (in second) for validating the assessment ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 18
- 19. The Facilitator’s tool The content of this editor is sortable by each column just by clicking on the corresponding header Plan It is sorted here in the ascending order on the probabilities assessed for the state Young Modeling by Brainstorming BAYESIALAB 5.0 Knowledge Elicitation Selecting the line allows Environment displaying the Expert’s picture Sorting the assessments by state probabilities can be used for: - detecting Experts’ misunderstanding - Knowledge sharing, especially by making the 2 “extremes” Experts debate If some useful knowledge comes out from the debate, the Facilitator can post again the question for a new Expert Assessment. Each Expert will then be allowed to update her/his assessment online (each Experts’ webpage is initialized ©2010 BAYESIA SAS All rights reserved. Forbidden with the information she/he set in the previous round) reproduction in whole or part without the Bayesia’s express written permission 19
- 20. The Consensus Once the assessments validated, a Mathematical consensus is computed by using the Experts’ credibility and their assessment’s confidence. This automatic consensus can be manually modified by the Facilitator to set a Behavioral consensus, i.e. one issued after a fruitful debate Plan Modeling by Brainstorming Hovering over this icon A small icon is added at the left of returns the minimum and the each probability to graphically BAYESIALAB 5.0 maximum assessments, and the represent the consensus degree: Knowledge number of assessments from a full transparency when there Elicitation all the Experts agree on the Environment probability, to no transparency when the range of the assessments is 1 ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 20
- 21. The Consensus An icon is added to the nodes for indicating the nodes that have Experts assessments. The darker the icon is, the lower the global consensus is Plan The probability distributions that have a set of assessments are framed with a green line Modeling by Brainstorming BAYESIALAB 5.0 Knowledge Elicitation Environment Here is the list of assessments corresponding to the second line of the table ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 21
- 22. The Consensus Pressing the “i” key while hovering over the expert icon allows displaying the information panel below Plan Modeling by Brainstorming BAYESIALAB 5.0 Knowledge iiiiiii Elicitation Environment This information panel contains: - the number of rows ((Conditional) probability distributions) that comes with Experts assessments - the total number of assessments that have been set in the probability table - the number of Experts that have assessed at least one probability distribution in the table - a measure of the global disagreement that takes into account the deviations from the mathematical consensus - the maximum disagreement corresponding to the greatest difference between two ©2010 BAYESIA SAS assessments in the probability table All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 22
- 23. The Assessment Report Right clicking on the Expert Icon in the lower left corner of the Graph window allows generating the following HTML report. This report first gives information on the Experts, Plan then returns a sorted list of the nodes wrt the global disagreements, and another one wrt the maximal disagreements. Finally, for each node, a summary contains all the Modeling by global information on the assessments of the Brainstorming (Conditional) Probability Table BAYESIALAB 5.0 Knowledge Elicitation Environment All these informations can be useful for the Model Validation, e.g. by checking first the nodes based on their associated disagreements (global and maximal), then based on the time for the assessments (that can reflect a difficulty, or, on the ©2010 BAYESIA SAS contrary, too prompt assessments) All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 23
- 24. Exportation of a Bayesian Network per Expert Plan Modeling by Brainstorming BAYESIALAB 5.0 Knowledge Elicitation Environment This exportation tool allows to create a Bayesian Belief Network for each Expert. The parameters (probabilities) are those assessed by the Expert. If the Expert has not assessed all the probabilities, the model will use either the consensual probabilities, or those manually entered by the Facilitator ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 24
- 25. Exportation of the Probability Assessments This exportation tool allows creating a CSV file with all the assessments of the probabilities. There is one column per variable to describe the context (yellow), one Plan column to indicate the assessed Node (green), the other columns describing the assessed probability, the confidence level, the Expert, and the assessment time. Each line describes one assessment of a Modeling by (Conditional) Probability Table cell Brainstorming MitA/TiPo MiAt TiPo Node Probability Confidence Expert Time BAYESIALAB 5.0 Weak Weak Strong MitA/TiPo 0,97 1 Hiro 56 Knowledge Strong Weak Strong MitA/TiPo 0,03 1 Hiro 56 Elicitation Weak Weak Strong MitA/TiPo 0,95 1 Haitien 210 Environment Strong Weak Strong MitA/TiPo 0,05 1 Haitien 210 Weak Weak Strong MitA/TiPo 0,8 0,58 Claire 145 Strong Weak Strong MitA/TiPo 0,2 0,58 Claire 145 Weak Weak Strong MitA/TiPo 0,85 0,77 Matt 65 Strong Weak Strong MitA/TiPo 0,15 0,77 Matt 65 Weak Weak Strong MitA/TiPo 0,4 0,8 Mohinder 76 Strong Weak Strong MitA/TiPo 0,6 0,8 Mohinder 76 Weak Weak Strong MitA/TiPo 0,75 0,9 Nathan 50 Strong Weak Strong MitA/TiPo 0,25 0,9 Nathan 50 ... ... ... ... ... ... ... Weak MiAt 0,7 0,2 Noah 76 ©2010 BAYESIA SAS Strong MiAt 0,3 0,2 Noah 76 All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express Weak MiAt 0,75 1 Matt 90 written permission 25 Strong MiAt 0,25 1 Matt 90
- 26. Exportation of the Expert Assessments This exportation tool generates a CSV file with all the assessments of the Experts. There is one column per Expert, one column per Expert’s Confidence (yellow), the last Plan column indicating the weight of the line (1/ number of states of the assessed variable) (green). Each line describes the Experts’ assessment Modeling by of a(Conditional) Probability Table cell Brainstorming BAYESIALAB 5.0 Knowledge Elicitation Environment Hiro Hiro Haitien Haitien .... Noah Noah Weight Confidence Confidence Confidence 0,97 1 0,95 1 .... 0,7 0,8 0,5 0,03 1 0,05 1 .... 0,3 0,8 0,5 0,3 0,81 0,05 1 .... 0,3 0,7 0,5 0,7 0,81 0,05 1 .... 0,7 0,7 0,5 0 1 0 1 .... 0 0,79 0,5 1 1 1 1 .... 1 0,79 0,5 0,65 0,79 0,71 1 .... 0,7 0,2 0,5 ©2010 BAYESIA SAS 0,35 0,79 0,29 1 .... 0,3 0,2 0,5 All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 26
- 27. Analysis of the Expert Assessments We then can used this file to analyze the direct probabilistic relationships that hold between the Experts’ assessments Plan Modeling by Brainstorming BAYESIALAB 5.0 Knowledge Elicitation Environment This network has been automatically learned with one of the BayesiaLab’s Association Each node represents the ©2010 BAYESIA SAS Discovering algorithms on a set discretized probabilities All rights reserved. Forbidden reproduction in whole or part of 120 Experts’ assessments assessed by the Expert without the Bayesia’s express written permission 27
- 28. Automatic Segmentation of the Experts Based on the obtained network, Experts can be clustered into homogeneous groups by using the BayesiaLab’s Variable Clustering algorithm Plan Modeling by Brainstorming BAYESIALAB 5.0 Knowledge Elicitation Environment Dendrogram Each color corresponds to a corresponding to that cluster. Three segments of Experts have segmentation been induced in that example. The real experts behind those anonymized experts have indeed three different profiles (functionally and geographically) Based on the obtained Expert Segments, one Bayesian network per segment can be generated (by using the Expert Editor). This can be useful for analyzing the sensibility of ©2010 BAYESIA SAS the model, but also to get specific networks (depending on the geographical localization All rights reserved. Forbidden for example) reproduction in whole or part without the Bayesia’s express written permission 28
- 29. Parameter Sensibility Analysis BayesiaLab also comes with an Assessment Sensitivity Analysis tool that allows measuring the uncertainty associated to the consensus. The general idea is to generate a set of networks by randomly drawing Experts’ assessments, and then measuring the uncertainty Plan associated to each probability distribution. Modeling by Brainstorming BAYESIALAB 5.0 Knowledge Elicitation Environment ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 29
- 30. Parameter Sensibility Analysis Plan Modeling by Brainstorming BAYESIALAB 5.0 Knowledge Elicitation Environment Three kinds of analysis are available, depending on the Random selection policy that is chosen to generate the set of networks (1000 networks in the above example): - One Expert per network: each network generated is parametrized by using the selected Expert (or the consensual probability if the selected Expert has not been involved in the assessment) - One Expert per node: each network generated is parametrized by selecting for each node ©2010 BAYESIA SAS one Expert. If the selected Expert is not involved, the consensual probability if the selected All rights reserved. Forbidden - One assessment per Conditional Probability Table’s row (if any) reproduction in whole or part without the Bayesia’s express written permission 30
- 31. Parameter Sensibility Analysis States of the Tabs for the selection of analyzed variable the variable under analysis Plan Modeling by Brainstorming BAYESIALAB 5.0 Knowledge Density function illustrating Elicitation Marginal the uncertainty associated to probability distribution of the Environment this node. The Mean over the Target node computed with 1000 networks (one Expert all the consensus per network) is 70.62% (versus 70.46% in the monitor), the Standard Deviation 2.13%. There are 62% of chance of having a probability comprise between 70 and 72% ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 31
- 32. Parameter Sensibility Analysis Plan Modeling by Brainstorming BAYESIALAB 5.0 Knowledge Elicitation Environment One Expert per node One Expert per Conditional ©2010 BAYESIA SAS All rights reserved. Forbidden Probability row reproduction in whole or part without the Bayesia’s express written permission 32
- 33. Contact Dr. Lionel JOUFFE Plan President / CEO Tel.: +33(0)243 49 75 58 Fax: +33(0)243 49 75 83 Modeling by Brainstorming BAYESIALAB 5.0 Knowledge Elicitation 6 rue Léonard de Vinci BP0119 Environment 53001 LAVAL Cedex FRANCE ©2010 BAYESIA SAS All rights reserved. Forbidden reproduction in whole or part without the Bayesia’s express written permission 33