Abstract-and-Compare: A Family of Scalable Precision Measures for Automated Process Discovery
Download as PPTX, PDF1 like264 views
The document discusses precision measures in process mining, highlighting their importance and the challenges in comparing process model behaviors against event log behaviors, particularly when one is infinite and the other is finite. It outlines five axioms that precision measures should satisfy and presents a family of scalable precision measures, including kth-order Markovian abstractions for capturing behavior effectively. The document evaluates existing measures against these axioms and introduces new approaches for precision comparisons based on behavioral abstraction.
Abstract-and-Compare: A Family of Scalable Precision Measures for Automated Process Discovery
- 1. Abstract-and-Compare: a Family of Scalable Precision Measures for Automated Process Discovery Adriano Augusto, Abel Armas-Cervantes, Raffaele Conforti, Marlon Dumas, Marcello La Rosa, and Daniel Reissner
- 2. Precision in Process Mining Precision captures the extent to which the behaviour allowed by a process model is observed in an event log: — How much behaviour of a process model can be found in an event log? Event Log Process Model Event Log Behaviour Process Model Behaviour Compare Precision 2
- 3. State of the Art Precision Measures Precision Name Authors Year Set Difference Precision Greco et Al. 2006 Advanced Behavioural Appropriateness Rozinat and van der Aalst 2008 Negative Events Precision De Weerdt et al. 2011 Alignments-based ETC precision (one-align) Adriansyah et al. 2015 Projected Conformance Checking Leemans et al. 2016 Anti-alignment Precision van Dongen et al. 2016 3
- 4. Five Axioms for Precision Measures Tax, N., Lu, X., Sidorova, N., Fahland, D. and van der Aalst, W. (2018). The imprecisions of precision measures in process mining. — Axiom 1: given a log L and a process model M, a precision measure is a deterministic function: Prec(L, M) in ℝ. 4
- 5. Five Axioms for Precision Measures Tax, N., Lu, X., Sidorova, N., Fahland, D. and van der Aalst, W. (2018). The imprecisions of precision measures in process mining. — Axiom 1: given a log L and a process model M, a precision measure is a deterministic function: Prec(L, M) in ℝ. — Axiom 2: given a log L, and two process models M1 and M2. If the behaviour of L is fully contained in M1 behaviour, and the behaviour of M1 is fully contained in M2 behavior, then Prec(L, M1) ≥ Prec(L, M2). M2 M1 L Axiom 2 5
- 6. Five Axioms for Precision Measures Tax, N., Lu, X., Sidorova, N., Fahland, D. and van der Aalst, W. (2018). The imprecisions of precision measures in process mining. — Axiom 1: given a log L and a process model M, a precision measure is a deterministic function: Prec(L, M) in ℝ. — Axiom 2: given a log L, and two process models M1 and M2. If the behaviour of L is fully contained in M1 behaviour, and the behaviour of M1 is fully contained in M2 behavior, then Prec(L, M1) ≥ Prec(L, M2). — Axiom 3: given a log L, and two process models M1 and M2. If the behaviour of L is fully contained in M1 and M2 is the flower process, then Prec(L, M1) > Prec(L, M2). M2 M1 L Axiom 2 6
- 7. Five Axioms for Precision Measures Tax, N., Lu, X., Sidorova, N., Fahland, D. and van der Aalst, W. (2018). The imprecisions of precision measures in process mining. — Axiom 1: given a log L and a process model M, a precision measure is a deterministic function: Prec(L, M) in ℝ. — Axiom 2: given a log L, and two process models M1 and M2. If the behaviour of L is fully contained in M1 behaviour, and the behaviour of M1 is fully contained in M2 behavior, then Prec(L, M1) ≥ Prec(L, M2). — Axiom 3: given a log L, and two process models M1 and M2. If the behaviour of L is fully contained in M1 and M2 is the flower process, then Prec(L, M1) > Prec(L, M2). — Axiom 4: given a log L, and two process models M1 and M2. If the behaviour of M1 is equal to the behaviour of M2, then Prec(L, M1) = Prec(L, M2). M2 M1 L Axiom 2 7
- 8. Five Axioms for Precision Measures Tax, N., Lu, X., Sidorova, N., Fahland, D. and van der Aalst, W. (2018). The imprecisions of precision measures in process mining. — Axiom 1: given a log L and a process model M, a precision measure is a deterministic function: Prec(L, M) in ℝ. — Axiom 2: given a log L, and two process models M1 and M2. If the behaviour of L is fully contained in M1 behaviour, and the behaviour of M1 is fully contained in M2 behavior, then Prec(L, M1) ≥ Prec(L, M2). — Axiom 3: given a log L, and two process models M1 and M2. If the behaviour of L is fully contained in M1 and M2 is the flower process, then Prec(L, M1) > Prec(L, M2). — Axiom 4: given a log L, and two process models M1 and M2. If the behaviour of M1 is equal to the behaviour of M2, then Prec(L, M1) = Prec(L, M2). — Axiom 5: given two logs L1 and L2, and a process model M. If the behaviour of L1 is fully contained in L2 behaviour, then Prec(L2, M) ≥ Prec(L1, M). M2 M1 L M L2 L1 Axiom 2 Axiom 5 8
- 9. State of the Art Precision Measures(2) Precision Axiom Satisfied Name Authors Year A1 A2 A3 A4 A5 Set Difference Precision Greco et Al. 2006 yes ? no yes yes Advanced Behavioural Appropriateness Rozinat and van der Aalst 2008 no ? no yes ? Negative Events Precision De Weerdt et al. 2011 no no ? ? ? Alignments-based ETC precision (one-align) Adriansyah et al. 2015 no no no no no Projected Conformance Checking Leemans et al. 2016 ? no ? ? no Anti-alignment Precision van Dongen et al. 2016 ? ? ? ? no Tax, N., Lu, X., Sidorova, N., Fahland, D. and van der Aalst, W. (2018). The imprecisions of precision measures in process mining. 9
- 10. Why so Challenging? Event Log Process Model Event Log Behaviour Process Model Behaviour Compare Precision 10
- 11. Why so Challenging? Event Log Process Model Event Log Behaviour Process Model Behaviour Compare Precision The process model behaviour may be infinite. The event log behaviour is always finite. 11
- 12. Why so Challenging? Event Log Process Model Event Log Behaviour Process Model Behaviour Compare Precision The process model behaviour may be infinite. The event log behaviour is always finite. How to fairly compare an infinite behaviour against a finite one? 12
- 13. Objectives-Driven Approach Design Event Log Process Model Event Log Behaviour Process Model Behaviour Compare Precision 13
- 14. Objectives-Driven Approach Design Event Log Process Model Event Log Behaviour Process Model Behaviour Compare Precision Abstract Behaviour Abstract Behaviour 14
- 15. Objectives-Driven Approach Design Event Log Process Model Event Log Behaviour Process Model Behaviour Compare Precision 1. Use the same Behavioural Abstraction 2. Capture only Chunks of Behaviour 3. Control Behavioural Approximation Abstract Behaviour Abstract Behaviour 15
- 16. Objectives-Driven Approach Design Event Log Process Model Event Log Behaviour Process Model Behaviour Compare Precision 1. Use the same Behavioural Abstraction 2. Capture only Chunks of Behaviour 3. Control Behavioural Approximation 4. Be Noise Tolerant and Rapid 5. Must satisfy the Five Axioms Abstract Behaviour Abstract Behaviour 16
- 17. kth-order Markovian Abstraction: a graphical representation of behavioural chunks (i.e. subtraces) of length k and their evolution. Traces # A, A, B x A, B, B y A, B, A, B, A, B z 1st Order Markovian Abstraction 17 Log Behaviour
- 18. kth-order Markovian Abstraction: a graphical representation of behavioural chunks (i.e. subtraces) of length k and their evolution. Traces # A, A, B x A, B, B y A, B, A, B, A, B z 1st Order Markovian Abstraction 18 Log Behaviour
- 19. kth-order Markovian Abstraction: a graphical representation of behavioural chunks (i.e. subtraces) of length k and their evolution. Traces # A, A, B x A, B, B y A, B, A, B, A, B z 1st Order Markovian Abstraction 19 Log Behaviour
- 20. kth-order Markovian Abstraction: a graphical representation of behavioural chunks (i.e. subtraces) of length k and their evolution. Traces # A, A, B x A, B, B y A, B, A, B, A, B z 1st Order Markovian Abstraction 20 Log Behaviour
- 21. kth-order Markovian Abstraction: example Traces # A, A, B x A, B, B y A, B, A, B, A, B z 2nd Order Markovian Abstraction 21 Log Behaviour
- 22. kth-order Markovian Abstraction: example Traces # A, A, B x A, B, B y A, B, A, B, A, B z 2nd Order Markovian Abstraction 22 Log Behaviour
- 23. kth-order Markovian Abstraction: example Traces # A, A, B x A, B, B y A, B, A, B, A, B z 2nd Order Markovian Abstraction 23 Log Behaviour
- 24. kth-order Markovian Abstraction: example Traces # A, A, B x A, B, B y A, B, A, B, A, B z 2nd Order Markovian Abstraction 24 Log Behaviour
- 25. 2nd Order Markovian Abstraction: from a Process Model 25
- 26. 26 1. Turn the process into an automaton 2nd Order Markovian Abstraction: from a Process Model
- 27. 27 1. Turn the process into an automatonAutomaton s0 s1 a b sf a b ba 2nd Order Markovian Abstraction: from a Process Model
- 28. 28 1. Turn the process into an automaton 2. Replay the automaton collecting all the subtraces of length k (and full traces of length < k) Automaton s0 s1 a b sf a b ba 2nd Order Markovian Abstraction: from a Process Model
- 29. 29 1. Turn the process into an automaton 2. Replay the automaton collecting all the subtraces of length k (and full traces of length < k) Automaton s0 s1 a b sf a b ba Subtraces: <a,b> <a,a> <b,a> <b,b> <a> <b> 2nd Order Markovian Abstraction: from a Process Model
- 30. 30 1. Turn the process into an automaton 2. Replay the automaton collecting all the subtraces of length k (and full traces of length < k) 3. Turn each subtrace into a node of the Markovian Abstraction Automaton s0 s1 a b sf a b ba Subtraces: <a,b> <a,a> <b,a> <b,b> <a> <b> 2nd Order Markovian Abstraction: from a Process Model
- 31. 31 1. Turn the process into an automaton 2. Replay the automaton collecting all the subtraces of length k (and full traces of length < k) 3. Turn each subtrace into a node of the Markovian Abstraction Subtraces: <a,b> <a,a> <b,a> <b,b> <a> <b> 2nd Order Markovian Abstraction 2nd Order Markovian Abstraction: from a Process Model
- 32. 32 1. Turn the process into an automaton 2. Replay the automaton collecting all the subtraces of length k (and full traces of length < k) 3. Turn each subtrace into a node of the Markovian Abstraction 4. Connect the nodes representing overlapping subtraces (e.g. <a,b> and <b,a>), and to the initial node (-) the traces’ suffixes and prefixes. Subtraces: <a,b> <a,a> <b,a> <b,b> <a> <b> 2nd Order Markovian Abstraction 2nd Order Markovian Abstraction: from a Process Model
- 33. 33 1. Turn the process into an automaton 2. Replay the automaton collecting all the subtraces of length k (and full traces of length < k) 3. Turn each subtrace into a node of the Markovian Abstraction 4. Connect the nodes representing overlapping subtraces (e.g. <a,b> and <b,a>), and to the initial node (-) the traces’ suffixes and prefixes. Subtraces: <a,b> <a,a> <b,a> <b,b> <a> <b> 2nd Order Markovian Abstraction 2nd Order Markovian Abstraction: from a Process Model
- 34. (Graph) Comparison — Comparator 1: Strict Graph Comparison (edges set difference) Precision (L, M) = 1 - | 𝑴 𝒆 𝑳 𝒆 | 𝑴 𝒆 𝐿 𝑒 = 𝑒𝑑𝑔𝑒𝑠 𝑠𝑒𝑡 𝑜𝑓 𝑡ℎ𝑒 𝐿𝑜𝑔 𝑀𝑎𝑟𝑘𝑜𝑣𝑖𝑎𝑛 𝐴𝑏𝑠𝑡𝑟𝑎𝑐𝑡𝑖𝑜𝑛 𝑀𝑒 = 𝑒𝑑𝑔𝑒𝑠 𝑠𝑒𝑡 𝑜𝑓 𝑡ℎ𝑒 𝑃𝑟𝑜𝑐𝑒𝑠𝑠 𝑀𝑜𝑑𝑒𝑙 𝑀𝑎𝑟𝑘𝑜𝑣𝑖𝑎𝑛 𝐴𝑏𝑠𝑡𝑟𝑎𝑐𝑡𝑖𝑜𝑛 34
- 35. (Graph) Comparison — Comparator 1: Strict Graph Comparison (edges set difference) — Comparator 2 (implemented): Hungarian Algorithm Graph Comparison (HGC), with Levenshtein Distance as cost function Precision (L, M) = 1 - | 𝑴 𝒆 𝑳 𝒆 | 𝑴 𝒆 𝐿 𝑒 = 𝑒𝑑𝑔𝑒𝑠 𝑠𝑒𝑡 𝑜𝑓 𝑡ℎ𝑒 𝐿𝑜𝑔 𝑀𝑎𝑟𝑘𝑜𝑣𝑖𝑎𝑛 𝐴𝑏𝑠𝑡𝑟𝑎𝑐𝑡𝑖𝑜𝑛 𝑀𝑒 = 𝑒𝑑𝑔𝑒𝑠 𝑠𝑒𝑡 𝑜𝑓 𝑡ℎ𝑒 𝑃𝑟𝑜𝑐𝑒𝑠𝑠 𝑀𝑜𝑑𝑒𝑙 𝑀𝑎𝑟𝑘𝑜𝑣𝑖𝑎𝑛 𝐴𝑏𝑠𝑡𝑟𝑎𝑐𝑡𝑖𝑜𝑛 Precision (L, M) = 1 - 𝐻𝐺𝐶 𝑐𝑜𝑠𝑡 𝑴 𝒆 𝐻𝐺𝐶𝑐𝑜𝑠𝑡 = 𝑚𝑎𝑡𝑐ℎ𝑖𝑛𝑔 𝑐𝑜𝑠𝑡 𝑜𝑓 𝑡ℎ𝑒 𝐻𝑢𝑛𝑔𝑎𝑟𝑖𝑎𝑛 𝐴𝑙𝑔𝑜𝑟𝑖𝑡ℎ𝑚 𝑎𝑝𝑝𝑙𝑖𝑒𝑑 𝑡𝑜 𝑡ℎ𝑒 𝑀𝑎𝑟𝑘𝑜𝑣𝑖𝑎𝑛 𝐴𝑏𝑠𝑡𝑟𝑎𝑐𝑡𝑖𝑜𝑛𝑠 𝑜𝑓 𝑡ℎ𝑒 𝑃𝑟𝑜𝑐𝑒𝑠𝑠 𝑀𝑜𝑑𝑒𝑙 𝑎𝑛𝑑 𝑜𝑓 𝑡ℎ𝑒 𝐿𝑜𝑔 35 note: comparator 2 is equal to comparator 1 when the process behaviour fully contains the log behaviour..
- 36. kth-order Markovian Abstraction-Based Precision: MAPk Traces # A, A, B x A, B, B y A, B, A, B, A, B z Process 1 (P1) Event Log (L) Flower Process (FP) 36
- 37. Traces # A, A, B x A, B, B y A, B, A, B, A, B z Process 1 (P1) Event Log (L) Flower Process (FP) MAP1 (L, P1) = 1 - 0 6 = 1.00 37 kth-order Markovian Abstraction-Based Precision: MAPk
- 38. Traces # A, A, B x A, B, B y A, B, A, B, A, B z Process 1 (P1) Event Log (L) Flower Process (FP) MAP1 (L, P1) = 1 - 0 6 = 1.00 38 kth-order Markovian Abstraction-Based Precision: MAPk
- 39. Traces # A, A, B x A, B, B y A, B, A, B, A, B z Process 1 (P1) Event Log (L) Flower Process (FP) MAP1 (L, P1) = 1 - 0 6 = 1.00 MAP1 (L, FP) = 1 - 2 8 = 0.75 39 kth-order Markovian Abstraction-Based Precision: MAPk
- 40. Traces # A, A, B x A, B, B y A, B, A, B, A, B z MAP2 (L, P1) = 1 - 4 12 = 0.66 MAP1 (L, P1) = 1 - 0 6 = 1.00 MAP2 (L, FP) = 1 - 12 20 = 0.40 MAP1 (L, FP) = 1 - 2 8 = 0.75 40 Process 1 (P1) Event Log (L) Flower Process (FP) kth-order Markovian Abstraction-Based Precision: MAPk
- 41. Satisfiability of the Five Axioms Precision Axiom Satisfied Name Authors Year A1 A2 A3 A4 A5 Set Difference Precision Greco et Al. 2006 yes yes no yes yes Advanced Behavioural Appropriateness Rozinat and van der Aalst 2008 no ? no yes ? Negative Events Precision De Weerdt et al. 2011 no no ? ? ? Alignments-based ETC precision (one-align) Adriansyah et al. 2015 no no no no no Projected Conformance Checking Leemans et al. 2016 ? no ? ? no Anti-alignment Precision van Dongen et al. 2016 ? ? ? ? no kth-order Markovian Abstraction Augusto et al. 2018 yes yes yes* yes yes *Axiom 3 is satisfied for a given order of k* (or higher orders). k* = 2 for any process having at least one activity that cannot be executed twice consecutively. 41
- 42. Qualitative Evaluation on Artificial Data (1) Traces # A, B, D, E, I 1207 A, C, D, G, H, F, I 145 A, C, G, D, H, F, I 56 A, C, H, D, F, I 23 A, C, D, H, F, I 28 van Dongen, B., Carmona, J. and Chatain, T. A unified approach for measuring precision and generalization based on anti-alignments, BPM 2016. 42 1. single trace 2. separate traces 3. flower model 4. optional G || optional H 5. G and H as self-loop activities 6. D as self-loop activity 7. all parallel activities 8. round robin
- 43. 43 Process Infinite Behaviuor Traces (max 2 loops) SD ETC NE PCC AA MAP1 MAP2 MAP3 … MAP7 original model no 6 7 7 9 8 7 7 7 7 = 7 single trace no 1 8 8 6 8 8 7 8 8 = 8 separate traces no 5 8 8 8 7 8 7 8 8 = 8 opt. G || opt. H no 12 6 3 7 6 6 5 6 6 = 6 all parallel no 362,880 1 2 2 2 3 2 2 2 = 2 round robin yes 27 1 4 3 3 1 4 5 5 = 5 D self-loop yes 118 1 6 4 5 4 5 4 4 = 4 G and H self-loops yes 362 1 5 5 4 5 3 3 3 = 3 flower model yes 986,410 1 1 1 1 1 1 1 1 = 1 processes ordered by precision (the higher the rank the more precise the process). Qualitative Evaluation on Artificial Data (2)
- 44. 44 Process Infinite Behaviuor Traces (max 2 loops) SD ETC NE PCC AA MAP1 MAP2 MAP3 … MAP7 original model no 6 7 7 9 8 7 7 7 7 = 7 single trace no 1 8 8 6 8 8 7 8 8 = 8 separate traces no 5 8 8 8 7 8 7 8 8 = 8 opt. G || opt. H no 12 6 3 7 6 6 5 6 6 = 6 all parallel no 362,880 1 2 2 2 3 2 2 2 = 2 round robin yes 27 1 4 3 3 1 4 5 5 = 5 D self-loop yes 118 1 6 4 5 4 5 4 4 = 4 G and H self-loops yes 362 1 5 5 4 5 3 3 3 = 3 flower model yes 986,410 1 1 1 1 1 1 1 1 = 1 processes ordered by precision (the higher the rank the more precise the process). Qualitative Evaluation on Artificial Data (2)
- 45. 45 Process Infinite Behaviuor Traces (max 2 loops) SD ETC NE PCC AA MAP1 MAP2 MAP3 … MAP7 original model no 6 7 7 9 8 7 7 7 7 = 7 single trace no 1 8 8 6 8 8 7 8 8 = 8 separate traces no 5 8 8 8 7 8 7 8 8 = 8 opt. G || opt. H no 12 6 3 7 6 6 5 6 6 = 6 all parallel no 362,880 1 2 2 2 3 2 2 2 = 2 round robin yes 27 1 4 3 3 1 4 5 5 = 5 D self-loop yes 118 1 6 4 5 4 5 4 4 = 4 G and H self-loops yes 362 1 5 5 4 5 3 3 3 = 3 flower model yes 986,410 1 1 1 1 1 1 1 1 = 1 processes ordered by precision (the higher the rank the more precise the process). Qualitative Evaluation on Artificial Data (2)
- 46. 46 Process Infinite Behaviuor Traces (max 2 loops) SD ETC NE PCC AA MAP1 MAP2 MAP3 … MAP7 original model no 6 7 7 9 8 7 7 7 7 = 7 single trace no 1 8 8 6 8 8 7 8 8 = 8 separate traces no 5 8 8 8 7 8 7 8 8 = 8 opt. G || opt. H no 12 6 3 7 6 6 5 6 6 = 6 all parallel no 362,880 1 2 2 2 3 2 2 2 = 2 round robin yes 27 1 4 3 3 1 4 5 5 = 5 D self-loop yes 118 1 6 4 5 4 5 4 4 = 4 G and H self-loops yes 362 1 5 5 4 5 3 3 3 = 3 flower model yes 986,410 1 1 1 1 1 1 1 1 = 1 processes ordered by precision (the higher the rank the more precise the process). Qualitative Evaluation on Artificial Data (2)
- 47. 47 Process Infinite Behaviuor Traces (max 2 loops) SD ETC NE PCC AA MAP1 MAP2 MAP3 … MAP7 original model no 6 7 7 9 8 7 7 7 7 = 7 single trace no 1 8 8 6 8 8 7 8 8 = 8 separate traces no 5 8 8 8 7 8 7 8 8 = 8 opt. G || opt. H no 12 6 3 7 6 6 5 6 6 = 6 all parallel no 362,880 1 2 2 2 3 2 2 2 = 2 round robin yes 27 1 4 3 3 1 4 5 5 = 5 D self-loop yes 118 1 6 4 5 4 5 4 4 = 4 G and H self-loops yes 362 1 5 5 4 5 3 3 3 = 3 flower model yes 986,410 1 1 1 1 1 1 1 1 = 1 processes ordered by precision (the higher the rank the more precise the process). Qualitative Evaluation on Artificial Data (2)
- 48. Real-Life Evaluation (setup) 48 SETUP — 20 Real-life logs: 12 publicly available (at the 4TU data centre), and 8 proprietary — Models discovered by three automated discovery algorithms (Split Miner, Inductive Miner, and Structured Heuristics Miner) — Qualitative comparison against ETC Precision (the only feasible in a real-life context) — Time performance comparison against ETC Precision
- 49. Real-Life Evaluation (results) 49 RESULTS — MAPk easily distinguishes between process models with poor precision and high precision — MAPk is suitable for quality assessment and (especially) comparison of process models — MAPk can be over 10 times faster than ETC (avg. time 3.7s vs 60.0s, on models discovered by Split Miner)
- 50. Inductive Miner – SEPSIS Log Split Miner – SEPSIS Log 50 Real-Life Evaluation qualitative example
- 51. Inductive Miner – SEPSIS Log Split Miner – SEPSIS Log 51 Real-Life Evaluation qualitative example
- 52. Real-Life Evaluation qualitative example Inductive Miner – SEPSIS Log Split Miner – SEPSIS Log 52 Split Miner Inductive Miner ETC 0.859 0.445 MAP2 1.000 0.226 MAP3 1.000 0.051 MAP4 1.000 0.009
- 53. Limitations 53 —The order k and the execution time are proportional —How to choose k? —The selection of the best k is not automated
- 54. Future Work —Designing a complementary Markovian Abstraction-Based Fitness —Exploring alternative comparison algorithms (e.g. graph bisimulation) —Using the Markovian Precision in automated process discovery for reinforcement learning 54