A Performance Study of BDD-Based Model Checking
Download as PPT, PDF0 likes302 views
This document presents a performance study on BDD-based model checking, highlighting the significant differences between combinational circuits and model checking computations. It details the evaluation methodology, experimental results, and the impact of both dynamic variable reordering and computed cache size on performance. The findings demonstrate that collaborative efforts and rigorous evaluation can lead to substantial improvements in model checking efficiency, with recommendations for future research directions.
A Performance Study of BDD-Based Model Checking
- 1. A Performance Study of BDD-Based Model Checking Bwolen Yang Randal E. Bryant, David R. O’Hallaron, Armin Biere, Olivier Coudert, Geert Janssen Rajeev K. Ranjan, Fabio Somenzi
- 2. 2 Motivation for Studying Model Checking (MC) An important application of BDD Representations Not well studied Packages are tuned using combinational circuits (CC) Qualitative differences between CC and MC computations CC: build outputs, constant time equivalence checking MC: build model, many fixed-points to verify the specs CC: BDD algorithms are polynomial MC: key BDD algorithms are exponential
- 3. 3 Outline BDD Overview Organization of this Study participants, benchmarks, metrics, evaluation process Experimental Results with and without dynamic variable reordering BDD Evaluation Methodology evaluation platform » various BDD packages » real workload metrics
- 4. 4 BDD Overview BDD DAG representation for Boolean functions fixed order on Boolean variables BDD Algorithms dynamic programming sub-problems (operations): Shannon decomposition memoization: computed cache
- 5. 5 BDD Overview (Cont’d) Garbage Collection recycle unreachable (dead) nodes Dynamic Variable Reordering BDD graph size depends on the variable order sifting based » nodes in adjacent levels are swapped
- 6. 6 Organization of this Study: Participants Armin Biere: ABCD Carnegie Mellon / Universität Karlsruhe Olivier Coudert: TiGeR Synopsys / Monterey Design Systems Geert Janssen: EHV Eindhoven University of Technology Rajeev K. Ranjan: CAL Synopsys Fabio Somenzi: CUDD University of Colorado Bwolen Yang: PBF Carnegie Mellon
- 7. 7 Organization of this Study: Setup Metrics: 17 statistics Benchmark: 16 SMV execution traces traces of BDD-calls from verification of » cache coherence, Tomasulo, phone, reactor, TCAS… size » 6 million - 10 billion sub-operations » 1 - 600 MB of memory Evaluation platform: trace driver “drives” BDD packages based on execution trace
- 8. 8 Organization of this Study: Evaluation Process Phase 1: no dynamic variable reordering Phase 2: with dynamic variable reordering hypothesize design experiments validate collect stats identify issues suggest improvements validate Iterative Process
- 9. 9 Phase 1 Results: Initial / Final Time Comparison Conclusion: collaborative efforts have led to significant performance improvements 1 6 13 76 1e+1 1e+2 1e+3 1e+4 1e+5 1e+6 1e+1 1e+2 1e+3 1e+4 1e+5 1e+6 final results (sec) initialresults(sec) 1x10x100x n/a n/a s
- 10. 10 Computed Cache effects of computed cache size amounts of repeat sub-problems across time Garbage Collection death rate rebirth rate Complement Edge Representation work space Memory Locality for Breadth-First Algorithms Computed Cache effects of computed cache size amounts of repeat sub-problems across time Garbage Collection death rate rebirth rate Complement Edge Representation work space Memory Locality for Breadth-First Algorithms Phase 1: Hypotheses / Experiments
- 11. 11 Phase 1: Hypotheses / Experiments (Cont’d) For Comparison ISCAS85 circuits (> 5 sec, < 1GB) c2670, c3540 13-bit, 14-bit multipliers based on c6288 Metrics depends only on the trace and BDD algorithms machine-independent implementation-independent
- 12. 12 Computed Cache: Repeated Sub-problems Across Time Source of Speedup increase computed cache size Possible Cause many repeated sub-problems are far apart in time Validation study the number of repeated sub-problems across user issued operations (top level operations).
- 13. 13 Hypothesis: Top-Level Sharing Hypothesis MC computations have a large number of repeated sub-problems across the top-level operations. Experiment measure the minimum number of operations with GC disabled and complete cache. compare this with the same setup, but cache is flushed between top-level operations.
- 14. 14 Results on Top-Level Sharing flush: cache flushed between top-level operations Conclusion: large cache is more important for MC 1 10 100 Model Checking Traces ISCAS85 #ofops (flush/min)
- 15. 15 Garbage Collection: Rebirth Rate Source of Speedup reduce GC frequency Possible Cause many dead nodes become reachable again (rebirth) » GC is delayed till the number of dead nodes reaches a threshold » dead nodes are reborn when they are part of the result of new sub-problems
- 16. 16 Hypothesis: Rebirth Rate Hypothesis MC computations have very high rebirth rate. Experiment measure the number of deaths and the number of rebirths
- 17. 17 Results on Rebirth Rate 0 0.2 0.4 0.6 0.8 1 Model Checking Traces ISCAS85 rebirths/deaths Conclusions delay garbage collection triggering GC should not base only on # of dead nodes delay updating reference counts
- 18. 18 BF BDD Construction Breadth-first based BDD construction is not a demonstrated advantage over traditional depth-first based techniques. Two packages (CAL and PBF) are BF based.
- 19. 19 BF BDD Construction Overview Level-by-Level Access operations on same level (variable) are processed together one queue per level Locality group nodes of the same level together in memory Good memory locality due to BF ==> # of ops processed per queue visit must be high
- 20. 20 Average BF Locality Conclusion: MC traces generally have less BF locality 0 1000 2000 3000 4000 5000 Model Checking Traces ISCAS85 #ofopsprocessed perlevel-visit
- 21. 21 Average BF Locality / Work Conclusion: For comparable BF locality, MC computations do much more work. 0 50 100 150 200 250 Model Checking Traces ISCAS85 avg.locality/#ofops (x1e-6)
- 22. 22 Phase 1: Some Issues / Open Questions Memory Management space-time tradeoff » computed cache size / GC frequency resource awareness » available physical memory, memory limit, page fault rate Top-Level Sharing
- 23. 23 Phase 2: Dynamic Variable Reordering BDD Packages Used CAL, CUDD, EHV, TiGeR improvements from phase 1 incorporated Variable Reordering is hard to study ==> limited experimental results
- 24. 24 Why is Variable Reordering Hard to Study Time-space tradeoff how much time to spent to reduce graph sizes Chaotic behavior e.g., small changes to triggering / termination algorithm can have significant performance impact Resource intensive reordering is expensive space of possible ordering is combinatorial Different variable order ==> different computation e.g., many “don’t-care space” optimization algorithms
- 25. 25 Quality of Variable Order Generated Variable Grouping Heuristic keep strongly related variables adjacent Reorder Transition Relation BDDs for the transition relation are used repeatedly Effects of Initial Variable Order for both with and without variable reordering Quality of Variable Order Generated Variable Grouping Heuristic keep strongly related variables adjacent Reorder Transition Relation BDDs for the transition relation are used repeatedly Effects of Initial Variable Ordering for both with and without variable reordering Phase 2: Experiments Only CUDD is used
- 26. 26 Variable Grouping Heuristic: Group Current / Next Variables Current / Next State Variables for transition relation, split variable into two copies: » current state and next state Hypothesis Grouping the corresponding current- and next-state variables is a good heuristic. Experiment compare results between with and without grouping » work (# of operations) » space (max # of live BDD nodes) » reorder cost (# of nodes swapped with their children)
- 27. 27 Results on Grouping Current / Next Variables Space 0 1 2 3 MC traces maxlivenodes Reorder Cost 0 1 2 3 MC traces nodesswapped Work 0 1 2 3 4 MC traces #ofops All results are normalized against no variable grouping. Conclusion: grouping is generally effective
- 28. 28 Effects of Initial Variable Order: Experimental Setup For each trace, find a good variable ordering O perturb O to generate new variable orderings » fraction of variables perturbed » distance moved use these new orderings to study the results with and without dynamic variable reordering
- 29. 29 Effects of Initial Variable Order: Perturbation Algorithm Perturbation Parameters (p, d) p: probability that a variable will be perturbed d: perturbation distance Properties in average, p fraction of variables is perturbed max distance moved is 2d Note (p = 1, d = infinity) ==> completely random variable order
- 30. 30 Effects of Initial Variable Order: Parameters For each perturbation level (p, d) generate a number (k) of random variable orders Parameter Values p: (0.1, 0.2, …, 1.0) d: (10, 20, …, 100, infinity) k: 10 ==> for each trace, 1100 orderings 2200 runs (w/ and w/o dynamic reordering)
- 31. 31 Effects of Initial Variable Order: Smallest Test Case Base Case (best ordering) time: 13 sec memory: 127 MB Limits on Generated Orders time: 128x base case memory: 500 MB
- 32. 32 Effects of Initial Variable Order: Result # of unfinished cases At 128x/500MB limit, “no reorder” finished 33%, “reorder” finished 90%. Conclusion: reordering alg. is effective 0 400 800 1200 >1x >2x >4x >8x >16x >32x >64x >128x time limit #ofcases no reorder reorder
- 33. 33 Phase 2: Some Issues / Open Questions Computed Cache Flushing cost Effects of Initial Variable Order determine sample size k Need New Better Experimental Design
- 34. 34 BDD Evaluation Methodology Trace-Driven Evaluation Platform real workload (BDD-call traces) study various BDD packages focus on key operations Evaluation Metrics more rigorous quantitative analysis
- 35. 35 BDD Evaluation Methodology Metrics: Time elapsed time (performance) CPU time page fault rate BDD Alg timeGC time reordering time # of GCs # of node swaps (reorder cost) memory usage # of ops (work) # of reorderings computed cache size
- 36. 36 BDD Evaluation Methodology Metrics: Space memory usage # of GCs # of reorderings computed cache size max # of BDD nodes
- 37. 37 Importance of Better Metrics Example: Memory Locality # of GCs # of reorderings computed cache size elapsed time (performance) CPU time page fault rate cache miss rate TLB miss rate memory locality # of ops (work) without accounting for work, may draw premature conclusions
- 38. 38 Summary Collaboration + Evaluation Methodology significant performance improvements » up to 2 orders of magnitude characterization of MC computation » computed cache size » garbage collection frequency » effects of complement edge » BF locality » reordering heuristic – current / next state variable grouping » effects of reordering the transition relation » effects of initial variable orderings other general results (not mentioned in this talk) issues and open questions for future research
- 39. 39 Conclusions Rigorous BDD evaluation can lead to dramatic results Adopt the Evaluation Methodology more benchmark traces » for IP issue, BDD-call trace is like assembly language use / improve on the metrics proposed for future evaluation For data, traces, trace-driver used in this study, http://www.cs.cmu.edu/~bwolen/fmcad98/