Model Checking-Enhanced Spectrum-Based Fault Localization

Model Checking-Enhanced Spectrum-Based Fault
Localization
Mohammed BEKKOUCHE
École Supérieure en Informatique, Sidi Bel Abbès, Algeria
LabRI-SBA Laboratory
m.bekkouche@esi-sba.dz
CSA 2024
April 23-24, 2024
Mohammed BEKKOUCHE (ESI-SBA) Fault Localization in Programs April 23-24, 2024 1 / 17

Presentation Outline
1 Introduction
2 State of the Art
3 Motivating Example
4 Approach
5 Results
6 Conclusion

Introduction
Introduction
Debugging software is time-consuming, especially when it comes to
pinpointing faulty instructions.
Accelerating fault localization can greatly reduce costs and improve software
quality.
Research focuses on : Spectrum-based and model-checking-based methods
for fault localization.
Proposal:
This presentation introduces a novel approach that combines testing and model
checking for improved fault localization.
Idea:
Our approach uses model checking to minimize suspicious instructions from
each failing test, reducing non-faulty statement coverage.
The core idea is to reduce failing test cases using model checking before
spectrum-based fault localization.

State of the Art
State of the Art
Model Checking-Based Fault Localization
→ Model checking evaluates program adherence to a specification by
traversing states.
Techniques : Explain, BugAssist, SNIPER, LocFaults, ...
Spectrum-Based Fault Localization
→ Spectrum-based approaches employ ranking metrics to assess program
elements’ suspicion levels.
Techniques : Tarantula, Ochiai, FLCN, ConsilientSFL, ...

Motivating Example
Motivating Example
Listing: The Mid program with an error
1 c l a s s Foo{
2 /∗@ r e q u i r e s t r u e ;
3 @ e n s u r e s r e s u l t == (max( min ( x , y ) , min (max( x , y ) , z ) ) ) ;
4 @∗/
5 s t a t i c void mid ( i n t a , i n t b , i n t c ) {
6 i n t m;
7 m=c ;
8 i f (b<c ) {
9 i f ( a<b )
10 m=b ;
11 e l s e i f ( a<c )
12 m=b ; // e r r o r , the i n s t r u c t i o n should be m=a ;
13 }
14 e l s e {
15 i f ( a>b )
16 m=b ;
17 e l s e i f ( a>c )
18 m=a ;
19 }
20 r e t u r n m;
21 }}
During the model checking phase, five suspicious instructions were identified:
{8, 12, 15, 17, 20} from the counterexample (a = 2, b = 1, c = 3)

Motivating Example
Motivating Example
During the testing phase :
Table: Example – Spectra, Four Counters, Scores, and Rankings in Our Approach
Statement
(3,3,5)
(3,2,1)
(5,5,5)
(5,4,3)
(2,1,3)
(4,5,3)
ef ep nf np Ochiai Rank
8 1 1 1 1 1 1 1 5 0 0 0.41 4
12 1 0 0 0 1 0 1 1 0 4 0.71 1
15 0 1 1 1 1 1 1 4 0 1 0.45 3
17 0 0 1 0 1 1 1 2 0 3 0.58 2
20 1 1 1 1 1 1 1 5 0 0 0.41 4
Result 0 0 0 0 1 0
Table: Ochiai and Tarantula Spectral Formulas Along with Their Equations
Ochiai
ef
p
(ef + nf ) · (ef + ep)
Tarantula
ef
ef + nf
ef
ef + nf
+
ep
ep + np

Motivating Example
Motivating Example
Utilizing traditional Spectrum-Based Fault Localization techniques :
Table: Illustrative Example – Four Counters, Scores, and Rankings in Pure
Spectrum-Based Fault Localization
Line Code ef ep nf np Ochiai Tarantula Rank
7 m=c; 1 5 0 0 0.41 0.5 3
8 if (b<c){ 1 5 0 0 0.41 0.5 3
9 if (a<b) 1 1 0 4 0.71 0.83 1
10 m=b; 0 0 1 5 0 0 7
11 else if (a<c) 1 1 0 4 0.71 0.83 1
12 m=b; 1 1 0 4 0.71 0.83 1
15 if (a>b) 0 0 1 1 0 0 7
16 m=b; 0 0 1 2 0 0 7
17 else if (a>c) 0 0 1 3 0 0 7
18 m=a; 0 0 1 4 0 0 7
20 return m; 1 5 0 0 0.41 0.5 3

Approach
Approach
Our approach encompasses three principal steps : Model Checking, Model
Checker-based Fault Localization (MCFL), and Spectrum-based Fault
Localization (SBFL).
Model Checking:
CPBPV/CBMC
Input: Faulty program,
specification
Output: Counterexamples,
successful test cases
MCFL:
LocFaults/BugAssist
postcondition,
failed test cases
Output: Suspicious
instructions
SBFL:
Ochiai/Tarantula/EWSR
suspicious instructions,
Output: Ranked instructions
Generate counterexamples
and successful test cases
Locate suspicious
instructions using
MCFL tools
Rank suspicious instructions
using SBFL and spectral
metrics
Figure: Synergistic Approach - Test and Model Checking for Fault Localization

Approach
Approach
Our approach encompasses three principal steps : Model Checking, Model
Checker-based Fault Localization (MCFL), and Spectrum-based Fault
Localization (SBFL).
Model Checking:
CPBPV/CBMC
specification
Output: Counterexamples,
MCFL:
LocFaults/BugAssist
postcondition,
failed test cases
Output: Suspicious
instructions
SBFL:
Ochiai/Tarantula/EWSR
suspicious instructions,
Output: Ranked instructions
Generate counterexamples
and successful test cases
Locate suspicious
instructions using
MCFL tools
Rank suspicious instructions
using SBFL and spectral
metrics
Figure: Synergistic Approach - Test and Model Checking for Fault Localization

Approach
Approach
Weighted Suspiciousness Ratio (WSR)
Novel spectral metric
Equation (refer to Eq. 1)
Higher execution frequency in failed tests indicates suspicion
WSR =
ef × np + nf × ep
ef × np + nf × ep + ef × ep + nf × np
(1)
Enhanced Weighted Suspiciousness Ratio (EWSR)
Advancement over WSR
Equation (refer to Eq. 2)
Normalization for refined fault localization
EWSR = WSR ×
1
p
(ef + ep) × (nf + np)
(2)

Results
Results
Introduction to TCAS Benchmark and Experiment Setup
Experiments conducted using the TCAS (Traffic Collision Avoidance System)
benchmark suite provided by Siemens.
TCAS task simulates an aircraft collision avoidance system with 173 lines of
code and 41 distinct versions with intentional faults.
Certain versions excluded due to array index out-of-bounds issues.
Benchmark includes 1608 test cases.

Results
Results
Methodology Overview and Comparison
BugAssist model checking tool utilized to pinpoint suspicious instructions in
faulty program.
→ using a single counterexample
Instructions classified using successful test cases and EWSR spectral metric.
Results compared with solely testing-based implementation using Ochiai,
Tarantula, and EWSR spectral formulas.
Average position of erroneous instruction calculated using Equation 3 to
address tie elements.
MID = S +

E − 1
2

(3)
S represents the starting position of the tie, and E denotes the tie size.

Results
Results
TCAS Benchmark Experiment Results
v
1
v
2
v
3
v
4
v
5
v
6
v
7
v
8
v
9
v
1
0
v
1
1
v
1
2
v
1
3
v
1
4
v
1
5
v
1
6
v
1
7
v
1
8
v
1
9
v
2
0
0
10
20
30
40
50
2.5
11.5
36.5
2.5
33.5
6.5
36.5
39.5
17.5
6.5
6.5
36.5
36.5
15.5
51.5
36.5
36.5
34.5
36.5
16.5
2.5
12.5
37.5
2.5
41.5
6.5
36.5
39.5
17.5
6.5
6.5
37.5
37.5
15.5
21.5
36.5
36.5
42.5
36.5
16.5
2.5
16.5
42.5
2.5
42.5
6.5
42.5
25.5
17.5
6.5
6.5
42.5
42.5
15.5
7.5
36.5
42.5
42.5
42.5
16.5
1
9
7
1
6.5
4
16
8.5
8
5
5
7.5
6.5
3
1.5
16
16.5
13.5
16
8.5
Versions
Line
numbers
Ochiai Tarantula EWSR MC+TEST
Figure: The Average Position of the Faulty Instruction in Each Program Version of the
TCAS Suite from Siemens

Results
Results
v
2
1
v
2
2
v
2
3
v
2
4
v
2
5
v
2
6
v
2
7
v
2
8
v
2
9
v
3
0
v
3
1
v
3
2
v
3
4
v
3
5
v
3
6
v
3
7
v
3
9
v
4
0
v
4
1
0
10
20
30
40
16.5
17.5
17.5
17.5
1.5
37.5
33.5
11.5
13.5
11.5
4
4
36.5
11.5
1
2.5
1.5
14
2.5
16.5
17.5
17.5
17.5
1.5
39.5
41.5
12.5
13.5
12.5
4
4
37.5
12.5
1
2.5
1.5
14
2.5
16.5
17.5
17.5
17.5
1.5
42.5
42.5
13.5
18.5
14.5
4
4
42.5
13.5
1
2.5
1.5
14
2.5
8.5
6.5
7
9
1
7
8.5
9
7
9.5
2
2
6
6.5
1
2
1
9.5
1
Versions
Line
numbers
Ochiai Tarantula EWSR BMC+TEST
Figure: The Average Position of the Faulty Instruction in Each Program Version of the
TCAS Suite from Siemens

Results
Results
Outcomes illustrated in histogram showing average position of faulty
instruction.
Comparison between pure spectrum-based methods and our integrated
approach (MC+TEST).
Ochiai consistently outperforms Tarantula in localizing faults.
EWSR spectral metric demonstrates variability in performance across versions.
Integrated approach consistently achieves lower average positions, indicating
improved fault localization accuracy.

Conclusion
Conclusion
Our integrated fault localization approach represents a significant
advancement in software debugging.
The combination of model checking and testing, as demonstrated in the
TCAS benchmark experiments, surpasses traditional spectrum-based
methods.
The novel EWSR spectral metric introduces valuable variability, addressing
challenges in certain versions.
This research contributes to automating and improving the fault localization
process, providing developers with a refined and efficient method for
identifying problematic instructions.
Overall, our integrated fault localization approach demonstrates a remarkable
improvement of approximately 65.47% in fault localization accuracy
compared to traditional spectrum-based methods.
Future research avenues may explore the applicability of this approach to
other benchmarks and further refine the integration of model checking and
testing for continued advancements in software debugging.

Conclusion
Thank you for your attention.
Questions ?

Model Checking-Enhanced Spectrum-Based Fault Localization

More Related Content

Similar to Model Checking-Enhanced Spectrum-Based Fault Localization (20)

Recently uploaded (20)

Model Checking-Enhanced Spectrum-Based Fault Localization