Automatically Customizing Static Analysis Tools to Coding Rules Really Followed by Developers

Automatically Customizing Static
Analysis Tools to Coding Rules
Really Followed by Developers
Yuki Ueda, Takashi Ishio and Kenichi Matsumoto
Nara Institute of Science and Technology
SANER2021

Static Analysis Tools or Linters
2
Static analysis tools or linters can detect:
1. Security Vulnerabilities,
2. Performance Issues,
3. Bad Programming Practices, Code Smells
These violations are defined by Coding Rules

Example of coding rule violations
3
if (a=b) {
return True
} else {
threw new
Error();
}
return False
JavaScript
Rule 1:
Limit the assignment in
conditional statement
a=b a==b

Example of coding rule violations
4
if (a=b) {
return True
} else {
threw new
Error();
}
return False
JavaScript
Rule 2:
Limit the unreachable code
return False

Linters in SE Research
5
Code Review [1]
[1] Carvalho, A., Luz, W., Marc´ılio, D., Bonif´acio, R.,Pinto, G. and Canedo, E. D.: C-3PR: A Bot for Fixing Static Analysis Violations via Pull Requests,
SANER’20
[2] Bavishi, Rohan, Hiroaki Yoshida, and Mukul R. Prasad. "Phoenix: Automated data-driven synthesis of repairs for static analysis violations." ESEC/FSE’19
[3] Trautsch, Alexander, Steffen Herbold, and Jens Grabowski. "Static source code metrics and static analysis warnings for fine-grained just-in-time defect
prediction." ICSME’20
Defect Prediction [3]
Bug Repair [2]

Linters in SE Research
6
Code Review [1]
These Studies Assume the Linters Are Correct
[1] Carvalho, A., Luz, W., Marc´ılio, D., Bonif´acio, R.,Pinto, G. and Canedo, E. D.: C-3PR: A Bot for Fixing Static Analysis Violations via Pull Requests,
SANER’20
[2] Bavishi, Rohan, Hiroaki Yoshida, and Mukul R. Prasad. "Phoenix: Automated data-driven synthesis of repairs for static analysis violations." ESEC/FSE’19
[3] Trautsch, Alexander, Steffen Herbold, and Jens Grabowski. "Static source code metrics and static analysis warnings for fine-grained just-in-time defect
prediction." ICSME’20
Defect Prediction [3]
Bug Repair [2]

Default Rules != Followed Rules
7
Dataset: 4 JavaScript Projects
• Used in BugsJS [4]
• Using ESLint [5]
[4] Gyimesi, Péter, et al. "BUGSJS: a benchmark and taxonomy of JavaScript
bugs." Software Testing, Verification and Reliability (2019): e1751.
[5] N. C. Zakas, B. Mills, T. Nagashima, and M. Djermanovic, “ES-Lint -
Pluggable JavaScript linter,” https://eslint.org/, 2020
N = 241 Default Rules: 57 Non-default Rules: 184
Followed by All Projects: 64 True Positives: 38 False Negatives: 26
Unfollowed by 1+ Projects: 177 False: Positives: 19 True Negatives: 158
1. 40% (26 / 64) of followed rules are not default

8
2. 33% (19 / 57) of default rules are ignored

9
2. 33% (19 / 57) of default rules are ignored
Linter Rules should be customized

Goal: Optimizing the Coding Rules
1. Reduce False Negative (non-default but followed) rules
Enabling the rules that are followed by the all of project files
2. Reduce False Positive (default but unfollowed) rules
Disabling the rules that are violated by the one or more files
10

Linter-Maintainer:
Tracing Followed Coding Rules
11
Project Code Files Coding Rule
Configuration File
Followed Rule List
When developer installed linter, execute
Input Output
$linter-maintainer --generate --eslint-js ./your/project/path > .eslintrc.json

Evaluation Process 1 / 4
12
Generate
Version 1.0.0
Configuration File 1.0.0

13
Apply and
Calc False Positive/Negative
Version 1.0.1
Version 1.0.0

14
Generate
Version 1.0.1 Version 1.0.2

15
Version 1.0.2
Apply both of
Major and Maintainance Rules
Version 1.0.1
Version 1.0.0
Configuration File 1.0.0 Configuration File 1.0.1

Rule Updates Reduce Median False Positives
by >49%
16
0.00-0.89
Precision 1.00
0.70-0.99
Maintainance
Major
0.00-0.39
Recall 1.00
0.88-0.99
Original
1.0.0 -> 1.0.2 1.0.1 -> 1.0.2

Rule Updates Reduce Median False Positives
by >49%
17
0.00-0.89
Precision 1.00
0.70-0.99
Maintainance
Major
0.00-0.39
Recall 1.00
0.88-0.99
Original
Rule updates for each major version are effective enough
1.0.0 -> 1.0.2 1.0.1 -> 1.0.2

Conclusion
• Linter-Maintainer traces the latest followed rules
• Linter-maintainer reduced false negatives by >49%
Future Plan
• Increasing evaluated languages and projects
• Improving rule selecting approach by using Git history
18
Install
Source
Unsupported Supported Evaluated
C/C++ Java JavaScript
Ruby Python
TypeScript

Automatically Customizing Static Analysis Tools to Coding Rules Really Followed by Developers

More Related Content

What's hot (20)

Similar to Automatically Customizing Static Analysis Tools to Coding Rules Really Followed by Developers (20)

Recently uploaded (20)

Automatically Customizing Static Analysis Tools to Coding Rules Really Followed by Developers

Editor's Notes