42- Using Templates to Elicit Implied Security Requirements from Functional Requirements − A Controlled Experiment

Using Templates to Elicit Implied Security Requirements from Functional Requirements − A Controlled Experiment
Maria Riaz, John Slankas, Jason King, Laurie Williams
Sep 18th, 2014
1

Agenda
•Motivation
•Prior Work
•Research Goal
•Experiment Design
•Results & Analysis
•Discussion
•Feedback
•Lessons Learned
•Contributions
2

Motivation
Security requirements challenges [Mellado10]
 Often overlooked due to lack of focus on security during early stages of system development
 Require sufficient level of security expertise Feedback from practitioners [Salini12]
Tool-assisted process to aid in security requirements elicitation may facilitate secure software development
3

Motivation
Systems that share security objectives, such as confidentiality and integrity, often have similar security requirements [Firesmith2003]
 Capture common security requirements that meet particular objectives as reusable templates
 Automatically suggest applicable templates based on the identified objectives from existing functional requirements
4

[ID & Authentication] Each user should be assigned a unique identifier that can be used for the purpose of authentication. [Confidentiality] The system shall enforce access privileges that enable HCP to modify or delete office visit. [Integrity] The system shall ensure that deletion of office visit is performed in accordance with the retention policy. [Accountability] The system shall log every time HCP modifies or deletes office visit. [Privacy] The system shall allow the owner of office visit to be notified when the office visit is modified or deleted by HCP.
[ID & Authentication] Each user should be assigned a unique identifier that can be used for the purpose of authentication.
[Confidentiality] The system shall enforce access privileges that enable HCP to modify or delete office visit.
[Integrity] The system shall ensure that deletion of office visit is performed in accordance with the retention policy.
[Accountability] The system shall log every time HCP modifies or deletes office visit.
[Privacy] The system shall allow the owner of office visit to be notified when the office visit is modified or deleted by HCP.
Prior Work [RE14] Security Discoverer (SD) Tool-assisted Process
5
•Input: Natural language requirements-related artifacts (requirements specification, use case scenarios, user stories)
•Output: Automatically suggested security requirements templates applicable for each input sentence
http://agile.csc.ncsu.edu/iTrust/wiki/doku.php?id=start
[ID & Authentication] Each user should be assigned a unique identifier that can be used for the purpose of authentication. [Confidentiality] The system shall enforce access privileges that enable <HCP> to <modify or delete> <office visit>. [Integrity: Delete Action] The system shall ensure that <deletion> of <office visit> is performed in accordance with the retention policy. [Accountability] The system shall log every time <HCP> <modifies or deletes> <office visit>. [Privacy] The system shall allow the owner of <office visit> to be notified when the <office visit> is <modified or deleted> by <HCP>.
“HCPs can return to an office visit and modify or delete the fields of the office visit.”
Subject
Resource
Action

Prior Work [RE14] Security Requirements Templates
6
 19 context-specific security requirements templates [Empirically derived from security-relevant sentences]

Research Goal
To improve the security requirements elicitation process by automatically suggesting appropriate security requirement templates implied by existing functional requirements.
7

Experiment Design Automatically Suggested Templates
8
•User Study Goal: Empirical evaluation of automatically- suggested security requirements templates in generating security requirements
•Task: Identify security requirements based on a given use case scenario
•Participants: 50 grad students [Software Security course]
•Groups: Between-subject design
 Treatment: assisted with automatically-suggested security requirements templates
 Control: manual, without assistance of suggested templates

Experiment Design Hypotheses
9
•Null Hypotheses
H01: The quality of elicited security requirements is unrelated to the use of automatically-suggested security requirements templates.
H02: The quantity of elicited security requirements is unrelated to the use of automatically-suggested security requirements templates.
H03: The relevance of elicited security requirements is unrelated to the use of automatically-suggested security requirements templates.
H04: The efficiency of the requirements elicitation process is unrelated to the use of automatically-suggested security requirements templates.

Experiment Design Metrics for Evaluation
Evaluation Criteria
Metric Type
Metric Used
Quality, of security requirements
Qualitative
Likert-like scale (1-5): lower score indicates lower quality
Quantity, of security requirements
Quantitative
Requirements coverage score: Percentage of security requirements covered in the oracle
Relevance, of security requirements
Quantitative
Relevance ratio: Ratio of relevant requirements to total requirements identified
Efficiency, of eliciting requirements
Quantitative
# of correct requirements identified per unit of time
10

Experiment Design Study Material
11
•All participants: 4 pages material, given 2 days in advance
 Textual clues that indicate when a security objective is implied
 40 example security requirements grouped by security objectives
•Additional material, treatment group: Example security requirements presented in two forms
16 reusable templates grouped by security objectives
 40 concrete example security requirements generated from the templates (also available to control group)
•Use cases: 2 use cases, electronic healthcare domain
 Document office visit (iTrust1 use case)
 Retrieve exam results by patient ID (VLER2 user story)
1http://agile.csc.ncsu.edu/iTrust/wiki/doku.php?id=requirements
2http://www.va.gov/vler/vlerdocs_userstories.asp

Experiment Design Task Screen – Treatment Group
12
Anonymous responses
Selected use case
sentence
Implied security objectives
Suggested templates
Selected templates
Track time on task
Links to ref material

Results & Analysis
Number of participants in each group
Use Case
Total
Requirements Process
UC1
UC2
Treatment
16
14
30
Control
10
10
20
Total
26
24
50
13
•Statistical analysis
 2x2 ANOVA
 Unbalanced number of participants in groups accounted for (Type III Sum of Squares)
•Balancing expertise among groups
 Participants randomly assigned to groups
Post-task survey: treatment and control groups balanced in terms of participants experience

Results & Analysis H01: Quality of Requirements
14
•Questions to guide quality assessment of requirements:
 Too general / too specific?
Logical inconsistencies?
 Necessary elements identified (subject, resource) / templates filled?
•Inter-rater agreement
 Moderate: weighted kappa = 0.588
 Paired t-test: difference in scores assigned by raters not statistically significant
•Finding
Treatment: 1/3rd did not fill the templates
Mean quality scores for each group (Range: 1-5)
Use Case
Overall
Req. Process
UC1
UC2
Treatment
2.78
2.96
2.87
Control
2.95
2.85
2.90
Overall
2.87
2.91
2.88
•Difference in group means not statistically significant at p <0.05
Treatment vs. Control (p-value=0.928)
UC1 vs. UC2 (p-value=0.891)

Results & Analysis H02: Quantity of Requirements
15
Mean coverage scores (%) for each group
Use Case
Overall
Req. Process
UC1
UC2
Treatment
24.26%
61.63%
42.95%
Control
12.45%
18.57%
15.51%
Overall
18.36%
40.1%
31.22%
•Statistically significant difference in group means at p <0.05
Treatment vs. Control (p-value<0.0001)
 Interaction (p-value=0.0054)
•Oracle of security requirements for evaluation:
 5 security researchers participated in creating the oracle
 Delphi method to generate consensus on applicable templates
•Findings
 Treatment group identified ~3 times as many requirements as control
 Overall, only a third of requirements in the oracle identified

Results & Analysis H03: Relevance of Requirements
16
•No statistically significant difference between group means at p <0.05
•Scoring:
 Ratio of relevant requirements to total requirements in each response
 Analogous to accuracy of response
•Findings
 Treatment group performed slightly better than control (90% vs. 85%)
 Overall, participants in both groups listed mostly relevant requirements
Mean ratio of relevant to total requirements for each group (%)
Use Case
Overall
Req. Process
UC1
UC2
Treatment
90
91
90
Control
86
84
85
Overall
88
87
88

Results & Analysis H04: Efficiency of Requirements Elicitation Process
17
•Statistically significant difference between group means at p <0.05
•Scoring:
 # of requirements identified per unit of time
 Measure of the process itself
•Findings
 Treatment group identified twice as many requirements per unit of time as control group
 More requirements for UC1 so more requirements to be identified
 Strong linear relation between task time and requirements identified (Pearson correlation coefficient=0.44)
Mean efficiency for each group (# of requirements identified per minute)
Use Case
Overall
Req. Process
UC1
UC2
Treatment
1.7
1
1.35
Control
1.15
0.4
0.77
Overall
1.43
0.7
1.14

Discussion
 Based on the results:
–Providing example security requirements helped all participants in improving quality of requirements
–Knowledge of security objectives and applicable templates led to higher coverage of security requirements
–Participants identified few (31%) but mostly relevant (88%) requirements
–Our process helped in identifying 2-3 times more security requirements than control in same time
18

Feedback on Templates
19
 Based on post-task survey, the templates
–Provide a good starting point
–Help in thinking about security
–Help in phrasing requirements
 Apply with caution
– Good but not exhaustive list
–People may tend to choose templates arbitrarily

Lessons Learned
 Based on design and execution of this study:
–Participation: Students willing to participate in a study when it is a relevant part of educational experience (50 of 54 enrolled students gave consent).
–Motivating diligent work: For motivating participants to produce quality responses, need further incentive than a participatory grade, and feedback on their performance.
–Task completeness: One-third participants did not fill in templates. Update tool to help in auto-filling templates.
20

Contributions
•Facilitate elicitation of a core set of security requirements, focus critical thinking around security concerns
–Empirical evaluation of use of security requirements templates in security requirements elicitation.
–Detailed study material and data to support future experimentation, replication and analysis.
–Lessons learned on how to improve security requirements elicitation.
21

References
[Firesmith03] Firesmith, D.G., 2003. Engineering Security Requirements. J. Object Technology 2, 1 (Jan-Feb.), 16. [Firesmith04] D. Firesmith, "Specifying Reusable Security Requirements," Jornal of Object Technology, vol. 3, p. 15, Jan- Feb. 2004. [Mellado10] D. Mellado, C. Blanco, L. E. Sánchez, and Fernández-Medina, E., 2010. A Systematic Review of Security Requirements Engineering. Computer Standards and Interfaces 32, 4 (Jun. ), 13 [McGraw06] G. McGraw. “Software Security: Building Security In”, Addison Wesley Professional, 2006. [RE14] Riaz, M., King, J., Slankas, J., and Williams, L., 2014. Hidden in Plain Sight: Automatically Identifying Security Requirements from Natural Language Artifacts. In Proceedings of the 22nd International Conference on Requirements Engineering (Karlskrona, Sweden, Aug 25-29 2014), IEEE. [Salini12] Salini, P. and Kanmani, S., 2012. Survey and Analysis on Security Requirements Engineering. Computers and Electrical Engineering 38, 13. [Schumacher06] M. Schumacher, E. Fernandez-Buglioni, D. Hyberston, F. Buschmann, and P. Sommerlad, Security Patterns: Integrating Security and Systems Engineering. West Sussex: John Wiley & Sons, Ltd, 2006. [Yskout12] Yskout, K., Scandariato, R., and Joosen, W., 2012. Does organizing security patterns focus architectural choices? In Proceedings of the International Conference on Software Engineering (ICSE '12) (Zurich, Switzerland, 2-9 June 2012), 617-627.
22

42- Using Templates to Elicit Implied Security Requirements from Functional Requirements − A Controlled Experiment

More Related Content

What's hot (20)

Viewers also liked (12)

Similar to 42- Using Templates to Elicit Implied Security Requirements from Functional Requirements − A Controlled Experiment (20)

More from ESEM 2014 (20)

Recently uploaded (20)

42- Using Templates to Elicit Implied Security Requirements from Functional Requirements − A Controlled Experiment