Improved Security Detection & Response via Optimized Alert Output: A Usability Study
0 likes463 views
The dissertation examines the impact of security alert output formats, specifically visual alert output (VAO) and text alert output (TAO), on the acceptance and effectiveness of security alerts among analysts, highlighting the issues of alert overload in organizations. Findings suggest that VAO is better accepted compared to TAO, particularly in terms of perceived usability, while no significant impact of TAO on the acceptance components was observed. Recommendations for future research include exploring analysts' preferences for visual alerts and the interactions with dynamic versus static visualizations.
Improved Security Detection & Response via Optimized Alert Output: A Usability Study
- 1. Improved Security Detection & Response via Optimized Alert Output: A Usability Study CapitolTechnology University Dissertation Defense by G. Russell McRee Dissertation Chair: Ian McAndrew PhD FRAeS Dissertation Committee: Dr. Atta-Ur-Rahman (Examiner), Allen H. Exner (Ex Officio) 17 AUG 2021
- 2. Statement of the Problem • Organizations risk data breach, loss of valuable human resources, reputation, and revenue due to excessive security alert volume and a lack of fidelity in security event data • These organizations face a large burden due to alert overload, where 99% of security professionals surveyed acknowledge that high volumes of security alerts are problematic
- 3. Rationale for the Study • This study addresses challenges inherent in data overload and complexity, using security data analytics derived from machine learning (ML) and data science models that produce alert output for analysts • Security analysts benefit in two ways: • Efficiency of results derived at scale via ML models • Benefit of quality alert results derived from the same models.
- 4. Literature Overview • Security data visualization can be used to address related human cognitive limitations (Rajivan, 2011) • Giacobe (2013) discussed the effectiveness of visual analytics and data fusion techniques on situation awareness in cyber-security, and focused on visual analytics, data fusion, and cybersecurity • Giacobe found that participants using the visual analytics (VA) interface performed better than those on the text-oriented interface, where the visual analytic interface yielded a performance that was quicker and more accurate that the text interface. • Giacobe conducted an experiment and survey separately • This study merged quasi-experiment in survey
- 5. Research Methodology/Design • Quantitative, quasi-experimental, explanatory study • TechnologyAcceptance Model (TAM) • Methodology utilized to statistically measure security analysts’ acceptance of two security alert output types: visual alert output (VAO) & text alert output (TAO) • A qualitative methodology & design was not considered as the business problem is one of data.The study’s data-driven findings can contribute to data-informed business decisions.
- 6. Data Analysis • DV: level of acceptance of the security alert output and is based on the four individual TAM components: PU, PEU, AU, and IU • Within-subjects IV: Scenario (3x), all participants subject to all scenarios • Between-subjects IV: Maximum Visual • Two levels: a preference forVAO in all three scenarios, and a preference forTAO in at least one of the scenarios • Mixed ANOVA to test level of acceptance of alert outputs as influenced by the within- subjects variable Scenario and the between-subjects variable Maximum Visual • Mann-Whitney U test performed to compare level of acceptance of alert outputs of the two levels of MaximumVisual • Friedman test performed to compare level of acceptance across the three scenarios
- 7. Findings (non-parametric) Significant difference (U = 863.5, p = 0.023) in level of acceptance of alert output between respondents who selected visual output across all scenarios (n = 59) compared to the respondents who provided mixed responses (n = 22). No significant difference between scenarios (𝑥^2 (2)=5.496, 𝑝< .064). Scenario mean ranks did not differ significantly from scenario to scenario when not also factoring for responses based on output preference (MaximumVisual).
- 8. Findings – Mixed ANOVA AllTAM measures (α = .05): a significant main effect of MaximumVisual scores (F(1, 79) = 4.111, p = .046, ηp2 = .049) on the level of acceptance of alert output as indicated by sum of participants' scores for allTAM components (PU, PEU, AU, and IU) between-subjects
- 9. Perceived Usability (α = .0125): a significant main effect of MaximumVisual scores (F(1, 79) = 7.643, p = .007, ηp2 = .088) on the level of acceptance of alert output as indicated by sum of participants' scores for Perceived Usability (PU) between-subjects Perceived Ease of Use (α = .0125): an insignificant main effect of MaximumVisual scores (F(1, 79) = .842, p = .362, ηp2 = .011) on the level of acceptance of alert output as indicated by sum of participants' scores for Perceived Ease of Use (PEU) between-subjects Findings: Mixed ANOVA
- 10. Findings: Mixed ANOVA AttitudeToward Using (α = .0125): an insignificant main effect of MaximumVisual scores (F(1, 79) = 4.566, p = .036, ηp2 = .055) on the level of acceptance of alert output as indicated by sum of participants' scores for Attitude Toward Using (AU) between-subjects Intention To Use (α = .0125): an insignificant main effect of MaximumVisual scores (F(1, 79) = 4.378, p = .040, ηp2 = .053) on the level of acceptance of alert output as indicated by sum of participants' scores for Intention to Use (IU) between-subjects
- 11. Findings – RQ1 • RQ1: Is there a difference in the level of acceptance of security alert output between those with a preference for visual alert outputs (VAO) and those with a preference for text alert outputs (TAO), withVAO andTAO generated via data science/machine learning methods, as predicted by the Technology Acceptance Model (TAM)? Yes. • Non-parametric (between-subjects): U = 863.5, p = 0.023 • Parametric: • Within-subjects: (F (1.455, 114.915) = 5.634, p = 0.010, ηp2 = .067) • Between-subjects: (F (1, 79) = 4.111, p = .046, ηp2 = .049)
- 12. Findings – SQ1 • SQ1: Does the adoption ofVAO have a significant impact on the four individualTAM components, perceived usefulness (PU), perceived ease of use (PEU), attitude toward using (AU), and intention to use (IU)? In part. • TheTAM components perceived usability (PU) and perceived ease of use (PEU) are not significantly influenced by the adoption ofVAO within-subjects while attitude toward using (AU), and intention to use (IU) are significantly influenced by the adoption ofVAO within-subjects. • TheTAM component perceived usability (PU) is significantly influenced by the adoption ofVAO between-subjects.
- 13. Findings – SQ2 • SQ2: Does the adoption ofTAO have a significant impact on the four individualTAM components, perceived usefulness (PU), perceived ease of use (PEU), attitude toward using (AU), and intention to use (IU)? No. • No individualTAM component is significantly influenced byTAO adoption, andTAO adoption trailedVAO in near totality.
- 14. Recommendations for Research • Security analysts likely seek an initial visual alert inclusive of the options to dive deeper into the raw data. A future study could expose the degree to which analysts seek multifaceted options • A future study could further explore the perceptions of, and interactions with, dynamic visualizations versus static visualizations • Further explore, even under online survey constraints, a framework that more robustly assesses user experience • Opportunity exists to develop more nuanced data where information specific to participant gender, location, age group, company or organization size, and business sector could lead to improved insights
- 15. Thank you Questions? Once in a while, you get shown the light In the strangest of places if you look at it right ~Garcia/Hunter