A Deep Dive into the Socio-Technical Aspects
of Delays in Security Patching
Ali Babar
CREST – Centre for Research on Engineering Software Technologies
Monash Cybersecurity Seminar, 18th October, 2022
Equifax – A Credit Assessment Service
NHS – A Health Service Agency
Optus – A Telco
Brief Bio
M. Ali Babar
• Cyber Security Cooperative Research Centre
(CSCRC) – Jemena, ACTewAGL, TCS, Cisco, NAB,
Defence SA, SA Health, Q-Labs, WA DGov, SA Gov,
TSS and ATO
• Lancaster – CPNI/GCHQ
• Denmark – Danish Strategic funding agency (5
industry partners)
• Lero, Ireland – focused on Irish industry scaling to
Robert Bosch, Finnish industry
• NICTA – CeBIT, DSTG, Mini MBA
• JRCASE - CSIRO-MacQ Uni – Linkage project
Trustworthy Digital Services
Data
Science
Integration &
Interoperability
Autonomy
Cyber
Security
Artificial
intelligence
Software System Engineering
DevOps
Peopl e, Pr oc es ses , and Tool s
MDE Design Space
Socio-
Technical
Technol ogi es
IoT / CPS Cloudlet Cloud Blockchain
Ap plications
Do mains
Health
Systems
AgFood
Systems
Defence
Systems
• Setting the context of our research
• Security Patch management as a socio-technical system
• Research questions stimulating our studies on this topic
• Methodological and logistical details
• Taxonomy of reasons for delays in security patching
• Strategies to avoid/minimize delays
• Takeaways for practitioners and researchers
5
Talk’s Roadmap
6
https://techcrunch.com/2019/05/12/wannacry-two-years-on/?guccounter
https://arstechnica.com/information-technology/2017/09/massive-equifax-breach-caused-by-failure-to-patch-two-month-old-bug/
https://www.nytimes.com/2020/09/18/world/europe/cyber-attack-germany-ransomeware-death.html?smid=tw-share
Setting the Context – Problem Statement
7
The application of security patches
to address the identified security
vulnerabilities in the software code
M. Souppaya, K. Scarfone, Guide to enterprise patch management technologies, NIST Special Publica-1060tion 800 (2013) 40.
S. Frei et al., Modeling the security ecosystem-the dynamics of (in) security, Economics of Information Security and Privacy, Springer, Boston, MA, 2010
Setting the Context - Security Patch Management
Retrieving new
patches
Deciding to patch
Preparing for
patch installation
Installing patches
Handling post-
deployment
issues
F. Li et al., Keepers of the machines: examining how system administrators manage software updates. USENIX Conference on Usable Privacy and Security, 2019.
C. Tiefenau et al., Security, Availability, and Multiple Information Sources: Exploring Update Behavior of System Administrators. In Sixteenth Symposium on Usable Privacy and Security ({SOUPS} 2020).
• Proactive search
for new patches
• Download patches
8
• Scan systems for
vulnerabilities
• Decide relevance to
managed systems
• Patch type and severity
• Organization
policies/compliance (e.g.,
Change Management)
• Patch pre-requisite
investigation
• Make backups or
snapshots
• Prepare machines (e.g.,
configurations, patch
dependencies)
• Patch testing
• Time update to
avoid disruptions
• Coordinate with
stakeholders
• Receive organization
approval
• Manual/automatic
deployment
• Monitor patch status
• Gather client feedback
• Decision by
organization policy
• Uninstall patch
• Revert to prior
software version
• Find workaround/
troubleshoot
Security Patch Management Process
• Emery & Trist coined the term,1960, socio-technical system: a system
with a complex interaction between humans, machines and context
• Such system has many interdependent internal and external parts to
be considered within their context
• Designers/engineers can avail more than one paths to reach the
system’s goals – design choices
• System performance is dependent upon a combination of technical
and social subsystems; ignore one results in poor outcomes
9
Security Patching as a Socio-Technical System
Baxter & Sommerville, From design methods to systems engineering, Interacting with Computers, 23 (2011) 4-17
• Why, how and where do delays occur in software security
patch management?
• How can the delays be mitigated?
• How are the interdependent activities coordinated?
• Is there any underlying theoretical model of coordinating
the socio-technical interactions/decisions?
• What are the automation needs and how to meet them?
• How would human-automation support work?
10
What did Stimulate our Studies on this Topic
Finding Some Answers
Methodological and Logistical Details
Observations
Post-meeting Discussions
• 51 patch meetings
• March 2020 - January 2021
• 2 organizations, 8 teams
• 90 min (avg), online meetings
Artifacts Analysis
• 11 discussions
• 30 - 45 mins (avg)
• Patch meeting minutes
• Patch mailing thread Figure: The studied context
Alpha Beta
EMR
Security
Windows
Non-
Windows
Change
Management
Pathology
Technical
(distributed teams)
Non-
Technical
Organisation-level interdependencies
Team-level interdependencies Team-level interdependencies
Internal stakeholder dependencies
External stakeholder dependencies
Customers
End-users
Customer dependencies
Vendor dependencies
Vendors
12
Methodological Details – Data Collection (1/2)
Methodological Details – Data Collection (2/2)
Analysis of
patching
tracker
Total no. of tasks = 232
No. of delayed tasks =
131 (56.5%)
Patch
Meetings
Patching
Tracker
mainartifact
EMR
Security
Win
Non-Win
Change
Management
Pathology
Server
FinanceandAudit
OrgA
13
OrgB
Glaser’s Grounded Theory data analysis procedure
Theory of the role of
coordination in software
security patch management
Selective
Coding
Theoretical
Coding
Observation and
discussion
transcripts
Open Coding
Key Points
Codes
Concepts
Categories
Core Category
14
Methodological Details – Data Analysis Stages
Methodological Details – Data Analysis Example
KeyPoints Concepts Sub-categories
Organisation-
relatedReasons
R
a
wdata
Reasonsfor
delays
OpenCoding AxialCoding
Categories
Codes
Technology-
relatedReasons Strategies
applied
Delaysin
securitypatch
management
Corecategory
SelectiveCoding
[L1TFspectreupdate]-WindowsID41
16/5/19- Task[T1]assignedto[P1-BT1]
forimplementation.
14/6/19 - This task is on hold due to
resourcebeingborrowedbythe[T6]team
Unavailabilityof
Resource
taskassignees
(workoverload) limitations
(human)
Performance Infrastructure Capacity
issues(server) limitations limitations
(hardware)
Periodicpatch
cycles(monthly Time
patchcycle) limitations
[Backupserverpatching]-E
M
RID49
24/1/20 - Backupserver patching failed
duetopatchloadimpactingserversbefore
reboot.Proposalsenttochangewindow.
[Storagefailoverissues]-E
M
RID43
07/02-Patchingfailedlastnight.Waiting
fornextfailuretologanothercaseto
vendor.NextpatchrundueinlateMarch.
People-related
Reasons
15
An Overview of the Key Aspects of Study
D e l a y s i n S o f t w a r e
S e c u r i t y P a t c h
M a n a g e m e n t
S e t o f R e a s o n s
c a u s e
S t r a t e g i e s
O v e r a l l p a t c h
m a n a g e m e n t p r o c e s s
a p p l i e d t o mi n i mi s e
a p p l i e d to a p p l i e d to
S p e c i f i c p a t c h
m a n a g e m e n t p r o c e s s
p h a s e
S 8 . D e f i n e c o m p l i a n c e p o l i c i e s a n d c o n t i n g e n c y
p l a n s
S 9 . P a t c h p r e - r e q u i s i t e s i n v e s t i g at i o n
S 1 0 . M o d i f y c o n f i g u r a t i o n s a n d d e p e n d e n c i e s
S 6 . P l a n a l t e r n a t i v e s f o r d e l a y e d p a t c h e s
S 7 . D e f i n e p rio rit ie s f o r v u l n e r a b i l i t y r e m e d i a t i o n
P 1
P 2
S 1 5 . E s t a b l i s h p o s t - d e p l o y m e n t v e rif icat ion
p r o c e d u r e s
S 1 6 . C o l l e c t i v e l y h a n d l e p o s t - d e p l o y m e n t i s s u e s
S 1 7 . D o c u m e n t d e p l o y m e n t s t a t u s o f e v e r y
p a t c h
P 3
S 1 1 . T i m e l y c o o r d i n a t i o n o f p a t c h s c h e d u l e s
S 1 2 . M a x i m i s e a va ila b ilit y: a p p l y w o r k a r o u n d s
S 1 3 . M i n i m i s e d a m a g e : s h if t t o m a n u a l
d e p l o y m e n t f o r c o m p l e x a n d f a u l t y p a t c h e s
S 1 4 . A g i l e d e p l o y m e n t : e x e c u t e c h a n g e s in
s m a l l it e ra t io n s P 4
P 5
S 5 . S e t s t ric t t i m e l i n e s f o r p a t c h d o w n l o a d
S 1 . F r e q u e n t c o m m u n i c a t i o n
S 2 . C o l l a b o r a t i v e d e ci si o n-
m a k i n g
S 3 . T a s k d e l e g a t i o n
S 4 . R e g u l a r l y u p d a t e p r o c e s s
d o c u m e n t a t i o n
R 1 . C o m p l e x i t y of
p a t c h e s
R 2 . L i m i t a t i o n s of
e x i s t i n g t o o l s
R 3 . C o o r d i n a t i o n d e l a y s
R 4 . D e l a y s i n i n p u t r e q u i r e m e n t s
R 5 . N e e d o f h u m a n i n t e r v e nt io n
R 6 . C a p a c i t y limit a t ions
R 7 . S e r v i c e a v a i la bi l i t y re s t ric t ions
R 8 . O r g a n i s a t i o n d e l a y s
R 9 . F a i l u r e s f r o m p o o r p l a n n i n g a n d e x e c u t i o n
T e c h n i c a l
R e a s o n s
S o c i o - T e c h n i c a l
R e a s o n s
P a t c h m a n a g e m e n t p r o c e s s p h a s e s
P 1 P a t c h I n f o r m a t i o n R e t r i e v a l P 2 V u l n e r a b i l i t y S c a n n i n g , A s s e s s m e n t & P rio rit is ation
P 3 P a t c h Te s t in g P 4 P a t c h D e p l o y m e n t P 5 P o s t - D e p l o y m e n t P a t c h V e rif icat ion
C o r e c a t e g o r y
16
C a t e g o r y
S u b - c a t e g o r y C o n c e p t
Why, How and Where of Delays in Software
Security Patch Management
A Taxonomy of the Identified Reasons for Delays
R e a s o n s f o r
d e l a y s
T e c h n o l o g y - r e l a t e d
R e a s o n s
R 1 . C o m p l e x i t y o f
p a t c h e s
R 2 . L i m i t a t i o n s o f
c u r r e n t t o o l s
P a t c h i n t e r d e p e n d e n c i e s ( 3 0 )
F a u l t y p a t c h e s ( 1 2 )
E x t e n s i v e m o n i t o r i n g f o r f a u l t y p a t c h f i x e s ( 8 )
P a t c h h e t e r o g e n e i t y ( 5 )
I n c r e a s i n g r a t e o f p a t c h r e l e a s e ( 2 )
L a c k o f a c c u r a c y ( 9 )
L a c k o f s c a l a b i l i t y ( 3 )
F u n c t i o n a l i t y l i m i t a t i o n s ( 7 )
T r o u b l e s h o o t i n g ( 2 3 )
M a n u a l p a t c h d e p l o y m e n t ( 1 7 )
D e c i s i o n a p p r o v a l s n e e d t h o r o u g h a s s e s s m e n t o f p a t c h i m p a c t ( 1 0 )
M a n u a l c o n f i g u r a t i o n s ( 6 )
D e l a y s i n o b t a i n i n g a p p r o v a l ( 3 7 )
L a c k o f a w a r e n e s s o f t a s k p r o g r e s s i o n ( 2 2 )
L a c k o f u n d e r s t a n d i n g o f r o l e s a n d r e s p o n s i b i l i t i e s ( 6 )
P o o r c o m m u n i c a t i o n a n d i n f o r m a t i o n m i s i n t e r p r e t a t i o n ( 5 )
M i s s i n g i n f o r m a t i o n d u e t o o v e r l o a d o f e m a i l s ( 2 )
D e l a y s i n o b t a i n i n g c u s t o m e r s ' a p p r o v a l ( 1 4 )
D e l a y s i n c o o r d i n a t i n g w i t h v e n d o r s f o r s u p p o r t ( 1 2 )
A d m i n i s t r a t i v e o v e r h e a d o f c o o r d i n a t i n g w i t h m u l t i p l e c u s t o m e r s ( 3 )
D e l e g a t i o n d e l a y s d u e t o c o n f l i c t s o f t a s k o w n e r s h i p ( 3 )
D e l a y s i n d e l i v e r i n g r e p o r t s ( 1 6 )
D e l a y s i n d e l i v e r i n g p a t c h s c h e d u l e i n f o r m a t i o n ( 1 5 )
D e l a y s i n p r o v i d i n g t e a m r e q u i r e m e n t s ( 4 )
D e l a y s i n p a t c h r e l e a s e b y v e n d o r s ( 1 1 )
D e l a y s i n p r o v i d i n g i n p u t f o r s u p p o r t c a s e s ( 8 )
M i s s i n g p a t c h p r e - r e q u i s i t e s d u r i n g i n s t a l l a t i o n ( 1 )
I n a c c u r a t e e s t i m a t e s o f p a t c h w i n d o w s ( 3 )
I n c o m p l e t e p a t c h d e p l o y m e n t ( 5 )
I n a d e q u a t e p o s t - p a t c h d e p l o y m e n t v e r i f i c a t i o n ( 6 )
D e l a y s i n g e t t i n g a p p r o v a l f r o m h i g h e r m a n a g e m e n t ( 1 8 )
D e l a y s d u e t o c h a n g e s i n c o m p a n y s c h e d u l e s ( 1 3 )
R e s o u r c e l i m i t a t i o n s ( e . g . , h u m a n r e s o u r c e s ) ( 2 4 )
I n f r a s t r u c t u r e l i m i t a t i o n s ( e . g . , p e r f o r m a n c e i s s u e s ) ( 1 1 )
P e o p l e - r e l a t e d
R e a s o n s
O r g a n i s a t i o n - r e l a t e d
R e a s o n s
C a t e g o r y S u b - c a t e g o r y C o n c e p t C o d e ( n u m b e r o f r e f e r e n c e s )
R 3 . N e e d o f h u m a n
i n t e r v e n t i o n
R 4 . C o o r d i n a t i o n
d e l a y s
R 5 . I n p u t
r e q u i r e m e n t d e l a y s
R 6 . F a i l u r e s d u e
t o p o o r p l a n n i n g
a n d e x e c u t i o n
R 7 . O r g a n i s a t i o n
d e l a y s
R 8 . C a p a c i t y
l i m i t a t i o n s
R 9 . S e r v i c e -
a v a i l a b i l i t y
r e s t r i c t i o n s
T i m e l i m i t a t i o n s ( e . g . , m o n t h l y p a t c h c y c l e s ) ( 5 )
I n a b i l i t y t o a l l o w s e r v i c e d o w n t i m e f r o m r e b o o t s ( 1 3 )
M u l t i r e b o o t s r e q u i r i n g l o n g e r a n d a d d i t i o n a l p a t c h w i n d o w s ( 8 )
C u s t o m e r r e q u e s t s t o p o s t p o n e p a t c h d e p l o y m e n t s c h e d u l e s ( 4 )
18
• Detecting and dealing with patch interdependencies – software,
hardware and firmware of new & legacy systems (1.5K Servers)
• Faulty patches causing unknown errors during patch testing,
deployment, and post-deployment
• Security patches usually require extensive post-deployment
monitoring to verify the fixes
• Frequent release of patches and their heterogeneity add to the
complexity of patches
19
TR1: Delays Caused by Complexity of Patches
• Lack of accuracy in the output of current tools (e.g., missing
some vulnerabilities during scanning, omitting patches during
patch deployment) – ASE 2022 paper (more details)
• Lack of scalability to handle diverse types of patches and their
features – disabling some of the tools’ functionalities
• Inability to detect patch compatibility and the lack of capability to
detect multi-reboot requirements
[Subject - Additional reboot required for .NET patching]
“7/2/20 - An investigation is needed around the number of required reboots for EMR patching and window requirements as a
result if more reboots are required. A new process needs to be fleshed out when patching is postponed to accommodate the
identification of the number of reboots required." - EMR, Task ID 35
20
TR2: Delays Caused by Limitations of Tools
• Human intervention emerges as full automation not available or
desirable – faulty patches causing unknown errors
• Manual configurations for selecting a suitable Group Policy
Object (GPO) configurations to avoid breakdowns
• Human support may be needed for deploying complex, erroneous
or business-critical patch installations, e.g., legacy systems
• Manual intervention for re-executing failed patch deployments
and re-planning patch schedules due to requirement changes.
21
31/10/19 - [B-T1] team putting in significant amounts of work, like 15-20 hours per month, to redo the schedules on custom
dates each time the deployments move off standard windows." - EMR, Task ID 30
TR3: Delays Caused by Need of Human Intervention
• A single patching task usually involves multiple interdependent
activities and several stakeholders – internal and external
• A lack of awareness of task progression and of understanding of
shared roles and responsibilities
• Email based communication about patches may result in lost
and/or misinterpretation of critical information
• Customers, End-users, Vendors inefficiently coordinating for
seeking and giving approvals for system downtime & verification
• FSE 2021 paper (More details)
22
PR4: Delays Caused by Coordination Issues
• Tightly coupled activities have input requirements be fulfilled in
timely fashion, e.g., vulnerability scan reports or prioritisation
• Non-delivery or incomplete delivery of the schedule-related
information resulting in poor planning for deploying patches
• No online repository for maintaining servers’ patching details
• Delays in receiving vendor’s support for patching errors and new
patch release information
[Subject - New zero-day vulnerability warning]
“12/6/20 - Monitor Microsoft patch release for critical vulnerability identified on [T1] servers. Font Type 1 expected as a zero-day
soon, full report not available yet.
24/7/20 - No update from Microsoft." - EMR, Task ID 43
23
PR5: Delays Caused by Input Requirements Issues
• Security patch management needs meticulous planning and
flawless execution to avoid system breakdowns
• Inaccurate estimates of patch windows may result in calling off
the whole process as mission cannot put on hold
• Unforeseen errors can become major risks to deploying within the
planned time frame if not considered during planning
• Incomplete patch deployment or insufficient verification needing
re-execution of patch deployment and operational disruption
24
PR6: Delays Caused by Poor Planning and Execution
• Ensuring full compliance with the organisational policies and
obtaining management approval for monthly patch schedules
• Changes in organisation schedules such as change freeze
periods, testing schedules like regression testing plans and
shutdown periods
25
[Subject - Patching for December 2019]
“18/10/19 - OOB for November patching from 4th December instead of December patching. 31/10/19 - [AT1] patching for December
month is off but November Microsoft patches will be applied in the first week of December instead to keep compliance up." - EMR,
Issue ID 29
OR7: Delays Caused by Policy & Procedures
• Lack/unavailability of qualified personnel experienced in handling
specific systems for patching, e.g., legacy system upgrades
• Insufficient infrastructure resources - hardware and network
limitations may hinder a patching task
• Testing the workarounds for failed deployments delayed for
weeks given the time-driven (i.e., monthly) patch cycle
26
• “24/1/20 - Patching cannot go ahead when the active backup is running. The patch load can impact servers before reboot.
Need a window change, proposal to be sent by [P1-BT1] to [P2-AT1]." - EMR, Issue ID 39
OR8: Delays Caused by Capacity Limitations
• Organisations’ inability to allow service downtime from reboots
required for patches to take effect after deployment
• Multi reboots requirements difficult to get prompt approvals out of
fear of service disruptions from longer patch windows
• Customers reluctant to agree to sufficiently large patch window;
rather requesting service continuity at all cost
27
[Subject - [Servers s1 and s2] patching]
“26/7/19 - OOB window is needed for the multi reboots to catch up.
9/8/19 - Waiting for the customer’s confirmation of the new patch window, pending information from [P1-AT1]." - EMR, Issue ID 20
OR9: Delays Caused by Service Level Agreements
Frequency of Reasons for Delays
28
Distribution of the Delays over the Process
29
Strategies for Avoiding/Minimizing Delays
An Overview of the Used Strategies
Common strategies relating to the overall patch management process
S1. Frequent communication (24)
S2. Collaborative decision-making (3)
S3. Task delegation (31)
S4. Regularly review and update patch management process-related documentation (3)
31
Strategies relating
to Patch Information
Retrieval (P1)
Strategies relating
to Vulnerability
Scanning, Assessment
& Prioritisation (P2)
Strategies relating
to Patch Testing (P3)
Strategies relating
to Patch
Deployment (P4)
S11. Timely coordination
of patch deployment
schedules (19)
S12. Apply workarounds
to maximise service
availability (18)
S13. Manual deployment
for complex patches to
minimise damage (12)
S14. Agile deployment
for executing changes (6)
Strategies relating
to Post-Deployment
Patch Verification (P5)
S15. Establish post-
deployment verification
procedures (10)
S16. Collectively handle
post-deployment issues
(9)
S17. Document
deployment status of
every patch (3)
S8. Define compliance
policies and contingency
plans for test failures (9)
S9. Patch pre-requisites
investigation (4)
S10. Modify software
configurations and
dependencies (3)
S6. Plan alternatives for
delayed patches (6)
S7. Define priorities for
vulnerability remediation
(15)
S5. Set strict timelines
for patch download (2)
• Setting tight timelines for patch download, e.g., within two days of
the release of patch by the relevant vendors
• Acquiring and analysing the list of the retrieved patches each
month before assessment and prioritisation of vulnerability
32
[Subject - Provide .NET report at the start of the patch cycle]
“15/3/19 - Org A requests BT1 to provide an extract of .NET released patches every month and a report including what patches will
be applied to what servers." - EMR, Task ID 53
Strategies – Patch Information Retrieval
• Plan alternatives for scheduled patching based on assessment of
the impact on other services and risks of cyber attacks
• Prioritised security patches based on the global vulnerability
rating; High-risk vulnerabilities fixed within 48-72 hours
• Defining priorities for patching vulnerability for reducing the risk of
exploitable attack vectors used successfully
• Prioritisation based on patch type – Patching OS earlier
[Subject - OS security patches need to be tracked separately in the vulnerability remediation]
“15/5/20 - [P1-AT1] requesting the OS security patches to be tracked separately from all other vulnerability remediation. Org B’s
report should only be addressing OS security patches anyway but can make sure to separate any non-OS remediation tasks." -
EMR, Task ID 45
33
Strategies – Vulnerabilities and Assessment
• Test run compliance policies and standards, e.g., reboot every
legacy server without patching; have contingency plans
• Allow a specific time to identify and modify the dependencies and
configurations during patch testing
• Investigate the availability of prerequisites for the patches
released every month as a separate task during patch testing
34
[Subject - Registry key missing for Knowledge Base (KB) ID [n] (LDAP)]
“2/10/20 - Patches not installed on [servers s1 and s2] due to missing a registry key. [P1-BT1] to check settings and apply where
missing." - Win, Task ID 24
Strategies – Patch Testing
• Clustering similar patches to reduce the time spent in testing,
deployment and rebooting
• Balancing workload on servers during patch deployment to avoid
unnecessary service disruptions
• Backup servers concurrently running the critical services while
being rebooted - Backup servers patched separately
• Manually patching business-critical servers having multiple
version dependencies, multi reboots and legacy systems
• Agile deployment - execute the changes in small iterations
35
Strategies – Patch Deployment
• Define a set of procedures for post-deployment patch verification
to reduce the risk of delays caused by poor execution
• Monitor a patched system for functional, performance or
unexpected issues; getting periodic scans to verify patching
• Collaborative problem handling for analysing the root causes for
post-deployment issues and finding workarounds
• Develop a knowledge base to keep track of every patch as a
reference in cases of errors encountered during the execution
[Subject - Automated second rescan for reboots]
“31/10/19 - [P1-BT1] raised this issue, he has configured the window to rescan for missing patches and conduct a second reboot
if required. No issues during patching, seeking client feedback for verification." - EMR, Task ID 28 36
Strategies – Post-Deployment Patch Verification
• Frequent communication reduces delays, strengthens
collaboration and improves mutual understanding
• Collectively making decisions about patch management helps
teams gain insight into plans, activities and alternatives
• Well-defined roles and responsibilities around patch management
activities resulting in delegation and accountability for actions
• Systematically and regularly review and update documents of
patch actions and decisions; test execute process changes
“13/12/19 - Finalising the documentation after testing internally for handover to 24x7. 10/1/20 - Documentation to be tested in
February, will be ready for handover in March." - EMR, Task ID 24
37
Strategies – Overall Patch Management Process
Takeaways for Practitioners & Researchers
• An understanding of the reasons for delays can enable security
staff to take measures for mitigating the potential delays
• Provided knowledge can help practitioners in suitable decision-
making, prioritisation and planning of patch management tasks
• Strategies can guide practitioners and organisations in better
planning and taking actions to mitigate the impact of the delays
• Developing new and/or innovative use of existing tools for
visualising dependencies, patch management knowledge
repositories, timely communication and collaboration
39
For Practitioners
• Our findings are context and domain specific (healthcare) –
extending and adapting case study for different domains
• Developing and executing interview guides and surveys to verify
the findings and discover variations
• Research on AI-based tools for detecting patch mismatches,
improved coordination across patching tasks and reducing delays
• Investigating the suitability of “human-AI collaboration" for
security patch management
• Evaluating the performance and accuracy of available tools
40
For Researchers
Nesara Dissanayake
@nesara_d
Asangi Jayatilaka
@DrAsangiJ
Mansooreh Zahedi
@MansoorehZ
Muhammad Ali Babar
@alibabar
Centre for Research on Engineering Software Technologies (CREST - @crest_uofa)
School of Computer Science, The University of Adelaide, Australia
The Research Team
Acknowledgements
• This talk is based on the research studies carried out by Nesara
Dissanayake, M. Ali Babar, Mansooreh Zahedi, Asangi Asangi Jayatilaka
• Partially Funded by the University of Adelaide and the CREST
• SA Health provided the access to the case studies
• We are grateful to the participants for enabling to collect a variety of data
for our research
42
CRICOS 00123M
Contact: Ali Babar
ali.babar@adelaide.edu.au

More Related Content

PDF
Security Education and Training1111.pdf
PDF
Overview of Identity and Access Management Product Line
PPTX
Chapter 8: Communications and Operations Security
PDF
How to implement effective ITSM System
PPTX
SLFC Healthcare APSS
PPTX
MS. Cybersecurity Reference Architecture
PPT
Cyber crime with privention
PDF
Paul C Brown S O A Governance
Security Education and Training1111.pdf
Overview of Identity and Access Management Product Line
Chapter 8: Communications and Operations Security
How to implement effective ITSM System
SLFC Healthcare APSS
MS. Cybersecurity Reference Architecture
Cyber crime with privention
Paul C Brown S O A Governance

Similar to A Deep Dive into the Socio-Technical Aspects of Delays in Security Patching (20)

PDF
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
PDF
An Exploration: Moving Your Enterprise to a Cloud Collaboration
PDF
LO2_CA(CL)_IT_(New)_22nd Batch (Section-B)_By Md.Monowar Hossain FCA,CISA_Par...
PDF
CYBER DEFENCE SCENARIOS - Part 2: Building The Blue Team
PDF
7 Mistakes of IT Security Compliance - and Steps to Avoid Them
PPTX
Enforcing SharePoint Governance
PDF
White Paper: Aligning application security and compliance
PDF
BetterCloud Whitepaper: Fixing IT's Blindspots – 8 Critical Security and Mana...
PDF
E-Mail Compliance Frameworks in the Real World
PPTX
Pronet for slideshare
PDF
Creating effective security controls
PDF
ReSoft Buyers Guide
PDF
Security Checkpoints in Agile SDLC
PDF
Upgrading OpenStack? Avoid these 3 Common Pitfalls
PPTX
Integrating security into Continuous Delivery
PDF
Emerging Trends in Information Privacy and Security
PDF
Emerging Trends in Information Privacy and Security
PDF
Service now vulnerability patching_move
DOCX
Many companies and agencies conduct IT audits to test and assess the.docx
PDF
Novell Access Governance Suite
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
An Exploration: Moving Your Enterprise to a Cloud Collaboration
LO2_CA(CL)_IT_(New)_22nd Batch (Section-B)_By Md.Monowar Hossain FCA,CISA_Par...
CYBER DEFENCE SCENARIOS - Part 2: Building The Blue Team
7 Mistakes of IT Security Compliance - and Steps to Avoid Them
Enforcing SharePoint Governance
White Paper: Aligning application security and compliance
BetterCloud Whitepaper: Fixing IT's Blindspots – 8 Critical Security and Mana...
E-Mail Compliance Frameworks in the Real World
Pronet for slideshare
Creating effective security controls
ReSoft Buyers Guide
Security Checkpoints in Agile SDLC
Upgrading OpenStack? Avoid these 3 Common Pitfalls
Integrating security into Continuous Delivery
Emerging Trends in Information Privacy and Security
Emerging Trends in Information Privacy and Security
Service now vulnerability patching_move
Many companies and agencies conduct IT audits to test and assess the.docx
Novell Access Governance Suite
Ad

More from CREST (20)

PDF
Mobile Devices: Systemisation of Knowledge about Privacy Invasion Tactics and...
PPTX
Making Software and Software Engineering visible
PPTX
Understanding and Addressing Architectural Challenges of Cloud- Based Systems
PPTX
DevSecOps: Continuous Engineering with Security by Design: Challenges and Sol...
PPTX
Mining Software Repositories for Security: Data Quality Issues Lessons from T...
PPTX
A Decentralised Platform for Provenance Management of Machine Learning Softwa...
PPTX
Privacy Engineering: Enabling Mobility of Mental Health Services with Data Pr...
PPTX
Falling for Phishing: An Empirical Investigation into People's Email Response...
PPTX
An Experience Report on the Design and Implementation of an Ad-hoc Blockchain...
PPTX
Gazealytics: A Unified and Flexible Visual Toolkit for Exploratory and Compar...
PPTX
Detecting Misuses of Security APIs: A Systematic Review
PPTX
Chen_Reading Strategies for Graph Visualizations that Wrap Around in Torus To...
PPTX
Data Quality for Software Vulnerability Dataset
PPTX
Mod2Dash Presentation
PDF
Run-time Patching and updating Impact Estimation
PDF
ECSA 2023 Ubuntu Case Study
PDF
Energy Efficiency Evaluation of Local and Offloaded Data Processing
PPTX
Designing Quality-Driven Blockchain Networks
PPTX
Privacy Engineering in the Wild
PPTX
Security Data Quality Challenges
Mobile Devices: Systemisation of Knowledge about Privacy Invasion Tactics and...
Making Software and Software Engineering visible
Understanding and Addressing Architectural Challenges of Cloud- Based Systems
DevSecOps: Continuous Engineering with Security by Design: Challenges and Sol...
Mining Software Repositories for Security: Data Quality Issues Lessons from T...
A Decentralised Platform for Provenance Management of Machine Learning Softwa...
Privacy Engineering: Enabling Mobility of Mental Health Services with Data Pr...
Falling for Phishing: An Empirical Investigation into People's Email Response...
An Experience Report on the Design and Implementation of an Ad-hoc Blockchain...
Gazealytics: A Unified and Flexible Visual Toolkit for Exploratory and Compar...
Detecting Misuses of Security APIs: A Systematic Review
Chen_Reading Strategies for Graph Visualizations that Wrap Around in Torus To...
Data Quality for Software Vulnerability Dataset
Mod2Dash Presentation
Run-time Patching and updating Impact Estimation
ECSA 2023 Ubuntu Case Study
Energy Efficiency Evaluation of Local and Offloaded Data Processing
Designing Quality-Driven Blockchain Networks
Privacy Engineering in the Wild
Security Data Quality Challenges
Ad

Recently uploaded (20)

PPTX
WiFi Honeypot Detecscfddssdffsedfseztor.pptx
PDF
The Dynamic Duo Transforming Financial Accounting Systems Through Modern Expe...
PPTX
Weekly report ppt - harsh dattuprasad patel.pptx
PDF
Product Update: Alluxio AI 3.7 Now with Sub-Millisecond Latency
PDF
Types of Token_ From Utility to Security.pdf
PPTX
CNN LeNet5 Architecture: Neural Networks
PPTX
Patient Appointment Booking in Odoo with online payment
PPTX
Oracle Fusion HCM Cloud Demo for Beginners
PDF
Ableton Live Suite for MacOS Crack Full Download (Latest 2025)
PPTX
Cybersecurity: Protecting the Digital World
PDF
How Tridens DevSecOps Ensures Compliance, Security, and Agility
PPTX
Computer Software - Technology and Livelihood Education
PDF
Salesforce Agentforce AI Implementation.pdf
PPTX
assetexplorer- product-overview - presentation
PPTX
Introduction to Windows Operating System
PDF
iTop VPN Crack Latest Version Full Key 2025
PDF
AI Guide for Business Growth - Arna Softech
PDF
How to Make Money in the Metaverse_ Top Strategies for Beginners.pdf
PPTX
Log360_SIEM_Solutions Overview PPT_Feb 2020.pptx
PPTX
Computer Software and OS of computer science of grade 11.pptx
WiFi Honeypot Detecscfddssdffsedfseztor.pptx
The Dynamic Duo Transforming Financial Accounting Systems Through Modern Expe...
Weekly report ppt - harsh dattuprasad patel.pptx
Product Update: Alluxio AI 3.7 Now with Sub-Millisecond Latency
Types of Token_ From Utility to Security.pdf
CNN LeNet5 Architecture: Neural Networks
Patient Appointment Booking in Odoo with online payment
Oracle Fusion HCM Cloud Demo for Beginners
Ableton Live Suite for MacOS Crack Full Download (Latest 2025)
Cybersecurity: Protecting the Digital World
How Tridens DevSecOps Ensures Compliance, Security, and Agility
Computer Software - Technology and Livelihood Education
Salesforce Agentforce AI Implementation.pdf
assetexplorer- product-overview - presentation
Introduction to Windows Operating System
iTop VPN Crack Latest Version Full Key 2025
AI Guide for Business Growth - Arna Softech
How to Make Money in the Metaverse_ Top Strategies for Beginners.pdf
Log360_SIEM_Solutions Overview PPT_Feb 2020.pptx
Computer Software and OS of computer science of grade 11.pptx

A Deep Dive into the Socio-Technical Aspects of Delays in Security Patching

  • 1. A Deep Dive into the Socio-Technical Aspects of Delays in Security Patching Ali Babar CREST – Centre for Research on Engineering Software Technologies Monash Cybersecurity Seminar, 18th October, 2022
  • 2. Equifax – A Credit Assessment Service NHS – A Health Service Agency Optus – A Telco
  • 3. Brief Bio M. Ali Babar • Cyber Security Cooperative Research Centre (CSCRC) – Jemena, ACTewAGL, TCS, Cisco, NAB, Defence SA, SA Health, Q-Labs, WA DGov, SA Gov, TSS and ATO • Lancaster – CPNI/GCHQ • Denmark – Danish Strategic funding agency (5 industry partners) • Lero, Ireland – focused on Irish industry scaling to Robert Bosch, Finnish industry • NICTA – CeBIT, DSTG, Mini MBA • JRCASE - CSIRO-MacQ Uni – Linkage project
  • 4. Trustworthy Digital Services Data Science Integration & Interoperability Autonomy Cyber Security Artificial intelligence Software System Engineering DevOps Peopl e, Pr oc es ses , and Tool s MDE Design Space Socio- Technical Technol ogi es IoT / CPS Cloudlet Cloud Blockchain Ap plications Do mains Health Systems AgFood Systems Defence Systems
  • 5. • Setting the context of our research • Security Patch management as a socio-technical system • Research questions stimulating our studies on this topic • Methodological and logistical details • Taxonomy of reasons for delays in security patching • Strategies to avoid/minimize delays • Takeaways for practitioners and researchers 5 Talk’s Roadmap
  • 7. 7 The application of security patches to address the identified security vulnerabilities in the software code M. Souppaya, K. Scarfone, Guide to enterprise patch management technologies, NIST Special Publica-1060tion 800 (2013) 40. S. Frei et al., Modeling the security ecosystem-the dynamics of (in) security, Economics of Information Security and Privacy, Springer, Boston, MA, 2010 Setting the Context - Security Patch Management
  • 8. Retrieving new patches Deciding to patch Preparing for patch installation Installing patches Handling post- deployment issues F. Li et al., Keepers of the machines: examining how system administrators manage software updates. USENIX Conference on Usable Privacy and Security, 2019. C. Tiefenau et al., Security, Availability, and Multiple Information Sources: Exploring Update Behavior of System Administrators. In Sixteenth Symposium on Usable Privacy and Security ({SOUPS} 2020). • Proactive search for new patches • Download patches 8 • Scan systems for vulnerabilities • Decide relevance to managed systems • Patch type and severity • Organization policies/compliance (e.g., Change Management) • Patch pre-requisite investigation • Make backups or snapshots • Prepare machines (e.g., configurations, patch dependencies) • Patch testing • Time update to avoid disruptions • Coordinate with stakeholders • Receive organization approval • Manual/automatic deployment • Monitor patch status • Gather client feedback • Decision by organization policy • Uninstall patch • Revert to prior software version • Find workaround/ troubleshoot Security Patch Management Process
  • 9. • Emery & Trist coined the term,1960, socio-technical system: a system with a complex interaction between humans, machines and context • Such system has many interdependent internal and external parts to be considered within their context • Designers/engineers can avail more than one paths to reach the system’s goals – design choices • System performance is dependent upon a combination of technical and social subsystems; ignore one results in poor outcomes 9 Security Patching as a Socio-Technical System Baxter & Sommerville, From design methods to systems engineering, Interacting with Computers, 23 (2011) 4-17
  • 10. • Why, how and where do delays occur in software security patch management? • How can the delays be mitigated? • How are the interdependent activities coordinated? • Is there any underlying theoretical model of coordinating the socio-technical interactions/decisions? • What are the automation needs and how to meet them? • How would human-automation support work? 10 What did Stimulate our Studies on this Topic
  • 11. Finding Some Answers Methodological and Logistical Details
  • 12. Observations Post-meeting Discussions • 51 patch meetings • March 2020 - January 2021 • 2 organizations, 8 teams • 90 min (avg), online meetings Artifacts Analysis • 11 discussions • 30 - 45 mins (avg) • Patch meeting minutes • Patch mailing thread Figure: The studied context Alpha Beta EMR Security Windows Non- Windows Change Management Pathology Technical (distributed teams) Non- Technical Organisation-level interdependencies Team-level interdependencies Team-level interdependencies Internal stakeholder dependencies External stakeholder dependencies Customers End-users Customer dependencies Vendor dependencies Vendors 12 Methodological Details – Data Collection (1/2)
  • 13. Methodological Details – Data Collection (2/2) Analysis of patching tracker Total no. of tasks = 232 No. of delayed tasks = 131 (56.5%) Patch Meetings Patching Tracker mainartifact EMR Security Win Non-Win Change Management Pathology Server FinanceandAudit OrgA 13 OrgB
  • 14. Glaser’s Grounded Theory data analysis procedure Theory of the role of coordination in software security patch management Selective Coding Theoretical Coding Observation and discussion transcripts Open Coding Key Points Codes Concepts Categories Core Category 14 Methodological Details – Data Analysis Stages
  • 15. Methodological Details – Data Analysis Example KeyPoints Concepts Sub-categories Organisation- relatedReasons R a wdata Reasonsfor delays OpenCoding AxialCoding Categories Codes Technology- relatedReasons Strategies applied Delaysin securitypatch management Corecategory SelectiveCoding [L1TFspectreupdate]-WindowsID41 16/5/19- Task[T1]assignedto[P1-BT1] forimplementation. 14/6/19 - This task is on hold due to resourcebeingborrowedbythe[T6]team Unavailabilityof Resource taskassignees (workoverload) limitations (human) Performance Infrastructure Capacity issues(server) limitations limitations (hardware) Periodicpatch cycles(monthly Time patchcycle) limitations [Backupserverpatching]-E M RID49 24/1/20 - Backupserver patching failed duetopatchloadimpactingserversbefore reboot.Proposalsenttochangewindow. [Storagefailoverissues]-E M RID43 07/02-Patchingfailedlastnight.Waiting fornextfailuretologanothercaseto vendor.NextpatchrundueinlateMarch. People-related Reasons 15
  • 16. An Overview of the Key Aspects of Study D e l a y s i n S o f t w a r e S e c u r i t y P a t c h M a n a g e m e n t S e t o f R e a s o n s c a u s e S t r a t e g i e s O v e r a l l p a t c h m a n a g e m e n t p r o c e s s a p p l i e d t o mi n i mi s e a p p l i e d to a p p l i e d to S p e c i f i c p a t c h m a n a g e m e n t p r o c e s s p h a s e S 8 . D e f i n e c o m p l i a n c e p o l i c i e s a n d c o n t i n g e n c y p l a n s S 9 . P a t c h p r e - r e q u i s i t e s i n v e s t i g at i o n S 1 0 . M o d i f y c o n f i g u r a t i o n s a n d d e p e n d e n c i e s S 6 . P l a n a l t e r n a t i v e s f o r d e l a y e d p a t c h e s S 7 . D e f i n e p rio rit ie s f o r v u l n e r a b i l i t y r e m e d i a t i o n P 1 P 2 S 1 5 . E s t a b l i s h p o s t - d e p l o y m e n t v e rif icat ion p r o c e d u r e s S 1 6 . C o l l e c t i v e l y h a n d l e p o s t - d e p l o y m e n t i s s u e s S 1 7 . D o c u m e n t d e p l o y m e n t s t a t u s o f e v e r y p a t c h P 3 S 1 1 . T i m e l y c o o r d i n a t i o n o f p a t c h s c h e d u l e s S 1 2 . M a x i m i s e a va ila b ilit y: a p p l y w o r k a r o u n d s S 1 3 . M i n i m i s e d a m a g e : s h if t t o m a n u a l d e p l o y m e n t f o r c o m p l e x a n d f a u l t y p a t c h e s S 1 4 . A g i l e d e p l o y m e n t : e x e c u t e c h a n g e s in s m a l l it e ra t io n s P 4 P 5 S 5 . S e t s t ric t t i m e l i n e s f o r p a t c h d o w n l o a d S 1 . F r e q u e n t c o m m u n i c a t i o n S 2 . C o l l a b o r a t i v e d e ci si o n- m a k i n g S 3 . T a s k d e l e g a t i o n S 4 . R e g u l a r l y u p d a t e p r o c e s s d o c u m e n t a t i o n R 1 . C o m p l e x i t y of p a t c h e s R 2 . L i m i t a t i o n s of e x i s t i n g t o o l s R 3 . C o o r d i n a t i o n d e l a y s R 4 . D e l a y s i n i n p u t r e q u i r e m e n t s R 5 . N e e d o f h u m a n i n t e r v e nt io n R 6 . C a p a c i t y limit a t ions R 7 . S e r v i c e a v a i la bi l i t y re s t ric t ions R 8 . O r g a n i s a t i o n d e l a y s R 9 . F a i l u r e s f r o m p o o r p l a n n i n g a n d e x e c u t i o n T e c h n i c a l R e a s o n s S o c i o - T e c h n i c a l R e a s o n s P a t c h m a n a g e m e n t p r o c e s s p h a s e s P 1 P a t c h I n f o r m a t i o n R e t r i e v a l P 2 V u l n e r a b i l i t y S c a n n i n g , A s s e s s m e n t & P rio rit is ation P 3 P a t c h Te s t in g P 4 P a t c h D e p l o y m e n t P 5 P o s t - D e p l o y m e n t P a t c h V e rif icat ion C o r e c a t e g o r y 16 C a t e g o r y S u b - c a t e g o r y C o n c e p t
  • 17. Why, How and Where of Delays in Software Security Patch Management
  • 18. A Taxonomy of the Identified Reasons for Delays R e a s o n s f o r d e l a y s T e c h n o l o g y - r e l a t e d R e a s o n s R 1 . C o m p l e x i t y o f p a t c h e s R 2 . L i m i t a t i o n s o f c u r r e n t t o o l s P a t c h i n t e r d e p e n d e n c i e s ( 3 0 ) F a u l t y p a t c h e s ( 1 2 ) E x t e n s i v e m o n i t o r i n g f o r f a u l t y p a t c h f i x e s ( 8 ) P a t c h h e t e r o g e n e i t y ( 5 ) I n c r e a s i n g r a t e o f p a t c h r e l e a s e ( 2 ) L a c k o f a c c u r a c y ( 9 ) L a c k o f s c a l a b i l i t y ( 3 ) F u n c t i o n a l i t y l i m i t a t i o n s ( 7 ) T r o u b l e s h o o t i n g ( 2 3 ) M a n u a l p a t c h d e p l o y m e n t ( 1 7 ) D e c i s i o n a p p r o v a l s n e e d t h o r o u g h a s s e s s m e n t o f p a t c h i m p a c t ( 1 0 ) M a n u a l c o n f i g u r a t i o n s ( 6 ) D e l a y s i n o b t a i n i n g a p p r o v a l ( 3 7 ) L a c k o f a w a r e n e s s o f t a s k p r o g r e s s i o n ( 2 2 ) L a c k o f u n d e r s t a n d i n g o f r o l e s a n d r e s p o n s i b i l i t i e s ( 6 ) P o o r c o m m u n i c a t i o n a n d i n f o r m a t i o n m i s i n t e r p r e t a t i o n ( 5 ) M i s s i n g i n f o r m a t i o n d u e t o o v e r l o a d o f e m a i l s ( 2 ) D e l a y s i n o b t a i n i n g c u s t o m e r s ' a p p r o v a l ( 1 4 ) D e l a y s i n c o o r d i n a t i n g w i t h v e n d o r s f o r s u p p o r t ( 1 2 ) A d m i n i s t r a t i v e o v e r h e a d o f c o o r d i n a t i n g w i t h m u l t i p l e c u s t o m e r s ( 3 ) D e l e g a t i o n d e l a y s d u e t o c o n f l i c t s o f t a s k o w n e r s h i p ( 3 ) D e l a y s i n d e l i v e r i n g r e p o r t s ( 1 6 ) D e l a y s i n d e l i v e r i n g p a t c h s c h e d u l e i n f o r m a t i o n ( 1 5 ) D e l a y s i n p r o v i d i n g t e a m r e q u i r e m e n t s ( 4 ) D e l a y s i n p a t c h r e l e a s e b y v e n d o r s ( 1 1 ) D e l a y s i n p r o v i d i n g i n p u t f o r s u p p o r t c a s e s ( 8 ) M i s s i n g p a t c h p r e - r e q u i s i t e s d u r i n g i n s t a l l a t i o n ( 1 ) I n a c c u r a t e e s t i m a t e s o f p a t c h w i n d o w s ( 3 ) I n c o m p l e t e p a t c h d e p l o y m e n t ( 5 ) I n a d e q u a t e p o s t - p a t c h d e p l o y m e n t v e r i f i c a t i o n ( 6 ) D e l a y s i n g e t t i n g a p p r o v a l f r o m h i g h e r m a n a g e m e n t ( 1 8 ) D e l a y s d u e t o c h a n g e s i n c o m p a n y s c h e d u l e s ( 1 3 ) R e s o u r c e l i m i t a t i o n s ( e . g . , h u m a n r e s o u r c e s ) ( 2 4 ) I n f r a s t r u c t u r e l i m i t a t i o n s ( e . g . , p e r f o r m a n c e i s s u e s ) ( 1 1 ) P e o p l e - r e l a t e d R e a s o n s O r g a n i s a t i o n - r e l a t e d R e a s o n s C a t e g o r y S u b - c a t e g o r y C o n c e p t C o d e ( n u m b e r o f r e f e r e n c e s ) R 3 . N e e d o f h u m a n i n t e r v e n t i o n R 4 . C o o r d i n a t i o n d e l a y s R 5 . I n p u t r e q u i r e m e n t d e l a y s R 6 . F a i l u r e s d u e t o p o o r p l a n n i n g a n d e x e c u t i o n R 7 . O r g a n i s a t i o n d e l a y s R 8 . C a p a c i t y l i m i t a t i o n s R 9 . S e r v i c e - a v a i l a b i l i t y r e s t r i c t i o n s T i m e l i m i t a t i o n s ( e . g . , m o n t h l y p a t c h c y c l e s ) ( 5 ) I n a b i l i t y t o a l l o w s e r v i c e d o w n t i m e f r o m r e b o o t s ( 1 3 ) M u l t i r e b o o t s r e q u i r i n g l o n g e r a n d a d d i t i o n a l p a t c h w i n d o w s ( 8 ) C u s t o m e r r e q u e s t s t o p o s t p o n e p a t c h d e p l o y m e n t s c h e d u l e s ( 4 ) 18
  • 19. • Detecting and dealing with patch interdependencies – software, hardware and firmware of new & legacy systems (1.5K Servers) • Faulty patches causing unknown errors during patch testing, deployment, and post-deployment • Security patches usually require extensive post-deployment monitoring to verify the fixes • Frequent release of patches and their heterogeneity add to the complexity of patches 19 TR1: Delays Caused by Complexity of Patches
  • 20. • Lack of accuracy in the output of current tools (e.g., missing some vulnerabilities during scanning, omitting patches during patch deployment) – ASE 2022 paper (more details) • Lack of scalability to handle diverse types of patches and their features – disabling some of the tools’ functionalities • Inability to detect patch compatibility and the lack of capability to detect multi-reboot requirements [Subject - Additional reboot required for .NET patching] “7/2/20 - An investigation is needed around the number of required reboots for EMR patching and window requirements as a result if more reboots are required. A new process needs to be fleshed out when patching is postponed to accommodate the identification of the number of reboots required." - EMR, Task ID 35 20 TR2: Delays Caused by Limitations of Tools
  • 21. • Human intervention emerges as full automation not available or desirable – faulty patches causing unknown errors • Manual configurations for selecting a suitable Group Policy Object (GPO) configurations to avoid breakdowns • Human support may be needed for deploying complex, erroneous or business-critical patch installations, e.g., legacy systems • Manual intervention for re-executing failed patch deployments and re-planning patch schedules due to requirement changes. 21 31/10/19 - [B-T1] team putting in significant amounts of work, like 15-20 hours per month, to redo the schedules on custom dates each time the deployments move off standard windows." - EMR, Task ID 30 TR3: Delays Caused by Need of Human Intervention
  • 22. • A single patching task usually involves multiple interdependent activities and several stakeholders – internal and external • A lack of awareness of task progression and of understanding of shared roles and responsibilities • Email based communication about patches may result in lost and/or misinterpretation of critical information • Customers, End-users, Vendors inefficiently coordinating for seeking and giving approvals for system downtime & verification • FSE 2021 paper (More details) 22 PR4: Delays Caused by Coordination Issues
  • 23. • Tightly coupled activities have input requirements be fulfilled in timely fashion, e.g., vulnerability scan reports or prioritisation • Non-delivery or incomplete delivery of the schedule-related information resulting in poor planning for deploying patches • No online repository for maintaining servers’ patching details • Delays in receiving vendor’s support for patching errors and new patch release information [Subject - New zero-day vulnerability warning] “12/6/20 - Monitor Microsoft patch release for critical vulnerability identified on [T1] servers. Font Type 1 expected as a zero-day soon, full report not available yet. 24/7/20 - No update from Microsoft." - EMR, Task ID 43 23 PR5: Delays Caused by Input Requirements Issues
  • 24. • Security patch management needs meticulous planning and flawless execution to avoid system breakdowns • Inaccurate estimates of patch windows may result in calling off the whole process as mission cannot put on hold • Unforeseen errors can become major risks to deploying within the planned time frame if not considered during planning • Incomplete patch deployment or insufficient verification needing re-execution of patch deployment and operational disruption 24 PR6: Delays Caused by Poor Planning and Execution
  • 25. • Ensuring full compliance with the organisational policies and obtaining management approval for monthly patch schedules • Changes in organisation schedules such as change freeze periods, testing schedules like regression testing plans and shutdown periods 25 [Subject - Patching for December 2019] “18/10/19 - OOB for November patching from 4th December instead of December patching. 31/10/19 - [AT1] patching for December month is off but November Microsoft patches will be applied in the first week of December instead to keep compliance up." - EMR, Issue ID 29 OR7: Delays Caused by Policy & Procedures
  • 26. • Lack/unavailability of qualified personnel experienced in handling specific systems for patching, e.g., legacy system upgrades • Insufficient infrastructure resources - hardware and network limitations may hinder a patching task • Testing the workarounds for failed deployments delayed for weeks given the time-driven (i.e., monthly) patch cycle 26 • “24/1/20 - Patching cannot go ahead when the active backup is running. The patch load can impact servers before reboot. Need a window change, proposal to be sent by [P1-BT1] to [P2-AT1]." - EMR, Issue ID 39 OR8: Delays Caused by Capacity Limitations
  • 27. • Organisations’ inability to allow service downtime from reboots required for patches to take effect after deployment • Multi reboots requirements difficult to get prompt approvals out of fear of service disruptions from longer patch windows • Customers reluctant to agree to sufficiently large patch window; rather requesting service continuity at all cost 27 [Subject - [Servers s1 and s2] patching] “26/7/19 - OOB window is needed for the multi reboots to catch up. 9/8/19 - Waiting for the customer’s confirmation of the new patch window, pending information from [P1-AT1]." - EMR, Issue ID 20 OR9: Delays Caused by Service Level Agreements
  • 28. Frequency of Reasons for Delays 28
  • 29. Distribution of the Delays over the Process 29
  • 31. An Overview of the Used Strategies Common strategies relating to the overall patch management process S1. Frequent communication (24) S2. Collaborative decision-making (3) S3. Task delegation (31) S4. Regularly review and update patch management process-related documentation (3) 31 Strategies relating to Patch Information Retrieval (P1) Strategies relating to Vulnerability Scanning, Assessment & Prioritisation (P2) Strategies relating to Patch Testing (P3) Strategies relating to Patch Deployment (P4) S11. Timely coordination of patch deployment schedules (19) S12. Apply workarounds to maximise service availability (18) S13. Manual deployment for complex patches to minimise damage (12) S14. Agile deployment for executing changes (6) Strategies relating to Post-Deployment Patch Verification (P5) S15. Establish post- deployment verification procedures (10) S16. Collectively handle post-deployment issues (9) S17. Document deployment status of every patch (3) S8. Define compliance policies and contingency plans for test failures (9) S9. Patch pre-requisites investigation (4) S10. Modify software configurations and dependencies (3) S6. Plan alternatives for delayed patches (6) S7. Define priorities for vulnerability remediation (15) S5. Set strict timelines for patch download (2)
  • 32. • Setting tight timelines for patch download, e.g., within two days of the release of patch by the relevant vendors • Acquiring and analysing the list of the retrieved patches each month before assessment and prioritisation of vulnerability 32 [Subject - Provide .NET report at the start of the patch cycle] “15/3/19 - Org A requests BT1 to provide an extract of .NET released patches every month and a report including what patches will be applied to what servers." - EMR, Task ID 53 Strategies – Patch Information Retrieval
  • 33. • Plan alternatives for scheduled patching based on assessment of the impact on other services and risks of cyber attacks • Prioritised security patches based on the global vulnerability rating; High-risk vulnerabilities fixed within 48-72 hours • Defining priorities for patching vulnerability for reducing the risk of exploitable attack vectors used successfully • Prioritisation based on patch type – Patching OS earlier [Subject - OS security patches need to be tracked separately in the vulnerability remediation] “15/5/20 - [P1-AT1] requesting the OS security patches to be tracked separately from all other vulnerability remediation. Org B’s report should only be addressing OS security patches anyway but can make sure to separate any non-OS remediation tasks." - EMR, Task ID 45 33 Strategies – Vulnerabilities and Assessment
  • 34. • Test run compliance policies and standards, e.g., reboot every legacy server without patching; have contingency plans • Allow a specific time to identify and modify the dependencies and configurations during patch testing • Investigate the availability of prerequisites for the patches released every month as a separate task during patch testing 34 [Subject - Registry key missing for Knowledge Base (KB) ID [n] (LDAP)] “2/10/20 - Patches not installed on [servers s1 and s2] due to missing a registry key. [P1-BT1] to check settings and apply where missing." - Win, Task ID 24 Strategies – Patch Testing
  • 35. • Clustering similar patches to reduce the time spent in testing, deployment and rebooting • Balancing workload on servers during patch deployment to avoid unnecessary service disruptions • Backup servers concurrently running the critical services while being rebooted - Backup servers patched separately • Manually patching business-critical servers having multiple version dependencies, multi reboots and legacy systems • Agile deployment - execute the changes in small iterations 35 Strategies – Patch Deployment
  • 36. • Define a set of procedures for post-deployment patch verification to reduce the risk of delays caused by poor execution • Monitor a patched system for functional, performance or unexpected issues; getting periodic scans to verify patching • Collaborative problem handling for analysing the root causes for post-deployment issues and finding workarounds • Develop a knowledge base to keep track of every patch as a reference in cases of errors encountered during the execution [Subject - Automated second rescan for reboots] “31/10/19 - [P1-BT1] raised this issue, he has configured the window to rescan for missing patches and conduct a second reboot if required. No issues during patching, seeking client feedback for verification." - EMR, Task ID 28 36 Strategies – Post-Deployment Patch Verification
  • 37. • Frequent communication reduces delays, strengthens collaboration and improves mutual understanding • Collectively making decisions about patch management helps teams gain insight into plans, activities and alternatives • Well-defined roles and responsibilities around patch management activities resulting in delegation and accountability for actions • Systematically and regularly review and update documents of patch actions and decisions; test execute process changes “13/12/19 - Finalising the documentation after testing internally for handover to 24x7. 10/1/20 - Documentation to be tested in February, will be ready for handover in March." - EMR, Task ID 24 37 Strategies – Overall Patch Management Process
  • 39. • An understanding of the reasons for delays can enable security staff to take measures for mitigating the potential delays • Provided knowledge can help practitioners in suitable decision- making, prioritisation and planning of patch management tasks • Strategies can guide practitioners and organisations in better planning and taking actions to mitigate the impact of the delays • Developing new and/or innovative use of existing tools for visualising dependencies, patch management knowledge repositories, timely communication and collaboration 39 For Practitioners
  • 40. • Our findings are context and domain specific (healthcare) – extending and adapting case study for different domains • Developing and executing interview guides and surveys to verify the findings and discover variations • Research on AI-based tools for detecting patch mismatches, improved coordination across patching tasks and reducing delays • Investigating the suitability of “human-AI collaboration" for security patch management • Evaluating the performance and accuracy of available tools 40 For Researchers
  • 41. Nesara Dissanayake @nesara_d Asangi Jayatilaka @DrAsangiJ Mansooreh Zahedi @MansoorehZ Muhammad Ali Babar @alibabar Centre for Research on Engineering Software Technologies (CREST - @crest_uofa) School of Computer Science, The University of Adelaide, Australia The Research Team
  • 42. Acknowledgements • This talk is based on the research studies carried out by Nesara Dissanayake, M. Ali Babar, Mansooreh Zahedi, Asangi Asangi Jayatilaka • Partially Funded by the University of Adelaide and the CREST • SA Health provided the access to the case studies • We are grateful to the participants for enabling to collect a variety of data for our research 42
  • 43. CRICOS 00123M Contact: Ali Babar ali.babar@adelaide.edu.au