Assessing and Mitigating the Log4j Vulnerability – A Case Study

Assessing and Mitigating the Log4j Vulnerability – A Vulnerability Management Case Study

Authors of this blog post include Subramani Sundar- Director, Threat & vulnerability Management, Tamika Miyashita- Security Project Manager, Tanveer Chowdhury- Security Manager, & Peter Bellarmine - Lead Security Engineer

This the 3rd blog post we have written on the Log4j or Log4Shell vulnerability. The reason is we wanted to not only provide you insight on the vulnerability itself and how impactful it could be now and going forward, but also provide you details on how we protected our clients.

Our first blog post, The Log4j Vulnerability is Likely to be a Significant Threat for Years discusses the vulnerability itself, the series of events that led to its discovery as a Zero-Day vulnerability that made billion of systems and technologies susceptible to attack.

Our second blog post, Lessons Learned in Defending Against the Log4j Vulnerability – A Case Study was written by our SOC managers at SecureOps and details the security technologies that were used to find the Log4j vulnerability, any attacks leveraging the vulnerability and lessons learned concerning how to defend more effectively.

In this blog post, our Authors, Subramani Sundar and Tamika Miyashita discuss a real-world case study in how they handled vulnerability management for a large client corporation who had nearly countless systems with one or more of the Log4j vulnerabilities.

We’ve segmented the blog post into the following sections to make the Vulnerability Management case study easy to follow:

1. App Sec - Asset Identification and Evaluation
2. App Sec – Vulnerability Scanning & Validation
3. App Sec - Vulnerability Reporting & Remediation
4. SecOps - Vulnerability Management Analysis & Response (VMAR)
5. SecOps - Vulnerability Scanning and Assessment

App Sec - Asset Identification and Evaluation

Asset identification is an important process for organizations that manage a variety of fixed or movable assets, and it’s an essential component of an overall asset tracking system. Asset identification is a critical process for any business because knowing which equipment you have is essential to being able to track it. If your assets are labeled incorrectly, or if you have duplicate labels, you risk facing compliance issues, falling behind with preventive maintenance, and putting your assets at risk of being stolen or lost. Utilizing asset identification best practices with fixed and movable physical assets is the foundation of efficient asset tracking for your business.

The 7 Steps of a Successful Risk Asset Management

Step 1: Identify Your Information Assets

Step 2: Identify the Asset Owners

Step 3: Identify Risks to Confidentiality, Integrity, and Availability of the Information Assets

Step 4: Identify the Risk Owners

Step 5: Analyze the Identified Risks and Assess the Likelihood and Potential Impact if the Risk Were to Materialize

Step 6: Determine the Levels of Risk

Step 7: Prioritize the Analyzed Risks for Treatment

How SecureOps Handled Risk Asset Management with Log4j

To make this simple and straightforward I’ll simply walk through the process we used to identify and value the assets in our client’s environment. As we suggested prior, this is a critical step because you can’t patch everything with Log4j because the vulnerability is on so many systems. Thus, we took the following steps:

• A hybrid approach to identify assets to add to scope using proprietary and open-source tools
• Publicly available information and internal records were combined to create scope for scanning
• Periodic queries for public facing assets ensured that the scope was up-to-date, even with the most recently deployed applications
• Scripting and automation efforts contributed to the fast response by:

o  Performing a discovery for active applications which respond to HTTP Requests

o  Redirection identification of from a URL leading to deduplication of scope

o  WAF identification that helped prioritize applications that lacked compensating controls

• Using this approach, 2000+ public facing applications were identified 

App Sec – Vulnerability Scanning & Validation

• Scans were performed using multiple proprietary and open-source tools in order to effectively scan using various detection methods and logic. Below, we walk through how we handled our scanning and validation for our client; however, it may be helpful to provide you a best practice 5-step process to provide you some context.

5-step checklist for web application security testing

1. What needs to be tested?

The scope of your security assessment is extremely important. You may have your own internal requirements, or you may have to follow the requirements of a business partner or customer. And you need to get all the right people on board.

2. What tools are best suited for the task?

At a minimum, web application security testing requires the use of a web vulnerability scanner, such as Netsparker or Acunetix Web Vulnerability Scanner. For authenticated testing, you'll want to use an HTTP proxy such as Burp Suite, which allows you to attempt to manipulate user logins, session management, application workflows and so on.

3. Vulnerability scanning

Rather than trying to create a checklist of every test you need to run for every vulnerability for web application security testing, it's easier to break it down into the important categories. Dedicated web vulnerability scanners should find most flaws, but I have found that you often need traditional network vulnerability scanners such as Nessus or Qualys to uncover everything.

4. Scanner validation and additional manual checks

As with vulnerability scanners, I can't possibly list all the checks you need to perform because there are so many potential areas for exploitation. The first thing you need to do is validate all your web vulnerability scanner findings to see what's exploitable and what matters in the context of your application and your business.

5. Document your findings and disseminate them to the proper people

• Unfortunately, a lot of application security testing stops with step four. One of the most undervalued aspects of a formal application testing program is codifying your findings into a formal security assessment report.

How we conducted our Vulnerability Assessments

• Various proprietary tools were utilized to scan for CVE-2021-44228. Vendors were engaged to fine tune detection logic and performance in an effort to achieve the most efficient detection mechanisms in the quickest time possible
• Open-source tools such as nuclei were leveraged to scan applications using publicly available detection mechanisms that were constantly updated by security researchers across the world. Each detection method was validated and vetted by our team before the inclusion in the scans and custom detections mechanisms were also created based on articles and reports of this versatile vulnerability.
• Validation using manual security testing tools allowed us to make adjustments to the payload and determine if the application is vulnerable from various perspective based on the response from the application

App Sec - Vulnerability Reporting & Remediation

An effective vulnerability assessment and remediation program must be able to prevent the exploitation of vulnerabilities by detecting and remediating vulnerabilities in systems and applications in a timely fashion. Proactively managing these vulnerabilities will reduce or eliminate the potential for exploitation and save on the resources otherwise needed to respond to incidents after exploitation has occurred.

Let’s walk through the process we executed in our strategy to mitigate the Log4j / Log4Shell vulnerabilities. 

A Fundamental 3 Step Process for Vulnerability Reporting, Validation, and Remediation

Step #1 - Keeping Your Open-Source Vulnerabilities in Check

With open-source security vulnerabilities, Software Composition Analysis (or SCA) tools are the way to go. This family of automated tracking tools enable software developers and security professionals to automatically detect all open-source components in an organization’s systems

Step #2 - Prioritize Your Vulnerabilities

Good news: you know what’s in your code, and what requires your attention.

Bad news: So many vulnerabilities, so little time. There are only so many hours in a developer’s day, and your team is most probably racing through a sprint. There is no way your team can attend to all the issues that your superior tracking processes have uncovered.

Step #3 - Who Goes First? How to Prioritize Open-Source Vulnerability Remediation

As we already know, the information about the vulnerability and its remediation might be found in a number of repositories, advisories, and bug trackers, so a tool that aggregates all open-source vulnerability information from across the open-source bazaar continuously will help your organization get the most accurate information.

Let’s walk through the 3 steps we took in identifying, validating, and remediating the vulnerabilities in the case of Log4j.

• Vulnerabilities identified from scans were manually validated to ensure validity before reporting to remediation teams
• A vulnerability scan report with vulnerability details along with a manual validation PDF report containing the exact validation actions, steps to reproduce the vulnerability and best practices/recommendations were sent to remediation teams.
• Our team also assisted with remediation directly with developers and application teams to help understand the method of exploitation and validate fixed vulnerabilities.

Vulnerability Management Analysis & Response (VMAR)

From a broad standpoint, vulnerability management is the process of identifying, evaluating, treating, and reporting on security vulnerabilities in systems and the software that runs on them. This, implemented alongside with other security tactics, is vital for organizations to prioritize possible threats and minimize their attack surface. Because Java and subsequently Log4j is on millions of machines the attack surface is massive.

So, specifically for Log4j and at SecureOps the Vulnerability Management Analysis and Response team handles the following:

• The mission of the Vulnerability Management Analysis & Response team is to analyze the likelihood/impact of the vulnerabilities and provide proper remediation/mitigation guidance.

o  Analysis of the vulnerabilities just based on CVSS score rating is considered as legacy approach.

o  We analyze vulnerabilities by assigning weight to multiple factors including attack vector, privileges required, attack complexity etc.

o  We include exploits availability, exploitability factors to assign criticality of each vulnerability.

o  We leverage threat intel, exploit maturity from different reputed sources to keep up with industry wide sightings of the vulnerability.

o  We help prioritize the vulnerabilities not only based on vulnerability factors, but also considering organization’s infrastructure and attack surface and other existing compensating security controls, which helps not overloading IT teams.

• For Remediation & Mitigation of Vulnerabilities:

o  We prepare an advisory with all required technical details to help IT teams to remediate the vulnerabilities or to mitigate to lessen the immediate risk.

o  We work with different security counterparts to make sure all the mitigations are applied wherever applicable utilizing network prevention and endpoint security solutions to their fullest capability.

o  We closely monitor exploits availability and make sure to communicate to Endpoint Detection and Response or EDR team so that they create detection rules and any anomaly activity monitored by SOC teams.

o  We follow up with respective stakeholders until the remediation is complete and guide wherever necessary.

Vulnerability Scanning and Assessment

Vulnerability scanning and assessment used to be a point in time exercise with patching often performed on a weekly or monthly schedule. However, for organizations that have a more mature security posture, scanning, assessment, and patching has evolved. The practice is continuous and as we suggested before based on the following considerations:

Four important considerations are:

1. The criticality of the affected system
2. The exploitability of the vulnerability
3. The potential impact of exploitation
4. Are there viable alternatives to patching?

Continuous Network Scanning

Yearly or quarterly vulnerability scanning is no longer sufficient to detect risks in your IT system. You need a proactive, 24x7 continuous defense to stand a chance against the hackers incessantly probing your network.

External Vulnerability Scans

This type of scan looks at your network from the hacker's perspective. It scans external IP addresses and domains, probing for vulnerabilities in internet-facing infrastructure to determine which ones can be exploited.

Internal Vulnerability Scans

Performed from a location with access to the internal network, internal vulnerability scans are more complex than external ones, because there are simply more potentially vulnerable assets within your organization. This scan will discover and catalog your core IP-connected endpoints, such as laptops, servers, peripherals, IoT-enabled machines, and mobile devices.

Vulnerability Scanning and Management to Detect Log4j

For our client we specifically utilized Tenable scanners and vulnerability assessment and management tools to detect the log4shell vulnerability on the client’s internal or on-prem and external space covering all major networks. We did by conducting the following:

• Scanning for log4j:

o  Network Wide vulnerability scans to uncover Log4j

o  Targeted scan covering all the assets vulnerable to Log4j twice daily till date

• Manual validation:

o  To perform validation on findings with partner teams and investigate False positives

o  Adhoc scans on remediated assets and confirm with partner teams

o  Fix authentication and scanner reachability issues for affected assets

• Automation:

o  Script to automate the reporting for log4j using proper filters

o  Script built to take the log4j affected hosts and populate all the required fields with relevant data pulling from multiple sources

Script built check on on-prem nodes to confirm tenable scanner reachability and authentication as well as agent presence

Conclusion and Lessons Learned

For most organizations, patching every vulnerability is not feasible, and, in the few cases where it may be, it likely is not financially viable. Companies need to take an intelligent approach to identifying and applying the updates that are most important to their organization’s security posture.

In the case of Log4j / Log4Shell it is impossible to apply patches to the thousands of systems and applications that leverage Java. In our case with our Fortune 50 client the important first question to ask is “when triaging potential updates is which system(s) do they affect?” Not every system in an organization's network is equally important, and their relative importance impacts the relative importance of the patches that they require.

Whether or not a vulnerability has been weaponized and is actively exploited should be a major factor in determining the speed at which an organization patches a vulnerability. Several major cyberattack campaigns, including those like the WannaCry ransomware outbreak and the Equifax data breach exploited vulnerabilities that had patches available and were officially announced as being actively exploited by cybercriminals.

Risk-based patch management is part of an overall vulnerability management program and is an IT security strategy in which organizations prioritize the patching or remediation of software vulnerabilities according to the risk they pose to the organization. 

To provide a recap; a risk-based vulnerability management or patching strategy has several key components:

• They assess the importance and the business context of systems; not all systems are created equal and an intrusion into some segments of a network may be more damaging or likely than others.
• They use intelligence from CVE to NIST to generate risk scores based on the likelihood of exploitation by attackers.
• They use threat intelligence to identify the vulnerabilities attackers are discussing, experimenting with, or using. Meaning have attackers created malicious code or an exploit to take advantage of the vulnerability.
• By combining asset criticality and intelligence risk-based patch management programs focus patching efforts on the vulnerabilities that are most likely to be exploited and that reside on the most critical systems.

To Learn More About How to Implement a Risk-Based Vulnerability Management Program Please Call Us – as Always, We Are Happy to Help - 1 (888) 982-0678.

You Can Also Fill Out Our Contact Us Form Here to Talk with a Security Specialist - https://secureops.com/contact-us/