Can't Touch This: Detecting Lateral Movement In Zero Touch Environments

Editor's Notes

• #2: Phillip Marlow is a cybersecurity and DevOps engineer. He helps organizations understand how to adopt DevOps practices to increase their security rather than sacrifice it in the name of speed. Often DevOps and security are thought of as separate fields. Sometimes organizations will consider one a foundation for the other. In other places, leadership will throw it all together as DevSecOps and hope their engineers can figure it out. But by thoughtfully bringing these two fields together, both groups can benefit and easily achieve outcomes that would be difficult for one field to accomplish on its own.
• #3: This research brings together tools and techniques from security with practices from DevOps to achieve improve confidentiality, integrity, and availability of deployed applications. Specifically, it will show that by using network monitoring and a SIEM, analysts can detect lateral movement and distinguish between policy violations and malicious intrusions.
• #4: In 2015, FireEye reported that 100% of incidents they investigated involved lateral movement, which made use of valid credentials harvested before or during the attack (FireEye, 2015). Recent reports continue to detect this trend, including the 2020 Verizon Data Breach Investigations Report (Verizon, 2020). This is in part due to requirements for employees to have privileged credentials to administer and update applications and the servers they run on. It is also due in part to the standard practice of leaving these credentials unencrypted on employee workstations, such as when using stored AWS API tokens or private SSH keys. One mitigation that we will explore in this presentation is removing those privileged credentials from employee workstations and instead requiring all updates to an application be made by automation. With the application server as the attacker’s target, there are 3 primary paths they could use: 1. A direct attack against the application server. This is a well studied path and generally the domain of IDS/IPS and firewalls. 2. Compromise an employee workstation, for example through phishing, and then pivot to the application server. This will be the primary attack path considered in this research. 3. Compromise an employee workstation, then use that access to modify the application source code or configuration. While dangerous, this path is harder due to the testing changes must go through before they are deployed to the application server.
• #5: Google’s definition of the zero-touch network and zero-touch production environment is vendor-neutral and applies to many highly automated environments. The lack of any manual interaction under normal circumstances gives zero-touch its name. Even under abnormal circumstances, all manual intervention must be highly audited (Koley, 2016). This takes the DevOps practice of automation to a logical, and useful, extreme. By defining certain network connections that should not happen under normal circumstances, we can create automated alerts for these connections. Analysts can then examine other useful details, such as application changes that occurred at that time, to determine if an attack is occurring.
• #6: To visualize how a zero-touch environment can make it easier to detect policy violations, including potential attacks, consider this scenario. An employee is working and making updates to the application server. As these updates are happening, they all flow through the DevOps process and are recorded and tested before the configuration change is allowed. During this normal operation, there is no traffic flowing through the bastion host to the application server.
• #7: When this traffic does occur, it can be easily and automatically identified. This can happen because the structure of this environment and the existence of a zero-touch policy makes lateral movement clearly stand out. This is a useful example of a security policy being implemented as code, an important DevSecOps concept. This anomaly and the alert it generates is the starting point an analyst needs to zero in on a particular interaction.
• #8: These alerts of zero-touch policy violations can be detected using any network monitoring tool capable of decoding the traffic in a given environment. For example, above is the Snort alert from an on-premise environment. These alerts draw analyst attention to the possibility of a problem but are not by themselves enough to know if an attack is occurring. To know what is really going on, analysts can use these alerts along with host-based logs correlated by a SIEM to detect common patterns that allow categorizing events. The next several slides will illustrate the major categories of policy violations and demonstrate patterns in the logs which can help an analyst correctly separate attacks from other violations.
• #9: Two different formats of timeline are generated for an analyst to use when categorizing zero-touch violations. The top image shows a 1-minute duration of aggregated metrics, which are useful in showing trends in the data rather than individual data points. This longer timeframe also allows better visualization of metrics that are continuous, but that the host-based agents do not report every second, such as the total number of containers present on the system. The bottom image shows the same metrics with a 1-second aggregation. This format allows individual events to stand out in the data and helps analysts see the exact sequence of events with increased precision.
• #10: Continued manual patching by development teams is likely to be one of the most common violations of a zero-touch policy immediately following implementation. The SIEM timelines show a few features which can indicate that this manual patching is going on. First, a temporary decrease in both the number of running containers (orange) and the total number of container images (green) is consistent with updating a running application which uses the latest tag. There are also a small number of HIDS alerts overall (yellow). Taken together, these all suggest a benign policy violation that can be referred to the business and not an attack.
• #11: Looking at individual events for this case also suggests a benign policy violation. First, multiple zero-touch violations (red spikes) between the same hosts in a short timeframe can suggest that an employee is copying files to the server, then logging into to perform an update. When the HIDS alerts are further filtered to identify command line data, it shows that no commands are logged that are indicative of either reconnaissance or incident response.
• #12: Following the initial zero-touch violation, there are a few patterns seen in these timelines that suggest an attack. First, we can see the total number of container images (green) increases, suggesting a new image has been installed. Immediately following that, the number of running containers (orange) increases, showing the new image is now being run. Once the new service is running, there is a spike in the overall number of HIDS alerts (yellow).
• #13: Looking at individual events, we see that early on multiple reconnaissance commands are run (blue). These are commands such as netstat or ps which are often used to determine what is available on a server. A Windows equivalent might include commands such as tasklist, ipconfig, or others from JP CERT’s list of commands abused by attackers (Tomonaga, 2016). Taken together with the trend data for containers and HIDS alerts, these patterns demonstrates typical attacker behavior in performing some reconnaissance, then downloading and running additional tools. While the HIDS alerts by themselves could alert an analyst to this attack, by beginning with a zero-touch violation this attack can be detected earlier and easily tracked back to the previous step in the attack chain.
• #14: The only allowed zero-touch violation is via an audited break-glass mechanism, such as when conducting incident response. Since this may happen, even long after the organization has adopted zero-touch, it is important to understand what it looks like so it can be quickly categorized. In this situation, we see a temporary decrease in the number of running containers (orange), which can indicate an image with no modification has been captured for forensic analysis. If there were modifications, we might also see an increase in the total number of container images (green).
• #15: In the individual events timeline, we see both reconnaissance commands (blue) and incident response commands (gray) being run to collect and preserve forensic evidence. Reconnaissance commands have a lot of overlap with incident response commands, so overall this data together with the trends in container images seen suggests incident response activity. Of course all of this can, and should, be correlated with the incident response team’s log of activity. If this activity isn’t supported by an incident response log entry, it is possible it instead represents an attack against the application server’s availability.
• #16: This table summarizes the experimental results seen over the last several slides. It can provide a quick reference for analysts when reviewing zero-touch violation alerts and assist in reaching an initial categorization. Trends in application data and privileged commands which occur immediately after a zero-touch violation can provide analysts a lot of value. By matching these patterns against the timelines generated for an event, analysts can reach not only an initial categorization, but also find helpful starting points for further investigation if needed.
• #17: The whole organization can benefit from monitoring for zero-touch policy violations. Development teams can get feedback on misconfigurations and security teams get another tool for detecting one of the most common attack techniques – lateral movement using valid credentials. Instead of just adding more alerts for analysts to sift through, this monitoring goes one step further and helps analysts determine whether an event requires further investigation by displaying related data on an easy to consume timeline. Even for those organizations who are not able to take advantage of this technique right away, this proof of concept can provide another example of the benefits of adopting DevOps and zero-touch policies.