Native cloud security monitoring
- 1. BAY AREA
- 2. Native Cloud Security Monitoring “A journey through bootstrapping security logging and monitoring in AWS”
- 3. ● Senior Security Engineer at Segment ● Author of Cloud-Based Rube Goldberg Machines ● 8 years experience building SIEM, and SOAR platforms for the enterprise Who am I and why should you care?
- 4. What this is about ● To get better visibility on events in our accounts. ● To automate incident detection and response ● Facilitate effective threat hunting Why are we here?
- 5. What this is about ● Configure requisite services (CloudTrail, GuardDuty, ElasticSearch) ● Assemble our services into monitoring infrastructure ● Apply some best practices How do we get there?
- 6. Security Monitoring on AWS CloudTrail - Consider this the audit and activity log for everything related to the events, users and resources in our account. ✅ Relatively easy to configure ✅ Covers the majority of AWS services
- 7. Security Monitoring on AWS CloudTrail - Consider this the audit and activity log for everything related to the events, users and resources in our account. ✅ Relatively easy to configure ✅ Covers the majority of AWS services ❌ Limited search capabilities ❌ Limited export capability
- 8. Security Monitoring on AWS CloudTrail - Consider this the audit and activity log for everything related to the resources, users and actions in your account. ✅ Relatively easy to configure ✅ Covers the majority of AWS services ❌ Limited search capabilities ❌ Limited export capability
- 9. Security Monitoring on AWS ✅ Easy to configure and deploy ✅ Monitors and alerts on events without additional configuration ✅ Supports external threat feeds ✅ Under active development GuardDuty - This is our Intrusion detection system. It generates alerts for suspicious activity in Cloudtrail, DNS, and VPC Flow logs.
- 10. Security Monitoring on AWS ✅ Easy to configure and deploy ✅ Monitors and alerts on events without additional configuration ✅ Supports external threat feeds ✅ Under active development ❌ Disabled by default ❌ No access to source log data ❌ No custom alerts GuardDuty - This is our Intrusion detection system. It generates alerts for suspicious activity in Cloudtrail, DNS, and VPC Flow logs.
- 11. Security Monitoring on AWS ✅ Easy to configure and deploy ✅ Monitors and alerts on events without additional configuration ✅ Supports external threat feeds ✅ Under active development ❌ Disabled by default ❌ No access to source log data ❌ No custom alerts GuardDuty - This is our Intrusion detection system. It generates alerts for suspicious activity in Cloudtrail, DNS, and VPC Flow logs.
- 12. Security Monitoring on AWS ✅ Industry Standard ✅ Helps contextualize data ✅ Search capabilities facilitate threat hunting AWS ElasticSearch - A managed SaaS search solution based on open source software, bundled with Kibana for data visualization (ELK)
- 13. Security Monitoring on AWS ✅ Industry Standard ✅ Helps contextualize data ✅ Search capabilities facilitate threat hunting ❌ Poor defaults ❌ Index management can be challenging AWS ElasticSearch - A managed SaaS search solution based on open source software, bundled with Kibana for data visualization (ELK)
- 14. Security Monitoring on AWS ✅ Industry Standard ✅ Helps contextualize data ✅ Search capabilities facilitate threat hunting ❌ Poor defaults ❌ Index management can be challenging AWS ElasticSearch - A managed SaaS search solution based on open source software, bundled with Kibana for data visualization (ELK)
- 15. Cloudtrail ❌ Limited search capabilities ❌ Limited export capability GuardDuty ❌ Disabled by default ❌ No access to source log data ❌ No custom alerts ElasticSearch ❌ Poor defaults ❌ Index management can be challenging Summary of Pain
- 16. ● Deploy all required services and components automatically ● Extend log retention beyond 90 days (configurable) ● Tune ElasticSearch for ingesting CloudTrail logs ● Allow custom alert generation ● Enhance search capabilities and eliminate weak defaults ✨ Fault Tolerant ✨ Scalable ✨ Self-Maintaining Easing the pain - our solution
- 17. The Shopping List ● AWS ElasticSearch ● AWS CloudTrail ● AWS GuardDuty ● Terraform 0.11+ ● Lambda* ● Cloudwatch Trigger ● Kinesis Streams ● Kinesis Firehose ● S3 Buckets ● IAM Roles
- 19. Infrastructure as Code ● All components will be arranged and configured by Terraform ● A resource name and an IP are all you need to get started Building the solution
- 20. Infrastructure as Code ● After about 15 minutes we have the full ELK ● Lambda replaces Logstash providing serverless log ingestion Building the solution
- 21. Terraform plan below 1. CloudTrail logs are written to S3 2. S3 ObjectCreated events trigger our log processing function 3. Lambda processes log and delivers it to a kinesis stream 4. Kinesis firehose reads from kinesis stream 5. Kinesis firehose delivers stream data to ElasticSearch index CloudTrail Logs
- 22. 1. CloudTrail logs are written to S3 2. S3 ObjectCreated events trigger our log processing function 3. Lambda processes log and delivers it to a kinesis stream 4. Kinesis firehose reads from kinesis stream 5. Kinesis firehose delivers stream data to ElasticSearch index CloudTrail Logs
- 23. 1. CloudTrail logs are written to S3 2. S3 ObjectCreated events trigger our log processing function 3. Lambda processes log and delivers it to a kinesis stream 4. Kinesis firehose reads from kinesis stream 5. Kinesis firehose delivers stream data to ElasticSearch index CloudTrail Logs ● Written in Go ● Highly efficient ● Cost effective
- 24. 1. Cloudtrail logs are written to S3 2. S3 ObjectCreated events trigger our log processing function 3. Lambda processes log and delivers it to a kinesis stream 4. Kinesis firehose reads from kinesis stream 5. Kinesis firehose delivers stream data to ElasticSearch index CloudTrail Logs
- 25. 1. Cloudtrail logs are written to S3 2. S3 ObjectCreated events trigger our log processing function 3. Lambda processes log and delivers it to a kinesis stream 4. Kinesis firehose reads from kinesis stream 5. Kinesis firehose delivers stream data to ElasticSearch index CloudTrail Logs
- 26. 1. New GuardDuty findings trigger cloudwatch event 2. Cloudwatch sends GuardDuty event to Kinesis Stream 3. Kinesis firehose reads from kinesis stream 4. Kinesis firehose delivers stream data to ElasticSearch index GuardDuty Logs
- 27. 1. New GuardDuty findings trigger cloudwatch event 2. Cloudwatch sends GuardDuty finding to Kinesis Stream 3. Kinesis firehose reads from kinesis stream 4. Kinesis firehose delivers stream data to ElasticSearch index GuardDuty Logs
- 28. 1. New GuardDuty findings trigger cloudwatch event 2. Cloudwatch sends GuardDuty event to Kinesis Stream 3. Kinesis firehose reads from kinesis stream 4. Kinesis firehose delivers stream data to ElasticSearch index GuardDuty Logs
- 29. 1. New GuardDuty findings trigger cloudwatch event 2. Cloudwatch sends GuardDuty event to Kinesis Stream 3. Kinesis firehose reads from kinesis stream 4. Kinesis firehose delivers stream data to ElasticSearch index GuardDuty Logs
- 30. Building the solution Logs are now flowing ● Data arrives in real time ● Kinesis streams allow us to retain log data and control flow rate ● Kinesis firehose allows us to retry delivery in case of downstream failure
- 31. Applying best practices ● Cloudtrail logs can potentially have thousands of fields ● Evenly distribute database shards to cluster hosts Building the solution
- 32. Raw log data is now searchable
- 34. ● Built in monitoring and alerting functionality ● Triggered by search results ● Alerts can be sent directly to Slack, webhooks or SNS topics Creating Alerts
- 35. Creating Alerts Monitors ● Define an execution schedule ● Define an extraction query
- 36. Triggers ● Define trigger conditions for your monitor ● Triggers define when we should be notified Creating Alerts
- 37. Actions ● Define alert action and destination Creating Alerts
- 38. ● Alert text is fully customizable via markdown ● Include links to source data ● Alerts can trigger other events via Lambda (webhook) Creating Alerts
- 39. ● Alert history is maintained automatically ● Alerts can easily be enabled/disabled ● Alert configurations can be exported Creating Alerts
- 40. Wrapping things up ✨ Extend visibility ■ Logs can be retained in ElasticSearch and S3 for a configurable period of time ✨ Facilitate threat hunting ■ ElasticSearch allows us to search and visualize all our log data ✨ Automate detection and response ■ ElasticSearch monitoring and alerting can notify us of conditions in our Cloudtrail logs ✨ Easily deploy all resources ■ The entire solution is deployed using Terraform, only requiring a name and client IP
- 41. Wrapping things up Terraform plan available here: https://github.com/louisbarrett/Eager-Locomotive git clone https://github.com/louisbarrett/eager-locomotive cd eager-locomotive terraform init terraform plan / terraform apply Terraform Plan QR
- 42. Native Cloud Security Monitoring “A journey through bootstrapping security logging and monitoring in AWS” Terraform plan below
- 43. Platform demo configuration and cost metrics Instance type: t2.medium.elasticsearch Number of instances: 1 Master instance type: n/a Number of master instances: 0 Storage type: EBS EBS volume type: General Purpose (SSD) EBS volume size: 30 GB ElasticSearch was the primary cost driver. ● Consuming 307mb of log data per week, cost approximately $200 per month ● Suitable for lab environments, or low traffic environments
- 44. Platform production configuration and cost metrics Instance type: m5.large.elasticsearch Number of instances: 20 Master instance type: c5.large.elasticsearch Number of master instances: 3 Storage type: EBS EBS volume type: General Purpose (SSD) EBS volume size: 512 GB ElasticSearch was the primary cost driver. ● Consuming 2TB of log data per week, cost approximately $3k per month ● Total platform storage in this configuration is 10TB ● Remarkably stable, fast, and reliable
- 45. Wrapping things up ElasticSearch New Defaults: Terraform Plan QR