Serverless log analytics with Amazon Kinesis
0 likes13 views
The document discusses the challenges of logging in AWS environments, particularly at Steamhaus, and introduces an efficient solution using the ELK stack (Elasticsearch, Logstash, and Kibana) with serverless technologies. It highlights the integration of various AWS services for log ingestion, storage, and analytics, along with an automated Terraform setup. The solution is a work-in-progress, intended for readability and ease of testing, with an invitation for further questions or contact.
Serverless log analytics with Amazon Kinesis
- 2. SteamhausKELK ON AWS Who am I? Rob Greenwood Tech Director @ Steamhaus
- 4. No centralised logging User needs OS knowledge Distribution Of keys Enemy of autoscaling Log rotation Users download logs unnecessarily Doesn’t scale To many servers Slow to find issues Alerting is hard Sshing to servers :(
- 5. SteamhausKELK ON AWS ELK is awesomE
- 6. SteamhausKELK ON AWS ELK on ec2
- 7. SteamhausKELK ON AWS KELK on AWS • Low maintenance - No ec2, Uses entirely AWS serverless technologies and services • ALB, Cloudfront and Cloudtrail logs are ingested as well as EC2 logs • Logs are archived in S3 for long term storage, and indexed in Elasticsearch for short term analytics • Automated with Terraform • Open source Kinesis: buffering and delivering instance logs Elasticsearch: Indexing and log storage Lambda: processing and delivering S3 logs Kibana: Search and analytics
- 8. SteamhausKELK ON AWS How does it work?
- 21. SteamhausKELK ON AWS Automation code Sample Web Stack VPC ALB EC2 Logging Stack Kinesis Elasticsearch Service Lambda S3 CloudfrontPython Terraform Do try this at home! github.com/steamhaus/kelk-example
- 22. SteamhausKELK ON AWS Callouts from the build • It’s not production ready, built for readability • Nailing iam and bucket policies can take a while! • Testing lambda - create a test event in the UI • Use Terraform, rinse and repeat
- 23. SteamhausKELK ON AWS Any Questions..?
- 24. Thank you :) Contact us hello@steamhaus.co.uk 0161 820 2020 @steamhausmcr Locate us Fourways House 57 Hilton Street Manchester M1 2EJ