Serverless Real-time Tracking & Analysis
Download as PPTX, PDF1 like187 views
We adopted a serverless architecture to build a real-time analytics solution for tracking website usage. This involved using AWS Lambda functions triggered by events in Amazon Kinesis streams to index data from API requests in Amazon Elasticsearch. The serverless approach allowed us to focus on solving business problems rather than managing infrastructure, and provided built-in monitoring, auto-scaling, and pay-per-use billing. While some services like API Gateway could become expensive at high volumes, we optimized costs by batching requests and retrieving data in batches from Kinesis. The resulting solution met our goals of speed, cost-effectiveness, and reduced maintenance.
Serverless Real-time Tracking & Analysis
- 1. We’re AGILE we embrace CHANGE we automate EVERYTHING we always value TOP TALENT
- 2. Serverless Real-time Tracking & Analysis @heisthedon
- 3. about://me dev manager @carsales head of retail platform @heisthedon
- 4. Agenda • Motivation behind this • Serverless.. What and Why? • The Architecture • What We Learnt
- 8. we want faster feedback
- 9. be autonomous and resourceful
- 10. take ownership and be accountable
- 11. so we build our own tracking solution
- 12. Our Success Metrics are Speed of Implementation Must be Cost Effectives No Reinvent the Wheels Less Maintenance
- 13. What Is Serverless Architectures? Serverless != No Server
- 14. Serverless is… • Orchestration. Event-driven - the "if this, then that" trigger • Microservices architecture • In the Cloud
- 15. Why Serverless Architectures? Focus on what matters
- 16. Serverless helps us to focus on what matters… • Business problem to solve • Infrastructure – less maintenance, auto-scale, capacity and underutilization servers not a big concern • Less code to write • Logging and monitoring built-in
- 17. Our Lego Building Blocks Amazon CloudWatch Amazon API Gateway Amazon S3 Amazon Elasticsearch Service Amazon Kinesis Streams AWS Lambda Amazon Redshift
- 18. Amazon API Gateway API Management Tools • REST Endpoint • Authentication • Caching • Monitoring • Request throttling • Cloudfront integration • Cloudwatch Monitoring • Stage features – dev vs prod Maintenance • No schedule downtime Pricing Model • Pay what you use
- 20. AWS Lambda Function as a Service (FAAS) • Execute in response to events • Deploy in millisecond • Live in second (max 300s) • Auto-scale by throughput (run 100 concurrent execution) • Built-in retry mechanism • Support versioning / aliasing • Cloudwatch Log & Monitoring Maintenance • No schedule downtime Pricing Model • Pay what you use on execution and compute (per execution on 100ms block)
- 22. Amazon Kinesis Stream Streaming Data Platform • Scale in / out by Shard • Data retention 1 – 7 days • One shard has capacity of 1MB/sec in & 2MB/sec out, and throughput of 1000 records / sec • Cloudwatch Monitoring Maintenance • No schedule downtime Pricing Model • Pay what you use on data in (PUT Payload Unit per 25KB block = $0.014 per million unit) • Shard Hour (1 Shard = $11 per month) • Data Retention
- 23. Amazon Elasticsearch Service The Power of Elasticsearch and Cloud Combined • The Full ELK Stacks • High availability • Failure detection, patch, snapshot are fully managed • Cloudwatch Monitoring Maintenance • No schedule downtime Pricing Model • Pay what you use on compute hours & EBS storage
- 24. AWS Lambda Amazon Elasticsearch Service Realtime Analytic The Architecture Amazon CloudWatch Amazon API Gateway Amazon Kinesis Streams
- 25. We are going to send… 100m requests / month
- 26. Amazon API Gateway Ajax / img 1x1 Amazon Kinesis Streams Integration with AWS Service Proxy Map API Payload into Kinesis Stream Payload Cost $350 Cost $18
- 27. Amazon Kinesis Streams AWS Lambda Amazon Elasticsearch Service POST to ES IndexPoll in batches Cost $94 Cost $50
- 29. Kibana Dashboard
- 30. Few Cost Optimisation • API Gateway and Lambda can be very expensive.. • 100m requests = $350 • 200m requests = $700 • Batch requests to API Gateway • Retrieve records in batch from Kinesis • Lambda execution time
- 31. Deployment Cloud Formation for rest Lambda utilises Serverless Framework
- 32. Monitoring and Logging For cost management and visibility CloudWatch integration with all services
- 34. CloudWatch Logs
- 36. Our Success Metrics Revisited Speed of Implementation Must be Cost Effectives No Reinvent the Wheels Less Maintenance
- 37. • Duplicate record on Kinesis Stream • Cold-start on Lambda
- 38. Question?
- 39. Stay in touch - careers.carsalesnetwork.com.au - linkedin.com/carsales.com.au - facebook.com/carsales.com.au - Twitter: @carsales