Building a Real-time Stream Processing Pipeline - Kinesis Data Firehose, Amazon Elasticsearch, Amazon Athena
Download as PPTX, PDF2 likes496 views
The document outlines a presentation on building a real-time stream processing pipeline for ad tracking and serving by Deltax, led by CTO Akshay Surve. It discusses the transition from batch processing to stream processing, highlighting the use of AWS tools like Kinesis Data Firehose, Amazon Elasticsearch, and Amazon Athena for data ingestion and analytics. Key challenges addressed include the need for low latency, efficient querying of streaming data, and integration of various cloud services.
Building a Real-time Stream Processing Pipeline - Kinesis Data Firehose, Amazon Elasticsearch, Amazon Athena
- 1. engineering.deltax.com Building a Real-time Stream Processing Pipeline Akshay Surve, CTO DeltaX akshay@deltax.com / @ak47surve Hastag: #awsblr #meetup
- 2. engineering.deltax.com ● 12 years ○ Shipping Ideas, Making Mistakes, GTD ○ Marathons / Hackathons / *-athon :) ● Co-founded DeltaX in 2013 ○ Ad-tech / Product Startup ○ 300+ advertisers across India, APAC and US. About Me 2
- 3. engineering.deltax.com Agenda ● Use-case ● Processing Models ● Old Batch Processing Architecture ○ Challenges ● Goals ● Moving Blocks for a Stream Processing Model ○ Kinesis Data Firehose ○ Amazon ElasticSearch ○ Amazon Athena ● Review New Stream Processing Architecture 3
- 4. engineering.deltax.com Use-case ● Ad Tracking & Ad Serving ● Cloud Architecture 4
- 5. engineering.deltax.com Use-case - Ad Tracking & Ad-serving 5
- 6. engineering.deltax.com Use-case - Ad Tracking & Ad-serving 6
- 7. engineering.deltax.com Use-case - Ad Tracking & Ad-serving Advertiser 7
- 8. engineering.deltax.com Use-case - Ad Tracking & Ad-serving Event 8
- 9. engineering.deltax.com Use-case - Ad Tracking & Ad-serving Timestamp 9
- 11. engineering.deltax.com ● Batch Processing ● Stream Processing Processing Models 11
- 12. engineering.deltax.com ● Batch Processing Processing Models Input OutputBatch Job(s) 12
- 13. engineering.deltax.com ● Stream Processing Processing Models Queue Stream Processor Output 13
- 14. engineering.deltax.com ● Batch vs Stream Processing Models Batch Stream High Latency Low Latency Static Files Event Streams Snapshot Continuous Window 14
- 17. engineering.deltax.com Batch Processing (Challenges) ● Modelled around batch processing and not stream processing ● Ingesting JSON files in bulk isn’t natural for SQL - JSON parsing > SQL tables ● Varied levels of aggregations - campaign, ad, device, geo + unique metrics ● Future roadmap - userid cookie pool across advertisers; exchange based cookie matching, etc. become challenges in itself 17
- 18. engineering.deltax.com ● Stream processing as a paradigm suits our use case the best ● Easy to maintain or managed service in the cloud would be ideal ● Developer friendly and peace of mind was of utmost importance ● Being able to ingest streaming data and query summaries was important ● Good to have a way to run batch processing framework for machine learning, data crunching, and analysis Goals 18
- 19. engineering.deltax.com ● Amazon Athena ● Amazon Elasticsearch ● Kinesis Data Firehose Moving Blocks 19
- 22. engineering.deltax.com Amazon Athena ● Persistent Store ● DDL ● Query 22
- 23. engineering.deltax.com Amazon Athena ● Persistent Store (AWS S3) ○ Text files, e.g., CSV, raw logs ○ Apache Web Logs, TSV files ○ JSON (simple, nested) ○ Compressed files ○ Columnar formats such as Apache Parquet & Apache ORC 23
- 24. engineering.deltax.com Amazon Athena ● Persistent Store (AWS S3) ○ JSON events 24
- 25. engineering.deltax.com ● DDL (Apache Hive) Amazon Athena 25
- 26. engineering.deltax.com ● DDL (Apache Hive) Amazon Athena 26
- 27. engineering.deltax.com Amazon Athena ● Query Engine (Presto query engine) ○ In Memory ○ ANSI SQL Compliant 27
- 28. engineering.deltax.com ● Query Engine (Presto query engine) ○ In Memory ○ ANSI SQL Compliant Amazon Athena 28
- 29. engineering.deltax.com ● Query Engine (Presto query engine) ○ In Memory ○ ANSI SQL Compliant Amazon Athena 29
- 30. engineering.deltax.com ● Serverless ● No spin-up time ● Query data directly from S3 ● ANSI SQL Amazon Athena (Advantages) 30
- 31. engineering.deltax.com ● Queries run fast Amazon Athena (Advantages) 31
- 33. engineering.deltax.com Amazon Elasticsearch ● ELK Stack (Searching, Log monitoring) ● Seamless Ingestion (Document-based model) ● Real-time queries (even during ingestion; 30s refresh window; immutability) ● Meant for search; Efficient for time-series (will discuss why?) 33
- 34. engineering.deltax.com Amazon Elasticsearch - Document that gets ingested 34
- 35. engineering.deltax.com Elasticsearch (Internals) ● Elasticsearch Index ○ Inverted Index ○ Doc Values 35
- 36. engineering.deltax.com Elasticsearch (Internals) Deeper into an Elasticsearch Index 36
- 37. engineering.deltax.com Elasticsearch (Internals) ● Deeper into an Elasticsearch Index - Inverted Index ○ The quick brown fox jumped over the lazy dog ○ Quick brown foxes leap over lazy dogs in summer 37
- 38. engineering.deltax.com Elasticsearch (Internals) Deeper into an Elasticsearch Index - Doc Values ● column-oriented fashion that is way more efficient for sorting and aggregations ● Filesystem optimized 38
- 39. engineering.deltax.com ● Integration with AWS ecosystem Amazon Elasticsearch (Advantages) 39
- 40. engineering.deltax.com Amazon Elasticsearch (Advantages) ● Cluster Management (scale out/up) 40
- 41. engineering.deltax.com Amazon Elasticsearch (Advantages) ● Monitoring & Alerts 41
- 42. engineering.deltax.com Amazon Elasticsearch (Advantages) ● Snapshot Recovery / Backup to S3 ● Elasticsearch Upgrades (could be made smoother) 42
- 43. engineering.deltax.com Amazon Elasticsearch (Advantages) ● Integration with AWS ecosystem ● Cluster Management (scale out/up) ● Monitoring & Alerts ● Snapshot Recovery / Backup to S3 ● Elasticsearch Upgrades 43
- 47. engineering.deltax.com Kinesis Data Firehose ● Streaming Data Processing ● Multiple destinations - S3, Redshift, ES ● Intermediate Record transformations (using AWS Lambda) before delivery to the destination ○ Ip2location ○ Enrich flow ○ Ua-parser ● Combine with Kinesis Analytics 47
- 48. engineering.deltax.com Kinesis Data Firehose (source) 48
- 49. engineering.deltax.com Kinesis Data Firehose (transformation) 49
- 50. engineering.deltax.com Kinesis Data Firehose (destination) 50
- 51. engineering.deltax.com Kinesis Data Firehose (ES config options) 51
- 52. engineering.deltax.com Kinesis Data Firehose (ES destination) Node.js (tracker) > 52
- 53. engineering.deltax.com Kinesis Data Firehose (Advantages) ● Cloud Offering 53 Source: https://blog.ippon.tech/spark-storm-s xd-comparison/
- 54. engineering.deltax.com Kinesis Data Firehose (Advantages) ● Pluggability 54 Source: https://www.slideshare.net/AmazonWebServices/aws-reinvent- 2016-analyzing-streaming-data-in-realtime-with-amazon-kinesis-analytics- bdm304
- 57. engineering.deltax.com Stats ● Data: ~12 GB / day (peaks of 32 GB/day) 57
- 58. engineering.deltax.com “The cloud is not a silver bullet” silver bullet ~ noun ‘a simple and seemingly magical solution to a complicated problem’ Twitter - @ak47suve #awsblr #meetup Email - akshay@deltax.com Blog - engineering.deltax.com 58