Ultimate journey towards realtime data platform with 2.5M events per sec
2 likes664 views
The document discusses the journey towards building a real-time data platform capable of handling 2.5 million events per second. It describes migrating Spark processing from on-premises CDH to AWS EMR to improve scalability. Fault tolerance was added through batch processing in Spark and auto-recovery capabilities. Backpressure was enabled through Spark streaming, HDFS, and pulling data into Vertica to prevent overloading downstream systems. Monitoring was enhanced with a separate application to track pipeline metrics. The final platform achieved the performance goals through these architectural changes.
![CAUTION, EMR!
EASY TO ALLOCATE AND EASY TO LOSE EMR NODE
• Concerns mostly m4.4xl as the most popular instance type
LOSING MASTER NODE – LOSING ENTIRE CLUSTER
• Hard to build reliable platform involving multiple AZ [see Fleets model]
• Develop one-step evacuation procedure to another EMR
LUCK OF LACK ON SPECIFIC INSTANCE TYPE
• Can be mitigated by fleets model](https://image.slidesharecdn.com/jeeconf-ultimatejourneytowardsrealtimedpwith2-180521183251/85/Ultimate-journey-towards-realtime-data-platform-with-2-5M-events-per-sec-26-320.jpg)
![DEPLOYMENT DETAILS
MASTER
TASK TASK TASK
…
EMR CLUSTER [YARN]
TASK TASK TASK
…](https://image.slidesharecdn.com/jeeconf-ultimatejourneytowardsrealtimedpwith2-180521183251/85/Ultimate-journey-towards-realtime-data-platform-with-2-5M-events-per-sec-27-320.jpg)
![DEPLOYMENT DETAILS
MASTER
S3
YARN
CONFIG
(zip)
TASK TASK TASK
…
EMR CLUSTER [YARN]
TASK TASK TASK
…](https://image.slidesharecdn.com/jeeconf-ultimatejourneytowardsrealtimedpwith2-180521183251/85/Ultimate-journey-towards-realtime-data-platform-with-2-5M-events-per-sec-28-320.jpg)
![DEPLOYMENT DETAILS
SPARK BINARIES
MASTER
S3
YARN
CONFIG
(zip)
TASK TASK TASK
…
EMR CLUSTER [YARN]DOCKER CONTAINER
DRIVER APP
TASK TASK TASK
…](https://image.slidesharecdn.com/jeeconf-ultimatejourneytowardsrealtimedpwith2-180521183251/85/Ultimate-journey-towards-realtime-data-platform-with-2-5M-events-per-sec-29-320.jpg)
![DEPLOYMENT DETAILS
SPARK BINARIES
MASTER
S3
YARN
CONFIG
(zip)
TASK TASK TASK
…
EMR CLUSTER [YARN]DOCKER CONTAINER
DRIVER APP
LOCAL
YARN
CONFIG
TASK TASK TASK
…](https://image.slidesharecdn.com/jeeconf-ultimatejourneytowardsrealtimedpwith2-180521183251/85/Ultimate-journey-towards-realtime-data-platform-with-2-5M-events-per-sec-30-320.jpg)
![DEPLOYMENT DETAILS
SPARK BINARIES
MASTER
S3
YARN
CONFIG
(zip)
TASK TASK TASK
…
EMR CLUSTER [YARN]DOCKER CONTAINER
DRIVER APP
TASK TASK TASK
…
LOCAL
YARN
CONFIG](https://image.slidesharecdn.com/jeeconf-ultimatejourneytowardsrealtimedpwith2-180521183251/85/Ultimate-journey-towards-realtime-data-platform-with-2-5M-events-per-sec-31-320.jpg)
![DEPLOYMENT DETAILS
SPARK BINARIES
MASTER
S3
YARN
CONFIG
(zip)
TASK TASK TASK
…
EMR CLUSTER [YARN]DOCKER CONTAINER
DRIVER APP
TASK TASK TASK
…
LOCAL
YARN
CONFIG
RANCHER](https://image.slidesharecdn.com/jeeconf-ultimatejourneytowardsrealtimedpwith2-180521183251/85/Ultimate-journey-towards-realtime-data-platform-with-2-5M-events-per-sec-32-320.jpg)
![SPARK PERFORMANCE
ONE EXECUTOR PER YARN NODE
• for better cpu and cache utilization, using 16 vcores (aligning to m4.4xl)
ALIGN RDD PARTITIONS TO VCORES
• Repartition data we read from Kakfa [address if there is a skew in kafka partitions]
SPLIT PROCESSING BATCH INTERVAL ONTO RESPONSIBILITY ZONES
• Control each interval separately
FETCH FROM KAFKA ENRICHMENT WRITE TO HIVE
1 minute
8 seconds 20 seconds 20 seconds
STUFF/OVERHEAD
12 seconds](https://image.slidesharecdn.com/jeeconf-ultimatejourneytowardsrealtimedpwith2-180521183251/85/Ultimate-journey-towards-realtime-data-platform-with-2-5M-events-per-sec-38-320.jpg)
![FAULT TOLERANCE
EVENT/BATCH SOURCING
• Spark associates microbatch with batch_id [timestamp]
• Batch_id is partitioned Hive column
• Aggregating and replicating only missed batches
• In case of failures after restart every component shall auto-recover without data losses
SPARK
BATCH 1
HDFS/HIVE
RAW FACT TABLE
BATCH 2
PRESTO
HDFS/HIVE
AGGREGATION TABLE
REPLICATOR
VERTICA
AGGREGATION TABLE
BATCH 1 BATCH 2 BATCH 1](https://image.slidesharecdn.com/jeeconf-ultimatejourneytowardsrealtimedpwith2-180521183251/85/Ultimate-journey-towards-realtime-data-platform-with-2-5M-events-per-sec-45-320.jpg)
![FAULT TOLERANCE
EVENT/BATCH SOURCING
• Spark associates microbatch with batch_id [timestamp]
• Batch_id is partitioned Hive column
• Aggregating and replicating only missed batches
• In case of failures after restart every component shall auto-recover without data losses
SPARK
BATCH 1
HDFS/HIVE
RAW FACT TABLE
BATCH 2
BATCH 3
PRESTO
BATCH 1
HDFS/HIVE
AGGREGATION TABLE
BATCH 2
REPLICATOR
VERTICA
AGGREGATION TABLE
BATCH 1](https://image.slidesharecdn.com/jeeconf-ultimatejourneytowardsrealtimedpwith2-180521183251/85/Ultimate-journey-towards-realtime-data-platform-with-2-5M-events-per-sec-46-320.jpg)
![FAULT TOLERANCE
EVENT/BATCH SOURCING
• Spark associates microbatch with batch_id [timestamp]
• Batch_id is partitioned Hive column
• Aggregating and replicating only missed batches
• In case of failures after restart every component shall auto-recover without data losses
SPARK
BATCH 1
HDFS/HIVE
RAW FACT TABLE
BATCH 2
PRESTO
HDFS/HIVE
AGGREGATION TABLE
REPLICATOR
VERTICA
AGGREGATION TABLE
BATCH 1 BATCH 2 BATCH 1
BATCH 3](https://image.slidesharecdn.com/jeeconf-ultimatejourneytowardsrealtimedpwith2-180521183251/85/Ultimate-journey-towards-realtime-data-platform-with-2-5M-events-per-sec-47-320.jpg)
![FAULT TOLERANCE
EVENT/BATCH SOURCING
• Spark associates microbatch with batch_id [timestamp]
• Batch_id is partitioned Hive column
• Aggregating and replicating only missed batches
• In case of failures after restart every component shall auto-recover without data losses
SPARK
BATCH 1
HDFS/HIVE
RAW FACT TABLE
BATCH 2
PRESTO
HDFS/HIVE
AGGREGATION TABLE
REPLICATOR
VERTICA
AGGREGATION TABLE
BATCH 3
BATCH 1 BATCH 2 BATCH 1
BATCH 3](https://image.slidesharecdn.com/jeeconf-ultimatejourneytowardsrealtimedpwith2-180521183251/85/Ultimate-journey-towards-realtime-data-platform-with-2-5M-events-per-sec-48-320.jpg)
![FAULT TOLERANCE
EVENT/BATCH SOURCING
• Spark associates micro-batch with batch_id [timestamp]
• Batch_id is partitioned Hive column
• Aggregating and replicating only missed batches
• In case of failures after restart every component shall auto-recover without data losses
SPARK
BATCH 1
HDFS/HIVE
RAW FACT TABLE
BATCH 2
BATCH 3
PRESTO
HDFS/HIVE
AGGREGATION TABLE
REPLICATOR
VERTICA
AGGREGATION TABLE
BATCH 1 BATCH 2 BATCH 1
BATCH 3](https://image.slidesharecdn.com/jeeconf-ultimatejourneytowardsrealtimedpwith2-180521183251/85/Ultimate-journey-towards-realtime-data-platform-with-2-5M-events-per-sec-49-320.jpg)
![FAULT TOLERANCE
EVENT/BATCH SOURCING
• Spark associates micro-batch with batch_id [timestamp]
• Batch_id is partitioned Hive column
• Aggregating and replicating only missed batches
• In case of failures after restart every component shall auto-recover without data losses
SPARK
BATCH 1
HDFS/HIVE
RAW FACTS
BATCH 2
BATCH 3
PRESTO
HDFS/HIVE
AGGREGATED FACTS
REPLICATOR
VERTICA
AGGREGATED FACTS
BATCH 1 BATCH 2 BATCH 1
BATCH 3
BATCH 2
BATCH 3](https://image.slidesharecdn.com/jeeconf-ultimatejourneytowardsrealtimedpwith2-180521183251/85/Ultimate-journey-towards-realtime-data-platform-with-2-5M-events-per-sec-50-320.jpg)
![MONITORING FUNDAMENTALS
FUNDAMENTAL REALTIME METRICS
• IN RATE
• OUT RATE
• CURRENT LAG
• ERRORS RATE
• BATCH PROCESSING TIME
• PIPELINE LATENCY
SEPARATED APP INTRODUCED [ aka BANDARLOG ]
• Tracks offsets for kafka, and Hive/Presto and Vertica
• Standalone application
• Open sourced soon
USING DATADOG
• Dashboards, monitors](https://image.slidesharecdn.com/jeeconf-ultimatejourneytowardsrealtimedpwith2-180521183251/85/Ultimate-journey-towards-realtime-data-platform-with-2-5M-events-per-sec-60-320.jpg)
![DASHBOARD EXAMPLE [INGESTION]](https://image.slidesharecdn.com/jeeconf-ultimatejourneytowardsrealtimedpwith2-180521183251/85/Ultimate-journey-towards-realtime-data-platform-with-2-5M-events-per-sec-61-320.jpg)
![DASHBOARD EXAMPLE [AGGREGATIONS]](https://image.slidesharecdn.com/jeeconf-ultimatejourneytowardsrealtimedpwith2-180521183251/85/Ultimate-journey-towards-realtime-data-platform-with-2-5M-events-per-sec-62-320.jpg)
Ultimate journey towards realtime data platform with 2.5M events per sec
- 1. ULTIMATE JOURNEY TOWARDS REALTIME DATA PLATFORM 2.5M / s BORIS TROFIMOV @ SIGMA SOFTWARE
- 2. Leading DWH @ Oath: Major expertise Big Data and Enterprise Cofounder of Odessa JUG Passionate follower of Scala Associate professor at ONPU ABOUT ME
- 3. WHERE IS BIG DATA?
- 4. API BACK OFFICE CUSTOMER WEB PORTAL MOBILE APPS INEGRATION POINTS INTRODUCING DATA PLATFORM
- 5. DOMAIN SERVICE API BACK OFFICE CUSTOMER WEB PORTAL MOBILE APPS MICROSERVICE CORE DOMAIN SERVICES INEGRATION POINTS DOMAIN SERVICE DOMAIN SERVICE DOMAIN SERVICE INTRODUCING DATA PLATFORM
- 6. DOMAIN SERVICE API BACK OFFICE CUSTOMER WEB PORTAL MOBILE APPS MICROSERVICE INFRASTRUCTURE SERVICES SERVICE DISCOVERY SHARED CONFIG DOMAIN DEPENDENCY MANAGEMENT ACL MANAGEMENT MICROSERVICE CORE DOMAIN SERVICES INEGRATION POINTS DOMAIN SERVICE DOMAIN SERVICE DOMAIN SERVICE INTRODUCING DATA PLATFORM
- 7. DOMAIN SERVICE API BACK OFFICE CUSTOMER WEB PORTAL MOBILE APPS MICROSERVICE INFRASTRUCTURE SERVICES SERVICE DISCOVERY SHARED CONFIG DOMAIN DEPENDENCY MANAGEMENT ACL MANAGEMENT MICROSERVICE CORE DOMAIN SERVICES INEGRATION POINTS DOMAIN SERVICE DOMAIN SERVICE DOMAIN SERVICE INTRODUCING DATA PLATFORM BIG DATA ?
- 8. API DATA PLATFORM BACK OFFICE CUSTOMER WEB PORTAL MOBILE APPS MICROSERVICE INFRASTRUCTURE SERVICES SERVICE DISCOVERY SHARED CONFIG DOMAIN DEPENDENCY MANAGEMENT ACL MANAGEMENT MICROSERVICE CORE DOMAIN SERVICES INEGRATION POINTS DOMAIN SERVICE DOMAIN SERVICE DOMAIN SERVICE DOMAIN SERVICE INTRODUCING DATA PLATFORM
- 9. DATA PLATFORM INTRODUCING DATA PLATFORM
- 10. DATA PLATFORM 3rd PARTY PROVIDERS PLATFORM COMPONENTS INTRODUCING DATA PLATFORM events
- 11. DATA PLATFORM INTRODUCING DATA PLATFORM 3rd PARTY PROVIDERS PLATFORM COMPONENTS REPORTING ANALYTICS Major mission: organize data events
- 12. ZOOMING IN DATA PLATFORM INGESTION MODULE REPORTING SERVICE WAREHOUSE VALIDATION ENRICHMENT MODULE RAW DATA AGGREGATIONS MODULE FACTS RAW DATA DIMENSIONS ANALYTICS MODULE CONFIGURATION MODULE DIMENSION UPDATER
- 13. based on © https://blogs.msdn.microsoft.com/agile/2012/07/26/cqrs-journey-guidance-project-released/
- 14. 7 JUNE
- 15. OUR DOMAIN CORE PLATFORM DATA PLATFORM VIDEO PLAYERS CONTENT OWNERS END USERS
- 16. OUR DOMAIN S3 Data Lake 5 PB Vertica 500 TB Raws/Table 600 B Events/Sec 2.5 M Files/Hour/Pipeline 15 K Data/Daly 25 TB DATA LAKE PROCESSING
- 17. ORIGINAL PIPELINE DATA PLATFORM VERTICAS3 HADOOPNGINX REPORTING SERVICE DATA LAG ~1h
- 18. UNITED LAMBDA PLATFORM DATA PLATFORM KAFKA SPARK MEMSQL VERTICAS3 HADOOPNGINX REPORTING SERVICE DATA LAG ~2m DATA LAG ~1h
- 19. UNITED LAMBDA PLATFORM DATA PLATFORM KAFKA SPARK MEMSQL VERTICAS3 HADOOPNGINX REPORTING SERVICE DATA LAG ~2m DATA LAG ~1h
- 20. WHAT WAS GOOD DATA DELIVERY TIME 2 min FINE ON PROD SCALE @ THAT TIME -- 150K/s PAINFUL SCALE UP TO 1M/s
- 21. НЕЛЬЗЯ ПРОСТО ТАК ВЗЯТЬ И ВЫРАСТИ В 20 РАЗ
- 22. WHY WE NEEDED CHANGES ROCKY SCALING Adding/removing nodes to CDH YARN requires yarn restart and downtime for apps Tricky to build quick sandboxes The latest Memsql release 5.X It was not able to operate cluster with > 80 nodes Max supported rate limit 1M events/s, while Business required 2.5M/s ZERO TOLERANCE EC2 faulty nodes could make Spark or Memsql get stuck for a while Buggy HA, even one faulty node could break entire Memsql cluster, make to recreate database and lose data PUSH approach to write data to Memsql MONITORING & ALERTING Find the most relevant metrics Eliminate FALSE POSITIVE and FALSE NEGATIVE errors
- 23. ON A WAY TO 2.5 M / S
- 25. MIGRATING SPARK TO EMR EASY CREATE, EASY DESTROY • easy to … make bill cost a fortune MULTIPLE EMR CLUSTERS • Separating concerns and Isolation • Better to run single application per EMR cluster • Simplified auto-scaling rules STATELESS EMR CLUSTERS • Do not use local HDFS
- 26. CAUTION, EMR! EASY TO ALLOCATE AND EASY TO LOSE EMR NODE • Concerns mostly m4.4xl as the most popular instance type LOSING MASTER NODE – LOSING ENTIRE CLUSTER • Hard to build reliable platform involving multiple AZ [see Fleets model] • Develop one-step evacuation procedure to another EMR LUCK OF LACK ON SPECIFIC INSTANCE TYPE • Can be mitigated by fleets model
- 27. DEPLOYMENT DETAILS MASTER TASK TASK TASK … EMR CLUSTER [YARN] TASK TASK TASK …
- 28. DEPLOYMENT DETAILS MASTER S3 YARN CONFIG (zip) TASK TASK TASK … EMR CLUSTER [YARN] TASK TASK TASK …
- 29. DEPLOYMENT DETAILS SPARK BINARIES MASTER S3 YARN CONFIG (zip) TASK TASK TASK … EMR CLUSTER [YARN]DOCKER CONTAINER DRIVER APP TASK TASK TASK …
- 30. DEPLOYMENT DETAILS SPARK BINARIES MASTER S3 YARN CONFIG (zip) TASK TASK TASK … EMR CLUSTER [YARN]DOCKER CONTAINER DRIVER APP LOCAL YARN CONFIG TASK TASK TASK …
- 31. DEPLOYMENT DETAILS SPARK BINARIES MASTER S3 YARN CONFIG (zip) TASK TASK TASK … EMR CLUSTER [YARN]DOCKER CONTAINER DRIVER APP TASK TASK TASK … LOCAL YARN CONFIG
- 32. DEPLOYMENT DETAILS SPARK BINARIES MASTER S3 YARN CONFIG (zip) TASK TASK TASK … EMR CLUSTER [YARN]DOCKER CONTAINER DRIVER APP TASK TASK TASK … LOCAL YARN CONFIG RANCHER
- 33. CDH vs EMR E M RC D H Cannot scale out/in on demand Is able to scale out/in on demand No extra cost (for community license) Extra ~30% to EC2 costs Per second billing (!) Adding machines to CDH requires restarting Yarn No Yarn restart Easy configuration management via CM Limited configuration available during EMR creation Classic Yarn cluster Ordinary Yarn under hood, imposes EMR-driven way to deploy apps Single CDH per AZ EMR cluster on demand as unit of clustering
- 35. MAKING SPARK WRITE FASTER USING CUSTOM HADOOP COMMITER • FileOutputCommiter committer with V2 option to exclude file moving in HDFS/S3 WRITE DATAFRAME TO HDFS FIRST • Spark writes to HDFS directly into partitioned folder and registers new partition in Hive
- 36. WRITING FASTER – FILE FORMATS MOST STABLE PERFORMANCE ON ORC UNCOMPRESSED • spark apps writes raw data in ORC • presto reads ORC and writes aggregations in ORC • replication uses ORC to send delta to Vertica BEST PERFORMANCE ON HDFS BLOCK SIZE AND STRIP 64M • Thankfully to strict retention policy 6 hours ENABLING hive.orc.use-column-names=true • simplifies Spark app, allowing to write dataframe as is, presto accesses columns by name • allows to evolve/modify schema for dataframe and database independently
- 38. SPARK PERFORMANCE ONE EXECUTOR PER YARN NODE • for better cpu and cache utilization, using 16 vcores (aligning to m4.4xl) ALIGN RDD PARTITIONS TO VCORES • Repartition data we read from Kakfa [address if there is a skew in kafka partitions] SPLIT PROCESSING BATCH INTERVAL ONTO RESPONSIBILITY ZONES • Control each interval separately FETCH FROM KAFKA ENRICHMENT WRITE TO HIVE 1 minute 8 seconds 20 seconds 20 seconds STUFF/OVERHEAD 12 seconds
- 40. FRIENDLY REMINDER DATA PLATFORM KAFKA SPARK MEMSQL NGINX REPORTING SERVICE
- 41. INTRODUCING PRESTO DATA PLATFORM KAFKA SPARK PRESTO VERTICANGINX REPORTING SERVICE DATA LAG ~3m
- 42. UNDER HOOD Aggregations and replications are running every minute Presto uses dimensions hosted outside. Using Memsql with realtime updates VERTICA NODE REPORTING SERVICE SPARK, EMR HDFS NODE NODE COLLOCATED HDFS/PRESTO PRESTO REPLICATORS JENKINS SCHEDULER MEMSQL
- 44. FAULT TOLERANCE EMR FLEETS MODEL • New feature • Allows to focus on cores instead of machines • Allows provisioning nodes over multiple AZ SPARK SPECULATION & BLACK LISTING • Faulty nodes is total disaster (c) • Spark Feature request to introduce minimal speculation interval (conflict with DirectCommiter)
- 45. FAULT TOLERANCE EVENT/BATCH SOURCING • Spark associates microbatch with batch_id [timestamp] • Batch_id is partitioned Hive column • Aggregating and replicating only missed batches • In case of failures after restart every component shall auto-recover without data losses SPARK BATCH 1 HDFS/HIVE RAW FACT TABLE BATCH 2 PRESTO HDFS/HIVE AGGREGATION TABLE REPLICATOR VERTICA AGGREGATION TABLE BATCH 1 BATCH 2 BATCH 1
- 46. FAULT TOLERANCE EVENT/BATCH SOURCING • Spark associates microbatch with batch_id [timestamp] • Batch_id is partitioned Hive column • Aggregating and replicating only missed batches • In case of failures after restart every component shall auto-recover without data losses SPARK BATCH 1 HDFS/HIVE RAW FACT TABLE BATCH 2 BATCH 3 PRESTO BATCH 1 HDFS/HIVE AGGREGATION TABLE BATCH 2 REPLICATOR VERTICA AGGREGATION TABLE BATCH 1
- 47. FAULT TOLERANCE EVENT/BATCH SOURCING • Spark associates microbatch with batch_id [timestamp] • Batch_id is partitioned Hive column • Aggregating and replicating only missed batches • In case of failures after restart every component shall auto-recover without data losses SPARK BATCH 1 HDFS/HIVE RAW FACT TABLE BATCH 2 PRESTO HDFS/HIVE AGGREGATION TABLE REPLICATOR VERTICA AGGREGATION TABLE BATCH 1 BATCH 2 BATCH 1 BATCH 3
- 48. FAULT TOLERANCE EVENT/BATCH SOURCING • Spark associates microbatch with batch_id [timestamp] • Batch_id is partitioned Hive column • Aggregating and replicating only missed batches • In case of failures after restart every component shall auto-recover without data losses SPARK BATCH 1 HDFS/HIVE RAW FACT TABLE BATCH 2 PRESTO HDFS/HIVE AGGREGATION TABLE REPLICATOR VERTICA AGGREGATION TABLE BATCH 3 BATCH 1 BATCH 2 BATCH 1 BATCH 3
- 49. FAULT TOLERANCE EVENT/BATCH SOURCING • Spark associates micro-batch with batch_id [timestamp] • Batch_id is partitioned Hive column • Aggregating and replicating only missed batches • In case of failures after restart every component shall auto-recover without data losses SPARK BATCH 1 HDFS/HIVE RAW FACT TABLE BATCH 2 BATCH 3 PRESTO HDFS/HIVE AGGREGATION TABLE REPLICATOR VERTICA AGGREGATION TABLE BATCH 1 BATCH 2 BATCH 1 BATCH 3
- 50. FAULT TOLERANCE EVENT/BATCH SOURCING • Spark associates micro-batch with batch_id [timestamp] • Batch_id is partitioned Hive column • Aggregating and replicating only missed batches • In case of failures after restart every component shall auto-recover without data losses SPARK BATCH 1 HDFS/HIVE RAW FACTS BATCH 2 BATCH 3 PRESTO HDFS/HIVE AGGREGATED FACTS REPLICATOR VERTICA AGGREGATED FACTS BATCH 1 BATCH 2 BATCH 1 BATCH 3 BATCH 2 BATCH 3
- 53. BACKPRESSURE ENABLED DATA PLATFORM KAFKA SPARK PRESTO VERTICANGINX REPORTING SERVICE
- 54. BACKPRESSURE ENABLED KAFKA SPARK PRESTO VERTICANGINX REPORTING SERVICE SPARK STREAMING BACKPRESSURE • MUST HAVE for variable rate • FEATURE contributed to Spark master with back pressure initial max rate for direct mode
- 55. BACKPRESSURE ENABLED KAFKA SPARK PRESTO VERTICANGINX REPORTING SERVICE HDFS VALVE • HDFS between Spark and Presto • Retention policy 12h
- 56. BACKPRESSURE ENABLED KAFKA SPARK PRESTO VERTICANGINX REPORTING SERVICE PULL WRITE • Using Vertica’s query COPY from HDFS to let Vertica read data with own rate
- 57. BACKPRESSURE ENABLED KAFKA SPARK PRESTO VERTICANGINX REPORTING SERVICE KAFKA OUTAGES • Lua writes events directly to Kafka • Unsent events stored locally and sent to S3 • NiFi periodically rends that data back to Kafka
- 59. I AM A GOD I HAVE NO IDEA WHAT’S GOING ON
- 60. MONITORING FUNDAMENTALS FUNDAMENTAL REALTIME METRICS • IN RATE • OUT RATE • CURRENT LAG • ERRORS RATE • BATCH PROCESSING TIME • PIPELINE LATENCY SEPARATED APP INTRODUCED [ aka BANDARLOG ] • Tracks offsets for kafka, and Hive/Presto and Vertica • Standalone application • Open sourced soon USING DATADOG • Dashboards, monitors
- 64. WHAT WE HAVE ACHIEVED SCLABLE PRODUCTION • Ability to grow further beyond 1M/s up to 2.5M STABLE PRODUCTION ENVIRONMENT • fault tolerant components, easier to recover LESS EXPENSIVE • Smaller Spark cluster (-50%) • Presto cluster is smaller than Memsql-driven one (30%) SIMPLIFIED MAINTENANCE • Auto recovery and scaling • No wakeups over night
- 65. THANK YOU