Deploying Apache Kylin on AWS and designing a task scheduler for it
- 1. Deploying Apache Kylin on AWS And designing a task scheduler for it Chase Zhang Strikingly
- 2. Outline Introduction Strikingly Analytics Service of Strikingly Deploy Apache Kylin on AWS Overview Containerizing Kylin Maintenance Scheduler for Kylin System Designing Goals Basic Idea & Implementation Tasks, Executors and Services Concurrency and Fault Tolerance Maintenance and Monitoring Conclusion
- 4. Introduction Strikingly At Strikingly, we are devoted to provide convenient and one stop website building solution to our customers.
- 5. Introduction Analytics Service of Strikingly The version 0 of our analytics service is Google Analytics Strikingly Google Analytics User Pages User Register / Get Track IDSet Track ID Generate User's website Collect Page Views Data Serve User Query Figure: Google Analytics
- 6. Introduction Analytics Service of Strikingly The version 1 of our analytics service is through Keen IO, a 3rd party service Strikingly User Pages User View Analytics Generate User's website Keen.IO Serve User Query Collect Page Views Data Figure: Keen IO
- 7. Introduction Analytics Service of Strikingly The version 2 of our analytics services is combining Keen IO and Apache Kylin Strikingly User Pages User View Analytics Generate User's website Keen.IO Collect Page Views Data Apache Kylin Serve User Query Figure: Keen IO + Apache Kylin
- 8. Deploy Apache Kylin on AWS Overview EMR ECS 7070 33345 33345Kylin (query) Kylin (query) Kylin (query) Kylin (job) Application Load Balancer Hadoop Hive HBase Hive HBase S3 33347 Target Group 80 Hadoop Hadoop Hive YARN HBase Query Requests Keen.IO Figure: Deploy Apache Kylin on AWS
- 9. Deploy Apache Kylin on AWS Containerizing Kylin Hive HBase HDFS Apache Kylin MapReduce / Spark Figure: Apache Kylin is “Stateless”
- 10. Deploy Apache Kylin on AWS Containerizing Kylin Problem We’d like to ▶ Deploy Kylin on multiple regions ▶ Customize behaviors with environment variables ▶ Build a single docker image and run everywhere
- 11. Deploy Apache Kylin on AWS Containerizing Kylin Hive HBase Hadoop YARN Kylin Configuration Files Templates Configuration Files Substitute variables Start running Figure: Launching Kylin with customized script
- 12. Deploy Apache Kylin on AWS Maintenance Problem Two problems while maintaining this system: ▶ Auto scale and dynamic ports ▶ Clean-up and back-up
- 13. Deploy Apache Kylin on AWS Maintenance ECS 33345 33345 xxxxx Kylin (query) Kylin (query) Kylin (query) Application Load Balancer 33347 Target Group 80 Query Requests Kylin (query)7070Kylin (job) Figure: Auto Scale and Dynamic Listening Ports
- 14. Deploy Apache Kylin on AWS Maintenance ./bin/metastore.sh backup ./bin/metastore.sh restore ./bin/metastore.sh clean ./bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob Figure: Clean-up and back-up tools
- 15. Deploy Apache Kylin on AWS Maintenance Solution A customized task scheduler.
- 16. Scheduler for Kylin System Designing Goals ▶ Customizing task scheduling ▶ Making system robust and fault tolerant ▶ Solving both previously mentioned maintenance problems
- 17. Scheduler for Kylin System Basic Idea & Overall Design The Systemd (Anti-UNIX) philosophy ▶ Scheduler works as a central service ▶ Other components work as RPC services
- 18. Scheduler for Kylin System Basic Idea & Overal Design Scheduler Target Group Kylin (query) Kylin (query) Kylin (query) Kylin (query) Kylin (job) HBase Hive DynamoDB S3 Keen.IO Kylin (query)
- 19. Scheduler for Kylin System Basic Idea & Overall Design Implementation details: ▶ Applying FP9 and Actor Model10 ideas ▶ Implemented with Scala11 and Akka12 ▶ Interact with Hadoop components through Java libraries 9 https://en.wikipedia.org/wiki/Functional_programming 10 https://en.wikipedia.org/wiki/Actor_model 11 http://scala-lang.org/ 12 https://akka.io/
- 20. Scheduler for Kylin System Basic Idea & Overall Design Control Actor Consistent Hashing Router Task Actor Executor Scheduler 1 2 1 22 3 3 3 1 2 3 1 Control Message Task Message Service Figure: Scheduler’s Actor System
- 21. Scheduler for Kylin System Tasks, Executors and Services ▶ Task = immutable message ▶ Task has a type for executor ▶ Executor call services to work ▶ Task categories: planning tasks, working tasks, maintaining tasks
- 22. Scheduler for Kylin System Tasks, Executors and Services PlanDataRefresh PlanCubeMaintenance HiveTableRefresh KylinCubeBuild KylinCubeRefresh KylinCubeMerge Hive Service Kylin Service Hourly Daily Need import new data? Need build a new segment? Need refresh old segments? Need fill holes between segments? Need merge segments? Need fill holes in hive table? Hive table has been refreshed, refresh segment Planning Tasks Working Tasks Services Message Storage Service Figure: Planning Tasks and Working Tasks
- 23. Scheduler for Kylin System Tasks, Executors and Services KylinMetadataBackup KylinMetadataCleanup KylinMetadataRestore KylinHBaseTableCleanup HBase Service Kylin Service AWS Service S3 Apache Kylin KYLIN_XWFQ12 kylin_metadata kylin-metadata-backups Update Cache Get Cube Info Delete Table Read MetadataDelete Row Write ZIP File Read ZIP File Write Table Get Cube Info Figure: Maintaining Tasks
- 24. Scheduler for Kylin System Concurrency and Fault Tolerance Problem We’d like to execute tasks in order ▶ Maintaining tasks run exclusively ▶ Tasks of the same cube run execlusively
- 25. Scheduler for Kylin System Concurrency and Fault Tolerance Solution Two manners to solve this problem: ▶ ReadWriteLock ▶ ConsistentHashingRouter
- 26. Scheduler for Kylin System Concurrency and Fault Tolerance Problem We’d like to be fault tolerant: 1. Recovering from failures 2. Filling missed segment gaps 3. Recording history
- 27. Scheduler for Kylin System Concurrency and Fault Tolerance Solution We’re taking multiple manners to solve this problem: 1. Assigning each task with a Unique ID 2. Persisting task message with progress to DynamoDB 3. Implementing planning and working tasks carefully to be issue aware
- 28. Scheduler for Kylin System Concurrency and Fault Tolerance ControlActor TaskActor Executor Consistent HashingRouterinit running finish error DynamoDB DynamoDB DynamoDB TaskMessage Acquire Lock Release Lock Figure: Concurrency and Message Persistent
- 29. Scheduler for Kylin System Maintenance and Monitoring Problem We still have two trival problems to solve: ▶ Manually performing actions ▶ Task monitoring and error notification
- 30. Scheduler for Kylin System Maintenance and Monitoring How to design the user interface of scheduler?
- 31. Scheduler for Kylin System Maintenance and Monitoring Introducing scheduler slack bot...
- 32. Scheduler for Kylin System Maintenance and Monitoring Event Bus Control Actor Consistent Hashing Router Task Actor Executor Service SlackBot Actor User Command Figure: Scheduler Slack Bot
- 33. Scheduler for Kylin System Maintenance and Monitoring Figure: List task status
- 34. Scheduler for Kylin System Maintenance and Monitoring Figure: List Kylin Job Progress
- 35. Conclusion ▶ With Apache Kylin, we’re providing a sub-second web analytics service ▶ With little effort, we managed to deploy Apache Kylin with docker container ▶ With the scheduler, we deployed the system on AWS without losses of features ▶ We’ve made the system concurrency safe and robust
- 36. Conclusion Version 3? But wait, we still have a problem, don’t we?
- 37. Conclusion Version 3? User Keen.IOS3 North America S3 Tokyo, Japan S3 Beijing, China User 5 minutes10 minutes20 minutes Page Views Figure: Data Transfer Delay of Keen IO
- 38. Conclusion Version 3? User S3 Tokyo, Japan Application Load Balancer S3 Beijing, China User 5 minutes Application Load Balancer 5 minutes Page Views Page Views Figure: Collecting Data with ALB?
- 39. Thank you! BTW, we’re still hiring Data Platform Engineer: 1. Writing Scala 2. Working on AWS 3. Working with Apache Kylin 4. Working on our “Project Manhattan”