Powering Interactive Analytics with Alluxio and Presto
0 likes146 views
The document discusses the integration of Alluxio and Presto for interactive analytics, outlining challenges related to metadata sync and memory contention. It presents various solutions, including exporting Glue tables to a Hive metastore and deploying custom Presto, while also emphasizing the importance of performance improvements and careful resource management. Key lessons learned include the significance of data locality, the potential growth of the Alluxio metastore, and the necessity of comprehensive monitoring.
![1313
™
© All Rights Reserved
™
Problem 2: Alluxio metadata sync for S3 changes
Solution:
alluxio fs ls [-R|-f]
Can be executed:
● Upon file change events from S3
● Periodically with crontab](https://image.slidesharecdn.com/walkme-201212222123/85/Powering-Interactive-Analytics-with-Alluxio-and-Presto-13-320.jpg)
Powering Interactive Analytics with Alluxio and Presto
- 1. 1 Powering interactive analytics with Alluxio and Presto Dmytro Dermanskyi DATA ORCHESTRATION SUMMIT 2020
- 2. 22 ™ © All Rights Reserved ™ WalkMe: Digital Adoption Platform
- 3. 33 ™ © All Rights Reserved ™ Insights: WalkMe’s web analytics solution
- 4. 44 ™ © All Rights Reserved ™ Funnels analysis
- 5. 55 ™ © All Rights Reserved User must perform these steps to submit a support ticket 1 2 3 Click “Contact support” link Visit “Submit a ticket” page Fill in the details and open the ticket1 2 3 Funnel example: “Submit support ticket” journey
- 6. 66 ™ © All Rights Reserved ™ Funnel example: UI
- 7. 77 ™ © All Rights Reserved ™ Funnel example: SQL behind the scenes
- 8. 88 ™ © All Rights Reserved ™ Initial architecture
- 9. 99 ™ © All Rights Reserved ™ Challenges on the way to Alluxio 1. Glue tables data location paths are fixed to s3:// “filesystem”. 2. Alluxio metadata needs to be periodically refreshed to sync changes in S3 since some S3 files are written not through Alluxio. 1. In a collocated deployment we need to carefully distribute memory allocations between Alluxio and Presto processes. By default EMR gives Presto JVMs most of available RAM.
- 10. 1010 ™ © All Rights Reserved ™ Problem 1: Pointing Glue tables to Alluxio Solution: 1. Export Glue catalog into a classic Hive metastore. 2. During the export fix table location paths to use alluxio:// instead of s3://. 3. Connect EMR Presto to the Hive metastore instead of Glue catalog. 4. Repeat the steps 1 and 2 periodically to sync changes in Glue(e.g. new partitions, new tables) to the Hive metastore.
- 11. 1111 ™ © All Rights Reserved ™ Problem 1: Pointing Glue tables to Alluxio Disadvantages of the solution: ● Glue-to-Hive export job is really complex. ● Necessary to keep two table catalogs: Glue and Hive metastore. Resolving and syncing schema changes is difficult. ● The export job depends on proprietary Glue API which can change with time and break the job.
- 12. 1212 ™ © All Rights Reserved ™ Problem 1: Pointing Glue tables to Alluxio Possible alternative solution: Alluxio Catalog Service Just delegate the problem to Alluxio https://docs.alluxio.io/os/user/stable/en/core-services/Catalog.html
- 13. 1313 ™ © All Rights Reserved ™ Problem 2: Alluxio metadata sync for S3 changes Solution: alluxio fs ls [-R|-f] Can be executed: ● Upon file change events from S3 ● Periodically with crontab
- 14. 1414 ™ © All Rights Reserved ™ Problem 3: Presto vs Alluxio memory contention By default EMR gives Presto JVM processes most available RAM: EC2 node type RAM EMR Presto -Xmx c5.2xlarge 16GB 12GB r5.4xlarge 128GB 100GB c5.4xlarge 32GB 24.5GB c4.8xlarge 60Gb 47GB c5.9xlarge 72GB 55GB Remaining memory may not be enough for Alluxio processes. Especially if you want Alluxio to cache data in RAM.
- 15. 1515 ™ © All Rights Reserved ™ Problem 3: Presto vs Alluxio memory contention EMR doesn’t provide means to adjust default Presto JVM config. But we need such control to appropriately distribute memory allocation for processes.
- 16. 1616 ™ © All Rights Reserved ™ Problem 3: Presto vs Alluxio memory contention Solution 1: Don’t use EMR Presto but deploy our own Presto distribution: ● This brings additional complexity and more work. ● Though also gives more flexibility for Presto version choice and configuration.
- 17. 1717 ™ © All Rights Reserved ™ Problem 3: Presto vs Alluxio memory contention Solution 2: Use EMR bootstrap script to setup a crontab script that fixes EMR Presto JVM config as soon as it provisioned by EMR: NOTE: EMR runs bootstrap scripts before installing Presto ● Allows reusing EMR Presto distribution (no need to deploy own Presto). ● “Hacky” solution.
- 18. 1818 ™ © All Rights Reserved ™ Alluxio-based architecture
- 19. 1919 ™ © All Rights Reserved ™ Performance improvement Average query execution time has dropped from ~20 sec to ~1 sec.
- 20. 2020 ™ © All Rights Reserved ™ Lessons learned: Memory cannot be overcommitted ● Make sure to not use default EMR Presto JVM config since there might be not enough memory left for Alluxio processes. Otherwise beware OOM crashes. ● Chose appropriate caching medium for Alluxio: ○ To cache in RAM use memory optimized instances. ○ Consider caching on SSD. Both NVMe and EBS gp2 disks can give significant performance boost.
- 21. 2121 ™ © All Rights Reserved ™ Lessons learned: Data locality matters We tried deploying Alluxio as a separate from Presto cluster: + This design allows decoupling of storage (S3), caching layer (Alluxio) and compute (Presto). You can give Alluxio as much resources (RAM/SSD) as necessary. - Query performance is at least 2-3 times worse comparing to the collocated deployment.
- 22. 2222 ™ © All Rights Reserved ™ Lessons learned: Alluxio metastore can grow large When the number of files in UFS is becoming huge (e.g. hundreds of thousand) consider the following: ● Alluxio journal will have gigabytes of size. Make sure the disk where it’s kept is large enough. Or use UFS for it (though it’s not so simple). ● It may be worth switching from HEAP to RocksDB-based metastore. This way gigabytes of master RAM taken by metastore can be freed up.
- 23. 2323 ™ © All Rights Reserved ™ Lessons learned: Monitor everything! What can go wrong: ● Presto workers can crash ● Alluxio workers can crash ● Presto/Alluxio masters can crash ● EC2 instance can run out of disk space ● Queries can start to fail because of lots of different reasons ● Queries can start to just hang and timeout ● You name it ...