Speeding Up Spark Performance using Alluxio at China Unicom
0 likes178 views
The document discusses the implementation of Alluxio at China Unicom to enhance Spark performance and address challenges in data orchestration. It highlights improvements in processing speed, resource efficiency, and stability of Spark jobs, leading to increased business agility. Key features of Alluxio 2.x, such as enhanced metadata operations and high availability, further contribute to operational efficiency and service reliability.
Speeding Up Spark Performance using Alluxio at China Unicom
- 1. DATA ORCHESTRATION SUMMI T Speeding Up Spark Performance Using Alluxio in China Unicom Ce Zhang | China Unicom BigData Engineer
- 2. DATA ORCHESTRATION SUMMIT Profile ▪Ce Zhang • China Unicom Big Data Engineer • Alluxio Contributor • Work with Spark, Flink, Alluxio, etc.
- 3. DATA ORCHESTRATION SUMMIT Agenda Background Alluxio at Unicom Alluxio 2.X at Unicom
- 5. DATA ORCHESTRATION SUMMIT Background We have workloads on different compute engines for different business requirements. • Deal with batch jobs with strict deadline • The cluster size reaches a bottleneck • Deal with N+1 batch jobs • Huge resource consumption. Slow in speed. • Deal with traditional data analysis jobs • Stand-alone performance bottlenecks and high cost
- 6. DATA ORCHESTRATION SUMMIT • Spark is more scalable than GreenPlum • Spark consumes less resource and is faster than Hive • Spark is cheaper and deals with much bigger data than Oracle • A unified computing platform has better resource utilization Background HDFS Building a unified computing platform with Spark ecosystem
- 7. DATA ORCHESTRATION SUMMIT • Performance on Spark doesn't compare with the speed on greenplum • Ad-hoc query are too slow for the business users • Lack of stability, and multiple rounds iterative computing tasks are prone to failure Background HDFS The challenges of building a unified computing platform
- 8. DATA ORCHESTRATION SUMMIT • Speed up data read and write performance • Improve data write stability • Taking over Spark cache improves Spark stability Background The unified computing platform architecture Metadata & data cache HDFS
- 9. DATA ORCHESTRATION SUMMIT 2020 Alluxio at Unicom Alluxio usage and business improvement in Unicom
- 10. DATA ORCHESTRATION SUMMIT Alluxio in Unicom-Architecture Alluxio Hive Yarn PrestoFlink Spark Jupyter Hue Zeppelin Hbase HDFS Oracle DRDS Kafka Text File TianYanUnifiedmonitoring platform TianGong Cloud Portal
- 11. DATA ORCHESTRATION SUMMIT Alluxio in Unicom Spark SQL Spark SQL Spark SQL Spark SQL Spark SQL Spark SQL Spark SQL Spark SQL Sink Source Sink Source Sink Sink SinkSink Source Source Sink Source Iterative computing: speed up downstream SQL reading data and improve data write stability • Most of sinks will be reused - cache hit • Sink data in Alluxio in memory
- 12. DATA ORCHESTRATION SUMMIT Share data between Spark Jobs • Taking over Spark cache Alluxio in Unicom Alluxio Spark SQL Spark SQL Spark SQL Data df.write.parquet(”/Data”) spark.read.parquet(”/Data”)
- 13. DATA ORCHESTRATION SUMMIT Alluxio in Unicom-Use replicas HDFS Alluxio Block Block Block Block Block Spark SQL Spark SQL Spark SQL ./bin/alluxio fs setReplication --max 3 --min 1 /foo ./bin/alluxio fs distributedLoad --replication 2 /foo ./bin/alluxio fs pin /foo FileSystem#free BlockBlock Having multiple replicas significantly improves hot data access speed
- 14. DATA ORCHESTRATION SUMMIT Alluxio in Unicom-Optimize memory usage Alluxio Client Alluxio Client Alluxio Client Alluxio Client Alluxio Worker Block Alluxio Worker Block Alluxio Worker Block Alluxio Worker Block We want to limit the number of replicas for a high concurrency access pattern 4 copies
- 15. DATA ORCHESTRATION SUMMIT Solution-DeterministicHashPolicy 2 copies Alluxio Client Alluxio Client Alluxio Client Alluxio Client Alluxio Worker Block Alluxio Worker Block Alluxio Worker Alluxio Worker Alluxio in Unicom-Optimize memory usage space
- 16. DATA ORCHESTRATION SUMMIT Alluxio in Unicom-Data load balance Available Used Alluxio Worker Alluxio Client Alluxio Client Alluxio Client Available Used Alluxio Worker Available Used Alluxio Worker Adjust the data distribution strategy for load balance MostAvailableFirstPolicy
- 17. DATA ORCHESTRATION SUMMIT Alluxio in Unicom-Data load balance Adjust the data distribution strategy for load balance RoundRobinPolicy Available Used Alluxio Worker Alluxio Client Available Used Alluxio Worker Available Used Alluxio Worker 1 2
- 18. DATA ORCHESTRATION SUMMIT Alluxio in Unicom-Performance improvements Alluxio brings performance with Increased stability Metrics Min 25% Med 75% Max GC Time 2s 5s 5s 6s 8s GC Time(+Alluxio) 54ms 0.9s 1s 2s 3s Task Time 1.1min 1.4min 1.5min 1.6min 2.0min Task Time(+Alluxio) 1s 16s 19s 22s 31s Job Time NA NA NA NA 2.1min Job Time(+Alluxio) NA NA NA NA 33s Job Restart Rate NA NA NA NA 100% Job Restart Rate(+Alluxio) NA NA NA NA 0
- 19. DATA ORCHESTRATION SUMMIT ▪Core batch applications • 50,000 Spark jobs -> 350,000 Spark jobs, The size of the mission has been expanded sixfold • 60% increase in the number of users • Business speed up by 100% ▪Core Ad-hoc business • Process 10TB of increasing data • Table with 2.5TB in minute-level queries ▪Oracle • User-level detail statistics Alluxio in Unicom-The value it brings to business
- 20. DATA ORCHESTRATION SUMMIT 2020 Alluxio 2.X at Unicom New features in Alluxio 2.X make a significant business improvement in Unicom
- 21. DATA ORCHESTRATION SUMMIT Alluxio 2.X-RAFT High Availability Distributed Storage (ie. HDFS) Alluxio Standby Master Distributed Quorum (Zookeeper) Alluxio Master Alluxio Standby Master Alluxio Standby Master Alluxio Master RAFT RAFT High Available getting rid of external dependencies
- 22. DATA ORCHESTRATION SUMMIT Alluxio 2.X-Enhanced metadata operations Java Heap RocksDB RocksDB storage metadata brings improvements • Improve the performance and stability of recursive operations in the 100G-plus directory • Java Heap: Operation large directory will lead to OOM error • RocksDB: Operation TB-level directory can be completed in seconds • Reduce the impact of file Inconsistency • Directory in TB-level which contains more than 10,000 files can be fixed within 10 seconds • The unrelated users are not aware
- 23. DATA ORCHESTRATION SUMMIT Alluxio 2.X-Spark write wait time optimization Alluxio Worker Spark SQL /foo/_temporary /foo/part-00001 /foo/part-00002 …… Blocking wait times for Spark writing to a large number of files have improved • 90% reduction in write blocking wait time for files over 100G
- 24. DATA ORCHESTRATION SUMMIT Alluxio 2.X-Multi-version HDFS mount alluxio:/// UserA UserB Path2 Path2Path1 Path1 hdfs:/// Path2Path1 hdfs:/// Path2Path1 Alluxio HDFS 2.7 HDFS 3.1 mount mount
- 26. DATA ORCHESTRATION SUMMIT Summary • Alluxio accelerates Unicom's Spark batch processing and ad-hoc services to ensure business stability and efficiency • Share Spark data with Alluxio to speed up business pipelines and reduce the chance of Job retrying • Performance improved by pinning, controlling replicas, and read/write strategies • Apps improve resource usage by actively free option • Read and write strategies can also be used to ensure that the Alluxio cluster is load balanced • Alluxio 2.X • Enhanced RAFT high availability greatly improves metadata operational performance and availability • Mount multiple versions of HDFS
- 27. DATA ORCHESTRATION SUMMIT Join Alluxio Community Join Slack channel alluxio.io/slack Wechat Public AccountJoin meetup groups near you alluxio-open-source-community/