How the Development Bank of Singapore solves on-prem compute capacity challenges with cloud bursting
1 like350 views
The Development Bank of Singapore (DBS) has evolved its data platforms over three generations to address big data challenges and the explosion of data. It now uses a hybrid cloud model with Alluxio to provide a unified namespace across on-prem and cloud storage for analytics workloads. Alluxio enables "zero-copy" cloud bursting by caching hot data and orchestrating analytics jobs between on-prem and cloud resources like AWS EMR and Google Dataproc. This provides dynamic scaling of compute capacity while retaining data locality. Alluxio also offers intelligent data tiering and policy-driven data migration to cloud storage over time for cost efficiency and management.
How the Development Bank of Singapore solves on-prem compute capacity challenges with cloud bursting
- 1. How the Development Bank of Singapore solves on-prem compute capacity challenges with cloud bursting Vitaliy Baklikov | VP at DBS Dipti Borkar | VP Product at Alluxio
- 2. About DBS • Headquartered in Singapore • Largest bank in South East Asia • Present in 18 markets globally, including 6 priority markets • Singapore, Hong Kong, China, India, Indonesia and Taiwan • We have a very cool digiBank app • And lots lots lots of data systems
- 3. AWS EnginesOnprem Engines HDFS Object Store Evolution of Data Platforms at DBS Generation 1 • Boxed data • Monolithic/Closed Systems • Proprietary HW/SW • Data for Targeted Use Cases Generation 2 • Big Data Explosion • Hadoop Data Lakes • Commodity HW and Hadoop Ecosystem • Compute tied to Storage Generation 3 • Data Democratization • Cloud Native platform… Hybrid! Multi! • Open Source Engines • AI/ML Centric Teradata Informatica SAS HadoopTeradata Informatica SAS Teradata Informatica SAS Hadoop
- 4. 4(Layer B) (Layer C/D)
- 5. Challenges 1. Data Lake built on local Object Store – Expensive rename operation – Object listing is slow – Variable performance – Data locality is gone 2. Multiple Data Silos 3. Limited on-premise compute capacity – Legacy ITIL processes for Infra provisioning – No dynamic scale out/in
- 6. Alluxio at DBS Mount HDFS from other platforms into common Alluxio cluster Unified Namespace Object store Analytics Hybrid cloud bursting Caching layer for hot data to speed up Presto and Spark jobs Extend Alluxio cluster into AWS VPC Run EMR for model training and bring the results back to on-prem
- 7. Advanced Analytics on Object Storage The Use Case • Cash-in-Transit use case • Forecast cash replenishment schedule for each ATM • Produce a delivery graph by 4am each morning The Challenges • Strict fixed SLA • Need to load all history data for the forecasting model… daily!
- 8. Alluxio at DBS Mount HDFS from other platforms into common Alluxio cluster Unified Namespace Object store Analytics Hybrid cloud bursting Caching layer for hot data to speed up Presto and Spark jobs Extend Alluxio cluster into AWS VPC Run EMR for model training and bring the results back to on-prem
- 9. Burst processing into the Cloud The Use Case – Call Center project – Millions of calls annually – Why do our customers call us? – What do they do before picking up the phone? • Reconstruct customer journey • Predict the reason for the call The Challenges – Transcript quality – Need lots of compute • >30TB of clickstream, transaction, customer, and product data • >20TB of audio files – Need dynamic compute for training and analysis
- 11. Next Steps 1. Use Cloud offerings for Speech-to-Text 2. Move more workloads to the cloud 3. Extend to other clouds
- 12. Data Orchestration for the Cloud
- 13. Data Orchestration for the Cloud Java File API HDFS Interface S3 Interface REST APIPOSIX Interface HDFS Driver Swift Driver S3 Driver NFS Driver Enable innovation with any frameworks running on data stored anywhere Data Analyst Data Engineer Storage Ops Data Scientist Lines of Business
- 14. Data Orchestration for the Cloud Java File API HDFS Interface S3 Interface REST APIPOSIX Interface HDFS Driver Swift Driver S3 Driver NFS Driver Enable innovation with any frameworks running on data stored anywhere
- 15. Problem: HDFS cluster is compute- bound & complex to maintain AWS Public Cloud IaaS Spark Presto Hive TensorFlow Alluxio Data Orchestration and Control Service On Premises Connectivity Datacenter Spark Presto Hive Tensor Flow Alluxio Data Orchestration and Control Service Barrier 1: Prohibitive network latency and bandwidth limits • Makes hybrid analytics unfeasible Barrier 2: Copying data to cloud • Difficult to maintain copies • Data security and governance • Costs of another silo Step 1: Hybrid Cloud for Burst Compute Capacity • Orchestrates compute access to on-prem data • Working set of data, not FULL set of data • Local performance • Scales elastically • On-Prem Cluster Offload (both Compute & I/O) Step 2: Online Migration of Data Per Policy • Flexible timing to migrate, with less dependencies • Instead of hard switch over, migrate at own pace • Moves the data per policy – e.g. last 7 days “Zero-copy” bursting to the cloud
- 16. Data Elasticity via Unified Namespace Enables effective data management across different Under Store - Uses Mounting withTransparent Naming
- 17. Unified Namespace: Global Data Accessibility Transparent access to understorage makes all enterprise data available locally SUPPORTS • HDFS • NFS • OpenStack • Ceph • Amazon S3 • Azure • Google Cloud IT OPS FRIENDLY • Storage mounted into Alluxio by central IT • Security in Alluxio mirrors source data • Authentication through LDAP/AD • Wireline encryption HDFS #1 Object Store NFS HDFS #2
- 18. Bursting on AWS EMR Presto Hive Cluster Metadata & Data cache Presto Hive Metadata & Data cache Compute-driven Continuous sync Compute-driven Continuous sync 18
- 19. Bursting on Google Cloud: Dataproc Presto Hive Metadata & Data cache Presto Hive Metadata & Data cache Compute-driven Continuous sync Compute-driven Continuous sync 19 § Google Dataproc with Alluxio (init action integration available) Google Dataproc Cluster
- 20. Alluxio MasterZookeeper / RAFT Standby Master WAN Alluxio Client Alluxio Client Alluxio Worker RAM / SSD / HDD Alluxio Worker RAM / SSD / HDD Alluxio Reference Architecture … … Application Application Under Store 1 Under Store 2
- 21. Spark Presto Hive TensorFlow RAM Framework Read file /trades/us Bucket Trades Bucket Customers Data requests Feature Highlight: Data Caching for faster compute Read file /trades/us again Read file /trades/top Read file /trades/top Variable latency with throttling Read file /trades/us again Read file /trades/top Read file /trades/top Read file /trades/us again Read file /trades/top Read file /trades/top Read file /trades/us again Read file /trades/top Read file /trades/top Read file /trades/us again
- 22. Spark Presto Hive TensorFlow RAM Framework Read file /trades/us Trades Directory Customers Directory Data requests ”Zero-copy” bursting under the hood Read file /trades/us again Read file /trades/top Read file /trades/top Variable latency with throttling Read file /trades/us again Read file /trades/top Read file /trades/top Read file /trades/us again Read file /trades/top Read file /trades/top Read file /trades/us again Read file /trades/top Read file /trades/top Read file /trades/us again
- 23. Spark Presto Hive TensorFlow RAM SSD Disk Framework Bucket Trades Bucket Customers Data requests Feature Highlight - Intelligent Tiering for resource efficiency Read file /customers/145 Out of memory Variable latency with throttling Data moved to another tier
- 24. Spark Presto Hive TensorFlow RAM SSD Disk Framework New Trades Policy Defined Move data > 90 days old to Feature Highlight – Policy-driven Data Management S3 Standard Policy interval : Every day Policy applied everyday
- 25. Questions?