Deploying Alluxio in the Cloud for Machine Learning
0 likes268 views
- Alluxio provides a POSIX-compatible API that allows machine learning workloads running on Kubernetes to access distributed data sources like HDFS, S3, and GCS as if they were local. - Alluxio can be deployed on Kubernetes using kubectl, Helm charts, or the Alluxio CSI driver to provide data caching and access control. - Recent developments include JNI-Fuse enhancements for improved performance, ongoing work on the Alluxio CSI driver, and collaborations to support Alluxio on Kubernetes.
Deploying Alluxio in the Cloud for Machine Learning
- 1. Deploying Alluxio in the Cloud for Machine Learning Lu Qiu @ Alluxio 06/24/2021 1
- 2. About Me – Lu Qiu ● Software Engineer @ Alluxio ● Email: lu@alluxio.com ● Master Data Science @ GWU ● Areas: Alluxio fault tolerant system, journal system, metrics system, and POSIX API. Alluxio integration with Cloud 2
- 3. Agenda ● What is Alluxio POSIX API ● Deploy Alluxio in K8S for ML workloads ● Updates in Alluxio POSIX Developments 3
- 4. What is Alluxio POSIX API 4
- 5. Apps Connecting to Alluxio via POSIX API 5
- 6. Accessing Remote/Distributed Data as Local Directories 6 HDFS #1 Obj Store NFS HDFS #2 Connecting to • HDFS • Amazon S3 • Azure • Google Cloud • Ceph • NFS • Many more
- 7. Alluxio Server Alluxio Server Model Training Distributed Caching w/ Unified Namespace Alluxio Server A B /path1/file1 /path2/file2 C A B C A Model Training Model Training 7
- 8. Deploy Alluxio in K8s for ML Workloads 8
- 9. Use Alluxio POSIX API in ML/AI Training 9 Alluxio Server Alluxio Server 9 ... GPU Instance
- 10. Train in Cloud Kubernetes with Alluxio - Provision the whole Kubernetes cluster on cloud - Deploy Alluxio services via Docker and Kubernetes - Launch distributed training job via Kubeflow and Arena in one command - Accelerate training speed by caching data locally - Focusing on the training logics instead of worry about the data preparation (No need to modify training script, use remote data like local data)
- 11. Deployment Variance - Kubectl - The default way of launching Alluxio cluster on Kubernetes - Use in testing environment, especially when your deployment is simple - If you have special requirements that is not fulfilled by other deploy ways, kubectl is your chose. 11 $ kubectl create -f alluxio-configmap.yaml $ kubectl create -f ./master/ $ kubectl create -f ./worker/
- 12. Deployment Variance - Helm - Package Manager for Kubernetes, preferred over Kubectl - Use Alluxio shared Helm charts, provide your desired configuration, and launch the Alluxio services in one command. - Define a common template for Alluxio services and deploy the same/similar applications across different environments without copy-pasting. 12 $ helm repo add alluxio-charts https://alluxio-charts.storage.googleapis.com/openSource/2.6.0 $ helm install alluxio -f config.yaml alluxio-charts/alluxio Deploy Alluxio on Kubernetes
- 13. Deployment Variance - Alluxio CSI Container Storage Interface - Expose Alluxio to container applications (e.g. Docker, Kubernetes) as persistent storage. - Support customizing storage features without the need to integrate Alluxio driver into Kubernetes package and wait for kubernetes releases. 13
- 14. Alluxio CSI - Mount point level access control - Separate mount points for different users or use cases - Access control via POSIX filesystem user group - Different configuration for different mount points - Separate read/write path - Enable metadata cache for input data folder - Disable metadata cache and set write option to Through for output folder. Customized mount option Path in Alluxio
- 15. Alluxio CSI Reference - Microsoft: Speed up large-scale ML/DL offline inference job with Alluxio - Alluxio/alluxio-csi (github.com) - Alluxio CSI integration and improvements (#13435) Special thanks to Binyang from Microsoft and Baolong from Tencent 15
- 16. Deployment Variance - Fluid Fluid is an open source Kubernetes-native Distributed Dataset Orchestrator and Accelerator for data-intensive applications, such as big data and AI applications. Apart from accelerating training, Fluid also supports - Access control in dataset level for data scientists - Auto launching and scaling internal Alluxio services. Users are not needed to worry about the deployment pain points of Alluxio services. 16
- 17. 17
- 18. Fluid Reference - Fluid: When Alluxio Meets Kubernetes - Speeding up Atlas deep learning platform with Alluxio + Fluid - cheyang/fluid (github.com) Special thanks to Cheyang from Alibaba 18
- 19. Updates in Alluxio POSIX Developments 19
- 20. Community Collaboration Community-driven collaboration - Contributors from NJU, Alibaba, Tencent, Microsoft, Alluxio Already in used by Microsoft, Analytics Aspects, BOSS 20
- 21. JNI-Fuse enhancements (available in 2.6.0) - Local RPC elimination by integrating Fuse into Worker process - Modularized JNI-Fuse library (github.com/maobaolong/jnifuse) - Implement Fuse flush() operation in write path (#13103) - Implement Fuse utimens() operation to support linux touch command (#13218) - Implement Fuse symlink() operation to avoid execution errors (#13429) - Support open file for overwrite (#13236) - Fix Fuse write then read problem in Fuse async release() operation (#13160) - Add metrics for Fuse operations (#13201) 21
- 22. JNI-Fuse enhancements (ongoing projects) - Alluxio CSI improvements (#13435) - Improve data loading speed by dynamically allocating loading jobs (#13485) - Support updating configuration during runtime to better support Alluxio on Kubernetes (#13643) - Support libfuse 3 (#12758) - Remote RPC optimization Join Alluxio weekly community sync to create solutions together! 22
- 23. We Are Hiring! (contact careers@alluxio.com) Join the Alluxio Community today www.alluxio.io/slack | @alluxio 23