Whamcloud - Lustre for HPC and Ai
Download as PPTX, PDF1 like2,150 views
The document outlines the advancements and features of the Lustre file system as developed by Whamcloud and DDN, emphasizing its open source model, performance optimizations, and management features. It details a roadmap for future releases with various capabilities such as persistent client caching, container integration, and data management options. Additionally, the document contains confidentiality notices and specifies that content is intended for DDN and Whamcloud employees only.
![whamcloud.com
IO-500 10 Client Node Challenge (Pre-ES5.0
Results)
[RESULT] BW phase 1 ior_easy_write 37.540 GB/s : time 343.38 seconds
[RESULT] IOPS phase 1 mdtest_easy_write 199.685 kiops : time 325.87 seconds
[RESULT] BW phase 2 ior_hard_write 0.262 GB/s : time 300.21 seconds
[RESULT] IOPS phase 2 mdtest_hard_write 24.348 kiops : time 395.55 seconds
[RESULT] IOPS phase 3 find 3332.110 kiops : time 21.96 seconds
[RESULT] BW phase 3 ior_easy_read 35.374 GB/s : time 364.41 seconds
[RESULT] IOPS phase 4 mdtest_easy_stat 527.669 kiops : time 124.08 seconds
[RESULT] BW phase 4 ior_hard_read 4.627 GB/s : time 17.03 seconds
[RESULT] IOPS phase 5 mdtest_hard_stat 79.476 kiops : time 106.64 seconds
[RESULT] IOPS phase 6 mdtest_easy_delete 226.094 kiops : time 288.22 seconds
[RESULT] IOPS phase 7 mdtest_hard_read 46.141 kiops : time 182.72 seconds
[RESULT] IOPS phase 8 mdtest_hard_delete 58.842 kiops : time 143.64 seconds
[SCORE] Bandwidth 6.33725 GB/s : IOPS 159.413 kiops : TOTAL
31.7843](https://image.slidesharecdn.com/11roberttriendlsc18usergroup-181201081305/85/Whamcloud-Lustre-for-HPC-and-Ai-27-320.jpg)
Whamcloud - Lustre for HPC and Ai
- 1. Lustre for HPC and AI
- 2. CONFIDENTIALITY NOTICE The contents of this document and any attachments are intended solely for the DDN and Whamcloud employees and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not a DDN or Whamcloud employee, or if this document has been addressed to you in error, please immediately alert the sender. If you receive this electronically by email or file sharing then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this document and content is strictly prohibited.
- 4. whamcloud.com Whamcloud & DDN ►Open Source Lustre ►Advanced Management Features ►Integration with DDN Hardware ►Performance Optimization ►New Application Areas ►Workload Optimization ►Packaged Solutions ►End-to-End Support
- 5. whamcloud.com DDN/Whamcloud Open Source Model Lustre Core Usability Layer Internal Management Jobstats NRS HSM Changelogs PFL/FLR & MDR Next Generation IML Granular Monitoring QoS HSM & PCC Data Management Data Replication D-RAID Z & ZFSDevice Management Logging APIsLog Interpretation
- 6. Official Lustre Community Roadmap Q2 Q3 Q4 Q2 Q3 Q4Q1 Q2 Q3Q1 LTS Updates Future releases 2.12 2.13 2.14 2.10.4 2.10.5 2.10.6 2.10.7 2.10.XLTS ReleaseStream LTSReleases continue… Expected Timeline Timeline TBD Completed LTSBranch 2.11 - Data on Metadata - FLR Delayed resync - Lock Ahead 2.12 - Lazysize on MDT - Lnet Health - DNEDirrestriping 2.13 - Persistent Client Cache - LNet Selection Policy - Self Extending Layouts 2.14 - FLR ErasureCoding - Health Monitoring - DNEAuto Restriping 2.12.XLTS ReleaseStream Estimates are not commitments and are provided for informational purposes only Fuller details of features in development are available at http://wiki.lustre.org/Projects 2.11.0 2018 2019 2020
- 7. whamcloud.com Lustre Core – Key Features Roadmap LustreCoreKeyFeatures 2.10.x 2.11 2.12 LTS 2.13 Progressive File Layouts NRS Delay Policy Project Quotas Multi-rail LNET Simplified User Space Snapshot (ZFS) Data on Metadata FLR Delayed Sync Lock Ahead LNET Multi-rail health DNE Directory restriping Lazy Size on Metadata T10 DIF support Support for Open ZFS 0.8.x Persistent Client Cache Lnet Selection Policy Self Extending Layouts FLR Erasure code striping DNE Auto Remote Dir striping Health Monitoring Future Versions Client metadata write back cache (CWBC) Client Container Image (CCI) Metadata Replication Dates and version are subjected to change without prior notification Possibly 2.14 Possibly 2.15 Features landed or about to belanded Features under development Features being designed and/or prototyped Not committed yet
- 8. whamcloud.com Lustre-on-Demand: Ephemeral Namespaces Dynamic File Systems Temporary fast Lustre filesystem across the compute nodes LOD creates Lustre on computes nodes dynamically Scheduler Integration User turn LOD on/off per job at job submission Currently integrated into SLURM’s Burst Buffer option, but other job scheduler also could work Automated Data Staging User can define file/directory list on stage-in/out to LOD at Job submission LOD automatically sync/migrate data from persistent Lustre to created temporary Lustre filesystem Flexibility Flexible MDT/OST configuration for advanced users
- 9. whamcloud.com Persistent Client Cache Lustre 2.13 Reduce latency, improve small IOPS, reduce network traffic PCC integrates Lustre with persistent per-client local cache devices Each client has own cache (SSD/NVMe/NVRAM) as a local filesystem (e.g. ext4/ldiskfs) No global/visible namespace is provided by PCC, data is local to client only Existing files pulled into PCC by HSM copytool per user directive, job script, or policy New files created in PCC is also created on Lustre MDS Kernel uses local file if in cache or normal Lustre IO Further file read/write access “directly” to local data No data/IOPS/attributes leave client while file in PCC File migrated out of PCC via HSM upon remote access Separate functionality read vs. write file cache Could later integrate with DAX for NVRAM storage
- 10. whamcloud.com Client Metadata Write Back Cache Lustre 2.15 or Later Metadata WBC creates new files in RAM in new directory Avoid RPC round-trips for each open/create/close Lock directory exclusively, avoid other DLM locking Cache file data only in pagecache until flush Flush tree incrementally to MDT/OST in background batches Could prefetch directory contents for existing directory Can integrate with PCC to avoid initial MDS create Early WBC prototype in progress Discussions underway for how to productize it Early results show 10-20x improvement for some workloads
- 11. whamcloud.com Client Container Image (CCI) Lustre 2.15 or later Filesystem images were used ad hoc with Lustre in the past Read-only cache of many small files manually mounted on clients Root filesystem images for diskless clients Container Image is local ldiskfs image mounted on client Holds a whole directory tree stored as a single Lustre file CCI integrates container handling with Lustre Mountpoint is registered with Lustre for automatic mount Automatic local loopback mount of image at client upon access Image file read on demand from OST(s) and/or cached in PCC Low I/O overhead, few file extent lock(s), high IOPS per client Access, migrate, replicate image with large read/write to OST(s) MDS can mount and re-export image files for shared use CCI can hold archive of whole directory tree for HSM Unregister/delete image and all files therein with a few ops
- 12. whamcloud.com Whamcloud Unified Roadmap Overview Management&MonitoringLustreCore NextGen Nov 2018 ES4.1 Sept 2019 ES5.0 April 2019 ES4.2 EX next gen v1.0 (MVP) – CLI and GUI ES5.x (sustaining support mode) Still Lustre 2.12 based Nov 2020 EX next gen Prototyping Dates and version are subjected to change without prior notification +5 years Dec 2019Jan 2019 Feature pathfinding LTS Releases Supported Releases Non LTS community releases EOL’d releases Prototype & Ideation May 2020July 2019 ES5.1 ES6.0 Lustre 2.12 IML 5.0 IML 5.x with SFA and ES integration Lustre 2.12.x Lustre 2.13 Lustre 2.14 Lustre 2.15
- 13. whamcloud.com ExaScaler – Key Features Roadmap ExascalerFeatures ES4.1 ES4.2 ES5.0 ES5.1 ES6.0 A3I – Support for AI200 (factory deployment) A3I – Support for AI200 (ES_Wizard) Lustre 2.12 LTS based with Support for all 2.12 Features IML Management for ExaScaler Call Home v1 IOPS Improvements Small IO Performance v1 Metadata Performance for “E” DNE2 Scaling Lustre-on-Demand T10-PI End-to-End Data Integrity Whamcloud Protocol Gateway Cluster v1: NFS & CIFS DDN Data Flow Support v1 Tech Preview: LiPE FS Accounting Tech Preview: DCS Platform LiPE GA v1 Transparent SSD Pools v1 SFA FStrim Support Optimized DDN Hardware support for Data on Metadata Lustre Persistent Client Cache (LPCC) Dsync Integration v1 Whamcloud Protocol Gateway Cluster v2: Object Support Tech Preview of New Management and Monitoring Framework Tech Preview: Object Support for Whamcloud Gateway Cluster Tech Preview: LWBC Integrated Management and monitoring New Lustre Management framework for High Availability, deployment and scalability Lustre Write Back Cache (LWBC) Lustre Client Container (LCC) Lustre Metadata Replication Small IO and Metadata Performance Improvements AI7990 and AI400 support ES7990, NV400 and ES18K support Exascaler Docker Container for A3I deployments IBM Power 9 client support ARM client support Large IO & Read Performance DDN Insight and DDN Performance Monitoring convergence Lustre on GCP LORIS v1 Tech Preview: NFS & NFS Gateway Reference Configuration Tech Preview: Lustre-on-Demand Dates and version are subjected to change without prior notification New Platform Support SFA Performance Features under development Features being designed and/or prototypedFeatures Ready NVMe &Data Management Lustre Next Gen
- 14. whamcloud.com Exascaler – Feature Development ES4.1 ES4.2 ES5.1 ES6.0ES5.0 • Ai200 A3I • AI400, AI7990 • A3I Client Docker containers • NV200 • ES14KX • ES7700 • NV400 • SFA7990 • SFA18K • Large IO & SFA Read Performance Increases • Whamcloud Call Home v0 • IML Management for SFA • Enhanced Lustre monitoring with Insight for Lustre • IML Monitoring for SFA • DDN Insight and DDN Perf Mon convergence • LORIS v1 • Whamcloud Lustre Manager: New Integrated Management and Monitoring for Lustre • Lustre on AWS • Lustre on GCP • CIFS and NFS Gateway Preview • Lustre-on-Demand Tech Preview • CIFS and NFS Gateway v1 • Lustre-on-Demand • New Management and Monitoring framework Tech Preview • SFA 18KX Support • SFA7990X, SFA 200/400NV X DDN SFA DDN Performance Monitoring& management Cloud& Enterprise Dates and version are subjected to change without prior notification • IOPS Optimizations • Small IO Optimizations • SFA “E” Platform Metadata Performance Optimizations • DNE Performance Scaling • T10-PI Support • DCS Hardware Tech Preview • Lustre WBC (LWBC) • Lustre Client Container (LCC) • Lustre PCC• Small IO Optimizations • Metadata Optimizations • Next Gen SFA (PCIe Gen4) • LustrePerfMon Framework Open Source • Metadata Performance with CWBC • Small File Performance with CCI • Metadata Replication • Local Caching with LPCC • LWBC Tech Preview • CIFS and NFS Gateway v2: Object Support
- 15. whamcloud.com DDN T10-PI End to End Data Integrity Fully transparent End-to-End Data integrity from Lustre client to disk Relies on open standard format T10PI/DIX Any T10PI/DIX supported hardware is usable Minimum performance impacts Keep compatibility for old Lustre version or non-T10PI supported hardware DDN SFA Ready!
- 16. whamcloud.com CIFS and NFS (Object in Future) Consistency & Cross-Protocol Locking UNIX passwords, LDAP and Microsoft AD High Availability Horizontal scalability • Blue Print – Architectural reference • Ready to run external servers • 2-node HA configuration Phase 1 • Integrated appliance model(s) • No need of external servers • Integration with Lustre to provide scalability ＆Performance • Multi-node HA configuration Phase 2 DDN CIFSand NFSGateway
- 17. whamcloud.com Broad CPU Support ►Server & Client •x86 Intel •x86 AMD •ARM (Fujitsu, Cavium) ►GPUs •NVIDIA, AMD ►Client Only •Power/PowerPC •Other Astra
- 18. whamcloud.com Lustre and Cloud Today: Lustre on Public Cloud Support IO-Intensive Applications (e.g. SAS Analytics) Easy Set-up Process DDN/Whamcloud Expert Support Import/Export Data via S3 Future: Hybrid Cloud for HPC Support data tiering between on-premise and public cloud instances Migrate temporary workloads to cloud quickly Stage-out cold data to public cloud storage
- 19. DDN Storage | ©2018 DataDirect Networks, Inc. Protect, vault, move and synchronize Access and store data on any storage platform Safeguard critical information and ensure availability Enrich data archives with advanced metadata collection Accelerate data workflows with scalable distributed architecture Structure data operations with management, monitoring and reporting Intuitive interfaces make data management simple and easy FILE OBJECT CLOUD TAPE WITH BACKUP ARCHIVE MOVE SYNC DDN DATAFLOW | ACCELERATE DATA WORKFLOWS AT SCALE Integrated with Lustre
- 20. whamcloud.com DDN ES18K – Block Performance (400 x HDD) 0 10,000 20,000 30,000 40,000 50,000 60,000 70,000 1 2 4 8 16 32 64 128 Bandwidth(MB/sec) Queuedepth WRITE 16 MB SGC=8 SGC=16 SGC=32 0 10,000 20,000 30,000 40,000 50,000 60,000 70,000 80,000 1 2 4 8 16 32 64 128 Bandwidth(MB/sec) Queuedepth Read 16 MB SGC=8 SGC=16 SGC=32
- 21. whamcloud.com DDN ES18K – IOR 1MB FPP (400 HDD) 0 10,000 20,000 30,000 40,000 50,000 60,000 70,000 1xVD(1xOSS_1xOST) 1xRP(2xOSS_2xOST) Half(4xOSS_4xOST) Full(8xOSS_8xOST) Bandwidth(MB/sec) Write Scaling SGC=8 SGC=16 SGC=32 0 10,000 20,000 30,000 40,000 50,000 60,000 70,000 1xVD(1xOSS_1xOST) 1xRP(2xOSS_2xOST) Half(4xOSS_4xOST) Full(8xOSS_8xOST) Bandwidth(MB/sec) Read Scaling SGC=8 SGC=16 SGC=32
- 22. whamcloud.com DDN ExaScaler vs. IBM ESS Max Spindle Efficiency ES400-NV IBMESS 152 %
- 23. * Cached AI/ES 200/400 VirtIO OSS 0 MDS 0 VM0 OSS 1 MDS 1 VM1 VirtIO OSS 2 MDS 2 VM2 OSS 3 MDS 3 VM3 FS in a Box Standard VM Stack Easy Deployment Management Monitoring
- 24. whamcloud.com AI-200/400 Random Read IOPS and B/W 0 10 20 30 40 50 60 0 200 400 600 800 1000 1200 1400 1600 1800 4KB 8KB 16KB 32KB 64KB 128KB Throughput(GB/sec) IOPS(Kops/sec) IO Size(Byte) AI200-Throughput AI400-Throughput AI200-IOPS AI400-IOPS 1.5M IOPS @4KB 50 GB/sec Throughput
- 25. whamcloud.com DDN ExaScaler vs. IBM ESS ES400-NV IBM ESS Max Spindle Efficiency Max IOPS 341 %ES400-NV IBMESS 152 %
- 26. whamcloud.com IO-500 on Pre-ES5.0 Test Configuration IB EDR 5 x SS9012 ES14K 1 x SFA7700FC 4 x MDS 10 x Client • 4 x MDS • 1x Platinum 8160, 96GB RAM, 1 x IB EDR • 1 x SFA7700 FC • 2 x MDT(2 x RAID1 SSD) per MDS • 1 x ES14K + 5 x SS9012 • 408 x NL-SAS 10TB • 8 x DCR POOL (51/MR=1) • 4 x vOSS(8 CPU Core, 90GB RAM, 1x IB EDR • 10 x Intel Server • 2 x E5-2650v4, 128GB RAM, 1x IB EDR • CentOS7.4 • Pre-ES5.0 • master branch for Lustre-2.12 MDSOSSCLIENTSW
- 27. whamcloud.com IO-500 10 Client Node Challenge (Pre-ES5.0 Results) [RESULT] BW phase 1 ior_easy_write 37.540 GB/s : time 343.38 seconds [RESULT] IOPS phase 1 mdtest_easy_write 199.685 kiops : time 325.87 seconds [RESULT] BW phase 2 ior_hard_write 0.262 GB/s : time 300.21 seconds [RESULT] IOPS phase 2 mdtest_hard_write 24.348 kiops : time 395.55 seconds [RESULT] IOPS phase 3 find 3332.110 kiops : time 21.96 seconds [RESULT] BW phase 3 ior_easy_read 35.374 GB/s : time 364.41 seconds [RESULT] IOPS phase 4 mdtest_easy_stat 527.669 kiops : time 124.08 seconds [RESULT] BW phase 4 ior_hard_read 4.627 GB/s : time 17.03 seconds [RESULT] IOPS phase 5 mdtest_hard_stat 79.476 kiops : time 106.64 seconds [RESULT] IOPS phase 6 mdtest_easy_delete 226.094 kiops : time 288.22 seconds [RESULT] IOPS phase 7 mdtest_hard_read 46.141 kiops : time 182.72 seconds [RESULT] IOPS phase 8 mdtest_hard_delete 58.842 kiops : time 143.64 seconds [SCORE] Bandwidth 6.33725 GB/s : IOPS 159.413 kiops : TOTAL 31.7843
- 28. whamcloud.com ExaScaler Perf Mon • Scalable Performance Monitoring for Lustre • Open Source • Based on CollectD and integrated into DDN Exascaler • More than 100 Lustre statistics gathered • Support for Job Stats • Configurable data retention • Support several NoSQL Time-Series databases
- 29. whamcloud.com DDNInsight for ExaScaler • Multi-solutions Monitoring • Centralized Web interface for comprehensive monitoring of all DDN solutions and platforms • Unprecedented Vision • Derive the most value of your storage infrastructure through a deep real-time analysis from file system to hardware platform • Open and Extensible • Extensively customizable DDN Insight open back end database and integration with Grafana allows you to correlate storage performance with metrics beyond the DDN system
- 30. whamcloud.com IML- Integrated Managementfor Lustre • Open source suite of tools for deploying, managing and monitoring Lustre • IML simplifies Lustre administration with intuitive interfaces and near real -time feedback • Works with new and existing Lustre installations • Monitors performance and system health
- 31. whamcloud.com Manage and Monitoring Strategy for Lustre Exascaler Perf Mon DDN Insight IML4.x Monitors and Manage DDN Hardware Scalable and Highly Available Monitors Lustre Scalable and extensible Some monitoring Capabilities Support all hardware and ZFS Exascaler Manage Lustre Support DDN platforms New Integrated Management Platform GUI entities from IML Support for DDN and 3PP DDN HA Agents Limited Monitoring MonitoringFor Lustre Integration of Statistics and Lustre aware data into Insight Continue developing Open Source ES Perf Mon new features 2018 2019 2020 Next Gen platform New HA Framework Integrated Monitoring capabilities Integrated Lustre features such as QoS and Policy Engine Integration with Job Schedulers GUI and CLI Elastic DB back-end ConvergencePath
- 32. whamcloud.com Future Lustre Events SC18 Tuesday, November 13th, 12:15-1:15pm – Room C140/142 May 14-17 Lustre User Group 2019 (OpenSFS) June 16-20 ISC Lustre User Group/BOF Mar 11-14 APAC Lustre User Event Sept at SCASIA19 (TBD) Sept Lustre Admins & Dev Workshop (EOFS) Oct Japan Lustre User Group (Whamcloud) Oct China Lustre User Group (Whamcloud) Nov SC19 Lustre User Group/BOF
- 33. Thank you!
Editor's Notes
- #13: This slide introduces to customers the overall time line. As any roadmap strategy, it is subjected to change. The slide is divided into three main development lines: - Exascaler: as we know today. The DDN’s Enterprise Lustre File system distribution, as it is. Lustre core: Lustre file system, Open Source releases Next Gen: what we envision for the 2, 3 years time span There are different icons and color coding, that are there to identify different type of development paths and stages: Prototype and Ideation: first phase, no firm commitments. It envolves feasibiity study, small development (prototype as per say), tests and product discussions LTS Release: this applies to Lustre. LTS stands for LONG TERM SUPPORT and it’s a policy defined by the LWG. There’s only ONE Lustre LTS release at a time. Currently, the LTS release is based on Lustre 2.10 Supported releases: This applies for the other products such as Exascaler, where there’s one or more versions under support. Non LST community releases: This applies to Lustre, intermediate releases, usually delivered every 7 months. These are smaller releases that carries new features that will be incorporate on the next LTS releases. EOL’d: End of Line, no further development but continuing support for 5 years. The goal of this slide is re-enforce the convergence of technologies towards an unified platform. ES5.0 will be the last ES release as we know today. Further releases will be merged into the next gen pathfinding where the plans is to get IML and other technologies integrated. However, to avoid an abrupt change, the first step is the concurrent lrelease of ES5.x and IML with support for DDN hardware platform. It’s valid to remember that today, IML has few monitoring capabilities and none management capabilities for DDN Hardware. In the same time, IML is the standard choice for non-DDN platform. The idea is to converge the best of the two worlds into a single platform.
- #20: 18 is vague and so is 19. Value? Use cases? Benefits?