Long Live Posix - HPC Storage and the HPC Datacenter
2 likes624 views
The document discusses the evolution of HPC (High-Performance Computing) storage, highlighting shifts from traditional shared POSIX file systems to more cloud-based, dynamic, and containerized storage solutions. It emphasizes trends such as the increasing importance of tiering, data management, and the need for systems that can handle diverse workloads, including analytics and deep learning. The predictions outline a future where storage integrates more seamlessly with compute platforms while adapting to the needs of various applications.
Long Live Posix - HPC Storage and the HPC Datacenter
- 1. ©2018 DataDirect Networks, Inc. DDN Confidential Death to the Shared POSIX File System!? Long Live the Shared POSIX File System!! HPC Storage and the HPC Data Center
- 3. this is a vendor presentation!
- 4. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. Classical HPC Data Centers • Everything is bare metal • Most data is stored in shared POSIX file systems that largely remain on HDDs • SSD-based systems are slowly coming, but capacities remain small • People have played with Hadoop and objects stores, but very few systems are actually in production • Special performance and capacity tiers are used mainly by the large centers, but not the broader market Cloud Data Centers • Everything is virtualized, containerized, dynamic, automated, etc. • Most date lives on local file systems in virtual machines (or containers), either on SSDs or HDDs • Monolithic web applications use large, distributed object stores • Enterprise storage still being used for various specific requirements, both on premise and hosted in the cloud • Tiers exists for redundancy, data back- up, data retention, etc.
- 5. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. Example: 2015 DDN User Analysis Work Data Mixed Use Archive Cloud HPC Work Weather Climate CAE Chemical General Academic Genomics Big Data Science Security Finance Energy Tier 2HPC Cloud Cloud Work 31% Data 33% Mixed 22% Cloud 2% Archive 12%
- 6. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. Example: 2018 Lustre End User Survey 0% 5% 10% 15% 20% 25% 30% 35% 40% Weather/Climate Other Media Manufacturing Life Sciences Government Financial Services Energy Education Defense AI/Machine Learning Primary Usage of Lustre
- 7. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. HPC Data Centers: The Past Decade ►“Research Big Data” is a very big market for storage (and a much larger than the classical tightly-coupled simulations) and it is quite different from classical HPC ►Many data center dedicated to analytics and deep learning look like large cloud environments, but their IO requirements are much closer to Research Big Data ►Some “Research Big Data” end users have moved to on-premise cloud environments using Open Stack etc. ►More HPC and Analytics customers are running applications in the cloud or in cloud-like environments, with shared file systems running in the cloud
- 8. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. Predictions From LANL in 2016: Serving Data to the Lunatic Fringe
- 9. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. Predictions From LANL in 2016: Serving Data to the Lunatic Fringe
- 10. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. This architecture was imperiled by SSD economics
- 11. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. Predictions From LANL in 2016: Serving Data to the Lunatic Fringe
- 12. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. This architecture becomes imperiled by tape economics
- 13. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. Predictions From LANL in 2016: Serving Data to the Lunatic Fringe
- 14. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. This architecture becomes imperiled by Lang’s Law! ”DOE doesn’t want tiers. Tiers are an unfortunate accident of economics. DOE wants infinite memory and a system without unplanned interrupts. Just remember this: The fewer tiers, the fewer tears.”
- 15. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. Predictions From LANL in 2016: Serving Data to the Lunatic Fringe
- 16. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. Predictions from NERSC: Storage 2020-2025 Four Tiers Three Tiers Two Tiers
- 17. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. The Foreseeable Future Remains Tiered
- 18. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
- 19. DDN ConfidentialDDN Storage | ©2019 DataDirect Networks Tiering Tiering – Data Schmiering All due respect to Lang’s Law (‘fewer tiers, fewer tears’), tiering is a (mostly) solved problem. Buffer-caching is a (mostly) solved problem! Russel Kirsch developed it for the SEAC in 1952.
- 20. DDN ConfidentialDDN Storage ©2019 DataDirect Networks, Inc. Flash Acceleration Layer Usage and Tiering Workflows in Traditional HPC Lee Ward, Use Cases or BB Roles, Informal Burst Buffer Presentation via Sandia National Laboratories, 2015. Challenges and Considerations for Utilizing Burst Buffers in High-Performance Computing, Melissa Romanus, Robert Ross, Manish Parashar, 2018. Development of a Burst Buffer System for Data-Intensive Applications, Teng Wang, Sarp Oral, Michael Pritchard, Kevin Vasko, Weikuan Yu, 2015. An Operational Perspective on a Hybrid and Heterogeneous Cray XC50 System. Sadaf Alam, Nicola Bianchi, Nicholas Cardo, Matteo Chesi, Miguel Gila, Stefano Gorini, Mark Klein, Colin McMurtrie, Marco Passerini, Carmelo Ponti, Fabio Verzelloni, 2017. 1. Checkpoint-Restart 2. In-situ/transit viz/analysis 3. Out-of-core 4. Accelerated reads (pre-stage) 5. Random-read centric applications (lots of them!)
- 21. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. Data Tiering: A Subtle Shift in Perception Applications Performance Storage Capacity Storage Unnecessarily Strict Tiering Relaxed Tiering Applications Capacity Storage Performance Storage
- 22. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. “What?” “Do you have any water?” “Build an object store!”
- 23. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. Typical Object Requirements (“Object Schmobject; long live POSIX”) ► Immutable, transactional get/put, trillions of objects ► Named objects ► Group objects into logical collections ► Nest logical collections within each other ► Have the same object appear within multiple collections ► Multi-threaded writes ► Tag objects ►Object is a subset of file • There is no application which uniquely requires object semantics • O_TMPFILE and rename are useful primitives ►Object requirements grow as humans use them • Eventually they become file requirements ►We do not live on a deserted desert island • We have two decades experience building parallel file systems ►RELEVANT LESSON FROM OBJECT STORES? • POSIX relaxation is useful
- 24. Bent’s Law for HPC Storage: The future is bright and mostly as we predicted it. Don’t be scared of POSIX; but embrace relaxations. Don’t be scared of tiering; but embrace relaxations.
- 25. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. ML/DL Storage Scenarios: Small-to-Large Local NVMe Node-Local “Shared” NVMe Global/Capacity Global NVMe OnDemand Namespaces Remote NVMe (NVMeoF)
- 26. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. Dynamically-Provisioned File Systems ►Job-Specific File Systems • LLNL Proposal (2014): file system per JOB with data staging • Isolation from other jobs • Increased metadata performance ► Loop-back devices for increased metadata Performance • K-Computer/Fujitsu: Very large-scale Implementation • NERSC Library (e.g. for use of SPARC on Lustre) • Lustre on Amazon/Azure/Google (since 2013) • Amazon Lustre Service (November 2018) ►Future • Client Container Image (CCI) Feature in Lustre (integrates look-back devices into Lustre) • Lustre-on-Demand Feature (first deployments in early 2019!)
- 27. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. Amazon FSx for Lustre ►Dynamically-provisioned Lustre file systems ►Application-specific ►Focused on analytics and DL applications ►Connector to Amazon S3 ►But: the usual limitations still apply;-)
- 28. DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc. Next Decade of HPC Storage: Back to the Future? ►Still PFS… • … but with increasing portions that are dynamically allocated and integrated into the compute platform • e.g. a file server turns into a containerized process run anywhere ►Still POSIX… • … but relaxed where needed ►Still Tiers… • … but relaxed to reflect actual application workflows Relaxed Tiering with Dynamically-Managed, System-integrated Storage Applications Capacity Storage Performance ENS ENS CapS