A Glimpse into the Future of I/O
1 like453 views
The document discusses the future challenges of data management and I/O performance as data scales to exabytes. It outlines growing issues with metadata, random access performance, multi-tenancy, and data retrieval. New workloads are more heterogeneous and driven by latency. Solutions proposed include improved file systems, analytics-driven administration, and technologies from DDN's acquisition of Tintri to address these challenges at extreme scales.
A Glimpse into the Future of I/O
- 1. DDN Storage | ©2018 DataDirect Networks, Inc. A Glimpse into the Future of I/O DDN Usergroup Meeting SC18 November, 2018 Sven Oehme – Chief Research Officer DDN
- 3. DDN Storage | ©2018 DataDirect Networks, Inc. Data At Scale HPC AI Analytics BigData Cloud Integration Data Management Distributed Object Advanced Data Management I/O Performance and Behavior Latency Latency-Tail Throughput Byte Addressable GPUs More Heterogeneous CPU Economic Model Cloud Integration Workloads SuperComputing Themes Meta-Analytics Simplicity Insight Scale Security Multi Tenancy Data Path Optimisation Tag&Search HW Accelerators Fat Nodes Orchestrators Containers Open Monitoring Openstack/ Kubernetes NVMe/SCM KV HDFS Trending Technologies Audit Data Views
- 4. DDN Storage | ©2018 DataDirect Networks, Inc. Challenges in Broader Multi-Tenanted Security and Flexibly Managing Data Collections ► Multi-Tenancy == Tenanted Isolation by • Data Collections ß Flexible allocation of collections of data to users, groups, organizations, allowing collaborative access, publication, etc • Performance ß The Noisy Neighbor problem - particularly for latency critical applications being impacted by throughput hungry workloads • Resilience ß failures in domain A should not impact domain B ► Virtualization and Containerization - Data should be accessible from within containers ► Multi tenancy needs to be a logical allocation, not a HW or physical separation unless desired ► Auditable - All events – both system and tenant in the system should be auditable ► One data platform needs to run on your own HW, the cloud and highly optimized integrated systems ► Need a more native approach to resolving all these challenges
- 5. DDN Storage | ©2018 DataDirect Networks, Inc. Challenges in Security and deployment – Starting down the path ► Lustre takes strong steps towards better multi-tenancy support ► This addresses some of the requirements ► dynamic ‘views’ on data based on policy and access control rather than namespace segregation
- 6. DDN Storage | ©2018 DataDirect Networks, Inc. Challenges in I/O Performance and Behavior ► Newer applications need to operate on byte addressable data ► Significant shift from sequential to random I/O ► Multifold increase of metadata to data ratio ► Average data sizes are less homogeneous and are now fractions or multiples of previous workloads. gap between small and large data seems to wide (bytes on one end , GB’s on the other end of the spectrum) ► Interactive, outcome and event driven analytics are driven by latency rather than bandwidth Big Data NoSQL Analytics Supercomputin g Multi- Physics Machine Learning Workflow s Adaptive Mesh Refinement Checkpointing Trend for ever more mixed, complex I/O Workloads
- 7. DDN Storage | ©2018 DataDirect Networks, Inc. I/O Performance several techniques to address them 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% Oakforest-PACS at JCAHPC (IME) Shaheen at Kaust (Datawarp) Mistral at DKRZ (Lustre) EMSL Cascade at PNNL (Lustre) Ratio of Easy:Hard IO500 Results Ratio of Easy:Hard (systems with 100 clients or more) Write Ratio Read Ratio ► Overlapping i/o and conflicting access to same resources are well addressed with IME ► Log structured filesystems mitigate i/o amplification and significant slowdown of random i/o patterns ► Leverage HDD for capacity tiers and IME for performance ► HAWC / LROC in Scale , HDFS flash accelerators are other examples
- 8. DDN Storage | ©2018 DataDirect Networks, Inc. Challenges in Data Management ► Metadata growth is substantial, people try to solve this with external data repositories with mixed success, but at current growth rate its not sustainable ► At Exabyte scale data reliability and availability will be substantially challenged by traditional recovery models ► We see requirements to support Quadrillions (millions of billions) of objects in storage systems and we are still at the beginning of this ► Data management at scale is complicated , snapshots, tiering (to disk, tape, cloud) and breaks in larger systems ► People can’t find their data, no easy way to tag and ‘search’ for it
- 9. DDN Storage | ©2018 DataDirect Networks, Inc. Challenges in Data Management – need for extensive analytics ► The acquisition of Tintri provides us technology to run analytics for systems at scale ► More insights into system to make decisions based on data and trends rather manual crafted administrative policies ► data placement based on historic data
- 10. DDN Storage | ©2018 DataDirect Networks, Inc. conclusions ► Already many strong inroads into these data challenges for 2019+ • IME • Lustre • SFAOS ► Heavy engineering investment underway to resolve this challenges in more native ways ► Come and talk to us and ask for the ‘secret’ presentation room ;-)
- 11. ddn.com©2018 DataDirect Networks, Inc. *Other names and brands may be claimed as the property of others. Any statements or representations around future events are subject to change. Thank You! Keep in touch with us. 9351 Deering Avenue Chatsworth, CA 91311 1.800.837.2298 1.818.700.4000 company/datadirect-networks @ddn_limitless sales@ddn.com