DAOS Middleware overview

Editor's Notes

• #3: Overall DAOS architecture Distributed object store built from the ground up for NVM technology. Includes 2 types of storage: SSD with different media, 3d-NAND, Optane. Available with NVMe interface, provide fast block-based storage Intel developing new user-space nvme driver, SPDK Pmem, Optane pmem. Fine grained storage directly memory mapped. Can do direct load store to the storage. No block abstraction Intel developing new storage stack PMDK to manage transactional update to pmem We don’t have any Object store to leverage these object stores. Lustre, GPFS built into kernel. Ceph still uses kernel IOs by going through Linux block layer. Here everything in userspace. DAOS new OS built on top of pmdk and spdk. highIOPS high BW. More advanced storage API, not just byte arrays, KV, etc. Still need to look into communication. Reuse several decade effort to optimize latency and BW for MPI. Re-use same communication middleware as MPI (libfabric). Client side, very lightweight with support to many middleware. All metadata stored on pmem. Small IOs land in pmem and later aggregated into SSD in the background to free space in Pmem. Application large data go in the SSD.
• #4: Baseline API based on Storage container -> Object address space inside a pool. Container can be encapsulated POSIX namespace, or just a flat one. HDF5 has it’s own representations for a single HDF5. KV store can be one. Database, ACG 1 graph Container provides a baseline API with object. Unit of storage management is the container not a file. When you have millions of file, data migration is done on all those. We Simplified that. Fewer number of containers than files, so we can index them. Data migration of entire container. unit of Snapshot. Snapshot not typically available traditionally (used to be at FS level) but now at container level and exposed to user
• #5: POSIX not the foundation any more. New key-array store API provided by DAOS with advanced features (ad-hoc concurrency control, snapshot support, asynchronous API, ..). DAOS API to be directly integrated with rich data model for better performance and extended capabilities. POSIX (i.e. File) support built as a middleware over the DAOS API. DAOS has a native KV API which means that the wire protocol is KV fetch/lookup. On the server, the tree structure is maintained in DCPM and values are either stored in DCPM or SSD depending on sizes. No file serialization is required. A client can directly lookup or insert a key/value pair over the wire and this can happen from many clients concurrently. Keys can be ordered, so we support range query. KV store are partitioned/sharded/replicated/erasure coded over multiple servers for performance & resilience. DAOS can also support complex query since it understand the key-value structure. The native indexing can also be augmented with ad-hoc application-driven index.
• #13: Spark Integration with DAOS – different team at intel Written Hadoop filesystem abstraction class for DAOS. Integrated in DAOS source code. Use DAOS in user-space through libdfs transparently with this Hadoop connector. More value/acceleration to Spark, by integrating DAOS as a datasource through parquet / apache arrow, or through native DAOS API.
• #14: Native Python bindings (pydaos) integrate KV store object directly with python dictionaries. Very pythonic API, you can have iterator, direct assignments, bulk insertion/retrieve Allow people who are not very familiar with C using high level language. Some integration with dbm, pyprob. More work to do