Breaking the Sound Barrier with Persistent Memory
1 like331 views
The document discusses the challenges of disk writes and performance issues in storage systems, highlighting the potential of persistent memory to address these problems. An experiment with bucketcache demonstrates significant performance improvements, achieving a throughput increase of approximately 6 times and reducing latency by 85%. Overall, persistent memory presents a promising solution for enhancing storage architecture and efficiency.
Breaking the Sound Barrier with Persistent Memory
- 1. Breaking the Sound Barrier with Persistent Memory Liqi Yi Shylaja Kokoori
- 2. Legal Disclaimer 2 Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance. Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Results have been simulated and are provided for informational purposes only. Results were derived using simulations run on an architecture simulator or model. Any difference in system hardware or software design or configuration may affect actual performance. Intel does not control or audit the design or implementation of third party benchmark data or Web sites referenced in this document. Intel encourages all of its customers to visit the referenced Web sites or others where similar performance benchmark data are reported and confirm whether the referenced benchmark data are accurate and reflect performance of systems available for purchase. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and other countries. *Other names and brands may be claimed as the property of others. Copyright © 2016 Intel Corporation. All rights reserved.
- 3. Problems at Hand Disk writes in burst fashion High bandwidth in flush, compaction Disk bandwidth inflation Write: Key/Value (KV) pairs written to disk many times in consecutive compactions Read: All KVs bring back to memory when HFile block was hit Data format changing Write: “serialize” maps to HFile blocks Read: linear scan within HFile blocks 3
- 4. Problems at Hand Disk writes in burst fashion High bandwidth in flush, compaction Disk bandwidth inflation Write: Key/Value (KV) pairs written to disk many times in consecutive compactions Read: All KVs bring back to memory when HFile block was hit Data format changing Write: “serialize” maps to HFile blocks Read: linear scan within HFile blocks 4 These problems come with Mem+Disk hardware architecture
- 5. Addressing Disk writes in burst fashion Faster drives: PCIe SSDs Disk bandwidth inflation Larger DRAM Data format changing None 5
- 6. Addressing Disk writes in burst fashion Faster drives: PCIe SSDs Disk bandwidth inflation Larger DRAM Data format changing None 6 Expensive, still not fast enough
- 7. Addressing Disk writes in burst fashion Faster drives: PCIe SSDs Disk bandwidth inflation Larger DRAM Data format changing None 7 Expensive, still not fast enough More expensive, small, volatile
- 8. Addressing Disk writes in burst fashion Faster drives: PCIe SSDs Disk bandwidth inflation Larger DRAM Data format changing None 8 Expensive, still not fast enough More expensive, small, volatile Do we have to persist on block device?
- 9. Addressing with Persistent Memory Disk writes in burst fashion Disk bandwidth inflation Data format changing 9 High bandwidth, low latency High bandwidth, non-volatile Could be eliminated
- 10. Persistent Memory 10 CPU caches (L1-L3) Register DRAM Persistent Memory NAND SSD HDD and other block devices CPU caches (L1-L3) Register DRAM NAND SSD HDD and other block devices Bandwidt h Latency Size Byte Addressable Performance Gap
- 11. Experiment on BucketCache • BucketCache on persistent memory • Code change in HBase • Introduce new IOEngine for BucketCache • Use libraries from http://pmem.io for persistent memory operations • Experiment • Persistent memory emulation with configurable latencies • Focus on performance impact 11
- 12. Experiment Design • Basic setup • HBase 2.0.0-SNAPSHOT, YCSB 0.6.0, Hadoop 2.5.2 • Preloaded table, 10 fields per row, 100 Bytes per field • 100% un-throttled uniform read, measures after BlockCache is filled • Experiments • Baseline: DRAM_LRU_BlockCache only • PM runs: DRAM_LRU_BlockCache + different_size_PM_BucketCache • Measure the throughput/latency impact 12
- 13. 1.00 1.06 1.13 1.23 1.44 1.65 1.98 2.44 3.13 4.22 5.95 0 1 2 3 4 5 6 7 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Normalizedspeedup Size of extra PM BucketCache Speed up with extra BucketCache Result: Throughput 5x 13
- 14. 1.00 0.98 0.92 0.83 0.73 0.63 0.52 0.42 0.33 0.23 0.14 0 0.2 0.4 0.6 0.8 1 1.2 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Normalized95%latencyvs.baseline Size of extra PM BucketCache (% of test table) 95% latency with extra BucketCache Result: 95% Latency Reduced by 85% 14
- 15. Summary • Significant performance improvement with persistent memory (~6x throughput, latency reduced by 85%) • Offers new possible solution from architecture side • Persist partially or completely on persistent memory • No more format changing overhead • Faster recovery 15
- 16. Acknowledging Anoop S John, Ramkrishna S Vasudevan 16
Editor's Notes
- #4: These are general issues where a two level LSM tree architecture is used. For 64K Hfile block, 100Byte Values, inflation for caching is about hundreds.
- #5: These are general issues where a two level
- #6: These are general issues where a two level
- #7: These are general issues where a two level
- #8: These are general issues where a two level
- #9: These are general issues where a two level
- #10: These are general issues where a two level
- #11: Dram ~100ns Nand SSD P3700 20us Nand SSD S3700 50-60us WD black Iometer: avg. 5-6ms, max 50ms