Intel - optimizing ceph performance by leveraging intel® optane™ and 3 d nandtlc ss-ds
3 likes1,057 views
The document discusses optimizing Ceph performance by leveraging Intel® Optane and 3D NAND TLC SSDs, achieving over 2.8 million IOPS with low latency. It highlights the configuration and performance analyses of Ceph clusters, emphasizing the importance of SSDs for enterprise applications and high-performance workloads. Suggestions for tuning configurations to enhance performance and scalability are provided, alongside a call to action for collaboration in the open-source community.
![27
[global]
pid_path = /var/run/ceph
auth_service_required = none
auth_cluster_required = none
auth_client_required = none
mon_data = /var/lib/ceph/ceph.$id
osd_pool_default_pg_num = 2048
osd_pool_default_pgp_num = 2048
osd_objectstore = bluestore
public_network = 172.16.0.0/16
cluster_network = 172.18.0.0/16
enable experimental unrecoverable data
corrupting features = *
bluestore_bluefs = true
bluestore_block_create = false
bluestore_block_db_create = false
bluestore_block_wal_create = false
mon_allow_pool_delete = true
bluestore_block_wal_separate = false
debug objectcacher = 0/0
debug paxos = 0/0
debug journal = 0/0
mutex_perf_counter = True
rbd_op_threads = 4
debug ms = 0/0
debug mds = 0/0
mon_pg_warn_max_per_osd = 10000
debug lockdep = 0/0
debug auth = 0/0
ms_crc_data = False
debug mon = 0/0
debug perfcounter = 0/0
perf = True
debug monc = 0/0
debug throttle = 0/0
debug mds_migrator = 0/0
debug mds_locker = 0/0
[mon]
mon_data = /var/lib/ceph/mon.$id
mon_max_pool_pg_num = 166496
mon_osd_max_split_count = 10000
mon_pg_warn_max_per_osd = 10000
[osd]
osd_data = /var/lib/ceph/mnt/osd-device-$id-data
osd_mkfs_type = xfs
osd_mount_options_xfs = rw,noatime,inode64,logbsize=256k
bluestore_extent_map_shard_min_size = 50
bluefs_buffered_io = true
mon_osd_full_ratio = 0.97
mon_osd_nearfull_ratio = 0.95
bluestore_rocksdb_options =
compression=kNoCompression,max_write_buffer_number=32,min_write_buffer_number_to_merge
=2,recycle_log_file_num=32,compaction_style=kCompactionStyleLevel,write_buffer_size=6710886
4,target_file_size_base=67108864,max_background_compactions=31,level0_file_num_compaction
_trigger=8,level0_slowdown_writes_trigger=32,level0_stop_writes_trigger=64,num_levels=7,max_b
ytes_for_level_base=536870912,max_bytes_for_level_multiplier=8,compaction_threads=32,flusher
_threads=8
bluestore_min_alloc_size = 65536
osd_op_num_threads_per_shard = 2
osd_op_num_shards = 8
bluestore_extent_map_shard_max_size = 200
bluestore_extent_map_shard_target_size = 100
bluestore_csum_type = none
bluestore_max_bytes = 1073741824
bluestore_wal_max_bytes = 2147483648
bluestore_max_ops = 8192
bluestore_wal_max_ops = 8192
Ceph All Flash Tunings
debug rgw = 0/0
debug finisher = 0/0
debug osd = 0/0
debug mds_balancer = 0/0
rocksdb_collect_extended_stats = True
debug hadoop = 0/0
debug client = 0/0
debug zs = 0/0
debug mds_log = 0/0
debug context = 0/0
rocksdb_perf = True
debug bluestore = 0/0
debug bluefs = 0/0
debug objclass = 0/0
debug objecter = 0/0
debug log = 0
ms_crc_header = False
debug filer = 0/0
debug rocksdb = 0/0
rocksdb_collect_memory_stats = True
debug mds_log_expire = 0/0
debug crush = 0/0
debug optracker = 0/0
osd_pool_default_size = 2
debug tp = 0/0
cephx require signatures = False
cephx sign messages = False
debug rados = 0/0
debug journaler = 0/0
debug heartbeatmap = 0/0
debug buffer = 0/0
debug asok = 0/0
debug rbd = 0/0
rocksdb_collect_compaction_stats = False
debug filestore = 0/0
debug timer = 0/0
rbd_cache = False
throttler_perf_counter = False
Intel Confidential](https://image.slidesharecdn.com/05-intel-optimizingcephperformancebyleveraginginteloptaneand3dnandtlcssds-170720075144/85/Intel-optimizing-ceph-performance-by-leveraging-intel-optane-and-3-d-nandtlc-ss-ds-27-320.jpg)
Intel - optimizing ceph performance by leveraging intel® optane™ and 3 d nandtlc ss-ds
- 1. OptimizingCephPerformancebyLeveraging Intel®Optane™and3DNANDTLCSSDs Kenny Chang, Storage Solution Architect, kenny.chang@intel.com July, 2017
- 2. 2 Agenda • Ceph* configuration with Intel® Non-Volatile Memory Technologies • 2.8M IOPS Ceph* cluster with Intel® Optane™ SSDs + Intel® 3D TLC SSDs • Ceph* Performance analysis on Intel® Optane™ SSDs based all-flash array • Summary Intel Confidential *Other names and brands may be claimed as the property of others. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
- 3. 3Intel Confidential *Other names and brands may be claimed as the property of others. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
- 4. 4 Intel®3DNANDSSDsandOPTANESSDTransformStorage Expand the reach of Intel® SSDs. Deliver disruptive value to the data center. 11 Refer to appendix for footnotes Capacity for Less Performance for Less Optimized STORAGE Intel Confidential *Other names and brands may be claimed as the property of others. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
- 5. 5 InnovationforCloudSTORAGE: Intel® Optane™ + Intel® 3D NAND SSDs New Storage Infrastructure: enable high performance and cost effective storage: Journal/Log/Cache Data Openstack/Ceph: ‒ Intel Optane™ as Journal/Metadata/WAL (Best write performance, Lowest latency and Best QoS) ‒ Intel 3D NAND TLC SSD as data store (cost effective storage) ‒ Best IOPS/$, IOPS/TB and TB/Rack Ceph Node (Yesterday) P3520 2TB P3520 2TB P3520 2TB P3520 2TB P3700 U.2 800GB Ceph Node (Today) Intel® Optane™ P4800X (375GB) 3D NAND P4500 4TB 3D XPoint™ P4500 4TB P4500 4TB P4500 4TB P4500 4TB P4500 4TB Transition to + P4500 4TB P4500 4TB Intel Confidential *Other names and brands may be claimed as the property of others. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
- 6. 6 Ceph* on all-flash array Storage providers are struggling to achieve the required high performance There is a growing trend for cloud providers to adopt SSD – CSP who wants to build EBS alike service for their OpenStack* based public/private cloud Strong demands to run enterprise applications OLTP workloads running on Ceph, tail latency is critical high performance multi-purpose Ceph cluster is a key advantage Performance is still an important factor SSD price continue to decrease *Other names and brands may be claimed as the property of others. Intel Confidential *Other names and brands may be claimed as the property of others. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
- 7. 7 Ceph* performance trend with SSD – 4K Random Write 38x performance improvement in Ceph all-flash array! 1.98x 3.7x 1.66x 1.23x 1.19x Intel Confidential *Other names and brands may be claimed as the property of others. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
- 8. 8 Who is using Ceph? Searchable Examples: http://www.slideshare.net/inktank_ceph Telecom CSP/IPDC OEM/ODM Enterprise, FSI, Healthcare, Retailers
- 9. Suggested Configurations for Ceph* Storage Node Standard/good (baseline): Use cases/Applications: that need high capacity storage with high throughput performance NVMe*/PCIe* SSD for Journal + Caching, HDDs as OSD data drive Better IOPS Use cases/Applications: that need higher performance especially for throughput, IOPS and SLAs with medium storage capacity requirements NVMe/PCIe SSD as Journal, High capacity SATA SSD for data drive Best Performance Use cases/Applications: that need highest performance (throughput and IOPS) and low latency/QoS (Quality of Service). All NVMe/PCIe SSDs More information at Ceph.com (new RAs update soon!) http://tracker.ceph.com/projects/ceph/wiki/Tuning_for_All_Flash_ Deployments 9 Ceph* storage node --Good CPU Intel(R) Xeon(R) CPU E5-2650v4 Memory 64 GB NIC Dual 10GbE Disks 1x 1.6TB P3700 + 12 x 4TB HDDs (1:12 ratio) P3700 as Journal and caching Caching software Intel(R) CAS 3.0, option: Intel(R) RSTe/MD4.3 Ceph* Storage node --Better CPU Intel(R) Xeon(R) CPU E5-2690v4 Memory 128 GB NIC Duel 10GbE Disks 1x Intel(R) DC P3700(800G) + 4x Intel(R) DC S3510 1.6TB Or 1xIntel P4800X (375GB) + 8x Intel® DC S3520 1.6TB Ceph* Storage node --Best CPU Intel(R) Xeon(R) CPU E5-2699v4 Memory >= 128 GB NIC Dual 40GbE Disks 1xIntel P4800X (375GB) + 8x Intel® DC P4500 4TB Intel Confidential *Other names and brands may be claimed as the property of others. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
- 10. 10
- 11. 11 Ceph* All Flash Optane configuration 8x Client Node • Intel® Xeon™ processor E5-2699 v4 @ 2.3GHz, 64GB mem • 1x X710 40Gb NIC 8x Storage Node • Intel Xeon processor E5-2699 v4 @ 2.3 GHz • 256GB Memory • 1x 400G SSD for OS • 1x Intel® DC P4800 375G SSD as WAL and DB • 8x 4.0TB Intel® SSD DC P4500 as data drive • 2 OSD instances one each P4500 SSD • Ceph 12.0.0 with Ubuntu 16.10 2x40Gb NIC Test Environment FIO FIO CLIENT 1 1x40Gb NIC FIO FIO CLIENT 2 FIO FIO FIO FIO ….. FIO FIO CLIENT 8 *Other names and brands may be claimed as the property of others. CEPH3 … CEPH8 Workloads • Fio with librbd • 20x 30 GB volumes each client • 4 test cases: 4K random read & write; 64K Sequential read & write Intel Confidential *Other names and brands may be claimed as the property of others. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. CLIENT 3 CEPH2CEPH1 MON OSD1 OSD16…
- 12. 12 Excellent performance on Optane cluster – random read & write hit CPU bottleneck Ceph* Optane Performance overview Throughput Latency (avg.) 99.99% latency 4K Random Read 2876K IOPS 0.9 ms 2.25 4K Random Write 610K IOPS 4.0 ms 25.435 64K Sequential Read 27.5 GB/s 7.6 ms 13.744 64K Sequential Write 13.2 GB/s 11.9 ms 215 Intel Confidential *Other names and brands may be claimed as the property of others. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
- 13. 13 • Good Node Scalability, poor disk scalability for 4K block workloads (CPU throttled!) • NUMA helps a lot on the Performance • Fine tune the # of OSD per node and Drive per Node. Ceph* Optane Performance – Tunings Intel Confidential *Other names and brands may be claimed as the property of others. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
- 14. 14 Ceph* Optane Performance – Performance improvement • The breakthrough high performance of Optane eliminated the WAL & rocksdb bottleneck • 1 P4800X or P3700 covers up to 8x P4500 data drivers as both WAL and rocksdb Intel Confidential *Other names and brands may be claimed as the property of others. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
- 15. 15 Ceph* Optane Performance – latency improvement • Significant tail latency improvement with Optane • 20x latency reduction for 99.99% latency 5.7 ms 14.7 ms 18.3 ms 317.8 ms Intel Confidential *Other names and brands may be claimed as the property of others. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
- 16. 16
- 17. 17 Ceph* Optane Performance Analysis - CPU utilizations • Random Read & Write performance are throttled by CPU • Unbalanced CPU utilization caused by HT efficiency for random workloads • Limiting # of drive scalability for small block random workloads • Need to optimize CPU utilization Intel Confidential *Other names and brands may be claimed as the property of others. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
- 18. Ceph* Optane Performance Analysis - CPU profiling for 4K RR • Perf record for 30 second • Ceph-osd: 34.1%, tp_osd_tp: 65.6% • Heavy network messenger overhead AsyncMsg 18Intel Confidential *Other names and brands may be claimed as the property of others. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
- 19. 19 Ceph* Optane Performance Analysis - CPU profiling for 4K RR Intel Confidential *Other names and brands may be claimed as the property of others. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
- 20. Ceph* Optane Performance Analysis - CPU profiling for 4K RR • The top three consumers of tp_osd_tp thread are • KernelDevice::read, • OSD::ShardedOpWQ::_ process • PrimaryLogPG::do_op. • Perf record 30s. 20Intel Confidential *Other names and brands may be claimed as the property of others. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
- 21. P3700 Optane P3700 Optane bluefs_bufferedIO bluefs_directIO IOPS(K) 389 418 322 429 0 50 100 150 200 250 300 350 400 450 500 KIOPS 4K RW Throughput Ceph Optane : Performance improvement 4K RW performance increased by 7% with Optane with BlueFS buffer IO enabled The date set used in the test is too small, so most of the metadata could be cached, so P3700 was not a bottleneck. But in the real environment with large data set, metadata cache miss will lead to lots of db-disk read, which will made P3700 a bottleneck. We can use bluefs_buffered_io = false to simulate the scenario. – Optane can bring 1.33x performance improvement 21 7% 33% Intel Confidential *Other names and brands may be claimed as the property of others. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
- 22. Ceph* Optane Performance Analysis - WAL tunings and optimizations • Key takeaways: • rocksdb memTable overhead lead to 50000 IOPS difference. (tuning2 vs. tuning3) • If we don’t clean bluestore WAL data in rocksdb, rocksdb overhead increase dramatically with the # of metadata increase: • Use an external WAL device to store WAL data, and just write WAL metadata into rocksdb! Tunings 4K RW IOPS comments Default: Baseline(NVMe as DB&& WAL drive) 340000 Separated DB&&WAL device Tuning1: DB on NVMe && WAL on Ramdisk 360000 Move WAL to Ramdisk Tuning2: Tuning1+disable rocksdb WAL 360000 RocksDB tuning Tuning3: Tuning2+omit WAL in deferred write mode 410000 Don’t write WAL in deferred write mode Tuning4: Tuning1+write WAL but don’t remove WAL from rocksdb 240000 Write WAL before write metadata into rocksdb, but will not clean WAL after write data to data device in deferred write mode Tuning5: Tuning1+external WAL 380000 Write WAL to an external WAL device, and write its metadata to rocksdb 22 Based on 5-OSD node cluster Intel Confidential *Other names and brands may be claimed as the property of others. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
- 23. 23
- 24. 24 Summary & Next Summary • Ceph* is awesome! • Strong demands for all-flash array Ceph* solutions • Optane based all-flash array Ceph* cluster is capable of delivering over 2.8M IOPS with very low latency! • Let’s work together to make Ceph* more efficient with all-flash array! Next • OLTP workloads over AFA Ceph • Client side cache on Optane Intel Confidential *Other names and brands may be claimed as the property of others. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
- 25. 25 Call for action • Participate in the open source community, and different storage projects • Try our tools – and give us feedback • CeTune: https://github.com/01org/CeTune • Virtual Storage Manager: https://01.org/virtual-storage-manager • COSBench: https://github.com/intel-cloud/cosbench • Optimize Ceph* for efficient SDS solutions! Intel Confidential *Other names and brands may be claimed as the property of others. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
- 26. 26 Backup
- 27. 27 [global] pid_path = /var/run/ceph auth_service_required = none auth_cluster_required = none auth_client_required = none mon_data = /var/lib/ceph/ceph.$id osd_pool_default_pg_num = 2048 osd_pool_default_pgp_num = 2048 osd_objectstore = bluestore public_network = 172.16.0.0/16 cluster_network = 172.18.0.0/16 enable experimental unrecoverable data corrupting features = * bluestore_bluefs = true bluestore_block_create = false bluestore_block_db_create = false bluestore_block_wal_create = false mon_allow_pool_delete = true bluestore_block_wal_separate = false debug objectcacher = 0/0 debug paxos = 0/0 debug journal = 0/0 mutex_perf_counter = True rbd_op_threads = 4 debug ms = 0/0 debug mds = 0/0 mon_pg_warn_max_per_osd = 10000 debug lockdep = 0/0 debug auth = 0/0 ms_crc_data = False debug mon = 0/0 debug perfcounter = 0/0 perf = True debug monc = 0/0 debug throttle = 0/0 debug mds_migrator = 0/0 debug mds_locker = 0/0 [mon] mon_data = /var/lib/ceph/mon.$id mon_max_pool_pg_num = 166496 mon_osd_max_split_count = 10000 mon_pg_warn_max_per_osd = 10000 [osd] osd_data = /var/lib/ceph/mnt/osd-device-$id-data osd_mkfs_type = xfs osd_mount_options_xfs = rw,noatime,inode64,logbsize=256k bluestore_extent_map_shard_min_size = 50 bluefs_buffered_io = true mon_osd_full_ratio = 0.97 mon_osd_nearfull_ratio = 0.95 bluestore_rocksdb_options = compression=kNoCompression,max_write_buffer_number=32,min_write_buffer_number_to_merge =2,recycle_log_file_num=32,compaction_style=kCompactionStyleLevel,write_buffer_size=6710886 4,target_file_size_base=67108864,max_background_compactions=31,level0_file_num_compaction _trigger=8,level0_slowdown_writes_trigger=32,level0_stop_writes_trigger=64,num_levels=7,max_b ytes_for_level_base=536870912,max_bytes_for_level_multiplier=8,compaction_threads=32,flusher _threads=8 bluestore_min_alloc_size = 65536 osd_op_num_threads_per_shard = 2 osd_op_num_shards = 8 bluestore_extent_map_shard_max_size = 200 bluestore_extent_map_shard_target_size = 100 bluestore_csum_type = none bluestore_max_bytes = 1073741824 bluestore_wal_max_bytes = 2147483648 bluestore_max_ops = 8192 bluestore_wal_max_ops = 8192 Ceph All Flash Tunings debug rgw = 0/0 debug finisher = 0/0 debug osd = 0/0 debug mds_balancer = 0/0 rocksdb_collect_extended_stats = True debug hadoop = 0/0 debug client = 0/0 debug zs = 0/0 debug mds_log = 0/0 debug context = 0/0 rocksdb_perf = True debug bluestore = 0/0 debug bluefs = 0/0 debug objclass = 0/0 debug objecter = 0/0 debug log = 0 ms_crc_header = False debug filer = 0/0 debug rocksdb = 0/0 rocksdb_collect_memory_stats = True debug mds_log_expire = 0/0 debug crush = 0/0 debug optracker = 0/0 osd_pool_default_size = 2 debug tp = 0/0 cephx require signatures = False cephx sign messages = False debug rados = 0/0 debug journaler = 0/0 debug heartbeatmap = 0/0 debug buffer = 0/0 debug asok = 0/0 debug rbd = 0/0 rocksdb_collect_compaction_stats = False debug filestore = 0/0 debug timer = 0/0 rbd_cache = False throttler_perf_counter = False Intel Confidential
- 28. Legalnotices Copyright © 2016 Intel Corporation. All rights reserved. Intel, the Intel logo, Xeon, Intel Inside, and 3D XPoint are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. FTC Optimization Notice Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice revision #20110804 The cost reduction scenarios described in this document are intended to enable you to get a better understanding of how the purchase of a given Intel product, combined with a number of situation-specific variables, might affect your future cost and savings. Nothing in this document should be interpreted as either a promise of or contract for a given level of costs. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance. 28