Ceph on Intel: Intel Storage Components, Benchmarks, and Contributions
- 1. CephonIntel IntelStorageComponents,Benchmarks, andContributions Daniel Ferber Open Source Software Defined Storage Technologist, Intel Storage Group
- 3. is Changing Source: IDC – The Digital Universe of Opportunities: Rich Data and the Increasing Value of the Internet of Things - April 2014 Fromnow until 2020, the size ofthe digital universe will aboutdouble everytwoyears Information Growth* 2X What we do with data ischanging, traditional storage infrastructuredoesnot solve tomorrow’sproblems Complexity ShiftingofIT servicesto cloud computing and next-generationplatforms Cloud Emergence offlash storage and software-defined environments New Technologies The World
- 4. Explosion EVERY MINUTE EVERY DAY* 300 HOURS VIDEO UPLOADED TO YOUTUBE 51,000 APPS DOWNLOADED 204 MILLION E-MAILS Source: TechSpartan.co.uk - 2013 vs 2015 In an Internet minute 48 HOURS VIDEO UPLOADED TO YOUTUBE 47,000 APPS DOWNLOADED 200 MILLIONE-MAILS 2013 2015 Information
- 5. TheImpactof theCloud Empowerment of the end-user through cloud services Emergence of new technologies and architectures Shifting the role of information technology professionals
- 6. Price + Performance = AffordableFlash Reliability and DurabilityStorage
- 7. Storage Compression Encryption Erasure coding Deduplication Flash (Non-volatile memory) Data tiering Workload Optimization
- 10. STORAGE TRANSFORMING KEY DRIVERS CLOUD CLOUD SERVICE PROVIDERS 1 STORAGE MEDIA TRANSITIONS 3 3D XPOINT™ TECHNOLOGY NEXT GENERATION ARCHITECTURES 2
- 11. WhereDataisCreated 0 10 20 30 40 50 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 (ZB) Unstructured Data Structured Data Sources: IDC, 2011 Worldwide Enterprise Storage Systems 2011-2015 Forecast update IDC, The Digital Universe Study In 2020 Forecasts 4.4ZB 44ZB 90% 10% 0% 20% 40% 60% 80% 100% 2012 2013 2014 2015 2016 2017 2018 2019 2020 % of Total Digital Universe Emerging Markets Mature Markets Sources: IDC,’s Digital Universe Study, 2014 USA, Canada, Western Europe, Australia, NZ, and Japan China, India, Mexico, Brazil, and Russia By 2020, about 90% of all data will be unstructured, driven by Consumer Images, Voice, and the Web Emerging Markets will Surpass Mature Markets before 2017 regarding data creation. Data Creation by Type (ZB) 11
- 12. AnnualGrowthRateforDataStorageCapacity 10% 25% 15% 13% 12% 8% 6% 3% 3% 3% 0% 2% 0% 5% 10% 15% 20% 25% 30% 1% to 10% annually 11% to 20% annually 21% to 30% annually 31% to 40% annually 41% to 50% annually 51% to 60% annually 61% to 70% annually 71% to 80% annually 81% to 90% annually 91% to 100% annually More than 100% annually Don't know Source: ESG Research Report: 2015 Data Storage Market Trends
- 13. SSD Use in Servers or External Storage Systems Source: ESG Research Report: 2015 Data Storage Market Trends
- 14. 14 IntelStorage
- 15. ProcessorD Delivering the performance and advanced intelligence of Intel® Xeon® processors to dense and low power storage designs Intel®Xeon®
- 16. Latency: ~100x Size of Data: ~1,000x 1000x faster than NAND 1000x higher endurance of NAND 10x denser than DRAM Technology claims are based on comparisons of latency, density, and write cycling metrics amongst memory technologies recorded on published specifications of in-market memory products against internal Intel specifications. 3DXpOINT™ TECHNOLOGY New Class of Non-Volatile Memory
- 17. 17 Intel®SolidStateDrives “The only SSDs that never ever gave me any issues like timeouts, task aborts… are Intel DC S3700s” From a post on ceph-devel* Source: http://ceph.com/r esources/maili ng-‐‑list-‐‑irc
- 18. 18 Intel’sroleinstorage Advance theIndustry OpenSource&Standards BuildanOpenEcosystem Intel®StorageBuilders Endusersolutions Cloud,Enterprise IntelTechnologyLeadership Storage Optimized CPU’s Intel® Xeon® E5v4 2600 Platform Intel® Xeon® Processor D-1500Platform Storage Optimized Software Intel® Intelligent AccelerationLibrary Intel® Storage Performance Development Kit Non-Volatile Memory 3D Xpoint™ Intel® Solid StateDrives for Datacenter >7 Cloud storage solutions architectures 70+ partners Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such asSYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. helpingcustomerstoenableNextgen storage Next gen solutions architectures>26 >10 Enterprise storage solution architectures
- 20. 20 Intel Ceph Contribution Timeline 2014 2015 2016 * Right Edge of box indicates approximate release date New Key/Value Store Backend (rocksdb) Giant* Hammer Infernalis Jewel CRUSH Placement Algorithm improvements (straw2 bucket type) BluestoreBackend Optimizations for NVM Bluestore SPDK Optimizations RADOS I/O Hinting (35% better EC Write erformance) Cache-tieringwith SSDs (Write support) PMStore (NVM-optimized backend based on libpmem) RGW, Bluestore Compression,Encryption (w/ ISA-L, QAT backend) VirtualStorageManager (VSM)Open Sourced CeTune Open Sourced ErasureCoding support with ISA-L Cache-tieringwith SSDs (Read support) Client-sideBlock Cache (librbd)
- 21. Ceph@Intel–2016CephFocusAreas 21 Optimize for Intel® platforms, flash and networking • Compression, Encryption hardware offloads (QAT & SOCs) • PMStore (for 3D XPoint DIMMs) • RBD caching and Cache tiering with NVM • IA optimized storage libraries to reduce latency (ISA-L, SPDK) Performance profiling, analysis and community contributions • All flash workload profiling and latency analysis • Streaming,Database and Analytics workload drivenoptimizations Ceph enterprise usages and hardening • Manageability (Virtual Storage Manager) • Multi Data Center clustering (e.g., async mirroring) End Customer POCs with focus on broad industry influence • CDN,Cloud DVR,Video Surveillance, Ceph Cloud Services, AnalyticsPOCs Ready to use IA, Intel NVM optimized systems & solutions from OEMs & ISVs • Ready to use IA, Intel NVM optimized systems & solutions from OEMs & ISVs • Intel system configurations, white papers,case studies • Industry events coverage Go to Market Intel® Storage AccelerationLibrary (Intel® ISA-L) Intel® StoragePerformance DevelopmentKit (SPDK) Intel® Cache Acceleration Software (Intel® CAS) Virtual StorageManager Ce-Tune Ceph Profiler
- 22. 22 Ceph Storage Cluster Hardware Environment Overview Ceph network (192.168.142.0/24) - 10Gbps CBT / Zabbix / Monitoring FIO RBD Client • OSD System Config: Intel Xeon E5-2699v3 2x@2.30 GHz, 72cores w/HT, 96GB, Cache 46080KB, 128GB DDR4 • Each system with 4xP3700 800GB NVMe, partitioned into 4 OSD’s each, 16OSD’stotal pernode • FIO Client Systems: Intel Xeon E5-2699 v3 2x@2.30 GHz,72 coresw/ HT, 96GB, Cache 46080KB, 128GB DDR4 • Cephv0.94.3 Hammer Release, CentOS7.1, 3.10-229 Kernel, Linked with JEMalloc 3.6 • CBT used for testing and data acquisition • Single 10GbE network for client & replication data transfer, Replication factor 2 FIO RBD Client FIO RBD Client FIO RBD Client FIO RBD Client FIO RBD Client FatTwin (4x dual-socket XeonE5 v3) FatTwin (4x dual-socket XeonE5 v3) CephOSD1 NVMe1 NVMe3 NVMe2 NVMe4 CephOSD2 CephOSD3 CephOSD4 CephOSD16 … CephOSD1 NVMe1 NVMe3 NVMe2 NVMe4 CephOSD2 CephOSD3 CephOSD4 CephOSD16 … CephOSD1 NVMe1 NVMe3 NVMe2 NVMe4 CephOSD2 CephOSD3 CephOSD4 CephOSD16 … CephOSD1 NVMe1 NVMe3 NVMe2 NVMe4 CephOSD2 CephOSD3 CephOSD4 CephOSD16 … CephOSD1 NVMe1 NVMe3 NVMe2 NVMe4 CephOSD2 CephOSD3 CephOSD4 CephOSD16 … SuperMicro 1028U SuperMicro 1028U SuperMicro 1028U SuperMicro 1028U SuperMicro 1028U Intel Xeon E5 v3 18 Core CPUs Intel P3700 NVMe PCI-e Flash Easily serviceable NVMe Drives
- 23. 23 4KRandomRead&WritePerformanceSummaryFirst Ceph cluster to break 1 Million 4K random IOPS Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Any difference in system hardware or software design or configuration may affect actual performance. See configuration slides in backup for details on software configuration and test benchmark parameters. Workload Pattern Max IOPS 4K 100% Random Reads (2TB Dataset) 1.35Million 4K 100% Random Reads (4.8TB Dataset) 1.15Million 4K 100% Random Writes (4.8TB Dataset) 200K 4K 70%/30% Read/Write OLTP Mix (4.8TB Dataset) 452K Source: Openstack Summit 2015: Accelerating Cassandraworkloadson ceph with all flashpcie ssds
- 24. Red Hat Ceph Reference Architecture Documents 24
- 25. MetaFormulaforCephDeployments • Have a general understanding of the use cases you want to support with Ceph • Understand the kind of performance or cost/performance you want to deliver • Refer to a reference architecture resource to match your use case(s) with known and measured reference architectures: • http://www.redhat.com/en/resources/performance-and-sizing-guide-red-hat- ceph-storage-qct-servers • https://www.redhat.com/en/files/resources/en-rhst-cephstorage-supermicro- INC0270868_v2_0715.pdf • These documents have Ceph config, tuning and best practices guidance • Additional help is available from Red Hat, including support and quick start 25
- 26. IntelExperience theStorage Revolution in accelerating next generation storage solutions Flash technology for high performance to accelerate software- defined storage Invest Embrace Align
- 27. Legalnotices Copyright © 2016 Intel Corporation. All rights reserved. Intel, the Intel logo, Xeon, Intel Inside, and3D XPoint are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names andbrands may be claimedas the property of others. FTC Optimization Notice Intel's compilers may or may not optimizeto the samedegreefor non-Intel microprocessors foroptimizations that arenot uniqueto Intel microprocessors. TheseoptimizationsincludeSSE2, SSE3, andSSSE3 instruction sets andotheroptimizations. Intel does not guarantee the availability, functionality, oreffectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for usewith Intel microprocessors. Certain optimizations not specific to Intel microarchitectureare reserved for Intel microprocessors. Pleasereferto the applicable product User and Reference Guides for moreinformation regarding the specific instruction sets coveredby this notice. Notice revision #20110804 The cost reduction scenarios describedin this document are intendedto enable you to get a better understanding of how the purchaseof a given Intel product, combinedwith anumberof situation-specific variables, might affect yourfuturecost and savings. Nothing in this document shouldbeinterpreted as either a promiseof or contract for agiven level of costs.
- 29. Backup
- 30. 30