QCT Ceph Solution - Design Consideration and Reference Architecture
Download as PPTX, PDF7 likes1,891 views
This document discusses QCT's Ceph storage solutions, including an overview of Ceph architecture, QCT hardware platforms, Red Hat Ceph software, workload considerations, reference architectures, test results and a QCT/Red Hat whitepaper. It provides technical details on QCT's throughput-optimized and capacity-optimized solutions and shows how they address different storage needs through workload-driven design. Hands-on testing and a test drive lab are offered to explore Ceph features and configurations.
![• Object Replication
– 1 Primary + 2 Replica (or more)
– CRUSH Allocation Ruleset
• Erasure Coding
– [k+m], e.g. 4+2, 8+3
– Better Data Efficiency
• k/(k+m) vs. 1/(1+replication)
20
Erasure Coding vs. Replication](https://image.slidesharecdn.com/07-qctforcephdayportland2016-160601131451/85/QCT-Ceph-Solution-Design-Consideration-and-Reference-Architecture-20-320.jpg)
QCT Ceph Solution - Design Consideration and Reference Architecture
- 1. QCT Ceph Solution – Design Consideration and Reference Architecture Gary Lee AVP, QCT
- 2. • Industry Trend and Customer Needs • Ceph Architecture • Technology • Ceph Reference Architecture and QCT Solution • Test Result • QCT/Red Hat Ceph Whitepaper 2 AGENDA
- 4. 4
- 5. • Structured Data -> Unstructured/Structured Data • Data -> Big Data, Fast Data • Data Processing -> Data Modeling -> Data Science • IT -> DT • Monolithic -> Microservice 5 Industry Trend
- 6. • Scalable Size • Variable Type • Longivity Time • Distributed Location • Versatile Workload 6 • Affordable Price • Available Service • Continuous Innovation • Consistent Management • Neutral Vendor Customer Needs
- 8. Ceph Storage Cluster 8 Cluster Network Ceph Linux CPU Memory SSD HDD NIC Ceph Linux CPU Memory SSD HDD NIC Ceph Linux CPU Memory SSD HDD NIC Ceph Linux CPU Memory SSD HDD NIC Object Block File Unified Storage Scale-out Cluster Open Source Software Open Commodity Hardware …..
- 9. 9 Block I/O Ceph Client RBD RADO SGW Ceph FS Object I/O File I/O RADOS/Cluster Network OSD File System I/O Disk I/O PublicNetwork End-to-end Data Path App Service
- 11. Public Network (ex. 10GbE or 40GbE) Cluster Network (ex. 10GbE or 40GbE) Ceph Monitor …... RCT or RCC Nx Ceph OSD Nodes Ceph OSD Node Clients Ceph OSD Node Ceph OSD Node Ceph Hardware Architecture
- 12. 12 Technology
- 13. 13 • 2x Intel E5-2600 CPU • 16x DDR4 Memory • 12x 3.5” SAS/SATA HDD • 4x SATA SSD + PCIe M.2 • 1x SATADOM • 1x 1G/10G NIC • BMC with 1G NIC • 1x PCIe x8 Mezz Card • 1x PCIe x8 SAS Controller • 1U QCT Ceph Storage Server D51PH-1ULH
- 14. 14 • Mono/Dual Node • 2x Intel E5-2600 CPU • 16x DDR4 Memory • 78x or 2x 35x SSD/HDD • 1x 1G/10G NIC • BMC with 1G NIC • 1x PCIe x8 SAS Controller • 1x PCIe x8 HHLH Card • 1x PCIe x16 FHHL Card • 4U QCT Ceph Storage Server T21P-4U
- 15. 15 • 1x Intel Xeon D SoC CPU • 4x DDR4 Memory • 12x SAS/SATA HDD • 4x SATA SSD • 2x SATA SSD for OS • 1x 1G/10G NIC • BMC with 1G NIC • 1x PCIe x8 Mezz Card • 1x PCIe x8 SAS Controller • 1U QCT Ceph Storage Server SD1Q-1ULH
- 16. • Standalone, without EC • Standalone, with EC • Hyper-converged, without EC • High Core vs. High Frequency • 1x OSD ~ (0.3-0.5)x Core + 2G RAM 16 CPU/Memory
- 17. • SSD: – Journal – Tier – File System Cache – Client Cache • Journal – HDD: SSD (SATA/SAS): 4~5 – HDD: NVMe: 12~18 17 SSD/NVMe
- 18. • 2x NVMe ~40Gb • 4x NVMe ~100Gb • 2x SATA SSD ~10Gb • 1x SAS SSD ~10Gb • (20~25)x HDD ~10Gb • ~100x HDD ~40Gb 18 NIC 10G/40G -> 25G/100G
- 19. • CPU Offload through RDMA/iWARP • Erasure Coding Offload • Allocate computing on different silicon areas 19 NIC I/O Offloading
- 20. • Object Replication – 1 Primary + 2 Replica (or more) – CRUSH Allocation Ruleset • Erasure Coding – [k+m], e.g. 4+2, 8+3 – Better Data Efficiency • k/(k+m) vs. 1/(1+replication) 20 Erasure Coding vs. Replication
- 21. Size/ Workload Small Medium Large Throughput Transfer Bandwidth Sequential R/W Capacity Cost/capacity Scalability IOPS IOPS/ per 4k Block Random R/W Hyper-converged ? Desktop Virtualization Latency Random R/W Hadoop ? 21 Workload and Configuration
- 22. 22 Red Hat Ceph
- 23. • Intel ISA-L • Intel SPDK • Intel CAS • Mellanox Accelio Library 23 Vendor-specific Value-added Software
- 24. 24 Ceph Reference Architecture and QCT Solution
- 25. • Trade-off among Technologies • Scalable in Architecture • Optimized for Workload • Affordable as Expected Design Principle
- 26. 1. Needs for scale-out storage 2. Target workload 3. Access method 4. Storage capacity 5. Data protection methods 6. Fault domain risk tolerance 26 Design Considerations
- 28. SMALL (500TB*) MEDIUM (>1PB*) LARGE (>2PB*) Throughput optimized QxStor RCT-200 16x D51PH-1ULH (16U) • 12x 8TB HDDs • 3x SSDs • 1x dual port 10GbE • 3x replica QxStor RCT-400 6x T21P-4U/Dual (24U) • 2x 35x 8TB HDDs • 2x 2x PCIe SSDs • 2x single port 40GbE • 3x replica QxStor RCT-400 11x T21P-4U/Dual (44U) • 2x 35x 8TB HDDs • 2x 2x PCIe SSDs • 2x single port 40GbE • 3x replica Cost/Capacity optimized IOPS optimized Future direction Future direction NA * Usable storage capacity QxStor RCC-400 Nx T21P-4U/Dual • 2x 35x 8TB HDDs • 0x SSDs • 2x dual port 10GbE • Erasure Coding 4:2 QCT QxStor Red Hat Ceph Storage Edition Portfolio Workload-driven Integrated Software/Hardware Solution
- 29. • Densest 1U Ceph building block • Best reliability with smaller failure domain • Scale at high scale 2x 280TB • At once obtain best throughput and density • Block or object storage • 3x replication • Video, audio, image repositories, and streaming media • Highest density 560TB raw capacity per chassis with greatest price/performance • Typically object storage • Erasure coding common for maximizing usable capacity • Object archive Throughput-Optimized RCC-400RCT-200 RCT-400 Cost/Capacity-Optimized USECASE QCT QxStor Red Hat Ceph Storage Edition Co-engineered with Red Hat Storage team to provide Optimized Ceph Solution
- 30. 30 Ceph Solution Deployment Using QCT QPT Bare Metal Privision Tool
- 31. 31 Ceph Solution Deployment Using QCT QPT Bare Metal Privision Tool
- 32. 32 QCT Solution Value Proposition • Workload-driven • Hardware/software pre-validated, pre-optimized and pre-integrated • Up and running in minutes • Balance between production (stable) and innovation (up-streaming)
- 33. 33 Test Result
- 34. Client 1 S2B Client 2 S2B Client 3 S2B Ceph 1 S2PH Ceph 2 S2PH Ceph 3 S2PH Ceph 5 S2PH Ceph 4 S2PH Client 8 S2B Client 9 S2B Client 10 S2B 10Gb 10Gb Public Network Cluster Network General Configuration • 5 Ceph nodes (S2PH) with each 2 x 10Gb link. • 10 Client nodes (S2B) with each 2 x 10Gb link. • Public network : Balanced bandwidth between Client nodes and Ceph nodes. • Cluster network : Offload the traffic from public network to improve performance. Option 1 (w/o SSD) a. 12 OSD per Ceph storage node b. S2PH (E5-2660) x2 c. RAM : 128 GB Option 2 : (w/ SSD) a. 12 OSD / 3 SSD per Ceph storage node b. S2PH (E5-2660) x2 c. RAM : 12 (OSD) x 2GB = 24 GB Testing Configuration (Throughput-Optimized)
- 35. Client 1 S2S Client 2 S2S Client 3 S2S Ceph 1 S2P Ceph 2 S2P Client 6 S2S Client 7 S2S Client 8 S2S 10Gb Public Network 40Gb 40Gb General Configuration • 2 Ceph nodes (S2P) with each 2 x 10Gb link. • 8 Client nodes (S2S) with each 2 x 10Gb link. • Public network : Balanced bandwidth between Client nodes and Ceph nodes. • Cluster network : Offload the traffic from public network to improve performance. Option 1 (w/o SSD) a. 35 OSD per Ceph storage node b. S2P (E5-2660) x2 c. RAM : 128 GB Option 2 : (w/ SSD) a. 35 OSD / 2 PCI-SSD per Ceph storage node b. S2P (E5-2660) x2 c. RAM : 128 GB Testing Configuration (Capacity-Optimized)
- 36. Level Component Test Suite Raw I/O Disk FIO Network I/O Network iperf Object API I/O librados radosbench Object I/O RGW Cosbench Block I/O RBD librbdfio 36 CBT (Ceph Benchmarking Tool)
- 39. 39 Price, in terms of Performance
- 40. 40 Price, in terms of Capacity
- 44. 44 http://www.qct.io/account/d ownload/download?order_ download_id=1022&dtype= Reference%20Architecture QCT/Red Hat Ceph Solution Brief
- 46. • The Red Hat Ceph Storage Test Drive lab in QCT Solution Center provides you a free hands-on experience. You'll be able to explore the features and simplicity of the product in real-time. • Concepts: Ceph feature and functional test • Lab Exercises: Ceph Basics Ceph Management - Calamari/CLI Ceph Object/Block Access 46 QCT Offer TryCeph (Test Drive) Later
- 47. 47 Remote access to QCT cloud solution centers • Easy to test. Anytime and anywhere. • No facilities and logistic needed • Configurations • RCT-200 and newest QCT solutions QCT Offer TryCeph (Test Drive) Later
- 48. • Ceph is Open Architecture • QCT, Red Hat and Intel collaborate to provide – Workload-driven, – Pre-integrated, – Comprehensive-tested and – Well-optimized solution • Red Hat – Open Software/Support Pioneer Intel – Open Silicon/Technology Innovator QCT – Open System/Solution Provider • Together We Provide the Best 48 CONCLUSION
- 50. www.QCT.io QCT CONFIDENTIAL50 Looking for innovative cloud solution? Come to QCT, who else?
Editor's Notes
- #29: Here is 3 skus based on small-medium-large scale. For larger scale, suggest customer to adopt RCT-400 or RCC-400. QCT is planning to do sku that is optimized for IOPS-intensive workloads. We’ll launch it in 2016H2