Mellanox High Performance Networks for Ceph
7 likes6,952 views
This document discusses how Mellanox networks enable high performance Ceph storage clusters. It notes that Ceph performance and scalability are dictated by the backend cluster network performance. It provides examples of customers deploying Ceph with Mellanox 40GbE and 10GbE interconnects, and highlights how these networks allow building scalable, high performing storage solutions. Specifically, it shows how 40GbE cluster networks and 40GbE client networks provide much higher throughput and IOPS compared to 10GbE. The document concludes by mentioning how RDMA offloads can free CPU for application processing, and how the Accelio library enables high performance RDMA for Ceph.
Mellanox High Performance Networks for Ceph
- 1. Building World Class Data Centers Mellanox High Performance Networks for Ceph Ceph Day, June 10th, 2014
- 2. © 2014 Mellanox Technologies 2 Leading Supplier of End-to-End Interconnect Solutions Virtual Protocol Interconnect Storage Front / Back-End Server / Compute Switch / Gateway 56G IB & FCoIB 56G InfiniBand 10/40/56GbE & FCoE 10/40/56GbE Virtual Protocol Interconnect Host/Fabric SoftwareICs Switches/GatewaysAdapter Cards Cables/Modules Comprehensive End-to-End InfiniBand and Ethernet Portfolio Metro / WAN
- 3. © 2014 Mellanox Technologies 3 The Future Depends on Fastest Interconnects 10Gb/s 40/56Gb/s1Gb/s
- 4. © 2014 Mellanox Technologies 4 From Scale-Up to Scale-Out Architecture Only way to support storage capacity growth in a cost-effective manner We have seen this transition on the compute side in HPC in the early 2000s Scaling performance linearly requires “seamless connectivity” (ie lossless, high bw, low latency, cpu offloads) Interconnect Capabilities Determine Scale Out Performance
- 5. © 2014 Mellanox Technologies 5 CEPH and Networks High performance networks enable maximum cluster availability • Clients, OSD, Monitors and Metadata servers communicate over multiple network layers • Real-time requirements for heartbeat, replication, recovery and re-balancing Cluster (“backend”) network performance dictates cluster’s performance and scalability • “Network load between Ceph OSD Daemons easily dwarfs the network load between Ceph Clients and the Ceph Storage Cluster” (Ceph Documentation)
- 6. © 2014 Mellanox Technologies 6 How Customers Deploy CEPH with Mellanox Interconnect Building Scalable, Performing Storage Solutions • Cluster network @ 40Gb Ethernet • Clients @ 10G/40Gb Ethernet Directly connect over 500 Client Nodes • Target Retail Cost: US$350/1TB Scale Out Customers Use SSDs • For OSDs and Journals 8.5PB System Currently Being Deployed
- 7. © 2014 Mellanox Technologies 7 CEPH Deployment Using 10GbE and 40GbE Cluster (Private) Network @ 40GbE • Smooth HA, unblocked heartbeats, efficient data balancing Throughput Clients @ 40GbE • Guaranties line rate for high ingress/egress clients IOPs Clients @ 10GbE / 40GbE • 100K+ IOPs/Client @4K blocks 20x Higher Throughput , 4x Higher IOPs with 40Gb Ethernet Clients! (http://www.mellanox.com/related-docs/whitepapers/WP_Deploying_Ceph_over_High_Performance_Networks.pdf) Throughput Testing results based on fio benchmark, 8m block, 20GB file,128 parallel jobs, RBD Kernel Driver with Linux Kernel 3.13.3 RHEL 6.3, Ceph 0.72.2 IOPs Testing results based on fio benchmark, 4k block, 20GB file,128 parallel jobs, RBD Kernel Driver with Linux Kernel 3.13.3 RHEL 6.3, Ceph 0.72.2 Cluster Network Admin Node 40GbE Public Network 10GbE/40GBE Ceph Nodes (Monitors, OSDs, MDS) Client Nodes 10GbE/40GbE
- 8. © 2014 Mellanox Technologies 8 CEPH and Hadoop Co-Exist Increase Hadoop Cluster Performance Scale Compute and Storage solutions in Efficient Ways Mitigate Single Point of Failure Events in Hadoop Architecture Name Node /Job Tracker Data Node Ceph NodeCeph Node Data Node Data Node Ceph Node Admin Node
- 9. © 2014 Mellanox Technologies 9 I/O Offload Frees Up CPU for Application Processing ~88% CPU Efficiency UserSpaceSystemSpace ~53% CPU Efficiency ~47% CPU Overhead/Idle ~12% CPU Overhead/Idle Without RDMA With RDMA and Offload UserSpaceSystemSpace
- 10. © 2014 Mellanox Technologies 10 Open source! • https://github.com/accelio/accelio/ && www.accelio.org Faster RDMA integration to application Asynchronous Maximize msg and CPU parallelism Enable > 10GB/s from single node Enable < 10usec latency under load In Next Generation Blueprint (Giant) • http://wiki.ceph.com/Planning/Blueprints/Giant/Accelio_RDMA_Messenger Accelio, High-Performance Reliable Messaging and RPC Library
- 11. © 2014 Mellanox Technologies 11 Summary CEPH cluster scalability and availability rely on high performance networks End to end 40/56 Gb/s transport with full CPU offloads available and being deployed • 100Gb/s around the corner Stay tuned for the afternoon session by CohortFS on RDMA for CEPH
- 12. Thank You