Scaling Ceph at CERN - Ceph Day Frankfurt
35 likes14,881 views
This document summarizes Dan van der Ster's experience scaling Ceph at CERN. CERN uses Ceph as the backend storage for OpenStack volumes and images, with plans to also use it for physics data archival and analysis. The 3PB Ceph cluster consists of 47 disk servers and 1,128 OSDs. Some lessons learned include managing latency, handling many objects, tuning CRUSH, trusting clients, and avoiding human errors when managing such a large cluster.
Scaling Ceph at CERN - Ceph Day Frankfurt
- 2. Scaling Ceph at CERN Dan van der Ster (daniel.vanderster@cern.ch) Data and Storage Service Group | CERN IT Department
- 3. CERN’s Mission and Tools ● CERN studies the fundamental laws of nature ○ Why do particles have mass? ○ What is our universe made of? ○ Why is there no antimatter left? ○ What was matter like right after the “Big Bang”? ○ … ● The Large Hadron Collider (LHC) ○ Built in a 27km long tunnel, ~200m underground ○ Dipole magnets operated at -271°C (1.9K) ○ Particles do ~11’000 turns/sec, 600 million collisions/sec ○ … ● Detectors ○ Four main experiments, each the size of a cathedral ○ DAQ systems Processing PetaBytes/sec Scaling Ceph at CERN - D. van der Ster 3
- 4. Big Data at CERN Physics Data on CASTOR/EOS ● LHC experiments produce ~10GB/s 25PB/year User Data on OpenAFS & DFS ● Home directories for 30k users ● Physics analysis development ● Project spaces for applications Service Data on AFS/NFS ● Databases, admin applications Tape archival with CASTOR/TSM ● RAW physics outputs ● Desktop/Server backups Scaling Ceph at CERN - D. van der Ster 4 Service Size Files OpenAFS 290TB 2.3B CASTOR 89.0PB 325M EOS 20.1PB 160M
- 5. IT Evolution at CERN Scaling Ceph at CERN - D. van der Ster 5 Cloudifying CERN’s IT infrastructure ... ● Centrally-managed and uniform hardware ○ No more service-specific storage boxes ● OpenStack VMs for most services ○ Building for 100k nodes (mostly for batch processing) ● Attractive desktop storage services ○ Huge demand for a local Dropbox, Google Drive … ● Remote data centre in Budapest ○ More rack space and power, plus disaster recovery … brings new storage requirements ● Block storage for OpenStack VMs ○ Images and volumes ● Backend storage for existing and new services ○ AFS, NFS, OwnCloud, Data Preservation, ... ● Regional storage ○ Use of our new data centre in Hungary ● Failure tolerance, data checksumming, easy to operate, security, ...
- 6. Ceph at CERN Scaling Ceph at CERN - D. van der Ster 6
- 7. 12 racks of disk server quads Wiebalck / van der Ster -- Building an organic block storage service at CERN with Ceph
- 8. Our 3PB Ceph Cluster Dual Intel Xeon L5640 24 threads incl. HT Dual 1Gig-E NICs Only one connected 2x 2TB Hitachi system disks RAID-1 mirror 1x 240GB OCZ Deneva 2 /var/lib/ceph/mon 48GB RAM Scaling Ceph at CERN - D. van der Ster 8 Dual Intel Xeon E5-2650 32 threads incl. HT Dual 10Gig-E NICs Only one connected 24x 3TB Hitachi disks Eco drive, ~5900 RPM 3x 2TB Hitachi system disks Triple mirror 64GB RAM 47 disk servers/1128 OSDs 5 monitors # df -‐h /mnt/ceph Filesystem Size Used Avail Use% Mounted on xxx:6789:/ 3.1P 173T 2.9P 6% /mnt/ceph
- 9. Use-Cases Being Evaluated 1. Images and Volumes for OpenStack 2. S3 Storage for Data Preservation / Public Dissemination 3. Physics data storage for archival and/or analysis Scaling Ceph at CERN - D. van der Ster 9 #1 is moving into production. #2 and #3 are more exploratory at the moment.
- 10. OpenStack Volumes & Images • Glance: using RBD for ~3 months now. • Only issue was to increase ulimit -n above 1024 (10k is good). • Cinder: testing with close colleagues. • 126 Cinder Volumes attached today – 56TB used Scaling Ceph at CERN - D. van der Ster 10 Growing # of volumes/images Usual traffic is ~50-100MB/s with current usage. (~idle)
- 11. RBD for OpenStack Volumes • Before general availability, we need to test and enable qemu iops/bps throttling • Otherwise VMs with many IOs can disrupt other users. • One ongoing issue is that a few clients are getting an (infrequent) segfault of qemu during a VM reboot. • Happens on VMs with many attached RBD’s. • Difficult to get a complete (16GB) core dump. Scaling Ceph at CERN - D. van der Ster 11
- 12. CASTOR & XRootD/EOS • Exploring RADOS backend for these two HEP-developed file systems • Gateway model, similar to S3 via RADOSGW • CASTOR needs raw throughput performance (to feed many tape drives at 250MBps each). • Striped RWs across many OSDs are important. • XRootD/EOS may benefit from the highly scalable namespace to store O(billion) objects • Bonus: XRootD also offers http/webdav with X509/kerberos, possibly even fuse mountable. • Developments are in early stages. Scaling Ceph at CERN - D. van der Ster 12
- 13. Operations & Lessons Learned Scaling Ceph at CERN - D. van der Ster 13
- 14. Configuration and Deployment • Dumpling 0.67.7 • Fully Puppet-ized • Automated server deployment, automated OSD replacement • Very few custom ceph.conf options à • Experimenting with the filestore wbthrottle • we find that disabling it completely gives better IOps performance • But don’t do this!!! Scaling Ceph at CERN - D. van der Ster 14 mon osd down out interval = 900 osd pool default size = 3 osd pool default min size = 1 osd pool default pg num = 1024 osd pool default pgp num = 1024 osd pool default flag hashpspool = true osd max backfills = 1 osd recovery max active = 1
- 15. Cluster Activity Scaling Ceph at CERN - D. van der Ster 15
- 16. General Comments… • In these ~7 months of running the cluster, there have been very few problems • No outages • No data losses/corruptions • No unfixable performance issues • Behaves well during stress tests • But now we’re starting to get real/varied/creative users, and this brings up many interesting issues... • “No amount of stress testing can prepare you for real users” - Unknown • (point being, don’t take the next slides to be too negative – I’m just trying to give helpful advice ;) Scaling Ceph at CERN - D. van der Ster 16
- 17. Latency & Slow Requests • Best latency we can achieve is 20-40ms • Slow SATA disks, no SSDs: hard to justify SSDs in a multi-PB cluster, but could in a smaller limited use-case cluster (e.g. for Cinder-only) • Latency can increase dramatically with heavy usage • Don’t mix latency-bound and throughput-bound users on the same OSDs • Local processes scanning the disks can hurt performance • Add /var/lib/ceph to the updatedb PRUNEPATH • If you have slow disks like us, you need to understand your disk IO scheduler – e.g. deadline prefers reads over writes: writes are given a 5 second deadline vs. 500ms for reads! • Scrubbing! • Kernel tuning: vm.* sysctl, dirty page flushing, memory reclaiming… • “Something is flushing the buffers, blocking the OSD processes” • Slow requests: monitor them, eliminate them. Scaling Ceph at CERN - D. van der Ster 17
- 18. Life with 250 million objects • Recently, a user decided to write 250 million 1kB objects • Not so unreasonable: 250M * 4MB = 1PB, so this simulates the cluster being full of RBD images, at least in terms of # objects • It worked – no big problems from holding this many objects. • Tested single OSD failure: ~7 hours to backfill, including a double-backfill glitch that we’re trying to understand. • But now we want to cleanup, and it is not trivial to remove 250M objects! • rados rmpool generated quite a load when we rm’d a 3 million object pool (some OSDs were temporarily marked down). • Probably due to a mistake in our wbthrottle tuning Scaling Ceph at CERN - D. van der Ster 18
- 19. Other backfilling issues • During a backfilling event (draining a whole server), we started observing repeated monitor elections • Caused by the mons’ LevelDBs being so active that the local SATA disks couldn’t keep up. • When a mon falls behind, it calls an election • Could be due to LevelDB compaction… • We moved /var/lib/ceph/mon to SSDs – no more elections during backfilling • Avoid double backfilling when taking an OSD out of service: • Start with ceph osd crush rm <osd id> !! • If you mark the OSD out first, then crush rm it, you will compute a new CRUSH map twice, i.e. backfill twice. Scaling Ceph at CERN - D. van der Ster 19
- 20. Fun with CRUSH • CRUSH is simple yet powerful, so it is tempting to play with the cluster layout • But once you have non-zero amounts of data, significant CRUSH changes will lead to massive data movements, which create extra disk load and may disrupt users. • Early CRUSH planning is crucial! • A network switch is a failure domain, so we should configure CRUSH to replicate across switches, right? • But (assuming we don’t have a private cluster network) that would send all replication traffic via the switch uplinks – bottleneck! • Unclear tradeoff between uptime and performance. Scaling Ceph at CERN - D. van der Ster 20
- 21. CRUSH & Data distribution • CRUSH may give your cluster an uneven data distribution • An OSD’s used space will scale with the number of PGs assigned to it • After you have designed your cluster, created your pools, started adding data, check the PG and volume distributions • reweight-‐by-‐utilization is useful to iron out an uneven PG distribution • The hashpspool flag is also important if you have many active pools Scaling Ceph at CERN - D. van der Ster 21 0 20 40 60 80 100 120 140 160 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30nOSDs n PGs Number of OSDs having N PGs (for pool = volumes)
- 22. RBD Reliability with 3 Replicas • RBD devices are chunked across thousands of objects: • A full 1TB volume is composed of 250,000 4MB objects • If any single object is lost, the whole RBD can be considered to be corrupted (obviously, it depends which blocks are lost!) • If you lose an entire PG, you can consider all RBDs to be lost / corrupted. • Our incorrect & irrational fears: • Any simultaneous triple disk failure in the cluster would lead to objects being lost – and somehow all RBDs would be corrupted. • As we add OSDs to the cluster, the data gets spread wider, and the chances of RBD data loss increase. • But this is wrong!! • The only triple disk failures that can lead to data loss are those combinations actively used by PGs – so having e.g. 4096 PGs for RBDs means that only 4096 combinations out of the 10^9 possible combinations matter. • N_PGs * ~(P_diskfailure^3) / 3! • We use 4 replicas for the RBD volumes, but this is probably overkill. Scaling Ceph at CERN - D. van der Ster 22
- 23. Trust your clients • There is no server-side per-client throttling • A few nasty clients can overwhelm an OSD, leading to slow requests for everyone. • When you have a high load / slow requests, it is not always trivial to identify and blacklist/firewall the misbehaving client • Could use some help in the monitoring: per-client perf stats? • One of our creative users found a way to make the mon’s generate 5*40 MBps of outbound network traffic • Could saturate the mon network, lead to disruptions • RADOS is not for end-users. A cephx keyring is for trusted persons only, not for Joe Random User. Scaling Ceph at CERN - D. van der Ster 23
- 24. Fat fingers • A healthy cluster is always vulnerable to human errors • We’ve thus far avoided any big mistakes • Used PG splitting to grow a pool from 8 to 2048 PGs • Leads to unresponsive OSDs who get marked down à degraded objs. • Safer & now-enforced to grow in 2x or 4x steps • ulimits, ulimits, ulimits • With a large number of OSDs (say, more than 500), you will hit num file and num process limits everywhere: • Glance, qemu, radosgw, ceph/rados CLI, … • If you use XFS, don’t put your OSD journal as a file on the disk • Use a separate partition, the first partition! • We still need to reinstall our whole cluster to re-partition the OSDs Scaling Ceph at CERN - D. van der Ster 24
- 25. Scale up and out • Scale up: we are demonstrating the viability of a 3PB cluster with O(1000) OSDs. • What about 10,000 or 100,000 OSDs? • What about 10,000 or 100,000 clients? • Many Ceph instances is always an option, but not ideal • Scale out: our growing data centre in Budapest brings many options: • Replicate over the WAN (though, 30ms RTT) • Tiering / Caching pools (new feature, need to get experience…) • Data locality – direct IOs to nearby replica or caching pool Scaling Ceph at CERN - D. van der Ster 25
- 26. Summary Scaling Ceph at CERN - D. van der Ster 26
- 27. Summary • CERN IT infrastructure is undergoing a private cloud revolution, and Ceph is providing the underlying storage. • Our CASTOR and XRootD physics data use- cases may exploit RADOS for improved performance/scalability. • In seven months with a 3PB cluster, we’ve not had any disasters. Actually it’s working quite well. • Presented some lessons learned, I hope they prove useful in your Ceph explorations. Scaling Ceph at CERN - D. van der Ster 27