Backup management with Ceph Storage - Camilo Echevarne, Félix Barbeira
1 like278 views
The document presents a comprehensive overview of backup management utilizing Ceph object storage, aimed at addressing challenges faced by Dinahosting, a web hosting and domain registration provider. It outlines the limitations of current backup systems, explores Ceph as a scalable, open-source solution, and details architecture, hardware planning, performance tuning, and monitoring strategies. Key aspects include the importance of resale options for storage, efficient data recovery processes, and future plans for service enhancement.
![RAID:
the end of an era
• Slow recovery.
• Hazardous recovery.
• Painful recovery.
• Disk incompatibility.
• Wasted disk for hot-spare.
• Expensive storage cards.
• Hard to scale.
• False sense of security.
[2]](https://image.slidesharecdn.com/backupmanagementwithcephstorage-dinahosting-180514152805/85/Backup-management-with-Ceph-Storage-Camilo-Echevarne-Felix-Barbeira-13-320.jpg)
![[3]
Upload backup
to the cloud](https://image.slidesharecdn.com/backupmanagementwithcephstorage-dinahosting-180514152805/85/Backup-management-with-Ceph-Storage-Camilo-Echevarne-Felix-Barbeira-20-320.jpg)
![Tuning
• Enable Jumbo Frames.
ping6 -M do -s 8972 <ip>
• Monitor options:
[mon]
mon osd nearfull ratio = .90
mon osd down out subtree limit = host
• OSD options:
[osd]
osd scrub sleep = .1
osd scrub load threshold = 1.0
osd scrub begin hour = 12
osd scrub end hour = 0
Overhead
Data
Standard Frames (1522 MTU)
Jumbo Frame (9000 MTU)
Overhead
Data
• Daily reweight:
ceph osd reweight-by-utilization [threshold]](https://image.slidesharecdn.com/backupmanagementwithcephstorage-dinahosting-180514152805/85/Backup-management-with-Ceph-Storage-Camilo-Echevarne-Felix-Barbeira-38-320.jpg)
![{
"user_id": "fbarbeira",
"display_name": "Felix Barbeira",
"email": "fbarbeira@dinahosting.com",
"suspended": 0,
"max_buckets": 100,
"auid": 0,
"subusers": [
{
"id": "fbarbeira:read-only",
"permissions": "read"
}
],
"keys": [
{
"user": “fbarbeira:read-only",
"access_key": "XXXXXXXXXXXXXXXXXXXX",
"secret_key": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
},
{
"user": "fbarbeira",
"access_key": "XXXXXXXXXXXXXXXXXXXX",
"secret_key": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
],
[...]
},
"user_quota": {
"enabled": true,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 1073741824,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw"
}
Limit max size object 1TB (default no-limit)
Ceph user
profile
Limit bucket max number (default 1000)
Subuser with read-only permission](https://image.slidesharecdn.com/backupmanagementwithcephstorage-dinahosting-180514152805/85/Backup-management-with-Ceph-Storage-Camilo-Echevarne-Felix-Barbeira-48-320.jpg)
![References:
[1] http://www.vacalouraestudio.es/
[2] https://www.krollontrack.co.uk/blog/survival-stories/24tb-of-confidential-data-recovered-from-raid-6-array/
[3] https://www.elempresario.com/noticias/economia/2017/09/27/
el_numero_billetes_500_euros_continua_minimos_desde_2003_54342_1098.html
¡Gracias!](https://image.slidesharecdn.com/backupmanagementwithcephstorage-dinahosting-180514152805/85/Backup-management-with-Ceph-Storage-Camilo-Echevarne-Felix-Barbeira-80-320.jpg)
Backup management with Ceph Storage - Camilo Echevarne, Félix Barbeira
- 1. Backup Management with Ceph Object Storage
- 2. ¿Who are we? Camilo Echevarne Félix Barbeira cechevarne@dinahosting.com fbarbeira@dinahosting.com Linux Sysadmin Head Linux Sysadmin
- 3. • Presentation. • The problem. • Alternatives. • Ceph to the rescue. • Hardware planification. • Architecture. • Tuning. • Backup management. • Clients. • Monitoring. • Upgrades. • Future plans. Agenda
- 4. • Presentation. • The problem. • Alternatives. • Ceph to the rescue. • Hardware planification. • Architecture. • Tuning. • Backup management. • Clients. • Monitoring. • Upgrades. • Future plans. Agenda
- 5. ¿What is Dinahosting? Our main business is web hosting and domain registration. We offer the user all tools needed to develop their project on Internet with guarantees: - Domain name for your site. - E-mail services. - Hosting plans: from the simplest ones to complex and powerful solutions like Cloud Hosting, as well as VPS and Dedicated Servers.
- 6. ¿Where are we? Presence on more than 130 international markets México, Argentina, Colombia, Chile, Portugal, Peru, Venezuela, USA, Brazil, Ecuador, France, United Kingdom, Italy, Denmark, Netherlands, Uruguay, Bolivia, Japan, China, Senegal, etc. Santiago
- 8. Revenue 0 3.500.000 7.000.000 10.500.000 14.000.000 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 Revenue
- 9. - Toll-free phone number. - Chat. - E-mail. - Social network presence. - 24/7 service. - No call-center auto-attendant. Customer service
- 10. • Only for clients of managed services. • Restorations at file level. • 30 days max retention. • Weekly complete backup, incremental rest of the week. • ~3000 machines. • ~1PB available space. • ~30 bare metal storage servers. • Complete backup size ~125TB. Backups Data size increases year by year an so their management complexity
- 11. Agenda • Presentation. • The problem. • Alternatives. • Ceph to the rescue. • Hardware planification. • Architecture. • Tuning. • Backup management. • Clients. • Monitoring. • Upgrades. • Future plans. Agenda
- 12. Current system NFS servers RAID storage
- 13. RAID: the end of an era • Slow recovery. • Hazardous recovery. • Painful recovery. • Disk incompatibility. • Wasted disk for hot-spare. • Expensive storage cards. • Hard to scale. • False sense of security. [2]
- 14. ¿Would we be protected against … ? hardware error network outage datacenter disaster power supply operating system error filesystem failure RAID: the end of an era
- 15. Problems managing files - Backwards compatibility. - Wasted space. - ¿Storage node partition table? - Corrupt files when disk is full. - Many hours spent by SysOps. - Forced to deploy and maintain an API.
- 16. IT support The bosses Problems managing files
- 17. Agenda • Presentation. • The problem. • Alternatives. • Ceph to the rescue. • Hardware planification. • Architecture. • Tuning. • Backup management. • Clients. • Monitoring. • Upgrades. • Future plans. Agenda
- 18. Upload backup to the cloud ¿When do we start to loose money?
- 19. Price per month 1PB cloud storage AWS Cost Blocking elements S3 Infrequent Access (IA) ~15.000 € Price S3 Glacier ~5.000 €* - Slow data retrieval** - Limited availability*** - 500TB upload limit**** * Files deleted before 90 days incur a pro-rated charge. ** Expedited retrievals: 1-5 min. 3 expedited retrievals can be performed every 5 minutes. Each unit of provisioned capacity costs 100$ per month. Retrieval Standard: 3-5 hours. *** Glacier inventory refresh every 24h. **** Increasing the upload limit is available contacting AWS support. AZURE Cost Blocking elements Storage Standard Cool ~9.000 € Price Storage Standard Archive ~5.000 € Restauraciones <15h Azure charges extra cost if files are deleted before 30 and 180 days respectively. GCP Cost Blocking elements Nearline Storage ~8.000 € Price Coldline Storage ~5.000 € Price In both types of storage, data access is measured in milliseconds. Upload backup to the cloud
- 20. [3] Upload backup to the cloud
- 21. • Presentation. • The problem. • Alternatives. • Ceph to the rescue. • Hardware planification. • Architecture. • Tuning. • Backup management. • Clients. • Monitoring. • Upgrades. • Future plans. Agenda
- 22. Unified, distributed storage system. Intelligence on software. - Open source. - Massively scalable. - Independent components. - No SPOF. - High performance. - S3 API. - Active community. - Use of commodity hardware.
- 23. Clients Object Storage Ceph Storage Cluster Block Storage File Storage Ceph Linux OS CPU Memory HDDs Network Ceph Ceph Linux OS Linux OS CPU Memory HDDs Network CPU Memory HDDs Network Linux OS CPU Memory HDDs Network Ceph … … … Ceph SDS Linux OS Hardware Distributed storage Server1 Server2 Server3 ServerN
- 24. OSD (object storage daemon) Monitors - From one to thousands. - Generally speaking, 1 OSD = 1 hard disk. - Communicate between them to replicate data and make recoveries. - Maintain cluster maps. - Provide consensus on the decisions of data distribution. - Small and odd number. - Do not store data. Gateways - Entry points to cluster.
- 25. OSD NODE Disk OSD OSD OSD DiskDiskDisk OSD Disk OSD OSD OSD DiskDiskDisk OSD Disk OSD OSD OSD DiskDiskDisk Server1 Server2 Server3 NODE RADOS sends the read request to the primary OSD. Primary OSD reads data from local disk and notifies Ceph client. WRITEREAD 1 2 1 2 Client writes data, RADOS creates object and sends data to the primary OSD. 1 4 3 2 Primary OSD finds the number of replicas and sends data to replica OSDs. Replica OSDs write data and send completion to primary OSD. Primary OSD signals write completion to Ceph client. 1 2 2 3 4 3 Data flow on OSDs
- 26. • Presentation. • The problem. • Alternatives. • Ceph to the rescue. • Hardware planification. • Architecture. • Tuning. • Backup management. • Clients. • Monitoring. • Upgrades. • Future plans. Agenda
- 27. Ceph OSDs • DELL R720XD / R730XD • CPU: 2xE5-2660 8 cores 2,20ghz • RAM: 64GB-96GB • Disks: • 12x8TB SATA • 1 SATA disk for OS • NIC: 10G • Controller: H730 / LSI JBOD Hardware planification Ceph monitors • VPS. • 4 vcores • RAM: 4GB • NIC: 10G Ceph gateways • DELL R230 • CPU: E3-1230v5 3.4GHz (8) • RAM: 8GB • NIC: 10G ¿Optimize for cost or performance? In our case, the principal objective is to optimize total cost per GB.
- 28. ¿What happen to the OSDs if the OS disk dies? “We recommend using a dedicated drive for the operating system and software, and one drive for each OSD Daemon you run on the host” So…¿where do I put the operating system of the OSD node? PROS CONS OS in RAID1 - Cluster protected against OS failures. - Hot-swap disks. - We do not have a RAID card*. - We would need 1 extra disk. OS single disk - Only 1 disk slot used. - High reliability. Monitor disk with SMART. If disk dies, all OSDs of that machine die too. OS in SATADOM All disk slots available for OSDs. They are not reliable after months of use. OS from SAN - All disk slots available for OSDs. - RAID protected. We depend on the network and remote storage. OS in SD All disk slots available for OSDs. Poor performance, not reliable. *PERC H730 supports RAID. Hardware planification
- 29. Rules of thumb for a Ceph installation. - 10G networking as a minimum. - Deeply knowledge of hardware you wish to use. - Always use at least 3 replicas - Try to use enterprise SSDs. - Don’t use configuration options you don’t understand. - Power loss testing. - Have a recovery plan. - Use a CM (configuration management) system. Hardware planification
- 30. https://github.com/ceph/ceph-ansible.git • Gradual learning curve. • Plain deploy, no lifecycle management. • No orchestration. • No server needed. • Evolution of ceph-deploy tool. http://docs.ceph.com/ceph-ansible/ TIPS: - Use ansible compatible version (no bleeding edge version supported). - Do not use master branch unless you like strong emotions.
- 31. • Presentation. • The problem. • Alternatives. • Ceph to the rescue. • Hardware planification. • Architecture. • Tuning. • Backup management. • Clients. • Monitoring. • Upgrades. • Future plans. Agenda
- 32. Cliente Public network Monitor01 Monitor02 Monitor03 RadosGW01 HTTP (S3) …RadosGW02 Ceph Architecture OSD OSD OSD OSD OSDn 10G Odd number (3) … RadosGWn 10G10G 10G 10G IPv6 IPv4 & IPv6 10G 10G
- 33. RadosGW01 …RadosGW02 RadosGWn Client HA Gateway Option 1: LB active/passive mode. LB-ON LB-OFF Inconveniences: - Bandwidth bottleneck at LB. - Mount 2 LB at least. - Increases TCO. - Increases complexity. Ceph Architecture
- 34. RadosGW01 …RadosGW02 RadosGWn Client HA Gateway Option 2: DNS Round Robin. Inconveniences: - DNS responses are stateless. - TTL dependency. - No instant fail-over. Ceph Architecture
- 35. HA Gateway Option 3 (selected): local anycast of gateway ip. Advantages: - Bandwidth of all nodes is added. - Instant fail-over. - Route is deleted if it fails node, RadosGW daemon or FrRouting daemon. OSPF Route RadosGW01 ©√FRROUTING IP RADOS-GATEWAY health-check RadosGW0n ©√FRROUTING IP RADOS-GATEWAY health-check… OSPF Route CPD RadosGW01 ©√FRROUTING IP RADOS-GATEWAY health-check Ceph Architecture
- 36. • Presentation. • The problem. • Alternatives. • Ceph to the rescue. • Hardware planification. • Architecture. • Tuning. • Backup management. • Clients. • Monitoring. • Upgrades. • Future plans. Agenda
- 37. Tuning No silver bullets. root@ceph:~# ceph --show-config | wc -l 1397 root@ceph:~# Default options are designed for general use cases. Most of the times you need to make some adjustments in order to achieve real performance. Ceph documentation is highly valuable and extensive: http://docs.ceph.com/docs/master/
- 38. Tuning • Enable Jumbo Frames. ping6 -M do -s 8972 <ip> • Monitor options: [mon] mon osd nearfull ratio = .90 mon osd down out subtree limit = host • OSD options: [osd] osd scrub sleep = .1 osd scrub load threshold = 1.0 osd scrub begin hour = 12 osd scrub end hour = 0 Overhead Data Standard Frames (1522 MTU) Jumbo Frame (9000 MTU) Overhead Data • Daily reweight: ceph osd reweight-by-utilization [threshold]
- 39. Erasure Code Replicated pool vs. Erasure code pool OBJECT COPY COPYCOPY Replicated pool CEPH STORAGE CLUSTER OBJECT Erasure coded pool CEPH STORAGE CLUSTER 1 2 3 X Y Full copies of stored objects • High durability. • 3x (200% overhead) • Quicker recovery • Admit all kind of operations. • Use less resources (CPU). One copy plus parity • Cost-effective durability. • 1.5x (50% overhead) • Expensive recovery. • Partial writes not supported*. • Higher CPU usage.
- 40. CEPH CLIENT Erasure coded pool CEPH STORAGE CLUSTER 1 OSD 2 OSD 3 OSD 4 OSD X OSD Y OSD READ Reads Erasure Code ¿HOW DOES ERASURE CODE WORKS?
- 41. Erasure coded pool CEPH STORAGE CLUSTER 1 OSD 2 OSD 3 OSD 4 OSD X OSD Y OSD READ READS Erasure Code CEPH CLIENT Reads ¿HOW DOES ERASURE CODE WORKS?
- 42. Erasure coded pool CEPH STORAGE CLUSTER 1 OSD 2 OSD 3 OSD 4 OSD X OSD Y OSD READ REPLY READS Erasure Code CEPH CLIENT Reads ¿HOW DOES ERASURE CODE WORKS?
- 43. CEPH STORAGE CLUSTER 1 OSD 2 OSD 3 OSD 4 OSD X OSD Y OSD WRITE Erasure coded pool Erasure Code CEPH CLIENT Writes ¿HOW DOES ERASURE CODE WORKS?
- 44. Erasure coded pool CEPH STORAGE CLUSTER 1 OSD 2 OSD 3 OSD 4 OSD X OSD Y OSD WRITE WRITES Erasure Code CEPH CLIENT Writes ¿HOW DOES ERASURE CODE WORKS?
- 45. CEPH STORAGE CLUSTER 1 OSD 2 OSD 3 OSD 4 OSD X OSD Y OSD WRITE ACK Writes Erasure coded pool Erasure Code CEPH CLIENT ¿HOW DOES ERASURE CODE WORKS?
- 46. Two variables: K + M K = data shards M = erasure code shards Usable space OSDs number allowed to fail 3+1 75 % 1 4+2 66 % 2 18+2 90 % 2 11+3 78.5 % 3 n=k+m n= total shards r=k/n r=encoding rate n=4+2=6 r=4/6=0.66 CERN Erasure Code ¿HOW DOES ERASURE CODE WORKS?
- 47. • Presentation. • The problem. • Alternatives. • Ceph to the rescue. • Hardware planification. • Architecture. • Tuning. • Backup management. • Clients. • Monitoring. • Upgrades. • Future plans. Agenda
- 48. { "user_id": "fbarbeira", "display_name": "Felix Barbeira", "email": "fbarbeira@dinahosting.com", "suspended": 0, "max_buckets": 100, "auid": 0, "subusers": [ { "id": "fbarbeira:read-only", "permissions": "read" } ], "keys": [ { "user": “fbarbeira:read-only", "access_key": "XXXXXXXXXXXXXXXXXXXX", "secret_key": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" }, { "user": "fbarbeira", "access_key": "XXXXXXXXXXXXXXXXXXXX", "secret_key": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" } ], [...] }, "user_quota": { "enabled": true, "check_on_raw": false, "max_size": -1, "max_size_kb": 1073741824, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw" } Limit max size object 1TB (default no-limit) Ceph user profile Limit bucket max number (default 1000) Subuser with read-only permission
- 49. ¿How do we sync backups? message broker AGENTBACKUP_DONE Consume n elements SERVER publish publish SERVER publish SERVER … User-RW WRITE
- 50. ¿How do we restore backups? PANEL AGENT Generate temporary links Method 1: Restorations ordered by control panel. SERVER GET User-RO Method 2: Restorations from the same machine.
- 51. • Presentation. • The problem. • Alternatives. • Ceph to the rescue. • Hardware planification. • Architecture. • Tuning. • Backup management. • Clients. • Monitoring. • Upgrades. • Future plans. Agenda
- 52. s3cmd minio Ceph client requirements • No dependencies. • Multi-OS compatible. • Low resources requirements. • Active development. • Bandwidth limit. We need to limit used bandwidth.
- 53. CPU and IO problems CPU NET 1G Limit bandwidth Powerful machine CPU NET 1G No limits Not-so-powerful machine CPU NET 1G No limits Powerful machine Elastic limit CPU NET 1G
- 54. Linux Traffic Control (TC) Flow1 Flow2 Flow3 Flow4 Port FIFO queue Ceph traffic Default behaviour Flow1 Flow2 Flow3 Flow4 Classifier Ceph traffic Port Ceph traffic FIFO queue Hierarchical Token Bucket (HTB) Applying tc policy CPU and IO adjustments
- 55. Linux Traffic Control (TC) Regulate outgoing traffic using system load. CPU NET Network limit Allowed CPU load range Reduce/increase transfer rate CPU and IO adjustments
- 56. • Presentation. • The problem. • Alternatives. • Ceph to the rescue. • Hardware planification. • Architecture. • Tuning. • Backup management. • Clients. • Monitoring. • Upgrades. • Future plans. Agenda
- 57. Prometheus Grafana Storage scrape metrics Alertmanager generate alerts E-mail XMPP NODE_EXPORTER MGR_EXPORTER NODE_EXPORTER MGR_EXPORTER NODE_EXPORTER MGR_EXPORTER NODE_EXPORTER NODE_EXPORTER Monitors OSDs and Gateways Monitoring
- 58. Monitoring user@prometheus:~$ curl --silent http://ceph-monitor:9283/metrics | head -20 # HELP ceph_osd_op_out_bytes Client operations total read size # TYPE ceph_osd_op_out_bytes counter ceph_osd_op_out_bytes{ceph_daemon="osd.6"} 192202.0 ceph_osd_op_out_bytes{ceph_daemon="osd.26"} 355345.0 ceph_osd_op_out_bytes{ceph_daemon="osd.30"} 99943.0 ceph_osd_op_out_bytes{ceph_daemon="osd.8"} 9687.0 ceph_osd_op_out_bytes{ceph_daemon="osd.20"} 6480.0 ceph_osd_op_out_bytes{ceph_daemon="osd.36"} 73682.0 ceph_osd_op_out_bytes{ceph_daemon="osd.22"} 497679.0 ceph_osd_op_out_bytes{ceph_daemon="osd.47"} 123536.0 ceph_osd_op_out_bytes{ceph_daemon="osd.34"} 95692.0 ceph_osd_op_out_bytes{ceph_daemon="osd.45"} 114504.0 ceph_osd_op_out_bytes{ceph_daemon="osd.10"} 8695.0 ceph_osd_op_out_bytes{ceph_daemon="osd.39"} 0.0 ceph_osd_op_out_bytes{ceph_daemon="osd.43"} 107303.0 ceph_osd_op_out_bytes{ceph_daemon="osd.12"} 199043.0 ceph_osd_op_out_bytes{ceph_daemon="osd.28"} 1165455.0 ceph_osd_op_out_bytes{ceph_daemon="osd.41"} 216581.0 ceph_osd_op_out_bytes{ceph_daemon="osd.14"} 124186.0 user@prometheus:~$ Prometheus exporter example:
- 62. • Presentation. • The problem. • Alternatives. • Ceph to the rescue. • Hardware planification. • Architecture. • Tuning. • Backup management. • Clients. • Monitoring. • Upgrades. • Future plans. Agenda
- 63. Upgrades overview Unattended upgrades • Mirror on-premises. • Upgrade policy: • Security: ASAP. • Updates: every Tuesday. • Maintenance window. • Package blacklist: ceph and ceph-* • Index results on Elasticsearch. Ceph upgrades sequence: - Monitors. - OSDs. - Gateways.
- 77. • Presentation. • The problem. • Alternatives. • Ceph to the rescue. • Hardware planification. • Architecture. • Tuning. • Backup management. • Clients. • Monitoring. • Upgrades. • Future plans. Agenda
- 78. Future plans • Metadata Search (Elasticsearch). • Search objects using tags. • Top 10 backup size. • Average size. • Crush Maps. • Use current datacenter configuration in Ceph: rack, row, room… • Increase availability. • EC adjustment. • Indexless buckets. • Incompatible with lifecycles.
- 79. ¿Questions?
- 80. References: [1] http://www.vacalouraestudio.es/ [2] https://www.krollontrack.co.uk/blog/survival-stories/24tb-of-confidential-data-recovered-from-raid-6-array/ [3] https://www.elempresario.com/noticias/economia/2017/09/27/ el_numero_billetes_500_euros_continua_minimos_desde_2003_54342_1098.html ¡Gracias!