Into the cold - Object Storage in SWITCHengines
Download as PPTX, PDF0 likes161 views
This document provides an overview of SWITCH and its use of Ceph object storage. Some key points: 1) SWITCH offers integrated ICT solutions to the Swiss academic community, including an OpenStack-based IaaS platform that uses Ceph for block and object storage. 2) Several SWITCH services like video hosting and a national MOOC platform now store assets in Ceph object storage for performance and cost reasons. 3) SWITCH's Ceph infrastructure currently includes two regions with over 4PB of total storage capacity used primarily for block and object storage. 4) SWITCH aims to build a national long-term storage service using Ceph erasure coding to provide 100GB/
![© 2018 SWITCH | 12
More Object Storage Use Cases
• National MOOC service uses Swift to store assets
• Swiss Data Science Center stores research data in S3
• A leading research university looks at Object Storage for
–Institutional research data repository
–Encrypted off-site storage of critical business data for DR etc.
–Replacing various internal storage systems
–“[…] most cat pictures/videos/etc will land in objects”
etc = research data & everything else ](https://image.slidesharecdn.com/into-the-cold-181116135153/85/Into-the-cold-Object-Storage-in-SWITCHengines-12-320.jpg)
Into the cold - Object Storage in SWITCHengines
- 1. © 2018 SWITCH | 1 Simon Leinen Ceph Day Berlin, 12 November 2018 Startseite Untertitel Into The Cold Object Storage in SWITCHengines simon.leinen@switch.ch
- 2. © 2018 SWITCH | 2 Our core beliefs Together for greater capability, convenience and security in the digital world.
- 3. © 2018 SWITCH | 3 Mission SWITCH is an integral part of the Swiss academic community. Based on our core competencies • Network • Security • Identity Management SWITCH offers collaboratively developed ICT solutions that empower users in and beyond the academic world to achieve leading edge results in a globally competitive environment.
- 4. © 2018 SWITCH | 4 Added value for customers Community Work Foundation Integrated Offer
- 5. © 2018 SWITCH | 5 Foundation purpose "The foundation has as its objective to create, promote and offer the necessary basis for the effective use of modern methods of telecomputing in teaching and research in Switzerland, to be involved in and to support such methods. It is a non-profit foundation that does not pursue commercial targets." Excerpt from the deed of foundation Berne, 22 October 1987
- 6. © 2018 SWITCH | 6 Our customers Extended community • Other organizations involved in research or education SWITCH Community • Swiss universities on tertiary level (academic sector) and their research institutions Commercial customers • Registrars of .ch- and .li- Domain-Names, Swiss financial institutions, research-related industry and government
- 7. © 2018 SWITCH | 7 Integrated Offer Video Management Collaboration Procurement Infrastructure & Data Services Network Registry Trust & Identity Security
- 8. © 2018 SWITCH | 8 Let’s build a cloud! • 2012: Pilot project to build OpenStack-based IaaS platform • Ceph adopted as VM (volume) backing storage –Initially using files on CephFS, soon replaced by RBD • Swift and S3 support were also enabled –“Opportunistically”, i.e. not very deep thought process • All this entered production in 2014/2015 as paid services
- 9. © 2018 SWITCH | 9 Excursion: Is this Reasonable? • We expose most features of OpenStack, Ceph etc. to users • “spaghetti approach to product development”? • This creates some risks –Feature may not work that well –When users need help it may be difficult to support them –Getting rid of features for any reason can be painful • But it also creates opportunities!
- 10. © 2018 SWITCH | 10 Users Discover Object Storage: SWITCHtube
- 11. © 2018 SWITCH | 11 Users Discover Object Storage: SWITCHtube • The developers of our video hosting service decided to store all assets (videos and thumbnails) via S3 • Client browsers download directly from RadosGW • They are authorized via AWS v4 Signatures –Under Jewel, those didn’t work with Keystone users –Fixed in Luminous
- 12. © 2018 SWITCH | 12 More Object Storage Use Cases • National MOOC service uses Swift to store assets • Swiss Data Science Center stores research data in S3 • A leading research university looks at Object Storage for –Institutional research data repository –Encrypted off-site storage of critical business data for DR etc. –Replacing various internal storage systems –“[…] most cat pictures/videos/etc will land in objects” etc = research data & everything else
- 13. © 2018 SWITCH | 13 Oopses (1) • That day when a SWITCHtube clip was shown on all “digital signage” screens at a university… • …and the video client retrieved the entire video at the playout speed (instead of in fast bursts)
- 14. © 2018 SWITCH | 14 Oopses (2) • Jewel: no RadosGW’s Keystone integration for AWS v4 sigs • Workaround: Extract credentials from Keystone, reprovision locally in RadosGW • This creates a liability, so we did this very sparingly • In Luminous, this started to work, but…
- 15. © 2018 SWITCH | 15 Oopses (3) • AWS signature/token validation using Keystone is slow • It adds ~250ms to every authenticated S3 request • High S3 request rates tend to DoS our Keystone • Workaround: … and reprovision credentials in RadosGW • Requires fix for #23089 (included in 12.2.8) and rgw_s3_auth_order = local, external
- 16. © 2018 SWITCH | 16 Current Ceph Infrastructure Status • Two regions, LS (Lausanne) and ZH (Zurich) • LS: 241 OSDs, 900TB raw • ZH: 580 OSDs, 3PB raw • Both clusters >50% utilized: ca. 80% RBD/20% Object Store • Each cluster has 3 RadosGW (on bare-metal) behind LB • Typical server config: 1U,2*10GE, 1*Xeon-D/Xeon Scalable, 64–128GB RAM, 12*4/8 TB HDD, 2*200GB SSD with PLP
- 17. © 2018 SWITCH | 17 Long-Term Storage Challenge • Can we build a national service for long-term storage at a (full) cost significantly below CHF (≈USD) 100/TB/year? We believe this possible using Ceph with EC, once we can amortize fixed cost (salaries etc.) over xx PB
- 18. © 2018 SWITCH | 18 Customer Requirements/Expectations • Integrity assurance (checksums, versioning, …) • Bucket policies (e.g. write-only users) • On the horizon: encryption at rest • “Cross-region replication” to/from external S3 clusters such as an institution’s (“Cloud sync”)
- 19. © 2018 SWITCH | 19 Cost/Performance • Target: low cost, high durability, acceptable performance for low-IOPS write-mostly use (but want high throughput) • One possible approach (that we won’t be taking ): –Extend existing (“warm” RBD/RadosGW) Ceph cluster –Continue to use similar servers • 1RU, Xeon-D, ~150W with 12*8TB HDD under average load –Add more disks—PMR like before, but bigger (8 -> 12TB) –Use aggressive Erasure Coding (EC) profiles for low cost & high durability
- 20. © 2018 SWITCH | 20 Performance by EC profile (courtesy Michael Bisig)
- 21. © 2018 SWITCH | 21 With some parallelism (iodepth=8)
- 22. © 2018 SWITCH | 22 “Moonshot” challenge for Ceph (or other SDS) • Smartly mix hot & cold storage users on the same spindles • Basic idea: cold data on inside, hot data on outside –Bin win: Energy to keep disks spinning no longer wasted • Next frontier: Host-controlled traditional/SMR split –See e.g. “Disks for Data Centers” paper https://ai.google/research/pubs/pub44830
- 23. © 2018 SWITCH | 23 Working for a better digital world
Editor's Notes
- #24: Endslide “Corporate”
- #25: Finanzierung Auf der linken Seite der Abbildung sieht man die Finanzierungsquellen bzw. woher die Finanzmittel stammen mit denen wir die Kosten unserer Dienstleistungen und (Innovations-)Projekte auf der rechten Seite decken. Die gesamten Kosten für Dienstleistungen und Projekte zusammen beliefen sich 2016 auf CHF 19.5 Mio. 79% davon wurden für den Betrieb existierender Dienstleistungen eingesetzt, 21% davon für (Innovations-)Projekte - also für die Entwicklung zukünftiger Dienstleistungen (= rechte Seite der Abbildung = Mittelverwendung) Die Herkunft der Mittel auf der linken Seite setzt sich wie folgt zusammen: 56% aus Beiträgen der Hochschulen, 7% aus Beiträgen der hochschulnahen Organisationen, 15% der Mittel stammen aus dem Reingewinn mit kommerziellen Kunden (Registry & Banken-Cert), 12 % aus Finanz- und sonstigen Nebenerträgen (bspw. von unserer Tochtergesellschaft switchplus) und 9% aus Förderbeiträgen der CH und der EU. Allfällige Verluste werden durch Reserven gedeckt (1% der Kosten mussten im Jahr 2016 aus Reserven gedeckt werden). Die Pfeile zwischen Mittelherkunft und Mittelverwendung zeigen die Zielsetzung der Finanzierungsgrundsätze, welche der Stiftungsrat 2016 verabschiedet hatte: Ziel bis 2020 ist es, dass die Betriebskosten für Dienstleistungen vollumfänglich aus Beiträgen der Hochschulen (SWITCH Community) und hochschulnahen Organisationen gedeckt werden können. 2016 machen die Beiträge der Hochschulen inkl. hochschulnahen Organisationen 63% der Mittelherkunft, aber 79% der Mittelverwendung aus. Diesen Gap von aktuell knapp 3 Mio. Franken gilt es zu schliessen.
- #26: Organisation
- #27: Organisation
- #28: Organisation