BeeGFS Enterprise Deployment
Download as PPTX, PDF3 likes561 views
The document summarizes Fred Hutch's transition to using BeeGFS for their file storage needs over recent years: 1) They initially deployed a scale-out NAS in 2012 to consolidate storage for hundreds of terabytes of data across multiple research groups. 2) In 2014, they deployed BeeGFS for scratch storage for up to 500TB of data across 10 research groups with 100% uptime for 3 years. 3) By 2019, they migrated all 150 research groups and over 700 million files totaling 2 petabytes to a redundant BeeGFS deployment for higher reliability, with backup to the cloud.
BeeGFS Enterprise Deployment
- 1. Posix file systems at Fred Hutch 2012: Scale Out NAS - 100s of TB, - Scaleout NAS replacing single filers - Consolidated HPC defines life sciences performance requirements 2014: BeeGFS Scratch - Up to 500TB Scratch - No redundancy - 100% uptime for 3 years - 10 research groups - 2 month project 2019: BeeGFS for all - Migrating to redundant deployment with HA - 150 research groups - 700M files, 2 PiB - Higher Expectations - Backup to cloud - 6 month project © Fred Hutchinson Cancer Research Center 0
- 2. BeeGFS Enterprise deployment – what does this mean? © Fred Hutchinson Cancer Research Center 1
- 3. “Fast File” service architecture - 3 racks each in 2 different DC pods - buddy mirrors of Meta and Storage - Clustered SMB/NFS NAS Gateway © Fred Hutchinson Cancer Research Center 2 - scale to 8 PiB usable capacity - Management server on Vmware - Flash Storage Pool
- 4. Standard Hardware - Supermicro standard workhorse - 34 x 14 TB drives = 476 TB RAW - WD HC530 and Seagate Exos X14 (mirror) - Micron SATA SSD for ZFS caching © Fred Hutchinson Cancer Research Center 3 - Intel Optane low latency NVMe for metadata - Metadata: cores, more cores - Cisco Nexus 9300 – 100G
- 5. A word about ZFS (ZoL) …..just use it! - 3 RAID groups with 11 drives each, one spare - RAIDZ3 most efficient, best resilver times - Resilver times 50% improved in v.0.8 - Large 14TB drives not a problem - Easy to use encryption in v.0.8 - Faster with LZ4 compression than without - 3.5 GB/s disk throughput - Comes with Ubuntu ! © Fred Hutchinson Cancer Research Center 4
- 6. Reliable monitoring! - Drives fail all the time - Benchmark while re- silvering ! - Notification to where you hangout (Slack?) © Fred Hutchinson Cancer Research Center 5
- 7. SMB, NFS, SFTP gateway - Clustered Samba 4.11.2 with CTDB - Latest Winbind scales to 100k users - CTDB also manages clustered NFS 4.2 - Kerberized NFS is finally working well - /fast -rw,fsid=101,no_subtree_check *(sec=krb5i:krb5p,root_squash) @cluster(sec=sys,no_root_squash) - Read only mounts to Backup/DR solution (ObjectiveFS) - S3 (minio) and http/WebDav are planned. © Fred Hutchinson Cancer Research Center 6
- 8. A shoutout to Samba+ from Sernet - Repositories with latest Samba for your OS - 6 Samba core developers on Staff - Outstanding support - HA with CTDB is stable & easy to install but hard to troubleshoot - Latest Winbind is much better ! © Fred Hutchinson Cancer Research Center 7
- 9. Some eye-candy with Prometheus & Grafana © Fred Hutchinson Cancer Research Center 8
- 10. More eye-candy with Prometheus & Grafana © Fred Hutchinson Cancer Research Center 9
- 11. Even more eye-candy with Prometheus & Grafana © Fred Hutchinson Cancer Research Center 10
- 12. Optane: 400k+ metadata read iops, 80-100k write iops © Fred Hutchinson Cancer Research Center 11
- 13. How much does it cost per month !? - Hardware cost for 5 years, divide by 60 - Support costs and license fees by month - Capacity is total file system space in TebiByte - Don’t forget sales tax ! - $1 per TiB per month for hardware, $2 mirrored - $1 per TiB per month for BeeGFS support - = $3 per TiB per month without compression - 1/3 LZ4 compression = $2 / TiB / month total cost mirrored © Fred Hutchinson Cancer Research Center 12
- 14. What does Fred Hutch want next from BeeGFS ? - ZFS based snapshots in BeeGFS - HA for Management server - --rebalance feature to ease cluster growth - Delta-sync for Metadata server - More explicit support for Ubuntu © Fred Hutchinson Cancer Research Center 13
- 15. Thank you ! Questions petersen@fredhutch.org Slides https://www.slideshare.net/dirkpetersen © Fred Hutchinson Cancer Research Center 14
Editor's Notes
- #5: Trick with Seagate: Unload the SAS driver