The basic concept of Linux FIleSystem
2 likes1,089 views
This document discusses file systems and distributed file systems. It describes how file systems work, including hardware, partitions, logical volume management (LVM), and basic and distributed file systems. It focuses on GlusterFS and NFS distributed file systems. GlusterFS allows various volume types including distributed, replicated, distributed-replicated and stripe. NFS provides network access but no redundancy. The document also discusses storage solutions for AI training workloads, including Pure Storage FlashBlade and AIRI systems optimized for high-performance needs of AI.
The basic concept of Linux FIleSystem
- 2. Outline What is File System What’s GlusterFS and NFS How aurora works
- 3. FileSystem User-Application File System Kernel Block I/O Device(HDD/ SDD) Device(HDD /SDD) Kernel Driver Hardware Kernel (OS) User-Space Partition Partition Partition Partition
- 4. Hardware HDD/SDD We won’t use the whole disk You need to plan how to use the disk ○ How many partition ○ Size of each partition /dev/sda (100G) ○ /dev/sda1 –(50G) ○ /dev/sda2 –(25G) ○ /dev/sda3 – (25G)
- 6. FileSystem Basic FS ○ EXT4/ZFS/NFS/NTFS/BTRFS Logical FS ○ LVM Distributed FS ○ GlusterFS/Ceph/OpenEBS/BeeGFS/Lizard FS Only focus on GlusterFS and NFS today
- 7. Write/READPath User-Application File System Kernel Block I/O Device(HDD/ SDD) Device(HDD /SDD) Kernel Driver Partition Partition Partition Partition Write Read
- 8. LVM It’s impossible to predict the usage of each partition. You need to re-partition the whole disk sometimes to fit the usage of your user. We can use the LVM ○ Logical Volume management
- 9. LVM
- 10. RAID RAID (software RAID) ○ MDADM ○ Block Level ○ Also integrate with LVM to support more flexible management File System implement ○ BTRFS ○ ZFS
- 11. RAID
- 13. NFS Network File System ○ No Any RAID Support Read/Write via Network NFS Client NFS Server NFS Client NFS Client NFS Client Switch
- 14. Theproblemoffilesystem I/OPS (Input / Output per second) ○ Slow Raid Mechanism isn’t for larger number of disk ○ 100+ (only broken 2/3)
- 15. WhyDFS Scale up V.S Scale out
- 16. Distributedfilesystem Allows access to files from multiple hosts sharing via a computer network. Multiple users on multiple machines to share files and storage resources. Components ○ Server ○ Client ○ Metadata Servers Different features ○ Security/Redundancy
- 20. GlusterFS Distributed File System Volume ○ Distributed ○ Replicated ○ Distributed-Replicated ○ Stripe ○ Disperse Metrics ○ Large/Small files ○ Random/Sequence location
- 21. Distributed File Based Like RAID-0 Volume = V1+V2+…Vn IOPS ○ Read: v * n ○ Write: v * n
- 22. Replicated File Based Like RAID-1 Assume Replication Factor =2 Volume = sum(V)/2 IOPS ○ Read: v*n ○ Write: v*n/2
- 23. Distribuuted-replicated File-Based The server will mirror by itself (in the same Replication) Volume: sum(V)/2 IOPS: ○ Read: n *v ○ Write: n*v/2
- 24. Stripe Object-Based Split file into many bricks Volume: sum(v) IOPS: ○ Read: n*v ○ Write: n*v
- 25. DiSPERSE Erasure Coding Volume: based on config
- 26. ErasureCoding Erasure Coding ○ N = k + m (data + redundancy) ○ Take 6=4 + 2 as example 10 MB File 2.5 MB 2.5 MB Server 1 Server 2 2.5 MB Server 3 2.5 MB Server 4 2.5 MB Server 5 2.5 MB Server 6 Data Data Data DataRedundancyRedundancy
- 27. Aurora How Kubernetes works with volume ○ local ○ GlusterFS ○ NFS
- 28. LocalMount Read/Write from/to local disk ○ Memory ○ Disk Can’t share data cross nodes Share data in the same node ○ Access control ○ Read-Write many ○ Read-Write once ○ Read-Only
- 29. Server Server MountLocalVolume Server Platform (K8S) Pod Pod Pod Pod Pod Pod Mount local volume Local Local Local No ShareShare
- 30. GlusterFS Read/Write from/to GlusterFS ○ Memory ○ Disk/Network Access Share data cross nodes (same name) ○ Access Control ■ Read-Write Once ■ Read-Write Many ■ Read-Only Ask the volume size from GlusterFS
- 31. Server Server MountGlusterFS Server Platform (K8S) Pod Pod Pod Pod Pod Pod Mount Remote Volume Local Local Local Gluster-FS Local LocalLocal 5G 10G 25G Server Local Loc al
- 32. NFS Read/Write from/to NFS ○ Memory ○ Network Access Share data cross nodes (same name) ○ Access Control (no…since the NFS doesn’t support those feature) Ask the data size ○ It don’t support this feature..lol
- 33. Server Server MountNFS Server Platform (K8S) Pod Pod Pod Pod Pod Pod NFS NFS Server Performance bottleneck
- 35. Storage systems available today are optimized for a design point that’s different to what AI truly requires They are optimized for structured workloads – predictable, sequential access, not random patterns.
- 36. Pure-Storage(FlashBlade) A single storage server for NVIDIA DGX-1 Based on NFS Storage ○ Flash-Array ○ PB ○ 17 GB/s bandwidth ○ 1.5M IOPS Network ○ 8 * 40Gb/s Price <$1 per GB
- 37. Pure-Storage(AIRI) Four DGX-1 systems Based on NFS Storage ○ Flash-Array ○ PB ○ 17 GB/s bandwidth ○ 1.5M IOPS Network ○ 100G ○ GPUDirect RDMA Price ○ > 1M
- 39. Pure-Storage You can use GPFS/Lustre and other HPC storage. Btu data-scientists don’t want to deal with data-center infrastructure