File Systems: Why, How and Where
1 like985 views
The document discusses various aspects and challenges of file systems, including scalability, reliability, metadata performance, and storage management. It explores different file system types, their architectures, and features such as journaling, recovery mechanisms, and modern enhancements like snapshotting and deduplication. The document also highlights the evolution of file systems, addressing bugs and performance issues while comparing notable file systems like ext2, ext3, ext4, and btrfs.
File Systems: Why, How and Where
- 1. File Systems: Why, How and Where Philip Derbeko enSilo 2017
- 3. The Tragedy of FileSystems •Scale and scalability •Reliability •Recovery •Complexity
- 4. The Tragedy of FileSystems
- 5. The Tragedy of FileSystems •Scale and scalability •Reliability •Recovery •Complexity •Flexibility for developers
- 6. Challenges 1. Metadata performance 2. Reliability and Recovery 3. Small files performance 4. Large files performance 5. Storage Management
- 7. Components •Block allocation •Directory Management •File and Directory operations •Inode handling •Transactions and journaling •Superblock handling •FS tree •Other
- 10. UNTIL …
- 11. Disk – The king of storage
- 12. Disk - Anatomy
- 13. You are here
- 14. Simplest possible FS – FAT
- 15. ext2
- 16. How it is done file file file dentry dentry inode inode File attributes Direct blocks Direct blocks … Indirect blocks Double Indirect blocks Triple Indirect blocks DirectDirectDirectDirect DirectDirectDirect DirectDirectDirectDirect
- 17. Ext2 - fsck •Unclean mount or mount counter •Not everything can be solved •Plan: • Superblock check • Free Blocks • Inode sanity • Inode links • Duplicates • Bad Blocks • Directory checks SLOW!!!
- 18. Other consistency options – Soft Updates •Dependency Rules: 1. Never point to uninitialized structure 2. Never reuse before nullifying the pointers 3. Never reset an old pointer before a new one was set
- 20. ext3 1. Journaling 2. Online file system growth 3. Directory indexing (not really, as was done for ext2 as well)
- 21. Ext3 - journaling TxB TxEInode Bit map WriteBack Data
- 22. Ext3 - journaling TxB TxEInode Bit map Ordered Data
- 23. Ext3 - journaling TxB TxEInode Bit map DataFull Journal Data
- 24. Ext3 – Journal final comments •Journal-assisted recovery: Redo Logging •Commit Batching •Journal Cleaning – mark the last checkpoint in journal superblock •Deletes and reuse
- 25. Ext4 •Backward and forward compatible – up to a certain point •Scalability •“Sequentiality” improvements: • Extent-based allocations • Journal checksum speed up • Delayed allocations •Transparent Encryption
- 26. Performance Optimizations 1. Synchronization of operations (the less is the better) 2. Locality of allocations 3. I/O Scheduling 4. Scalability 5. Caching 6. Pre-fetching
- 28. New Features •Snapshotting •Versioning •Backups •Deduplication •Data and meta data checksums
- 32. Newer is better?
- 33. FS Sizes
- 34. FS Patches (Linux 2.6 over 5079 patches) •Maintenance (45%) •Bugs (35%) – constant bug fixing over the life of FS •Performance •Reliability •Features FAST 2013 – “A Study of Linux File System Evolution”
- 35. Bug Consequences •Corruption •Crash •Failure of operation •Deadlock •Hang •Memory leak •Other FAST 2013 – “A Study of Linux File System Evolution” 38% of bugs are on failure paths frequency
- 36. “Timeline” – facts should not mess a story Berkley FFS ext2 ext3 ReiserFS ext4 ZFS BTRFS FAT FAT32 NTFS WinFS (dead) ReFS HFS HFS+ APFS
- 37. What was not covered •Shared, network, distributed and clustered file systems: • WAFL • AFS • GFS and DFS • WebDav •Volume Management •UnionFS (Knoppix CD+HDD, Docker layers)
- 38. The End Keep in touch: philip@ensilo.com