Optimizing FUSE for Cloud Storage
This document discusses optimizations made to FUSE for improving performance when using Parallels Cloud Storage. It summarizes the key optimizations made as asynchronous direct I/O, which allows I/O requests to be submitted asynchronously without blocking; synchronous direct I/O, which batches small writes into larger requests to reduce overhead; and writeback caching, which caches writes in memory before writing to storage asynchronously in the background. Performance testing showed the optimized FUSE implementation using Parallels Cloud Storage outperformed an iSCSI SAN storage system on 10 nodes despite using only commodity hardware. Future improvements discussed include allowing variable message sizes, eliminating global locks, using multiple queues, and setting CPU and NUMA affinities.
Optimizing FUSE for Cloud Storage
- 1. Vault 2015 Maxim Patlasov Linux Kernel Developer, Parallels Inc. Optimizing FUSE for Cloud Storage
- 2. 2 Agenda 1. Parallels Cloud Storage 2. FUSE concept 3. FUSE optimizations 4. Performance achieved 5. Future improvements
- 4. Parallels Hypervisor Virtualization • OS flexibility • HW emulation • Bare metal installation
- 5. Parallels Containers • More efficient memory management • More efficient caching reduces I/O • CT resource management • Easy migration • Easy backups&snapshots
- 6. 6 Storage requirements Key requirements for VM and Container’s needs: • Strong consistency • High performance • Fault tolerance • Fast recovery • Address all space from any node • Commodity hardware • In-flight reconfiguration and update
- 7. 7 Parallels Cloud Storage solution Key decisions made: • Optimize for big files • Union of all local storages • Replication for fault tolerance • Keep data and metadata separately • Multiple metadata servers
- 9. FUSE
- 10. 10 FUSE framework Kernel FUSE /fuse /dev/fuse Client application FUSE daemon req reqack ack
- 11. 11 FUSE: Containers on PStorage Kernel FUSE /fuse /dev/fuse FUSE daemon Client application per-container FS virtual block device image file libpstorage Parallels Cloud Storage MDS CS CS CS MDS MDS
- 13. 13 Asynchronous direct IO io_submit Application: io_submit(&iocb1); io_submit(&iocb2); Before: After: kernel fuse fuse daemon actual IO io_submit kernel fuse fuse daemon actual IO time io_submit kernel fuse io_submit kernel fuse fuse daemon actual IO
- 14. 14 Synchronous direct IO write Application: fd = open(O_DIRECT); write(fd, buf, 1<<20); Before: After: kernel fuse: 128K fuse daemon actual IO . . . kernel fuse: 128K fuse daemon actual IO time 8x: write kernel fuse: 8 x 128K fuse daemon actual IO
- 15. 15 Writeback cache write Application: buffered write(fd, buf, 1<<20); Before: After: kernel fuse: 128K fuse daemon actual IO . . . kernel fuse: 128K fuse daemon actual IO time 8x: write kernel fuse: populate page cache fuse daemon actual IO kernel writeback kernel fuse: 8 x 128K Key benefits: • Lower latency of write(2) • Parallel processing writeback
- 16. 16 Performance Comparison :: HW iSCSI SAN Storage DELL EqualLogic PS6510E 48 SATA Disks: 1TB 7200rpm (Seagate ST31000524NS) 30 SATA Disks: 2TB 7200rpm (Seagate ST2000DM001) + 10 SSD for caching (Intel SSD 520) x1 HW SAN EQL PS6510E x10 compute nodes Network: 10Gbit (Dell Force10 S4810) Network: 10Gbit (Brocade FastIron SuperX SX-F42XG) vs. (FUSE based) Parallels Cloud Storage
- 17. 17 PCS FASTER than HW SAN Just 10 nodes PCS cluster faster than DELL EQL SAN ($97000) in most workloads
- 19. 19 FUSE: future improvements Kernel FUSE /fuse /dev/fuse FUSE daemon 0 App 0 … FUSE daemon 1 FUSE daemon N App 1 App M … queue 0 queue 1 queue N… • Variable message size (currently 128K) • Eliminate global lock • Multi-queue • CPU and NUMA affinity
- 20. Q&A 20