Leases and-caching final
1 like746 views
This document summarizes a presentation about implementing lease locks and client-side caching in GlusterFS. It discusses quick introductions to lease locks and why they should be implemented in GlusterFS. The design section outlines how lease locks would work, storing lease info on servers and using upcalls. Challenges discussed include network partitions, rebalancing, and inconsistencies. Future enhancements proposed are supporting more lease types, migration/healing, and lock recovery. Client-side caching benefits from lease locks through cache coherency and reducing response times.
Leases and-caching final
- 1. Soumya Koduri Poornima G Lease Locks and Caching Gluster design Summit May. 2015
- 2. 05/17/16 2GlusterFS Meetup Agenda ● Quick intro to Lease locks ● Why implement in Gluster ● Design ● Network Partitions ● Challenges / limitations ● Future enhancements ● Client side caching with leases
- 3. 05/17/16 3GlusterFS Meetup ● Locking mechanism, which guarentees the client, certain semantics with respect to the sharing of the file/directory with other clients. ● Granted on regular files (file/directory). ● Protocol for cache consistency and aggressive caching. ● Enhances performance. Quick intro to Lease locks
- 4. 05/17/16 4GlusterFS Meetup Quick intro to Lease locks Client 1 Server Request for lease on file1 file1 file1 Granted, cache file1 Client 2 Request for access to file1 Lease Break file1 Flush all the cached data Flush file1, Unlock lease 6.Granted access to file1 1 2 6 4 3 5
- 5. 05/17/16 5GlusterFS Meetup Why support it in Gluster ● NFS-Ganesha/Samba applications ● Multi-protocol access on Gluster ● A new client side caching xlator based on leases – File data – Directory entries (+ readdirplus) – Metadata
- 6. 05/17/16 6GlusterFS Meetup Lease locks and its manifestations ● Linux lease locks: - Read lease - Write lease - Process based ● NFS V4 delegations: - Read delegations - Write delegations - Directory delegations - Client based
- 7. 05/17/16 7GlusterFS Meetup Lease locks and its manifestations ● SMB - Oplocks (opurtunistic locking) Level 1 (exclusive lock) Level 2 (Read lock) Batch (handle caching + exclusive lock) Fd based Leases introduced in SMB 2.1 Read Read/Handle Read/Write Read/Write/handles Client based
- 8. 05/17/16 8GlusterFS Meetup On Gluster ● Currently supported - Read Lease - Write Lease - fd based ● Plan is to support a super set of all standards: - Handle Lease, Read handle, Read Write Handle Lease - Lease Key support i.e. client based. - Lease upgrade/downgrade. - Directory Lease
- 9. 05/17/16 9GlusterFS Meetup Design Gluster Server Locks xlator Lease xlator libgfapi Protocol clnt Protocol server glfs_adv_lock() Lk Request Lk Response Recall Lease Uses Upcall Infra - Stores the lease info - Check conflict,grant/ Recall lease - Timer based recall
- 10. 05/17/16 10GlusterFS Meetup Design ● New API introduced in libgfapi to request lease locks. int glfs_*advanced*_lock (); ● Lease xlator on server side above Posix locks: - Maintains the lease state of a file - Checks for conflicts, Grants, Recall, lease locks ● Recall lease is implemented using Gluster file system notifications. ● Timer based recall
- 11. 05/17/16 11GlusterFS Meetup Network Parition ● Gluster Server disconnects: ➢ Replica brick will serve the lease state. ➢ Non-replica - the lease state is lost forever. ● Gluster Client disconnects: ➢ All the locks held by this client will be unlocked. ● Gluster Server/Client Reconnects: ➢ The client will replay all the lease locks it has held.
- 12. 05/17/16 12GlusterFS Meetup Challenges / Limitations ● Rebalance/Tiering ● Self-Heal ● Client lock replay not enabled. ● Inconsistency when used along with performance xlators ● Delegation throttling ● Tracking non fd based fops and the actual client associated
- 13. 05/17/16 13GlusterFS Meetup Future Enhancements ● Super set of all leases ● Upgrade and downgrade leases ● Heuristics based lease grant – Conflict frequency – Limit on leases granted – Quota limits ● Migrate/heal lease state ● Locks recovery during network disconnects
- 14. 05/17/16 14GlusterFS Meetup Client side caching
- 15. 05/17/16 15GlusterFS Meetup Common challenges of caching ● Challenges with caching ➢ Mostly in memory hence less aggressive ➢ Not distributed cache coherent ➢ Performance hit with cache-validation requests ➢ Not tolerant with file-system notification failures
- 16. 05/17/16 16GlusterFS Meetup Benifits of using leases ● Cache Coherency ● Aggressive persistent data caching ● Reduces application-level response time ● Suitable for distributed systems of larger scale with their larger ratio of processor speed to network delay ● Performance improvement in the workloads with little write-sharing
- 17. 05/17/16 17GlusterFS Meetup Benifits of using leases ● Directory leases will help in caching readdir and readdirplus coherently ● Well suited for home directory use cases ● Decreases file creation latency
- 18. 05/17/16 18GlusterFS Meetup Client cache based on leases Gluster Server Locks xlator Lease xlator Libgfapi/FUSE Protocol clnt Protocol server Open file1 Open + Lease req Lease Granted rsp Lease Cache Open Open+ lease Cache file1 if lease granted Gluster Client
- 19. 05/17/16 19GlusterFS Meetup Thank You!
Editor's Notes
- #8: Oplocks was introduced in SMB 2? Leases were introduced in SMB 2.1 – considered as oplocks done right. Major differences being: - Introduction of new typrs of leases - Introduction of lease Key
- #9: Doesn't adhere to the Posix standard of one unlock per process.
- #10: Command to enable Lease lock support for a volume: Gluster volume set <volname> leaeses on Recall Lease implemented via the file system notification feature in Gluster The lease lock requests are not sent to the Posix locks xlator, and hence it should be placed above the Posix locks, for it to function right.
- #11: Command to enable Lease lock support for a volume: Gluster volume set <volname> leaeses on Recall Lease implemented via the file system notification feature in Gluster The lease lock requests are not sent to the Posix locks xlator, and hence it should be placed above the Posix locks, for it to function right.
- #12: Command to enable Lease lock support for a volume: Gluster volume set <volname> leaeses on Recall Lease implemented via the file system notification feature in Gluster The lease lock requests are not sent to the Posix locks xlator, and hence it should be placed above the Posix locks, for it to function right.
- #16: architectures; this work relies on reliable, synchronous broadcast communication as provided by the system bus. A distributed system, however, can experience partial fail- ures: a host may crash or messages may be lost. Existing approaches to consistency for file caches fall into two cat- egories: those that assume reliable broadcast, and so do not tolerate communication failures, and those that require a consistency check for every read, and so fail to deliver good performance.
- #17: architectures; this work relies on reliable, synchronous broadcast communication as provided by the system bus. A distributed system, however, can experience partial fail- ures: a host may crash or messages may be lost. Existing approaches to consistency for file caches fall into two cat- egories: those that assume reliable broadcast, and so do not tolerate communication failures, and those that require a consistency check for every read, and so fail to deliver good performance.
- #19: Command to enable Lease lock support for a volume: Gluster volume set <volname> leaeses on Recall Lease implemented via the file system notification feature in Gluster The lease lock requests are not sent to the Posix locks xlator, and hence it should be placed above the Posix locks, for it to function right.