Gluster d thread_synchronization_using_urcu_lca2016
Download as ODP, PDF0 likes540 views
This document discusses using user space RCU (URCU) for thread synchronization in GlusterD, the management daemon for the Gluster distributed file system. It begins with an introduction to Gluster and GlusterD, noting that GlusterD was initially single-threaded but was changed to be multi-threaded. This introduced the need for thread synchronization, which was initially implemented with a "big lock" but had issues. The document then provides an overview of RCU and how it can provide advantages over read-write locks for thread synchronization. It covers the key mechanisms of RCU, various URCU flavors, URCU APIs, and examples of when URCU would be useful.
Gluster d thread_synchronization_using_urcu_lca2016
- 2. 2 GlusterD Thread Synchronization using user space RCU Atin Mukherjee SSE-Red Hat Gluster Maintainer IRC : atinm on freenode Twitter: @mukherjee_atin Mail: amukherj@redhat.com
- 3. 3 Agenda ● Introduction to Gluster & GlusterD ● Big lock in thread synchronization in GlusterD ● Issues with Big Lock approach ● Different locking primitives ● What is RCU ● Advantage of RCU over read-write lock ● RCU mechanisms – Insertion, Deletion, Reader ● URCU flavors ● URCU APIs ● URCU use cases ● Q&A
- 4. 4 Gluster ● Open-source general purpose scale-out distributed file system ● Aggregates storage exports over network interconnect to provide a single unified namespace ● Layered on disk file systems that support extended attributes
- 5. 5 GlusterD ● Manages the cluster configuration for Gluster ● Responsible for – Peer membership management – Elastic volume management – Configuration consistency – Distributed command execution (orchestration) – Service management (manages GlusterFS daemons)
- 6. 6 Thread synchronization in GlusterD ● GlusterD was initially designed as single threaded ● Single threaded → Multi threaded to satisfy usecases like snapshot ● Big lock – A coarse grained lock – Only one transaction can work inside big lock – Protects all the shared data structures
- 7. 7 Issues with Big Lock ● Threads contend for even unrelated data ● Can end up in a deadlock – RPC request's callback also needs big lock ● Shall we release big lock in between a transaction to get rid of above deadlock? Yes we do, but…. ● Here come's the problem - a small window of time when the shared data structures are prone to updates leading to inconsistencies
- 8. 8 Different locking primitives ● Fine grained locks – Mutex – Read-write lock – Spin lock – Seq lock – Read-Copy-Update (RCU)
- 9. 9 What is RCU ● Synchronization mechanism ● Not new, added to Linux Kernel in 2002 ● Allows reads to occur concurrently with update ● Maintains multiple version of objects for read coherency ● Almost zero over heads in read side critical section
- 10. 10 Advantages of RCU over read-write lock ● Concurrent readers & writers – writer writes, readers read ● Wait free reads – RCU readers have no wait overhead. They can never be blocked by writers ● Existence guarantee – RCU guarantees that RCU protected data in a readers critical section will remain in existence till the end of the critical section ● Deadlock immunity – RCU readers always run in a deterministic time as they never block. This means that they can never become a part of a deadlock. ● No writer starvation – As RCU readers don't block, writers can never starve.
- 11. 11 RCU mechanism ● RCU is made up of three fundamental mechanisms – Publish-Subscribe Mechanism (for insertion) – Wait For Pre-Existing RCU Readers to Complete (for deletion) – Maintain Multiple Versions of Recently Updated Objects (for readers)
- 12. 12 Publish-Subscribe model ● rcu_assign_pointer () for publication 1 struct foo { 2 int a; 3 int b; 4 int c; 5 }; 6 struct foo *gp = NULL; 7 8 /* . . . */ 9 10 p = malloc (...); 11 p->a = 1; 12 p->b = 2; 13 p->c = 3; 14 gp = p; 1 struct foo { 2 int a; 3 int b; 4 int c; 5 }; 6 struct foo *gp = NULL; 7 8 /* . . . */ 9 10 p = malloc (...); 11 p->a = 1; 12 p->b = 2; 13 p->c = 3; 14 rcu_assign_pointer(gp, p); ● rcu_dereference () for subscription 1 p = gp; 2 if (p != NULL) { 3 do_something_with(p->a, p->b, p->c); 4 } 1 rcu_read_lock(); 2 p = rcu_dereference(gp); 3 if (p != NULL) { 4 do_something_with(p->a, p->b, p->c); 5 } 6 rcu_read_unlock();
- 13. 13 Publish-Subscribe Model (ii) ● rcu_assign_pointer () & rcu_dereference () embedded in special RCU variants of Linux's list-manipulation API ● rcu_assign_pointer () → list_add_rcu () ● rcu_dereference () → list_for_each_entry_rcu ()
- 14. 14 Wait For Pre-Existing RCU Readers to Complete ● Approach used for deletion ● Synchronous – synchronize_rcu () ● Asynchronous – call_rcu () q = malloc(...); *q = *p; q->b = 2; q->c = 3; list_replace_rcu(&p->list, &q->list); synchronize_rcu(); free(p) q = malloc(...); *q = *p; q->b = 2; q->c = 3; list_replace_rcu(&p->list, &q->list); call_rcu (&p->list, cbk); /* cbk will free p */
- 15. 15 Maintain multiple version objects ● Used for existence gurantee 1. p = search(head, key); 2. list_del_rcu(&p->list); 3. synchronize_rcu(); 4. free (p); 1. p = search(head, key); 2. list_del_rcu(&p->list); 3. synchronize_rcu(); 4. free (p); 1. p = search(head, key); 2. list_del_rcu(&p->list); 3. synchronize_rcu(); 4. free (p); Maintain multiple version objects ● Used for existence gurantee 1. p = search(head, key); 2. list_del_rcu(&p->list); 3. synchronize_rcu(); 4. free (p); 1. p = search(head, key); 2. list_del_rcu(&p->list); 3. synchronize_rcu(); 4. free (p); 1. p = search(head, key); 2. list_del_rcu(&p->list); 3. synchronize_rcu(); 4. free (p);
- 16. 16 URCU flavors ● QSBR (quiescent-state-based RCU) – each thread must periodically invoke rcu_quiescent_state() – Thread (un)registration required ● Memory-barrier-based RCU – Preemptible RCU implementation – Introduces memory barrier in read critical secion, hence high read side overhead ● “Bullet-proof” RCU (RCU-BP) – Similar like memory barrier based RCU but thread (un)registration is taken care – Primitive overheads but can be used by application without worrying about thread creation/destruction
- 17. 17 URCU flavors (ii) ● Signal-based RCU – Removes memory barrier – Can be used by library function – requires that the user application give up a POSIX signal to be used by synchronize_rcu() in place of the read-side memory barriers. – Requires explicit thread registration ● Signal-based RCU using an out-of-tree sys_membarrier() system call – sys_membarrier() system call instead of POSIX signal
- 18. 18 URCU APIs ● Atomic-operation and utility APIs – caa_: Concurrent Architecture Abstraction. – cmm_: Concurrent Memory Model. – uatomic_: URCU Atomic Operation. – https://lwn.net/Articles/573435/ ● The URCU APIs – https://lwn.net/Articles/573439/ ● RCU-Protected Lists – https://lwn.net/Articles/573441
- 19. 19 When is URCU useful
- 20. 20 References ● https://lwn.net/Articles/262464/ ● https://lwn.net/Articles/263130/ ● https://lwn.net/Articles/573424/ ● http://www.efficios.com/pub/lpc2011/Presentation- lpc2011-desnoyers-urcu.pdf ● http://www.rdrop.com/~paulmck/RCU/RCU.IISc- Bangalore.2013.06.03a.pdf ● http://urcu.so/
- 21. 21 Q&A