Winter is coming? Not if ZooKeeper is there!

Editor's Notes

• #3: What I mean by Winter are all the problems that we face when we work with distributed systems like the faults, failures, crashes, concurrency issues, synchronization issues that we face in distributed systems.
• #4: Earlier Life was simpler when most applications were a single program running on a single computer with a single CPU Today, things have changed. In the Big Data and Cloud Computing world, applications are made up of many independent programs running on an ever-changing set of computers. Distributed systems have brought their own set of challenges along with them. Like Concurrency. Security. Scalability. Resilience to failure But as an application developer I should not be concerned about these. That brings me to this very interesting slide that talks about the fallacies of DS.
• #5: These are the assumptions that most the programmers make while developing or coding for distributed applications. And we all know none of them is true…
• #6: To tackle these problems, we need coordination among the distributed components. Coordination can be for the purposes of cooperation or to regulate contention. For example: ………… master worker example for coordination Now let me tell you distributed coordination is really hard. So who should take care of these coordination needs. Who should save us from the winter? This is where zookeeper comes to our rescue.
• #7: Definition So it can be used for all coordination needs that we talked about yet in a simple way. Separates Coordination tasks from application. Hence: coordination task , loosely coupled Makes apps simple to maintain Makes development of apps independent to design and develop How it does this?
• #8: 2 ways of communication in DS. Shared storage and message passing Provides shared storage over ensemble of servers. Client API: Java and C Application use these Client apis to access ZooKeeper Service ZooKeeper Quorums – Minimum no. of servers that replicate data tree before sending acknowledgement to client. A leader is elected out of the quorum Clients connect to one of the servers through a TCP connection and maintain a session All servers have a copy of the data tree and ones which do not eventually catch up Read from any ZooKeeper Server Write goes through the leader and voting happens between members of quorum called followers Observers do not vote but just commit updates A Client connects to only one zookeeper server at a time using a TCP connection and establishes a Session. After establishing session, clients use these apis to create and manipulate Znodes. So what is znodes? It is a simple file system like structure which can store coordination data
• #9: Znode – special data structure, can store data as well as children. Like a file as well as directory. Clients can create, update and delete these znodes and their children. They can also update the data of a znode. Also clients can read the data of a znode and check whether znode exists. Using these operation, coordination can be achieved between multiple clients. For example how can we achieve a lock on a resource using znodes?
• #10: A Resource in you application is represented by /resource znode in ZooKeeper ensemble. Any process looking to access the resource would have to first acquire a lock on it which means create a znode /lock processName. Other processes keep on checking the /resource znode to find out if a lock znode already exists. If it exists, then they may want to read who/which process has the lock. If not, then they may themselves create a /lock znode
• #11: Once the process having the lock, finishes accessing the resource or its session expires then the /lock znode is deleted by ZooKeeper ensemble. Other processes can now acquire the lock. ZooKeeper does not delete the lock arbitrarily but because the znode /lock is an ephemeral znode, which is destined to be deleted.
• #13: There are 4 types of Znodes based on their properties: Persisitent – Continues to live on the ensemble even after creating node dies. In previous example, /resource is a candidate for persistent znode. Ephemeral – Deleted once the client session has ended, either explicitly or due to expiry. In previous example, /lock is a candidate for ephemeral znode. Persistent Sequential – Is a persistent znode with a monotonically increasing number appended to its name. Ephemeral Sequential – Is an ephemeral znode with a monotonically increasing number appended to its name like /lock-1,/lock-2 to show , maybe, order in which the different processes acquired the locks.
• #14: Now instead of 3 processes, there can be thousands of these processes waiting to acquire the lock. All these processes would be polling the ZooKeeper ensemble to check whether lock exists or not. This can cause a spike in server activity in which most of its time would be spent in fulfilling the client requests of exists() call. Due to this a client can set a one time trigger called watch on a znode when the following happens: Znode is created Znode is deleted Data of a znode changes Znode Children changes A watch is triggered at most once and generates a notification to the client who set the watch on matching the condition of watch
• #15: Like a Lannister, the ZooKeeper always pays its debts! It means that once you set a watch on a znode and an update on the znode happens, you are guaranteed to receive a notification before the next update happens on the znode. It does not mean that the clients will see all changes because there can be a time lag between the client getting the notification and reading the znode. By that time there can be multiple updates to the znode, for which no watch was set and a watch being a one-time trigger, will only service client once after it was set and when the first update happens. In any case, client will see the latest state when reading from zookeeper ensemble and may set a watch optionally.
• #17: sync call – waits for data to be propagated
• #18: Working of a Master-Worker system to process some tasks submitted by a client. Remember all znodes are conceptual representations and master, worker, task and assign are at same level under /zookeepermain znode. These are a group of systems working together to accomplish some task and managing their distributed coordination need using the znode naming system. Master, workers and tasks are all created by different clients to manage the processing. These are steps in order of appearance of the animation: A client creates /master ephemeral znode and keeps running to process client task completion requests. Others Clients who double as both worker and backup master may set a watch on /master znode to receive notification when master goes down. The master creates three persistent znodes namely /worker, /task and /assign to represent the worker queues, task queues and assignment of a task to a worker. The master sets a watch each on /worker and /task to listen for new workers and tasks. Once a worker becomes available, it creates a persistent znode under /assign to be assigned task to and an ephemeral znode under /worker to tell zookeeper that it is available to take up a task. Remember, the znode under /assign is created first so that as soon as it makes itself available, zookeeper will search under /assign to allocate a task. If not found, zookeeper will not be able to assign even if a worker is available. Next worker client sets a watch on /assign/worker-1 to listen for any changes under it viz. task assignment. Once /worker/worker-1 is created zookeeper will get notified and will check /task to assign one based on availability. Now a client creates a new /task-1 under /task and sets a watch on /status to watch for task completion. Zookeeper gets notification of new task, check for available workers and assigns by creating znode /task-1 under /assign/worker-1 Now worker gets a notification, since it set a watch on /assign/worker-1, and finds the task details under /task After completing task, worker creates a znode /status with data “DONE”/any other failure msg under /task/task-1 Client watching on the status now receives a notification and reads status to find task is completed or error message.
• #23: Two Small Changes in Priority Queues: Add to the /queue with pathname /queue-YY where YY is the priority of the element with lower numbers representing higher priority. When removing elements from queue, a client uses an up-to-date children list meaning that the client that the client will invalidate previously obtained children lists if a watch notification triggers for the queue node.