Accumulo Summit 2016: Accumulo Client Failover
0 likes112 views
The document discusses the Accumulo Client Failover (ACF) API developed to automate failover handling for client applications using multiple geographically distributed data centers. It details how ACF allows clients to seamlessly switch between Accumulo clusters during outages, maintenance, or unforeseen events while maintaining data consistency. The presentation outlines the motivation, design, and implementation of ACF, highlighting its integration with third-party libraries and features for enhanced client usability.
Accumulo Summit 2016: Accumulo Client Failover
- 1. Noe Detore Automated data center selection for a client Accumulo Client Failover Copyright 2016 Bloomberg Finance L.P. All rights reserved.
- 2. Abstract Most large organizations using Accumulo, such as Bloomberg, have two or more data centers distributed geographically. Replication was recently added to Accumulo providing automated eventual consistency between these data centers. On the other hand clients are still left with a need for an automated way to handle a failover from one Accumulo cluster to another. A client application needs the ability to continue processing and operating regardless of data center outages or failovers. An operations manager of an Accumulo data center needs the capability to control which Accumulo data center should be used by clients. At any given point in time a data center failover could be triggered by routine maintenance, system upgrade, or any other unforeseen event. To solve these use cases, the Accumulo Client Failover (ACF) API was developed. This presentation will cover the motivation, design, and use of ACF. We will show how ACF is used by an Accumulo client, and how it can be used alongside an existing third-party Accumulo client library.
- 4. Multi Data Center Support • Accumulo 1.7 supports eventually consistent replication – Write-ahead-log (WAL) files are sent when closed – Multiple configurations supported
- 6. Accumulo Users & Multiple DCs • No automated way to know active DC • Not able to failover from one DC to other • Maintenance outages require coordination with all clients • No automated failover for catastrophic event
- 7. Accumulo Client Failover (ACF) Goals • A client library that can handle arbitrary failover of Accumulo clusters • Provide failover capable implementations of Accumulo interfaces – Connector, Instance, Scanner, BatchScanner, BatchWriter – Use an eventingmodel to notify clients of an active/preferred Accumulo cluster • Upon receiving a failover event, client code will be interrupted so that it can do cleanup before continuing on with the new Accumulo cluster
- 9. ACF Initial Release • Failover is controlled and triggered by ZooKeeper • Current Accumulo instance information from znode • Update of znode triggers failover • Watcher on znode creates FailoverEvent when changed • Ultimately results in reader or writer throwing a FailoverException for a client to handle
- 10. • FailoverController triggers FailoverEvent • FailoverHandler and FailoverFlag • FailFailoverInstance like ZooKeeperInstance with a handle • FailoverConnector type behaves in a similar fashion – Once notified will notify all readers/writers it created • FailoverScanner, FailoverBatchScanner, FailoverBatchWriter… – Will throw a FailoverException on the next method call ACF Design
- 11. ACF Code Example Connector con = new ZooKeeperFailoverController(znode,zookeepers) .getInstance().getConnector(principal,token); while(true) { try { Scanner s = con.createScanner(table, auths); s.setRange(range); for(Entry<Key, Value> e : s) System.out.println(e); break; }catch(FailoverException e) { System.println(“failover, going to try again”); } }
- 12. Retryable • An API to abstract away the while loop • Retry any actions in case they fail • Allows users to focus solely on user code retry(() -> { Scanner s = con.createScanner(table, auths); s.setRange(range); for(Entry<Key, Value> e : s) System.out.println(e); })
- 14. Questions ?