Hadoop HDFS NameNode HA
1 like2,581 views
This document outlines and compares two NameNode high availability (HA) solutions for HDFS: AvatarNode used by Facebook and BackupNode used by Yahoo. AvatarNode provides a complete hot standby with fast failover times of seconds by using an active-passive pair and ZooKeeper for coordination. BackupNode has limitations including slower restart times of 25+ minutes and supporting only two-machine failures. While it provides hot standby for the namespace, block reports are sent only to the active NameNode, making it a semi-hot standby solution. The document also briefly mentions other experimental HA solutions for HDFS.
Hadoop HDFS NameNode HA
- 1. Anty Rao April 10, 2011
- 2. Outline Architecture of HDFS Available NN HA options
- 3. HDFS architecture NN is SPOF, need some kind of HA for NN.
- 4. NN HA Currently two main available HA options: AvatarNode (facebook) BackupNode(yahoo!) (available?)
- 5. AvatarNode
- 6. AvatarNode (AN) Active-Standby Pair Client Coordinated via ZooKeeper Failover in few seconds Client retrieves block location from Wrapper over NameNode Primary or Standby Active AvatarNode Write Read Active transaction Standby Writes transaction log to AvatarNode transaction AvatarNode NFS filter (NameNode) (NameNode) Standby AvatarNode Reads/Consumes transactions from NFS filter Block Block Processes all messages from Location Location DataNodes messages messages Latest metadata in memory DataNodes
- 7. Four steps to failover Wipe ZooKeeper entry. Clients will know the failover is in progress. (0 seconds) Stop the primary NameNode. Last bits of data will be flushed to Transaction Log and it will die. (Seconds) Switch Standby to Primary. It will consume the rest of the Transaction log and get out of SafeMode ready to serve traffic. (Seconds) Update the entry in ZooKeeper. All the clients waiting for failover will pick up the new connection (0 seconds) After: Start the first node in the Standby Mode (Takes a while, but the cluster is up and running)
- 8. AvatarNode @Facebook Diagram from Facebook Contrib@hadoop 0.20 (HDFS-976)
- 9. Conclusions Complete Hot Standby NFS for storage of fsimage and editlogs. (no data loss) Standby node Consumes transactions from editlogs on NFS continuously. (namespace hot standby) DataNodes send message to both primary and standby node. (block reports hot standby) Fast Switchover Less than a minute Make sense!
- 10. BackupNode
- 11. BackupNode (BN) NN synchronously streams Client transaction log to Client retrieves block location BackupNode from NN BackupNode applies log Synchronous NN to in-memory and disk stream transacton (NameNode) logs to BN image BN always commit to disk BN Block (BackupNode before success to NN Location ) If BN restarts, it has to messages catch up with NN Available in HDFS 0.20.1 release DataNodes
- 12. Limitations of BackupNode(BN) Maximum of one BackupNode per NN Support only two-machine failure NN doesn’t forward block reports to BackupNode Time to restart from 12GB image, 70M files + 100M blocks 3-5 minutes to read the image from the disk 20 min to process block reports BN will still take 25+ minutes to failover!
- 13. Conclusions Incomplete Hot Standby / Semi-Hot Standby Namespace: hot standby Block reports: cold standby Still-Slow Switchover
- 14. Other HA solutions DRDB + Linux HA http://www.cloudera.com/blog/2009/07/hadoop-ha- configuration/ metadata backup http://wiki.apache.org/hadoop/NameNodeFailover