Status of Hadoop 0.23 Operations at Yahoo
3 likes801 views
The document summarizes the process Yahoo! undertook to upgrade its Hadoop clusters from version 0.20 to 0.23. It involved providing a 420-node sandbox cluster running 0.23 for customer testing, configuring queues and memory settings, and adding Kerberos authentication. Init scripts were modified for the DataNode, NameNode, SecondaryNameNode, HistoryServer, ResourceManager and NodeManager to start the new Hadoop versions.
![DataNode/NameNode
start_20(){
. . .
}
start_next(){
. . .
}
if [ -x /home/gs/hadoop/current/bin/hdfs ] ; then
start_next $@
else
start_20 $@
fi](https://image.slidesharecdn.com/statusofhadoop23opsatyahoo-120620141901-phpapp01/85/Status-of-Hadoop-0-23-Operations-at-Yahoo-14-320.jpg)
![SecondaryNameNode
function clean_checkpoint_dir {
CHECKPOINT_DIR=/grid/0/tmp/hadoop-
hdfs/dfs/namesecondary/current
if [ -d "$CHECKPOINT_DIR" ] ; then
DELETE_DIR=`mktemp -p /grid/0/tmp -d delete-XXXXXX`
if [ $? -eq 0 ] ; then
echo "moving $CHECKPOINT_DIR to ${DELETE_DIR}/ "
mv $CHECKPOINT_DIR ${DELETE_DIR}/
cat<<EOF | at now+1min 2>/dev/null
if [ -d $DELETE_DIR ] ; then
rm -rf --preserve-root $DELETE_DIR
fi
EOF
fi
fi
}](https://image.slidesharecdn.com/statusofhadoop23opsatyahoo-120620141901-phpapp01/85/Status-of-Hadoop-0-23-Operations-at-Yahoo-15-320.jpg)
Status of Hadoop 0.23 Operations at Yahoo
- 1. Status of Hadoop 0.23 Operations at Yahoo! From Inception to Customer Validation Charles Wimmer, Staff Site Reliability Engineer at LinkedIn
- 2. Summary of This Talk ● Includes ○ Operational changes required to support 0.23 ● Does not include ○ Specifics about customer testing ○ Deployment into Research or Production clusters
- 3. Scope of This Change at Yahoo! ● 42,000+ Hadoop servers ● 20+ clusters ● Three tiers: Sandbox, Research, Production ● 0.20.205.x
- 4. Overview of the Process ● Provide customers a 0.23 Sandbox cluster ● Provide customers enough data to test their applications ● Provide developer support to address application issues quickly ● Upgrade Research and Production clusters as applications are certified to work with 0.23
- 5. Test Cluster ● 420 Nodes ● 2 x Westmere 4 core processors ● 24G RAM ● 12 x 2T Disks ● No Federation
- 6. Configuration ● Hierarchical Queues ● Memory Configuration ● Kerberos
- 7. Hierarchical Queues <property> <name>yarn.scheduler.capacity.root.queues</name> <value>BIZUNIT-A,BIZUNIT-U,BIZUNIT-C,unfunded</value> </property> <property> <name>yarn.scheduler.capacity.root.capacity</name> <value>100</value> </property>
- 8. Hierarchical Queues <property> <name>yarn.scheduler.capacity.root.BIZUNIT-A.capacity</name> <value>50</value> </property> <property> <name>yarn.scheduler.capacity.root.BIZUNIT-U.capacity</name> <value>30</value> </property> <property> <name>yarn.scheduler.capacity.root.BIZUNIT-C.capacity</name> <value>15</value> </property> <property> <name>yarn.scheduler.capacity.root.unfunded.capacity</name> <value>5</value> </property>
- 9. Hierarchical Queues <property> <name>yarn.scheduler.capacity.root.BIZUNIT-U.proj-a. capacity</name> <value>50</value> </property> <property> <name>yarn.scheduler.capacity.root.BIZUNIT-U.proj-b. capacity</name> <value>50</value> </property>
- 11. Memory Configuration <property> <name>yarn. nodemanager.resource.memory- mb</name> <value>21504</value> </property>
- 12. Kerberos Configuration <property> <name>yarn.resourcemanager.principal</name> <value>mapred/clustername-jt1.domain.name.com@REALM. NAME.COM</value> </property> <property> <name>yarn.nodemanager.principal</name> <value>tt/_HOST@REALM.NAME.COM</value> </property>
- 13. Init Scripts ● DataNode/NameNode ● SecondaryNameNode ● HistoryServer ● NodeManager ● ResourceManager
- 14. DataNode/NameNode start_20(){ . . . } start_next(){ . . . } if [ -x /home/gs/hadoop/current/bin/hdfs ] ; then start_next $@ else start_20 $@ fi
- 15. SecondaryNameNode function clean_checkpoint_dir { CHECKPOINT_DIR=/grid/0/tmp/hadoop- hdfs/dfs/namesecondary/current if [ -d "$CHECKPOINT_DIR" ] ; then DELETE_DIR=`mktemp -p /grid/0/tmp -d delete-XXXXXX` if [ $? -eq 0 ] ; then echo "moving $CHECKPOINT_DIR to ${DELETE_DIR}/ " mv $CHECKPOINT_DIR ${DELETE_DIR}/ cat<<EOF | at now+1min 2>/dev/null if [ -d $DELETE_DIR ] ; then rm -rf --preserve-root $DELETE_DIR fi EOF fi fi }
- 16. HistoryServer case "$1" in start) su $HADOOP_USER -s /bin/sh -c "$HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver" RET=$? ;;
- 17. ResourceManager/NodeManager case "$1" in start) su $HADOOP_USER -s /bin/sh -c "$HADOOP_PREFIX/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start $PROC" RET=$? ;;