Reduce Storage Costs by 5x Using The New HDFS Tiered Storage Feature

Editor's Notes

• #19: In this presentation, I will first describe the motivation behind archiving some of our datasets and benefits of using tiered storage to archive the data. Then we will go over the setup of tiered storage on one of our clusters. I’ll describe our Archival process to identify the data to archive and their movement to archival storage. Let us first review eBay’s requirements for Archival Storage by taking a look at our clusters.
• #20: The Apollo cluster was setup in 2012 with around 20 Petabytes of storage, then we needed more storage and computing, increased the capacity to 36 Peta bytes in 2013. Towards the end of 2013, we were getting close to capacity, again . So we added some more data nodes and got the capacity up to 40 Peta bytes.
• #21: Our computing capacity requirements for Apollo Cluster were satisfied at this point, but data kept on growing. So it was evident that we needed to increase the storage capacity, to store the the existing data as well as the new data that’s coming in. So far, to increase capacity, we added the machines which has both storage and computing power. But this time we wanted to take advantage of the condition that we needed more storage , but no compute. We wanted to reduce costs by paying only for storage. We decided to take a look at the data we store in our cluster to identify the data which are rarely used.
• #22: A major class of data that we store in the Apollo cluster is User behavior data. There are different types of user behavior data. There is experimentation data, buyer and seller related data etc.These data will be used by various teams at eBay. They write their own map reduce jobs, hive queries, spark jobs etc to aggregate/summarize the data to derive insights or make decisions. For most of the datasets mentioned here, we expected that the data is heavily used initially and then the usage would go down. . We decided to verify this assumption by scanning our audit logs. The audit logs show who accessed which data and when. At eBay, we store our HDFS Audit logs in HDFS itself and we keep it for a long time.
• #23: This chart shows the access pattern of files under a dataset which has information about user’s sessions. The session data was stored in HDFS in early January , 2014. The X axis shows the age of the data in months and Y axis shows the access count. In the first months, there were 35K accesses for files under the directory. This dropped to 30 K in February. In March , it was around 7.5 K. Starting from the 8th month onwards, there was not much access. For this data, there is not much computation happening after 8 months and so there is no benefit to keep it on a node which has computing power. We can move it to a data node which has little computing power
• #24: There are many options for Archival storage. We could create an HDFS only cluster using machines with low computing power. The data can be moved using distcp. Another option is to have a NON-HDFS cheap storage and copy the data from the original cluster to this external storage. But if you want to keep the data in same cluster, HDFS tiered storage is the right choice. HDFS tiered storage introduces the concept of Storage types. There is a DISK storage type and Archive storage type. The default DISK storage type refers to normal storage where datanode has both compute and storage so that we can take advantage of data locality for data processing. The ARCHIVE storage type refers to data nodes which does not have computing power. Since the computing power is less, we pay mainly for the storage only and not for computing. This lowers the cost per GB for archival node compared to a DISK node. Once you have DISK and ARCHIVE storage types in the cluster, you archive data by moving the blocks from DISK Storage to ARCHIVE Storage.
• #25: There are many benefits of Archiving using tiered storage . Most benefits are due to the fact that the dataset continues to reside on the HDFS in the original cluster. The applications which operate on the data does not need to change at all. The hdfs location remains same. The attributes on the dataset like ACLS or permissions remains same. The Audit Logs will continue record the information as if nothing has changed for these datasets. So the audit log history will contain the full history from the time it was stored till you delete it. Now in case, data access pattern changes and archived data gets heavily used , then we can move the data back to compute nodes to achieve data locality during computation. There is also operational simplicity since you do not need to manage a separate form of Archival Storage.
• #26: Setting up Tiered storage on our Apollo Cluster. As I mentioned we already had 40 PB of storage with storage Type DISK. This comprised of 2000 datanodes. We added a new set of data nodes which are heavy in storage, but with limited computing power. There were 48 nodes with a total storage capacity of 10 Peta bytes. This formed our tiered storage, which is 40 peta bytes of DISK and 10 peta bytes of ARCHIVE storage.
• #27: Let us do comparison between a Disk data node and an Archive datanode.Our regular Disk nodes have a capacity ranging from 20 TB to 40 TB. We have a lot of 20 TB and few 40 TB nodes. But the details in slide refers to 40TB data node since they are the recent ones.The ARCHIVE nodes have 210 TB storage capacity on one datanode. The DISK nodes have 12 drives whereas ARCHIVE nodes have 60 drives. The regular DISK nodes run computations on the data. There are a total of 32 processing units on the regular DISK nodes. The ARCHIVE nodes are to store data only. The processing power is needed to mainly run the operating system and datanode process. So they have only 4 processing units. The regular datanode have 128GB of main memory where as ARCHIVAL nodes havea main memory of only 64 GB. The cost per GB of Archival storage is about one fifth of that DISK storage. There is no node manager running in the Archival datanode since the data is rarely accessed. It runs only the data node . The DISK nodes run both Node Manager to provision containers and data node.
• #28: The Archival nodes run only the data node process on them. There are few key differences in the data node configuration. The datanode directories should be marked as ARCHIVE. The major operation that happens on these data nodes is the move operation from a DISK storage. So we need to tune the Archival data node to support the move operations efficiently. There are two parameters which influence the move operations on a data node. dfs.datanode.balance.max.concurrent.moves specifies how many moves are concurrently allowed to or from a data node. The second parameter is the allowed bandwidth for block moves to or from a data node. In our cluster, we had 2000 DISK data nodes moving blocks to 48 ARCHIVE data nodes. The ratio between the source DISK nodes to the destination ARCHIVE nodes is 1:40 In our clusters, for DISK data nodes we have set concurent moves to 10. For ARCHIVE nodes, we have set this value to 400, following he 1:40 ration In our clusters, for DISK data nodes we have set this value to 10 MB.For Archival nodes, we have set this value 400 MB. With these values , we moved around 1 Peta byte of data from DISK to ARCHIVE storage in 12 hours without impacting the normal cluster operations.
• #29: So far we have moved around 8 petabytes of data to Archival storage. Going forward, we plan to derive an Archival policy for each dataset. To derive an Archive policy, we analyze the HDFS audit logs to determine the data access pattern for the dataset. Consider a dataset which is accessed heavily in the initial 3 months, then accessed moderately in the next 6 months and then rarely accessed beyond 9 months. For this dataset, we will define an archival policy where the directory’s storage policy set to HOT initially , then changed to WARM for the next 6 months and after 9 months , the policy is set to cold so that all replicas are moved to the ARCHIVAL storage. Our archival process will have the following steps : for each dataset. Identify data access pattern, Derive the Archival policy based on the data access pattern, Have a process which sets the storage policies according to the derived Archival policy and run mover to enforce the storage policies.
• #30: A few additional points to note about the tiered storage. The speed of data movement from DISK nodes to Archive nodes is proportional to the number of DISK nodes and ARCHIVE nodes. With 2000 DISK nodes and 48 ARCHIVE nodes, we could archive 1 PB of data in 12 hours. FSCK provides vital information for a given directory. We have enhanced FSCK so that it provides information on the storage tiers for the data blocks belonging to the directory. The work on displaying storage tier information on the Name node UI is in progress.
• #31: Summarizing my presentation, eBay uses tiered storage to store hot data and cold data. The storage costs are reduced by lowering compute capability for nodes which store cold data. HDFS tiered storage is operated via storage types and storage policies associated to directories. An archival policy needs to be setup for datasets based on their access patterns. Audit logs are useful identifying the access patterns. So collect them and store them for a long time. The archive process should change storage policies on directories to trigger their movement. FSCK provides information about tiered storage.