HDFS Analysis for Small Files
5 likes3,182 views
This document summarizes a presentation about analyzing small files in HDFS clusters. It outlines the problems small files can cause, such as inefficient data access and slower jobs. It then describes the architecture of the small files analysis solution, which processes the HDFS fsimage to attribute and aggregate file information. This information is stored and used to power dashboards showing metrics like small file counts and distributions over time. Future work includes improving performance and developing a customizable compaction utility.
HDFS Analysis for Small Files
- 1. Analyzing Small Files in HDFS Cluster Presenters: Rohit Jangid Presenters: Raman Goyal HDFS Analysis for Small Files
- 2. Outline ▪ What are small files and their problems? ▪ Small Files Analysis ▪ Architecture ▪ FsImage Processing and Aggregation ▪ Implementation and tool ▪ Dashboards and Results ▪ Dashboards ▪ Results ▪ Future Work ▪ Conclusions 2
- 4. Hdfs Doesn’t Like Lots Of Small Files… 4 Problem?
- 5. INEFFICIENT DATA ACCESS PATTERN 5
- 11. ARCHITECTURE HDFS Cluster RAW FsImage Interpreted FsImage Attributed Files and Directory information Aggregated Files and Directory information Dashboard Storage 11 LSR
- 12. LSR FsIMAGE PROCESSING MeProcessed 20gb FsImage In ~20 Minutes Custom OIV Interpreter For Reduced Memory Usage Fetched from Name node OIV to LSR Interpreter HDFS Cluster RAW FsImage Interpreted FsImage 12
- 13. LSR ARCHITECTURE HDFS Cluster RAW FsImage Interpreted FsImage Attributed Files and Directory information Aggregated Files and Directory information Dashboard Storage 13
- 14. Attributes Found Directly Owner Name Group Name Size of File Replication Factor Number of Direct File objects Last Modified Date Level of File Is File or Is Directory? Attribution and Aggregation Aggregated Attributes Number of Small File objects Number of Namespace objects Smallest, Largest, Avg File size Difference in Size since Last run If Directory 14
- 15. Attribution and Aggregation Generate Small Files / Total Files Metrics Roll-up Attributes to Parent Directories Custom UDF’s and PIG Scripts Using Sqoop Stored in HDFS Attributed Files and Directory information Aggregated Files and Directory information Storage 15
- 16. ARCHITECTURE HDFS Cluster RAW FsImage Interpreted FsImage Attributed Files and Directory information Aggregated Files and Directory information Dashboard Storage 16 LSR
- 17. STORAGE AND REPORTING DashboardStorage Relational Database and Rest API Dashboards Different Dashboards Showing User Level and Overall Level REST API Powered by Cyclotron: http://cyclotron.io 17
- 18. Implementation and Tool Files and Directories Attributed Small file & Directory information Download and Interpret HDFS NameNode At Directory level Statistics like Smallest File calculated Using OIV Interpreter By splitting FsImage rows Storage, REST API and Dashboards Can easily add new Clusters in Tool 18
- 20. Dashboards Information For file size less than 10 MB For file size between 10 MB to 70 MB For file size between 70 MB to ~100 MB 3 possible bucketing models Goes upto all levels in HDFS Distribution of owners of small Top 10 Directories to be investigated for deletion, re-partition, compaction 3 2 1 20
- 21. Overall Dashboard containing all Information 21
- 22. Distribution of Owners of Small Files 22
- 23. Sample Directories Containing Small Files 23
- 24. Top 10: Files vs Small Files 24
- 25. Daily Small Files per Directory 25
- 26. Doesn’t have real time analysis! with alerting Cluster has 200+ million namespace objects that we get as memory dump from Hadoop server. Future Work Translating and attributing each directory and file is a time consuming process. Developing Customisable Compaction Utility 1 2 26