Apache hadoop
Download as ODP, PDF0 likes242 views
The document discusses Apache Hadoop, an open-source Java software framework used for storing and processing large data sets across clusters of commodity machines using HDFS (Hadoop Distributed File System) and MapReduce. HDFS ensures reliable, scalable, and fault-tolerant storage, while MapReduce enables efficient data processing through a model inspired by functional programming. The document highlights the architecture of HDFS, its limitations, and the responsibilities of different components, such as namenodes and datanodes, as well as the task management in MapReduce jobs.
Apache hadoop
- 1. Apache Hadoop Sheetal Sharma Intern At IBM Innovation Centre
- 2. Why Data? Get insights to offer a better product “More data usually beats better algorithms” Get of insights to make better decisions Avoid “guesstimates”
- 3. What Is Challenging? Store data reliably Analyze data quickly Cost-effective way Use expressible and high-level language
- 4. Fundamental Ideas A big system of machines, not a big machine Failures will happen Move computation to data, not data to computation Write complex code only once, but right
- 5. Apache Hadoop An open-source Java software Storing and processing of very large data sets A clusters of commodity machines A simple programming model
- 6. Apache Hadoop Two main components: HDFS - a distributed file system MapReduce – a distributed processing layer
- 7. HDFS The Purpose Of HDFS ● Store large datasets in a distributed, scalable and fault- tolerant way ● High throughput ● Very large files ● Streaming reads and writes (no edits)
- 8. HDFS Mis-Usage Do NOT use, if you have Low-latency requests Random reads and writes Lots of small files Then better to consider RDBMs,
- 9. Splitting Files And Replicating Blocks Split a very large file into smaller (but still large) blocks Store them redundantly on a set of machines
- 10. Spiting Files Into Blocks ● The default block size is 64MB ● Minimize the overhead of a disk seek operation (less than 1%) ● A file is just “sliced” into chunks after each 64MB (or so)
- 11. Replicating Blocks The default replication factor is 3 ● It can be changed per a file or a directory ● It can be
- 12. Master And Slaves The Master node keeps and manages all metadata information The Slave nodes store blocks of data and serve them to the client Master node (called NameNode) Slave nodes (called DataNodes
- 13. Classical* HDFS Cluster *no NameNode HA, no HDFS Replication Manages metadata Does some “house-keeping” operations for NameNode Stores and retrieves blocks of data
- 14. HDFS NameNode Performs all the metadata- related operations Keeps information in RAM (for fast look up) The file system tree Metadata for all files/directories (e.g. ownership, permissions) Names and locations of blocks
- 15. HDFS DataNode Stores and retrieves blocks of data Data is stored as regular files on a local filesystem (e.g. ext4) e.g. blk_-992391354910561645 (+ checksums in a separate file) A block itself does not know which file it belongs to! Sends a heartbeat message to the NN to say that it is still alive Sends a block report to the NN periodically
- 16. HDFS Secondary NameNode NOT a failover NameNode Periodically merges a prior snapshot (fsimage) and editlog(s) (edits) Fetches current fsimage and edits files from the NameNode Applies edits to fsimage to create the up-to-date fsimage Then sends the up-to-date fsimage back to the NameNode
- 17. Reading A File From HDFS Block data is never sent through the NameNode The NameNode redirects a client to an appropriate DataNode The NameNode chooses a DataNode that is as “close” as possible Lots of data comes from DataNodes to a client Blocks locations $ hadoop fs -cat /toplist/2013-05-15/poland.txt
- 18. HDFS And Local File System ● Runs on the top of a native file system (e.g. ext3, ext4, xfs) ● HDFS is simply a Java application that uses a native
- 19. HDFS Data Integrity HDFS detects corrupted blocks ● When writing Client computes the checksums for each block Client sends checksums to a DN together with data ● When reading Client verifies the
- 20. HDFS NameNode Scalability Stats based on Yahoo! Clusters ● An average file 1.5≈ blocks (block size = 128 MB) ● An average file 600≈ bytes in RAM (1 file and 2 blocks objects) ● 100M files 60 GB of≈ metadata
- 21. HDFS NameNode Performance Read/write operations throughput limited by one machine ● ~120K read ops/sec ● ~6K write ops/sec ● MapReduce tasks are also HDFS clients Internal load increases as the cluster grows ●
- 22. HDFS Main Limitations Single NameNode ● Keeps all metadata in RAM ● Performs all metadata operations ● Becomes a single
- 23. MapReduce MapReduce Model Programming model inspired by functional programming map() and reduce() functions processing <key, value> pairs Useful for processing
- 24. Map And Reduce Functions ● Map and Reduce
- 25. Map And Reduce Functions - Counting Word
- 26. MapReduce Job Input data is divided into splits and converted into <key, value> pairs Invokes map() function multiple times Keys are sorted, values not (but could be) Invokes reduce() Function multiple times
- 27. MapReduce Example: ArtistCount Artist, Song, Timestamp, User Key is the offset of the line from the beginning of the line We could specify which artist goes to which reducer (HashParitioner is default one)
- 28. MapReduce Example: ArtistCount map(Integer key, EndSong value, Context context): context.write(value.artist, 1) reduce(String key, Iterator<Integer> values, Context context): int count = 0 for each v in values: count += v context.write(key, count) Pseudo-code in non-existing language ;)
- 29. MapReduce Combiner Make sure that the Combiner combines fast and enough (otherwise it adds overhead only)
- 30. MapReduce Implementation ● Batch processing system ● Automatic parallelization and distribution of computation ● Fault-tolerance ● Deals with all messy details related to distributed processing ● Relatively easy to use for programmers
- 31. JobTracker Reponsibilities ● Manages the computational resources Available TaskTrackers, map and reduce slots ● Schedules all user jobs Schedules all
- 32. TaskTracker Reponsibilities ● Runs map and reduce tasks ● Reports to JobTracker Heartbeats saying that it is still alive Number of free map and reduce slots Task progress,
- 33. Apache Hadoop Cluster ● It can consists of 1, 5, 100 and 4000 nodes
- 34. Thank You!