Hadoop Interacting with HDFS

Editor's Notes

• #5: Commodity Hardware: -affordable and easy to obtain -capable of running Windows, Linux, or MS-DOS without requiring any special devices or equipment -broadly compatible and can function on a plug and play basis -low-end but functional product without distinctive features
• #6: BLOCK: A physical storage disk has a block size - minimum amount of data it can read or write. Normally 512 bytes. File systems for a single disk also deal with data in blocks. Normally few kilo bytes (4 kb). Hadoop has a much larger block size. By default it is 64 mb. Files in HDFS are broken down into block sized chunks and are stored as independent units. However, files smaller than a block size do not occupy the entire block. Why so large blocks??
• #12: > NameNode does not store the actual data or the dataset. The data itself is actually stored in the DataNodes. > NameNode knows the list of the blocks and its location for any given file in HDFS. > With this information NameNode knows how to construct the file from blocks.
• #14: JobTracker finds the best TaskTracker nodes to execute tasks based on: -data locality -available slots to execute a task on a given node
• #17: HDFS High Availability Namenode metadata is written to a shared storage (Journal Manager) Only one active NN can write to shared storage Passive NNs read & replay metadata from shared storage When active NN fails, one of the passive NNs is promoted to active Snapshot: Able to store a checkpointed stage of hdfs () Improved performance: Multithreaded random read HDFS v1: 264MB/sec HDFS v2: 1395MB/sec (about 5X !!) Federation Namenode stores metadata in memory For very large files, namenode could exaust memory Spread metadata over multiple namenodes
• #18: Details about HDFS Federation.
• #22: The general command line syntax is bin/hadoop command [genericOptions] [commandOptions] Generic options supported are -conf <configuration file> specify an application configuration file -D <property=value> use value for given property -fs <local|namenode:port> specify a namenode -jt <local|resourcemanager:port> specify a ResourceManager -files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster -libjars <comma separated list of jars> specify comma separated jar files to include in the classpath. -archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
• #24: The general command line syntax is bin/hadoop command [genericOptions] [commandOptions] Generic options supported are -conf <configuration file> specify an application configuration file -D <property=value> use value for given property -fs <local|namenode:port> specify a namenode -jt <local|resourcemanager:port> specify a ResourceManager -files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster -libjars <comma separated list of jars> specify comma separated jar files to include in the classpath. -archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
• #25: Everything you need to know about hdfs commands: http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html
• #26: Description: List the contents that match the specified pattern. If path is not specified, the contents of /user/<current_user> are listed Options: -d Directories are listed as plain files. -h Formats the sizes of files in a human-readable fashion, rather than a number of bytes. -R Recursively list the contents of directories. Output: (<permissions> <-/#replicas> <userid> <groupid> <size(in bytes)> <modification_date> <directoryName/fileName>)
• #27: Description: Create a directory in specified location. Options: -p : Create directories in the specified path, if does not exist
• #28: Description: Copy files into fs. Options: -f : If the file already exists, copying does not fail & the destination is overwritten. -p : Preserves access time, modification time, ownership and the mode.
• #29: Description: Copy files from hdfs. When copying multiple files, the destination must be a directory. Options: -p : Preserves access time, modification time, ownership and the mode. -ignorecrc : Files that fail the CRC check may be copied with this option. -crc : Files and CRCs may be copied using this option.
• #30: Description: Copy files from local filesystem into hdfs. It is same as “put” command but more specific w.r.t local filesystem Options: -f : If the file already exists, copying does not fail & the destination is overwritten. -p : Preserves access time, modification time, ownership and the mode.
• #31: Description: Copy files from hdfs to local filesystem. When copying multiple files, the destination must be a directory. It is same as “get” command but more specific w.r.t local filesystem Options: -p : Preserves access time, modification time, ownership and the mode. -ignorecrc : Files that fail the CRC check may be copied with this option. -crc : Files and CRCs may be copied using this option.
• #32: Description: Same as -copyFromLocal, except that the source is deleted after it's copied.
• #33: Description: Copy files from hdfs to the same hdfs. File pattern can be specified. When copying multiple files, the destination must be a directory. Options: -f : If the file already exists, copying does not fail & the destination is overwritten. -p : Preserves access time, modification time, ownership and the mode.
• #34: Description: Move files from hdfs to the same hdfs. File pattern can be specified. When moving multiple files, the destination must be a directory.
• #35: Description: Get all the files in the directories that match the source file pattern and merge and sort them to only one file on local fs. <src> is kept. Options: -nl Add a newline character at the end of each file.
• #36: -cat [-ignoreCrc] <src> ... : Fetch all files that match the file pattern <src> and display their content on stdout. -tail [-f] <file> : Show the last 1KB of the file. -f Shows appended data as the file grows. -text [-ignoreCrc] <src> ... : Takes a source file and outputs the file in text format. The allowed formats are zip and TextRecordInputStream and Avro.
• #37: Description: Delete all files that match the specified file pattern. Equivalent to the Unix command "rm <src>" Options: -skipTrash option bypasses trash, if enabled, and immediately deletes <src> -f If the file does not exist, do not display a diagnostic message or modify the exit status to reflect an error. -[rR] Recursively deletes directories
• #38: -chgrp [-R] GROUP PATH... : This is equivalent to -chown ... :GROUP ... -chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH... : Changes permissions of a file. This works similar to the shell's chmod command with a few exceptions. -R modifies the files recursively. This is the only option currently supported. <MODE> Mode is the same as mode used for the shell's command. -chown [-R] [OWNER][:[GROUP]] PATH... : Changes owner and group of a file. This is similar to the shell's chown command with a few exceptions. -R modifies the files recursively. This is the only option currently supported.
• #39: -du [-s] [-h] <path> ... : Show the amount of space, in bytes, used by the files that match the specified file pattern. The following flags are optional: -s Rather than showing the size of each individual file that matches the pattern, shows the total (summary) size. -h Formats the sizes of files in a human-readable fashion rather than a number of bytes. Note that, even without the -s option, this only shows size summaries one level deep into a directory. The output is in the form size disk space consumed name(full path)
• #40: -touchz <path> ... : Creates a file of zero length at <path> with current time as the timestamp of that <path>. An error is returned if the file exists with non-zero length
• #41: Options: -d: f the path is a directory, return 0. -e: if the path exists, return 0. -f: if the path is a file, return 0. -s: if the path is not empty, return 0. -z: if the file is zero length, return 0.
• #42: -stat [format] <path> ... : Print statistics about the file/directory at <path> in the specified format. Format accepts filesize in blocks (%b), group name of owner(%g), filename (%n), block size (%o), replication (%r), user name of owner(%u), modification date (%y, %Y)
• #43: -count [-q] [-h] [-v] <path> ... : Count the number of directories, files and bytes under the paths that match the specified file pattern. The -h option shows file sizes in human readable format. The -v option displays a header line. The output columns are: DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME or, with the -q option: QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME
• #44: -setrep [-R] [-w] <rep> <path> ... : Set the replication level of a file. If <path> is a directory then the command recursively changes the replication factor of all files under the directory tree rooted at <path>. -w It requests that the command waits for the replication to complete. This can potentially take a very long time. -R It is accepted for backwards compatibility. It has no effect.
• #45: The block size specified by dfs.blocksize should be multiple of 512 -copyFromLocal error if block size is not valid: Invalid values: dfs.bytes-per-checksum (=512) must divide block size (=104857601).
• #46: -expunge : Empty the Trash To enable hdfs thrash set fs.trash.interval > 1 in core-site.xml Deleted data goes in hdfs folder at : /user/<username>/.Trash/
• #48: Options: -move move corrupted files to /lost+found -delete delete corrupted files -files print out files being checked -openforwrite print out files opened for write -includeSnapshots include snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it -list-corruptfileblocks print out list of missing blocks and files they belong to -blocks print out block report -locations print out locations for every block -racks print out network topology for data-node locations
• #50: Command: hdfs dfsadmin -help report Description: Reports basic filesystem information and statistics. The dfs usage can be different from "du" usage, because it measures raw space used by replication, checksums, snapshots and etc. on all the DNs. Optional flags may be used to filter the list of displayed DNs. Options: -report [-live] [-dead] [-decommissioning]
• #53: hadoop getconf [-namenodes] gets list of namenodes in the cluster. [-secondaryNameNodes] gets list of secondary namenodes in the cluster. [-backupNodes] gets list of backup nodes in the cluster. [-includeFile] gets the include file path that defines the datanodes that can join the cluster. [-excludeFile] gets the exclude file path that defines the datanodes that need to decommissioned. [-nnRpcAddresses] gets the namenode rpc addresses [-confKey [key]] gets a specific key from the configuration