Hadoop File System Shell Commands,
5 likes4,314 views
The document provides a comprehensive overview of various Hadoop file system commands for interacting with the Hadoop distributed file system (HDFS), covering operations such as creating directories, copying files, listing files, and managing permissions and ownership. It includes details on how to represent file paths in different installation modes and commands for monitoring disk usage and checking file existence. Additionally, it addresses commands for file manipulation like moving, deleting, and setting replication factors, as well as managing the trash and viewing command usage.
![3. Hadoop fs du:
Hadoop fs –du displays disk usage of file or directory in HDFs
Show s the amount of space in Bytes occupied by each file or directory in HDFS.
[cloudera@localhost ~]$ hadoop fs -du hdfs://localhost:8020/user/cloudera/Ajay
1333 hdfs://localhost:8020/user/cloudera/Ajay/pig_1403050574637.log
2341 hdfs://localhost:8020/user/cloudera/Ajay/pig_1407465524489.log
22 hdfs://localhost:8020/user/cloudera/Ajay/test
0 hdfs://localhost:8020/user/cloudera/Ajay/test3
114 hdfs://localhost:8020/user/cloudera/Ajay/wctest1
–s to display total no of bytes used by the directory in HDFS
[cloudera@localhost ~]$ hadoop fs -du -s hdfs://localhost:8020/user/cloudera/Ajay
3810 hdfs://localhost:8020/user/cloudera/Ajay
-h shows the amount of space in kilo bytes occupied by each file or directory in HDFS
[cloudera@localhost ~]$ hadoop fs -du -h hdfs://localhost:8020/user/cloudera/Ajay
1.3 K hdfs://localhost:8020/user/cloudera/Ajay/pig_1403050574637.log
2.3 K hdfs://localhost:8020/user/cloudera/Ajay/pig_1407465524489.log
22 hdfs://localhost:8020/user/cloudera/Ajay/test
0 hdfs://localhost:8020/user/cloudera/Ajay/test3
114 hdfs://localhost:8020/user/cloudera/Ajay/wctest1
8](https://image.slidesharecdn.com/hadoopcommands-140818020903-phpapp02/85/Hadoop-File-System-Shell-Commands-8-320.jpg)
![4. Hadoop fs -put
Used to copy a file from your systems Folder to Hdfs.
In this Example I am copying the files from my Documents.
pig_1403050574637.log,
pig_1407465524489.log,
wctest1
Hdfs Directory Ajay.
You can use any type of URI to represent a directory in HDFS.
[cloudera@localhost ~]hadoop fs -put pig_1403050574637.log
hdfs://localhost:8020/user/cloudera/Ajay
[cloudera@localhost ~]$ hadoop fs -put pig_1407465524489.log
/user/cloudera/Ajay
[cloudera@localhost ~]$ hadoop fs -put wctest1
hdfs://127.0.0.1:8020/user/cloudera/Ajay
9](https://image.slidesharecdn.com/hadoopcommands-140818020903-phpapp02/85/Hadoop-File-System-Shell-Commands-9-320.jpg)
![5. hadoop fs chgrp:
hadoop fs -chgrp used to change the group association of files. Optionally you can use the
-R option to change recursively through the directory structure.
hadoop fs -chgrp [-R] <NewGroupName> <file or directory name>
hadoop fs -chgrp -R Hadoopusr hdfs://localhost:8020/user/cloudera/prem
6. hadoop fs chmod:
hadoop fs -chmod is used to change the permissions of files. The -R option can be used to
recursively change the permissions of a directory structure.
hadoop fs -chmod [-R] <mode | octal mode> <file or directory name>
hadoop fs -chmod -R 777 hdfs://localhost:8020/user/cloudera/prem
7. hadoop fs chown:
hadoop fs -chown used to change the ownership of files. The -R option can be used to
recursively change the owner of a directory structure.
hadoop fs -chown [-R] <NewOwnerName>[:NewGroupName] <file or directory name>
11](https://image.slidesharecdn.com/hadoopcommands-140818020903-phpapp02/85/Hadoop-File-System-Shell-Commands-11-320.jpg)
![11. Hadoop fs get:
Hadoop fs –get is command which copies a file to Local File system from HDFS.
Hadoo fs –get hdfs://localhost:8020/user/cloudera/Ajay/test3 /home/Documents/
12. Hadoop fs getmerge
-getmerge [-nl] <src> <localdst>
Get all the files in the directories that
match the source file pattern and merge and sort them to only
one file on local fs. <src> is kept.
-nl Adds a newline character at the end of each file.
[cloudera@localhost Documents]$ touch test4
[cloudera@localhost Documents]$ hadoop fs -getmerge hdfs://localhost:8020
/user/cloudera/Ajay/test /home/cloudera/Documents/test4
14](https://image.slidesharecdn.com/hadoopcommands-140818020903-phpapp02/85/Hadoop-File-System-Shell-Commands-14-320.jpg)
![14. Hadoop fs test:
Hadoop fs –test [edz] tests whether the file or directory exists in Hdfs or not.
--e to check file exists or not
-d to check directory exists or not
-z to check the file is Zero size or not.
[cloudera@localhost ~]$ hadoop fs -test -e hdfs://localhost:8020/user/cloudera/Ajay/test
[cloudera@localhost ~]$ hadoop fs -test -d hdfs://localhost:8020/user/cloudera/Ajay
[cloudera@localhost ~]$ hadoop fs -test -z hdfs://localhost:8020/user/cloudera/Ajay/test3
[cloudera@localhost ~]$
It returns 0 if exists
16](https://image.slidesharecdn.com/hadoopcommands-140818020903-phpapp02/85/Hadoop-File-System-Shell-Commands-16-320.jpg)
![19
17.Hadoop fs rm
hadoop fs -rm used to delete files in HDFS
Hadoop fs –rm –r to delete directories in HDFS
Hadoop fs -rm [-f] [-r|-R] [-skipTrash] filepath
Hadoop fs -rm
-skipTrash The file can not be moved to trash
-f If the file does not exist, do not display a diagnostic
message or modify the exit status to reflect an error.
-[rR] Recursively deletes
[cloudera@localhost ~]$ hadoop fs -mkdir /user/cloudera/trashdemo
[cloudera@localhost ~]$ hadoop fs -touchz /user/cloudera/trashdemo/temp
[cloudera@localhost ~]$ hadoop fs -rm -r -skipTrash
hdfs://localhost:8020/user/cloudera/trashdemo
Deleted hdfs://localhost:8020/user/cloudera/trashdemo](https://image.slidesharecdn.com/hadoopcommands-140818020903-phpapp02/85/Hadoop-File-System-Shell-Commands-19-320.jpg)
Hadoop File System Shell Commands,
- 1. 1
- 3. Hadoop File system Commands: The File system(FS) shell includes various shell like commands that directly interact With the Hadoop Distributed File system, as well with other file system That hadoop Supports such as Local Fs,Hftp Fs, S3 Fs and others. 3
- 4. 4
- 5. The File or Directory which created in HDFS can be represented using two ways. We can access using Local uri and HDFS uri representation. The command…. hadoop fs –mkdir /user/cloudera/Ajay Creates Ajay directory in Hdfs and can be represented as /user/hadoop/Ajay. We use this type of representation if you using hadoop is local mode installation. (check ppt hadoop installation Types) we can represent same directory with Hdfs URI hdfs://localhost:8020/user/cloudera/Ajay We use this type of representation if you are using Hadoop in Psuedo Distributed Mode installation We can also represent the Same Directory Hdfs://128..0.0.1:8020/user/cloudera/Ajay We use this type of representation if you are using Fully Distributed mode Installation. 5
- 6. To copy a File sample.txt which is in your system Drive or Folder to Hdfs.. Hadoop fs –put sample.txt /user/root/Ajay Hadoop fs –put sample.txt hdfs://localhost:8020/user/root/Ajay Localhost is we have configured all the nodes in one machine Which is like single node Cluster. Hadoop fs –put sample.txt hdfs://128.0.0.1:8020/user/root/Ajay 128.0.0.1 is the Node IP address, in which you want to copy the file. We use this type in Multinode Hadoop Cluster 6
- 7. 1. Hadoop fs ls: Hadoop fs –ls used to list files and directories in HDFS. -d Directories are listed as plain files. -h Formats the sizes of files in kilo bytes format. -R Recursively list the contents of directories. Hadoop fs –ls Hadoop fs –ls hdfs://localhost:8020/user/cloudera/Ajay Hadoop fs –ls –R displays directories and its contents recursively. Hadoop fs –ls –R Hadoop fs –ls –R /user/cloudera/Ajay 2. Hadoop fs cat Hadoop fs –cat used to displays the content of a file. Hadoop fs –put hdfs://localhost:8020/user/cloudera/Ajay/test 7
- 8. 3. Hadoop fs du: Hadoop fs –du displays disk usage of file or directory in HDFs Show s the amount of space in Bytes occupied by each file or directory in HDFS. [cloudera@localhost ~]$ hadoop fs -du hdfs://localhost:8020/user/cloudera/Ajay 1333 hdfs://localhost:8020/user/cloudera/Ajay/pig_1403050574637.log 2341 hdfs://localhost:8020/user/cloudera/Ajay/pig_1407465524489.log 22 hdfs://localhost:8020/user/cloudera/Ajay/test 0 hdfs://localhost:8020/user/cloudera/Ajay/test3 114 hdfs://localhost:8020/user/cloudera/Ajay/wctest1 –s to display total no of bytes used by the directory in HDFS [cloudera@localhost ~]$ hadoop fs -du -s hdfs://localhost:8020/user/cloudera/Ajay 3810 hdfs://localhost:8020/user/cloudera/Ajay -h shows the amount of space in kilo bytes occupied by each file or directory in HDFS [cloudera@localhost ~]$ hadoop fs -du -h hdfs://localhost:8020/user/cloudera/Ajay 1.3 K hdfs://localhost:8020/user/cloudera/Ajay/pig_1403050574637.log 2.3 K hdfs://localhost:8020/user/cloudera/Ajay/pig_1407465524489.log 22 hdfs://localhost:8020/user/cloudera/Ajay/test 0 hdfs://localhost:8020/user/cloudera/Ajay/test3 114 hdfs://localhost:8020/user/cloudera/Ajay/wctest1 8
- 9. 4. Hadoop fs -put Used to copy a file from your systems Folder to Hdfs. In this Example I am copying the files from my Documents. pig_1403050574637.log, pig_1407465524489.log, wctest1 Hdfs Directory Ajay. You can use any type of URI to represent a directory in HDFS. [cloudera@localhost ~]hadoop fs -put pig_1403050574637.log hdfs://localhost:8020/user/cloudera/Ajay [cloudera@localhost ~]$ hadoop fs -put pig_1407465524489.log /user/cloudera/Ajay [cloudera@localhost ~]$ hadoop fs -put wctest1 hdfs://127.0.0.1:8020/user/cloudera/Ajay 9
- 10. 10
- 11. 5. hadoop fs chgrp: hadoop fs -chgrp used to change the group association of files. Optionally you can use the -R option to change recursively through the directory structure. hadoop fs -chgrp [-R] <NewGroupName> <file or directory name> hadoop fs -chgrp -R Hadoopusr hdfs://localhost:8020/user/cloudera/prem 6. hadoop fs chmod: hadoop fs -chmod is used to change the permissions of files. The -R option can be used to recursively change the permissions of a directory structure. hadoop fs -chmod [-R] <mode | octal mode> <file or directory name> hadoop fs -chmod -R 777 hdfs://localhost:8020/user/cloudera/prem 7. hadoop fs chown: hadoop fs -chown used to change the ownership of files. The -R option can be used to recursively change the owner of a directory structure. hadoop fs -chown [-R] <NewOwnerName>[:NewGroupName] <file or directory name> 11
- 12. 8. Hadoop fs copyFromLocal: This command only copies a file from Local file system to Hdfs File system. Is case Sensitive. I am copying a file test from my Documents. hadoop fs -copyFromLocal test hdfs://localhost:8020/user/cloudera/Ajay 9 . Hadoop fs moveFromLocal: This command only Moves a file from Local file system to Hdfs File System. I am copying a file test3 from my Documetnts Folder. hadoop fs -moveFromLocal test3 hdfs://localhost:8020/user/cloudera/Ajay 10. Hadoop fs –copyToLocal: This command only copies a file from Hdfs to Local File system. I am copying a file in Hdfs /user/cloudera/Ajay/test3 to my Documents. hadoop fs -copyToLocal hdfs://localhost:8020/user/cloudera/Ajay/test3 /home/cloudera/Documents 12
- 13. 13
- 14. 11. Hadoop fs get: Hadoop fs –get is command which copies a file to Local File system from HDFS. Hadoo fs –get hdfs://localhost:8020/user/cloudera/Ajay/test3 /home/Documents/ 12. Hadoop fs getmerge -getmerge [-nl] <src> <localdst> Get all the files in the directories that match the source file pattern and merge and sort them to only one file on local fs. <src> is kept. -nl Adds a newline character at the end of each file. [cloudera@localhost Documents]$ touch test4 [cloudera@localhost Documents]$ hadoop fs -getmerge hdfs://localhost:8020 /user/cloudera/Ajay/test /home/cloudera/Documents/test4 14
- 15. 13. Hadoop setrep: Hadoop fs –setrep to specify the replication factor for a file. The default replication factor is 3, which is specified in core-site.xml With the property dfs.replication=3. It means all the blocks in Hdfs replicated 3 times. The file test with replication factor 3, can be changed to replication factor 2 With the command setrep: Hadoop fs –setrep -w 2 –R hdfs://localhost:8020/user/cloudera/Ajay/test 15
- 16. 14. Hadoop fs test: Hadoop fs –test [edz] tests whether the file or directory exists in Hdfs or not. --e to check file exists or not -d to check directory exists or not -z to check the file is Zero size or not. [cloudera@localhost ~]$ hadoop fs -test -e hdfs://localhost:8020/user/cloudera/Ajay/test [cloudera@localhost ~]$ hadoop fs -test -d hdfs://localhost:8020/user/cloudera/Ajay [cloudera@localhost ~]$ hadoop fs -test -z hdfs://localhost:8020/user/cloudera/Ajay/test3 [cloudera@localhost ~]$ It returns 0 if exists 16
- 17. 15. Hadoop fs stat: Print statistics information about the file/directory . We can check the information Blocks with %b Groupname or owner with %g Filename with %n Replication with %r User name of owner %u Modification date with %y %u), modification date (%y, %Y) Hadoop fs -stat %r hdfs://localhost:8020/user/cloudera/Ajay/test specifies the replication factor of a file test---2 hadoop fs -stat %u hdfs://localhost:8020/user/cloudera/Ajay/test Output the user name—Ajay hadoop fs -stat hdfs://localhost:8020/user/cloudera/Ajay Outputs date and time when the directory Ajay is created 17
- 18. 18 16 .Hadoop fs count: Hadoop fs –count used to count the number of directories, files and bytes for specified directory. it displays DIR_COUNT ,FILE_COUNT, CONTENT_SIZE FILE_NAME Hadoop fs –count hdfs://localhost:8020/user/cloudera/Ajay Hadoop fs –count -q Displays the Quota information about directory, (Refer dfsadmin PPT to assign quota to a directory) It displays QUOTA REMAINING_QUATA SPACE_QUOTA REMAINING_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE FILE_NAME Hadoop fs –count –q hdfs://localhost:8020/user/cloudera/Ajay
- 19. 19 17.Hadoop fs rm hadoop fs -rm used to delete files in HDFS Hadoop fs –rm –r to delete directories in HDFS Hadoop fs -rm [-f] [-r|-R] [-skipTrash] filepath Hadoop fs -rm -skipTrash The file can not be moved to trash -f If the file does not exist, do not display a diagnostic message or modify the exit status to reflect an error. -[rR] Recursively deletes [cloudera@localhost ~]$ hadoop fs -mkdir /user/cloudera/trashdemo [cloudera@localhost ~]$ hadoop fs -touchz /user/cloudera/trashdemo/temp [cloudera@localhost ~]$ hadoop fs -rm -r -skipTrash hdfs://localhost:8020/user/cloudera/trashdemo Deleted hdfs://localhost:8020/user/cloudera/trashdemo
- 20. 20 18.Hadoop fs expunge: Hadoop uses trash to store deleted files in HDFS core-site.xml contains property fs.trash.interval If you set Fs.trash.interval=0, means trash is disabled. To enable trash specify amount of time file resides in Trash Fs.trash.interval=100 I am creating a directory trashdemo and creating a file trashing in that directory. And I deleted it with hadoop fs –rmr. Trashdemo with the filele trashing moved to Trash. To clear the trash Use hadoop fs –expunge. hadoop fs -mkdir trashdemo hadoop fs -touchz hdfs://localhost:8020/user/cloudera/trashdemo/trashing hadoop fs -rm -r hdfs://localhost:8020/user/cloudera/trashdemo/trashing Moved: 'hdfs://localhost:8020/user/cloudera/trashdemo/trashing‘ to trash at: hdfs://localhost:8020/user/cloudera/.Trash/Current hadoop fs -expunge 14/08/17 22:52:24 INFO fs.TrashPolicyDefault: Created trash checkpoint: /user/cloudera/.Trash/140817225224 hadoop fs –expunge
- 21. 21
- 22. 22 19.Hadoop fs touchz Hadoop fs –touchz creates a zero length file in a specified directory. Hadoop fs –touchz hdfs://locahost:8020/user/cloudera/ajay/zerosize a file with the name zerosize with length zero bytes is create in the Directory ajay 20. Hadoop fs text Hadoop fs –text takes a source file and outputs the file in text format. The allowed formats are zip and TextRecordInputStream. hadoop fs -text hdfs://localhost:8020/user/cloudera/Ajay/pig_1403050574637.log
- 23. 23 21. Hadoop fs usage Hadoop fs –usage Displays the usage for given command or all commands if none is specified. Hadoop fs –usage copyFromLocal Displays how to use copyFromLocal command. 22. Hadoop fs tail: Hadoop fs –tail Show the last 1KB of the file. hadoop fs -tail hdfs://localhost:8020/user/cloudera/Ajay/pig_1403050574637.l og