Lecture 11 file management

Introduction to
computational thinking
Module 11 : File Management
Asst Prof Michael Lees
Office: N4‐02c‐76
email: mhlees[at]ntu.edu.sg
Module 11 : File Management 1 of 53

Contents
• File basics
• File interaction
– writing, reading
• File facts
• Other file operations
• OS module
Chapter 5 & 14

FILES BASICS

What is a File?
• A file is a collection of data that is
stored on secondary storage like
a disk or a thumb drive.
• Accessing a file means
establishing a connection
between the file and the program
and moving data between the
two.

Two types of file
• Files come in two general types:
– Text files: files where control characters such as
“/n” are translated. These are generally human
readable
– Binary files: all the information is taken directly
without translation. Not readable and contains
non‐readable info.

Binary vs. plain text
• Plain text
+ human readable, useful for certain file types.
‐ inefficient storage (each character requires ?
bytes) : 256 combinations for ASCII = 8 bits:1byte.
Unicode could be 32 bits : 4 bytes
• Binary
+ More efficient storage, custom format
‐ Not human readable

Example
• Storing all the ages of the class (assume 500 students)
ASCII
‘20’
‘18’
‘21’
‘19’
How many bytes per entry? ASCII = 2
2 x 500 = 1000 bytes
Binary
20
18
21
19
How many bytes per entry? Binary= 1 byte
1 byte is 0‐255 (enough for age)
1 x 500 = 500 bytes

File Objects or Stream
• When opening a file, you create a file object
or file stream that is a connection between
the file information on disk and the program.
• The stream contains a “buffer” of the
information from the file, and provides the
information to the program

Buffering
• Reading from a disk is very slow. Thus the
computer will read a lot of data from a file in
the hope that, if you need the data in the
future, it will be “buffered” in the file object.
• This means that the file object contains a copy
of information from the file called a cache
(pronounced “cash”).

Buffering
File Buffer (Cache)Program Disk
write
read
write
read

Creating a file object
myFile = open(“myFile.txt”, “r”)
• myFile is the file object.
• It contains the buffer of information.
• The open function creates the connection
between the disk file and the file object.
• The first quoted string is the file name on disk,
the second is the mode to open it (here,“r”
means to read).

File location
• When opened, the name of the file can come
in one of two forms:
– “file.txt” assumes the file name is file.txt, and it is
located in the current program directory.
– “c:billfile.txt” is the fully qualified file name and
includes the directory information.
– ‘/Users/michaellees/python/CZCE1003’

File modes
Mode Description
‘r’ read a text file
‘w’ write a text (wipes contents)
‘a’ append to existing file
‘b’ binary file
‘+’ both read and write
• Be careful if you open a file with the ‘w’ mode. It sets an
existing file’s contents to be empty, destroying any existing
data.
• The ‘a’ mode is nicer, allowing you to write to the end of an
existing file without changing the existing contents.

FILE INTERACTION

Everything is a string
• If you are interacting with plain text files
(which is all we will do for this semester),
remember that everything is a string:
–everything read is a string
–if you write to a file, you can only write a
string

File contents
• Once you have a file object:
• fileObject.read()
– Reads the entire contents of the file as a string
and returns it. It can take an optional argument
integer to limit the read to N bytes, that is
fileObject.read(N).
• fileObject.readline()
– Delivers the next line as a string.

More contents
• fileObject.readLines()
–Returns a single list of all the lines from the
file.
• for line in fileObject:
–Iterator to go through the lines of a file.

Close the door behind you
• When done, you close the file. Closing is
important because the information in the
fileObject buffer is “flushed” out of the buffer
and into the file on disk, making sure that no
information is lost.
fileObject.close()

Python elegance
• Python is often praised for it’s simplicity and
elegance.
for line in file(“fileToRead.txt”):
print(line)
• File is automatically opened (by file( )).
• File is automatically closed at the end of the for
loop.
• Defaults are read and text.

Writing
• Once opened, you can write to a file (if
the mode is appropriate):
• fileObject.write(s)
–writes the string s to the file
• fileObject.writelines(list)
–write a list of strings (one at a time) to the
file

Errors?
• What if the file doesn’t exist?
• Your program should behave gracefully if the
file can’t be opened.
• When writing software, treat others as you
would like to be treated.
• In later chapters we will describe “exception,”
but for now we will just assume that you can
get the file.

Challenge 11.1 File Copy
Write a program to copy the contents of a file but removing all vowels

Thought process
• Open input and output file (‘r’, ‘w’)
• Process each line of the input file.
• Take each line and replace any vowel with
empty string “”
• Write new string to output file
• Close both files!!

Copy without vowels
vowels= [‘a’, ‘e’, ‘i’, ‘o’, ‘u’, ‘A’, ‘E’, ‘I’,
‘O’, ‘U’]
# File reading and writing
inFile = open("input.txt", "r")
outFile = open("output.txt", "w")
for line in inFile:
for letter in line:
if letter in vowels:
line = line.replace(letter,’’)
outFile.write(line) # written to the output file
inFile.close()
outFile.close()

FILE FACTS

Newline character
• Each operating system (Windows, OS X, Linux)
developed certain standards for representing
text.
• In particular, they chose different ways to
represent the end of a file, the end of a line,
etc.
• This can confuse our text readers!

Universal new line
• To get around this, Python provides a special
file option to deal with variations of OS text
encoding.
• The ‘U’ option means that Python deals with
the problem so you don’t have to!
fileObj = open(‘myFile.txt’, ‘rU’)

Current file position
• Every file maintains a “current file position.”
• It is the current position in the file and
indicates what the file will read next.
• It is set by the mode table above.

File buffer
• When the disk file is opened, the contents of
the file are copied into the buffer of the file
object.
• Think of the file object as a very big list where
every index is one of the pieces of information
of the file.
• The current position is the present index in
that list.

OTHER FILE OPERATIONS

tell()
• The tell() method tells you the current file
position.
• The positions are in bytes (think characters for
ASCII) from the beginning of the file:
fileObject.tell() => 42L

seek()
• The seek() method updates the current file
position to where you like (in bytes offset from
the beginning of the file):
– fd.seek(0) # to the beginning of the file
– fd.seek(100) # 100 bytes from beginning
• Counting bytes is a pain.
• Seek has a optional second argument:
– 0: count from the beginning
– 1: count for the current file position
– 2: count from the end (backwards)
e.g., fd.seek(-100,2)
100 bytes from end of file

Reading forward
• Every read/readline/readlines moves the
current pos forward.
• When you hit the end, every read will just
yield “” since you are at the end.
• You need to seek to the beginning to start
again (or close and open, seek is easier).

The power of pickle
• Everything is a string. So how about things
that aren’t?
• Python provides a standard module called
pickle. This is an amazing module that can
take almost any Python object and transfer
it to a file (converting to string in the
process); this process is called pickling.
(import pickle – it’s a module)
x = pickle.load(f)pickle.dump(x, f)
Pickle object x to file f Unpickle object x to file f
Remember:
str(2)=>
‘2’

Challenge 11.2 MP3 ID3 tag
Take an mp3 file and output the song name and artist name from the ID3
tag

Thought process
• MP3 is a binary file (can see that if you load in
text editor)
• Generally file headers (like ID3) are at specific
locations.
• Use the internet to find this location (Wiki)
• Open file, seek to correct bytes and then print
out.

OS MODULE

What is the OS module
• The os module in Python is an interface
between the operating system and the Python
language.
• As such, it has many sub‐functionalities
dealing with various aspects.
• We will look mostly at the file‐related stuff.
import os # to use os

What is a directory/folder
• Whether in Windows, Linux or on OS X, all
OS’s maintain a directory structure.
• A directory is a container of files and other
Directories.
• These directories are arranged in a hierarchy
or tree.

Different paths styles
• It turns out that each OS has its own way of
specifying a path:
– C:billpythonmyFile.py
– /Users/bill/python/myFile.py
• Nicely, Python knows that and translates to
the appropriate OS.

Some OS methods
• os.getcwd(): Returns the full path of the
current working directory.
• os.chdir(pathString): Change the current
directory to the path provided.
• os.listdir(pathString): Return a list of the files
and directories in the path (including ‘.’).

More OS methods
• os.rename(sourcePathStr, destPathStr):
Renames a file or directory.
• os.mkdir(pathStr): make a new directory. So
os.mkdir(‘/Users/bill/python/new’) creates
the directory new under the directory python.
• os.remove(pathStr). Removes the file.
• os.rmdir(pathStr). Removes the directory, but
the directory must be empty.

Take home lessons
• Files are important! (Obviously)
• Binary vs. Plain text
• Buffers – why and how.
• Reading/Writing files (binary & plain text)
• Elegant degradation (more in next module)

Further reading
• http://docs.python.org/tutorial/inputoutput.h
tml
• http://diveintopython3.org/files.html

Lecture 11 file management

More Related Content

Viewers also liked (18)

Similar to Lecture 11 file management (20)

Recently uploaded (20)

Lecture 11 file management