File Input/output, Database Access, Data Analysis with Pandas

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING
VELAGAPUDI RAMAKRISHNA SIDDHARTHA ENGINEERING COLLEGE
20CSH4801A
ADVANCED PYTHON PROGRAMMING
UNIT 2
Lecture By,
Prabu. U
Assistant Professor,
Department of Computer Science and Engineering.

UNIT 2:
File Input/output: Introduction to Files, Paths and IO, Reading and
Writing Files, Stream IO, Working with CSV Files, Working with Excel
Files.
Database Access: Introduction to Databases, Python DB-API,
PyMySQL Module.
Data Analysis with Pandas: The data structure of Pandas: Series, Data
Frame, Panel; Inserting and Exploring data: CSV, XLS, JSON, Database.
20CSH4801A ₋ ADVANCED PYTHON PROGRAMMING

1. Introduction to Files, Paths and IO
2. Reading and Writing Files
3. Stream IO
4. Working with CSV Files
5. Working with Excel Files
FILE INPUT/OUTPUT

1. Introduction to Files, Paths and IO
(i) Introduction
(ii) File Attributes
(iii) Paths
(iv) File Input/Output
(v) Sequential Access Versus Random Access
(vi) Files and I/O in Python

(i) Introduction
 The operating system is a critical part of any computer systems.
 It is comprised of elements that manage the processes that run on the CPU,
how memory is utilized and managed, how peripheral devices are used (such
as printers and scanners), it allows the computer system to communicate with
other systems and it also provide support for the file system used.
 The File System allows programs to permanently store data.

 This data can then be retrieved by applications at a later date; potentially after
the whole computer has been shut down and restarted.
 The File Management System is responsible for managing the creation, access
and modification of the long-term storage of data in files.
 This data may be stored locally or remotely on disks, tapes, DVD drives, USB
drives etc.
 Although this was not always the case; most modern operating systems
organize files into a hierarchical structure, usually in the form of an inverted
tree.

 For example, in the following diagram the root of the directory structure is
shown as ‘/’.
 This root directory holds six subdirectories. In turn the Users subdirectory
holds 3 further directories and so on:

 Each file is contained within a directory (also known as a folder on some
operating systems such as Windows). A directory can hold zero or more files
and zero or more directories.
 For any give directory there are relationships with other directories as shown
below for the directory jhunt:

 The root directory is the starting point for the hierarchical directory tree
structure. A child directory of a given directory is known as a subdirectory.
 The directory that holds the given directory is known as the parent directory.
At any one time, the directory within which the program or user is currently
working, is known as the current working directory.
 A user or a program can move around this directory structure as required. To
do this the user can typically either issue a series of commands at a terminal or
command window.
 Such as cd to change directory or pwd to print the working directory.
Alternatively Graphical User Interfaces (GUIs) to operating systems usually
include some form of file manager application that allows a user to view the
file structure in terms of a tree.

(ii) File Attributes
 A file will have a set of attributes associated with it such as the date that it
was created, the date it was last updated/modified, how large the file is etc. It
will also typically have an attribute indicating who the owner of the file is.
 This may be the creator of the file; however the ownership of a file can be
changed either from the command line or through the GUI interface. For
example, on Linux and Mac OS X the command chown can be used to change
the file ownership.
 It can also have other attributes which indicate who can read, write or execute
the file. In Unix style systems (such as Linux and Mac OS X) these access
rights can be specified for the file owner, for the group that the file is
associated with and for all other users.

 The file owner can have rights specified for reading, writing and executing a
file. These are usually represented by the symbols ‘r’, ‘w’ and ‘x’ respectively.
For example, the following uses the symbolic notation associated with Unix
files and indicates that the file owner is allowed to read, write and execute a
file:
-rwx------
 Here the first dash is left blank as it is to do with special files (or directories),
then the next set of three characters represent the permissions for the owner,
the following set of three the permissions for all other users.
 As this example has rwx in the first group of three characters this indicates that
the user can read ‘r’, write ‘w’ and execute ‘x’ the file. However the next six
characters are all dashes indicating that the group and all other users cannot
access the file at all.

 The group that a file belongs to is a group that can have any number of users as
members. A member of the group will have the access rights as indicated by
the group settings on the file. As for the owner of a file these can be to read,
write or execute the file. For example, if group members are allowed to read
and execute a file, then this would be shown using the symbolic notation as:
----r-x---
 Now this example indicates that only members of the group can read and
execute the file; note that group members cannot write the file (they therefore
cannot modify the file).
 If a user is not the owner of a file, nor a member of the group that the file is
part of, then their access rights are in the ‘everyone else’ category. Again this
category can have read, write or execute permissions. For example, using the
symbolic notation, if all users can read the file but are not able to do anything
else, then this would be shown as:
-------r--

 Of course a file can mix the above permissions together, so that an owner may
be allowed to read, write and execute a file, the group may be able to read and
execute the file but all other users can only read the file. This would be shown
as:
-rwxr-xr—
 In addition to the symbolic notation there is also a numeric notation that is
used with Unix style systems. The numeric notation uses three digits to
represent the permissions. Each of the three rightmost digits represents a
different component of the permissions: owner, group, and others.
 Each of these digits is the sum of its component bits in the binary numeral
system. As a result, specific bits add to the sum as it is represented by a
numeral:

 The read bit adds 4 to its total (in binary 100),
 The write bit adds 2 to its total (in binary 010), and
 The execute bit adds 1 to its total (in binary 001).
 This the following symbolic notations can be represented by an equivalent
numeric notation:

 Directories have similar attributes and access rights to files. For example, the
following symbolic notation indicates that a directory (indicated by the ‘d’) has
read and execute permissions for the directory owner and for the group. Other
users cannot access this directory:
dr-xr-x---
 The permissions associated with a file or directory can be changed either using
a command from a terminal or command window (such as chmod which is
used to modify the permissions associated with a file or directory) or
interactively using the file explorer style tool.

(iii) Paths
 A path is a particular combination of directories that can lead to a specific sub
directory or file.
 This concept is important as Unix/Linux/Max OS X and Windows file systems
represent an inverted tree of directories and files., It is thus important to be
able to uniquely reference locations with the tree.
 For example, in the following diagram the path /Users/jhunt/workspaces/
pycharmprojects/furtherpython/chapter2 is highlighted:

File Input/output, Database Access, Data Analysis with Pandas

 A path may be absolute or relative. An absolute path is one which provides a
complete sequence of directories from the root of the file system to a specific
sub directory or file.
 A relative path provides a sequence from the current working directory to a
particular subdirectory or file.
 The absolute path will work wherever a program or user is currently located
within the directory tree. However, a relative path may only be relevant in a
specific location.
 For example, in the following diagram, the relative path pycharmprojects/
furtherpython/chapter2 is only meaningful relative to the directory
workspaces:

 Note that an absolute path starts from the root directory (represented by ‘/’)
where as a relative path starts from a particular subdirectory (such as
pychamprojects).

(iv) File Input/Output
 File Input/Output (often just referred to as File I/O) involves reading and
writing data to and from files.
 The data being written can be in different formats.
 For example, a common format used in Unix/Linux and Windows systems is
the ASCII text format.
 The ASCII format (or American Standard Code for Information Interchange) is
a set of codes that represent various characters that is widely used by
operating systems.

 The following table illustrates some of the ASCII character codes and what
they represent:

(v) Sequential Access Versus Random Access
 Data can be read from (or indeed written to) a file either sequentially or via a
random-access approach.
 Sequential access to data in a file means that the program reads (or writes)
data to a file sequentially, starting at the beginning of a file and processing
the data an item at a time until the end of the file is reached.
 The read process only ever moves forward and only to the next item of data to
read.
 Random Access to a data file means that the program can read (or write) data
anywhere into the file at any time.

 That is the program can position itself at a particular point in the file (or
rather a pointer can be positioned within the file) and it can then start to read
(or write) at that point.
 If it is reading, then it will read the next data item relative to the pointer rather
than the start of the file.
 If it is writing data, then it will write data from that point rather than at the
end of the file.
 If there is already data at that point in the file, then it will be over written.
 This type of access is also known as Direct Access as the computer program
needs to know where the data is stored within the file and thus goes directly to
that location for the data.

 In some cases, the location of the data is recorded in an index and thus is also
known as indexed access.
 Sequential file access has advantages when a program needs to access
information in the same order each time the data is read.
 It is also is faster to read or write all the data sequentially than via direct
access as there is no need to move the file pointer around.
 Random access files however are more flexible as data does not need to be
written or read in the order in which it is obtained.
 It is also possible to jump to just the location of the data required and read
that data (rather than needing to sequentially read through all the data to find
the data items of interest).

(vi) Files and I/O in Python
 We will explore the basic facilities provided for reading and writing files in
Python.
 We will also look at the underlying streams model for file I/O.
 After this we will explore the widely used CSV and Excel file formats and
libraries available to support those.

2. Reading and Writing Files
(i) Introduction
(ii) Obtaining References to Files
(iii) Reading Files
(iv) File Contents Iteration
(v) Using Files and with Statements
(vi) The Fileinput Module
(vii) Renaming Files
(viii)Deleting Files
(ix) Random Access Files
(x) Directories
(xi) Temporary Files
(xii) Working with Paths

(i) Introduction
 Reading data from and writing data to a file is very common within many
programs.
 Python provides a large amount of support for working with files of various
types.

(ii) Obtaining References to Files
 Reading from, and writing to, text files in Python is relatively straightforward.
 The built in open()function creates a file object for you that you can use to
read and/or write data from and/ or to a file.
 The function requires as a minimum the name of the file you want to work
with.
 Optionally you can specify the access mode (e.g. read, write, append etc.). If
you do not specify a mode, then the file is open in read-only mode.

 You can also specify whether you want the interactions with the file to be
buffered which can improve performance by grouping data reads together.
 The syntax for the open() function is
file_object = open(file_name, access_mode, buffering)
 where,
 file_name indicates the file to be accessed.
 access_mode The access_mode determines the mode in which the file is to
be opened, i.e. read, write, append, etc. A complete list of possible values is
given below in the table. This is an optional parameter and the default file
access mode is read (r).

 buffering If the buffering value is set to 0, no buffering takes place. If the
buffering value is 1, line buffering is performed while accessing a file.
The access_mode values are given in the following table.

 The file object itself has several useful attributes such as
 file.closed returns True if the file has been closed (can no longer be
accessed because the close() method has been called on it).
 file.mode returns the access mode with which the file was opened.
 file.name The name of the file.
 The file.close()method is used to close the file once you have finished
with it. This will flush any unwritten information to the file (this may occur
because of buffering) and will close the reference from the file object to the
actual underlying operating system file.

 This is important to do as leaving a reference to a file open can cause problems
in larger applications as typically there are only a certain number of file
references possible at one time and over a long period of time these may all be
used up resulting in future errors being thrown as files can no longer be
opened.
 Refer FileReferences.py

(iii) Reading Files
 Of course, having set up a file object we want to be able to either access the
contents of the file or write data to that file (or do both). Reading data from a
text file is supported by the read(), readline() and readlines()
methods:
 The read() method This method will return the entire contents of the file
as a single string.
 The readline() method reads the next line of text from a file. It returns
all the text on one line up to and including the newline character. It can be
used to read a file a line at a time.
 The readlines() method returns a list of all the lines in a file, where each
item of the list represents a single line.

 The following illustrates using the readlines() method to read all the text in
a text file into a program and then print each line out in turn:
 Refer ReadingFile.py
 Notice that within the for loop we have indicated to the print function that we
want the end character to be ' ' rather than a newline; this is because the line
string already possesses the newline character read from the file.

(iv) File Contents Iteration
 It is very common to want to process the contents of a file one line at a time.
 In fact, Python makes this extremely easy by making the file object support
iteration. File iteration accesses each line in the file and makes that line
available to the for loop.
 It is also possible to use the list comprehension to provide a very concise way
to load and process lines in a file into a list. It is similar to the effect of
readlines() but we are now able to pre-process the data before creating the
list
 Refer FileContentIteration.py

(v) Writing Data to Files
 Writing a string to a file is supported by the write() method.
 Of course, the file object we create must have an access mode that allows
writing (such as 'w’).
 Note that the write method does not add a newline character (represented as
'n') to the end of the string—you must do this manually.
 Refer WritingToFile.py

(vi) Using Files and with Statements
 Like several other types where it is important to shut down resources; the file
object class implements the Context Manager Protocol and thus can be used
with the with statement.
 It is therefore common to write code that will open a file using the with as
structure thus ensuring that the file will be closed when the block of code is
finished with.
 Refer WithStatementExample.py

(vii) The Fileinput Module
 In some situations, you may need to read the input from several files in one
go.
 You could do this by opening each file independently and then reading the
contents and appending that contents to a list etc.
 However, this is a common enough requirement that the fileinput module
provides a function fileinput.input() that can take a list of files and treat
all the files as a single input significantly simplifying this process.
 Refer FileInputModule.py

 Features provided by the fileinput module include
 Return the name of the file currently being read.
 Return the integer “file descriptor” for the current file.
 Return the cumulative line number of the line that has just been read.
 Return the line number in the current file. Before the first line has been read
this returns 0.
 A boolean function that indicates if the current line just read is the first line
of its file.

(viii) Renaming Files
 A file can be renamed using the os.rename()function.
 This function takes two arguments, the current filename and the new
filename.
 It is part of the Python os module which provides methods that can be used to
perform a range of file-processing operations (such as renaming a file).
 To use the module, you will first need to import it.
 Refer RenameFiles.py

(ix) Deleting Files
 A file can be deleted using the os.remove() method.
 This method deletes the file specified by the filename passed to it.
 Again, it is part of the os module and therefore this must be imported first.
 Refer DeleteFiles.py

(x) Random Access Files
 All the examples presented so far suggest that files are accessed sequentially,
with the first line read before the second and so on.
 Although this is (probably) the most common approach it is not the only
approach supported by Python; it is also possible to use a random-access
approach to the contents within a file.
 To understand the idea of random file access it is useful to understand that we
can maintain a pointer into a file to indicate where we are in that file in terms
of reading or writing data.

 Before anything is read from a file the pointer is before the beginning of the file
and reading the first line of text would for example, advance the point to the
start of the second line in the file etc. This idea is illustrated below:

 When randomly accessing the contents of a file the programmer manually
moves the pointer to the location required and reads or writes text relative to
that pointer. This means that they can move around in the file reading and
writing data.
 The random-access aspect of a file is provided by the seek method of the file
object:
file.seek (offset, whence) this method determines where the next
read or write operation (depending on the mode used in the open() call)
takes place.
 In the above the offset parameter indicates the position of the read/ write
pointer within the file. The move can also be forwards or backwards
(represented by a negative offset).

 The optional whence parameter indicates where the offset is relative to. The
values used for whence are:
 0 indicates that the offset is relative to start of file (the default).
 1 means that the offset is relative to the current pointer position.
 2 indicates the offset is relative to end of file.
 Thus, we can move the pointer to a position relative to the start of the file, to
the end of the file, or to the current position.
 For example, in the following sample code we create a new text file and write a
set of characters into that file. At this point the pointer is positioned after the ‘z’
in the file.

 However, we then use seek()to move the point to the 10th character in the file
and now write ‘Hello’, next we reposition the pointer to the 6th character in
the file and write out ‘BOO’.
 We then close the file. Finally, we read all the lines from the file using a with as
statement and the open() function and from this we will see that the text is
the file is now abcdefBOOjHELLOpqrstuvwxyz:

(xi) Directories
 Both Unix like systems and Windows operating systems are hierarchical
structures comprising directories and files. The os module has several
functions that can help with creating, removing and altering directories. These
include:
 mkdir() This function is used to create a directory, it takes the name of
the directory to create as a parameter. If the directory already exists
FileExistsError is raised.
 chdir() This function can be used to change the current working
directory. This is the directory that the application will read from/ write to
by default.
 getcwd() This function returns a string representing the name of the
current working directory.

 rmdir() This function is used to remove/ delete a directory. It takes the
name of the directory to delete as a parameter.
 listdir() This function returns a list containing the names of the entries
in the directory specified as a parameter to the function (if no name is given
the current directory is used).
 Refer DirectoriesExample.py

(xii) Temporary Files
 During the execution of many applications, it may be necessary to create a
temporary file that will be created at one point and deleted before the
application finishes.
 It is of course possible to manage such temporary files yourself however, the
tempfile module provides a range of facilities to simplify the creation and
management of these temporary files.
 Within the tempfile module TemporaryFile, NamedTemporaryFile,
TemporaryDirectory, and SpooledTemporaryFile are high-level file
objects which provide automatic cleanup of temporary files and directories.
These objects implement the Context Manager Protocol.

 The tempfile module also provides the lower-level function mkstemp()
and mkdtemp() that can be used to create temporary files that require the
developer to management them and delete them at an appropriate time.
 The high-level feature for the tempfile module are:
 TemporaryFile(mode=‘w+b’) Return an anonymous file-like object that
can be used as a temporary storage area. On completion of the managed
context (via a with statement) or destruction of the file object, the temporary
file will be removed from the filesystem. Note that by default all data is
written to the temporary file in binary format which is generally more
efficient.
 NamedTemporaryFile(mode=‘w+b’)This function operates exactly as
TemporaryFile() does, except that the file has s visible name in the file
system.

 SpooledTemporaryFile(max_size=0, mode=‘w+b’)This function
operates exactly as TemporaryFile() does, except that data is spooled in
memory until the file size exceeds max_size, or until the file’s fileno()
method is called, at which point the contents are written to disk and
operation proceeds as with TemporaryFile().
 TemporaryDirectory(suffix=None, prefix=None, dir=None)
This function creates a temporary directory. On completion of the context or
destruction of the temporary directory object the newly created temporary
directory and all its contents are removed from the filesystem.

 The lower level functions include:
 mkstemp() Creates a temporary file that is only readable or writable by
the user who created it.
 mkdtemp() Creates a temporary directory. The directory is readable,
writable, and searchable only by the creating user ID.
 gettempdir() Return the name of the directory used for temporary files.
This defines the default value for the default temporary directory to be used
with the other functions in this module.
 Refer TemporaryFiles.py

(xiii) Working with Paths
 The pathlib module provides a set of classes representing filesystem paths;
that is paths through the hierarchy of directories and files within an operating
systems file structure. It was introduced in Python 3.4. The core class in this
module is the Path class.
 A Path object is useful because it provides operations that allow you to
manipulate and manage the path to a file or directory.
 The Path class also replicates some of the operations available from the os
module (such as mkdir, rename and rmdir) which means that it is not
necessary to work directly with the os module.

 A path object is created using the Path constructor function; this function
actually returns a specific type of Path depending on the type of operating
system being used such as a WindowsPath or a PosixPath (for Unix style
systems).
 The Path() constructor takes the path to create for example ‘D:/mydir’ (on
Windows) or ‘/Users/user1/mydir’ on a Mac or ‘/var/temp’ on Linux etc.
 You can then use several different methods on the Path object to obtain
information about the path such as:
 exists() returns True of False depending on whether the path points to an
existing file or directory.
 is_dir() returns True if the path points to a directory. False if it references
a file. False is also returned if the path does not exist.

 is_file() returns True of the path points to a file, it returns False if the path does
not exist or the path references a directory.
 absolute() A Path object is considered absolute if it has both a root and (if
appropriate) a drive.
 is_absolute() returns a Boolean value indicating whether the Path is absolute or
not.
 Refer PathsExample.py
 There are also several methods on the Path class that can be used to create and
remove directories and files such as:
 mkdir() is used to create a directory path if it does not exist. If the path already
exists, then a FileExistsError is raised.
 rmdir() remove this directory; the directory must be empty otherwise an error
will be raised.

 rename(target) rename this file or directory to the given target.
 unlink() removes the file referenced by the path object.
 joinpath(*other) appends elements to the path object e.g.
path.joinpath(‘/temp’).
 with_name(new_name) return a new path object with the name changed.
 The ‘/’ operator can also be used to create new path objects from existing paths
for example path/ ‘test’/ ‘output’ which would append the directories test and out to
the path object.
 Two Path class methods can be used to obtain path objects representing key directories
such as the current working directory (the directory the program is logically in at that
point) and the home directory of the user running the program:
 Path.cwd() return a new path object representing the current directory.
 Path.home() return a new path object representing the user’s home directory.
 Refer NewDirectory.py

 A very useful method in the Path object is the glob(pattern) method.
 This method returns all elements within the path that meet the pattern
specified. For example path.glob(‘*.py’) will return all the files ending .py
within the current path.
 Note that ‘**/*.py’ would indicate the current directory and any sub
directory.
 For example, the below code will return all files where the file name ends with
‘.txt’ for a given path.
 Refer DisplayingFiles.py

 Paths that reference a file can also be used to read and write data to that file.
For example the open() method can be used to open a file that by default
allows a file to be read:
 open(mode=‘r’)this can be used to open the file referenced by the path
object.
 However, there are also some high-level methods available that allow you to
easily write data to a file or read data from a file. These include the Path
methods write_text and read_text methods:
 write_text(data) opens the file pointed to in text mode and writes the
data to it and then closes the file.
 read_text() opens the file in read mode, reads the text and closes the file;
it then returns the contents of the file as a string.
 Refer PathMethods.py

3. Stream IO
(i) Introduction
(ii) What is a Stream?
(iii) Python Streams
(iv) IOBase
(v) Raw IO/UnBuffered IO Classes
(vi) Binary IO/Buffered IO Classes
(vii) Text Stream Classes
(viii)Stream Properties
(ix) Closing Streams
(x) Returning to the open() Function

(i) Introduction
 The Stream I/O model under pins the way in which data is read from and
written to data sources and sinks.
 One example of a data source or sink is a file, but another might be a byte
array.
 This model is actually what sits underneath the file access mechanisms.
 It is not actually necessary to understand this model to be able to read and
write data to and from a file, however in some situations it is useful to have an
understanding of this model so that you can modify the default behaviour
when necessary.

(ii) What is a Stream?
 Streams are objects which serve as sources or sinks of data.
 At first this concept can seem a bit strange. The easiest way to think of a
stream is as a conduit of data flowing from or into a pool.
 Some streams read data straight from the “source of the data” and some
streams read data from other streams.
 These latter streams then do some “useful” processing of the data such as
converting the raw data into a specific format.

 The following figure illustrates this idea.
 In the above figure the initial FileIO stream reads raw data from the actual
data source (in this case a file).
 The BufferedReader then buffers the data reading process for efficiency.

 Finally, the TextIOWrapper handles string encoding; that is it converts strings
from the typical ASCII representation used in a file into the internal
representation used by Python (which uses Unicode).
 A stream can read or write data to or from a source of data rather than just
from a file.
 Of course, a file can be a source of data but so can a socket, a pipe, a string, a
web service etc.
 It is therefore a more flexible data I/O model.

(iii) Python Streams
 The Python io module provides Python’s main facilities for dealing with data
input and output. There are three main types of input/output these are text
I/O, binary I/O and raw I/O. These categories can be used with various types
of data source/sinks.
 Whatever the category, each concrete stream can have a number of properties
such as being read-only, write-only or read-write. It can also support
sequential access or random access depending on the nature of the underlying
data sink.
 For example, reading data from a socket or pipe is inherently sequential
whereas reading data from a file can be performed sequentially or via a
random-access approach.

 Whichever stream is used however, they are aware of the type of data they can
process. For example, attempting to supply a string to a binary write-only
stream will raise a TypeError. As indeed will presenting binary data to a text
stream etc.
 As suggested by this there are a number of different types of stream provided
by the Python io module and some of these are presented below:

(iv) IOBase
 This is the abstract base class for all I/O stream classes.
 The class provides many abstract methods that subclasses will need to
implement.
 The IOBase class (and its subclasses) all support the iterator protocol.
 This means that an IOBase object (or an object of a subclass) can iterate over
the input data from the underling stream.
 IOBase also implements the Context Manager Protocol and therefore it can be
used with the with and with-as statements.

 The IOBase class defines a core set of methods and attributes including:
 close()flush and close the stream.
 closed an attribute indicating whether the stream is closed.
 flush()flush the write buffer of the stream if applicable.
 readable()returns True if the stream can be read from.
 readline(size=-1)return a line from the stream. If size is specified at
most size bytes will be read.
 readline(hint=-1)read a list of lines. If hint is specified, then it is used
to control the number of lines read.
 seek(offset[, whence])This method moves the current the stream
position/pointer to the given offset. The meaning of the offset depends on
the whence parameter. The default value for whence is SEEK_SET.

 SEEK_SET or 0: seek from the start of the stream (the default); offset
must either be a number returned by TextIOBase.tell(), or zero. Any other
offset value produces undefined behaviour.
 SEEK_CUR or 1: “seek” to the current position; offset must be zero,
which is a no-operation (all other values are unsupported).
 SEEK_END or 2: seek to the end of the stream; offset must be zero (all
other values are unsupported).
 seekable()does the stream support seek().
 tell()return the current stream position/pointer.
 writeable()returns true if data can be written to the stream.
 writelines(lines)write a list of lines to the stream.

(v) Raw IO/UnBuffered IO Classes
 Raw IO or unbuffered IO is provided by the RawIOBase and FileIO classes.
 RawIOBase This class is a subclass of IOBase and is the base class for raw
binary (aka unbuffered) I/O.
 Raw binary I/O typically provides low-level access to an underlying OS
device or API and does not try to encapsulate it in high-level primitives (this is
the responsibility of the Buffered I/O and Text I/O classes that can wrap a raw
I/O stream).

 The class adds methods such as:
 read(size=-1)This method reads up to size bytes from the stream and
returns them. If size is unspecified or -1 then all available bytes are read.
 readall() This method reads and returns all available bytes within the
stream.
 readint(b) This method reads the bytes in the stream into a pre-
allocated, writable bytes-like object b (e.g. into a byte array). It returns the
number of bytes read.
 write(b) This method writes the data provided by b (a bytes -like object
such as a byte array) into the underlying raw stream.

 FileIO The FileIO class represents a raw unbuffered binary IO stream linked
to an operating system level file.
 When the FileIO class is instantiated, it can be given a file name and the
mode (such as ‘r’ or ‘w’ etc.).
 It can also be given a flag to indicate whether the file descriptor associated
with the underlying OS level file should be closed or not.
 This class is used for the low-level reading of binary data and is at the heart of
all file-oriented data access (although it is often wrapped by another stream
such as a buffered reader or writer).

(vi) Binary IO/Buffered IO Classes
 Binary IO aka Buffered IO is a filter stream that wraps a lower level
RawIOBase stream (such as a FileIO stream). The classes implementing
buffered IO all extend the BufferedIOBase class and are:
 BufferedReader When reading data from this object, a larger amount of data
may be requested from the underlying raw stream and kept in an internal
buffer. The buffered data can then be returned directly on subsequent reads.
 BufferedWriter When writing to this object, data is normally placed into an
internal buffer. The buffer will be written out to the underlying RawIOBase
object under various conditions, including:
 when the buffer gets too small for all pending data;
 when flush()is called;
 when the BufferedWriter object is closed or destroyed.

 BufferedRandom A buffered interface to random access streams. It supports
seek()and tell()functionality.
 BufferedRWPair A buffered I/O object combining two unidirectional
RawIOBase objects – one readable, the other writeable—into a single
bidirectional endpoint.
 Each of the above classes wrap a lower-level byte oriented stream class such as
the io.FileIO class, for example:
f = io.FileIO('data.dat’)
br = io.BufferedReader(f)
print(br.read())

 This allows data in the form of bytes to be read from the file ‘data.dat’. You can
of course also read data from a different source, such as an in memory BytesIO
object:
binary_stream_from_file =io.BufferedReader(io.BytesIO(b'starship.png'))
bytes = binary_stream_from_file.read(4)
print(bytes)
 In this example the data is read from the BytesIO object by the BufferedReader.
The read() method is then used to read the first 4 bytes, the output is:
b ‘star’
 Note the ‘b’ in front of both the string ‘starship.png’ and the result ‘star’. This
indicates that the string literal should become a bytes literal in Python 3. Bytes
literals are always prefixed with ‘b’ or ‘B’; they produce an instance of the bytes
type instead of the str type.

 They may only contain ASCII characters.
 The operations supported by buffered streams include, for reading:
 peek(n) return up to n bytes of data without advancing the stream pointer.
The number of bytes returned may be less or more than requested depending
on the amount of data available.
 read(n) return n bytes of data as bytes, if n is not supplied (or is negative)
the read all available data.
 readl(n) read up to n bytes of data using a single call on the raw data
stream.
 The operations supported by buffered writers include:
 write(bytes) writes the bytes-like data and returns the number of bytes
written.
 flush() This method forces the bytes held in the buffer into the raw
stream.

(vii) Text Stream Classes
 The text stream classes are the TextIOBase class and its two subclasses
TextIOWrapper and StringIO.
 TextIOBase This is the root class for all Text Stream classes. It provides a
character and line-based interface to Stream I/O. This class provides several
additional methods to that defined in its parent class:
 read(size=-1)This method will return at most size characters from the
stream as a single string. If size is negative or None, it will read all
remaining data.
 readline(size=-1)This method will return a string representing the
current line (up to a newline or the end of the data whichever comes first).
If the stream is already at EOF, an empty string is returned. If size is
specified, at most size characters will be read.

 seek(offset, [, whence])change the stream position/pointer by the
specified offset. The optional whence parameter indicates where the seek
should start from:
 SEEK_SET or 0: (the default) seek from the start of the stream.
 SEEK_CUR or 1: seek to the current position; offset must be zero, which
is a no-operation.
 SEEK_END or 2: seek to the end of the stream; offset must be zero.
 tell()Returns the current stream position/pointer as an opaque number.
The number does not usually represent a number of bytes in the underlying
binary storage.
 write(s)This method will write the string s to the stream and return the
number of characters written.

 TextIOWrapper. This is a buffered text stream that wraps a buffered binary stream and
is a direct subclass of TextIOBase. When a TextIOWrapper is created there are a
range of options available to control its behaviour:
io.TextIOWrapper(buffer, encoding=None, errors=None,
newline=None,line_buffering=False,
write_through=False)
 Where
 buffer is the buffered binary stream.
 encoding represents the text encoding used such as UTF-8.
 errors defines the error handling policy such as strict or ignore.
 newline controls how line endings are handled for example should they be
 ignored (None)or represented as a linefeed, carriage return or a newline/carriage
return etc.
 line_buffering if True then flush() is implied when a call to write contains a
newline character or a carriage return.
 write_through if True then a call to write is guaranteed not to be buffered.

 The TextIOWrapper is wrapped around a lower-level binary buffered I/O
stream, for example:
f = io.FileIO('data.txt')
text_stream = io.TextIOWrapper(br, 'utf-8’)
 StringIO This is an in-memory stream for text I/O. The initial value of the buffer held
by the StringIO object can be provided when the instance is created, for example:
in_memory_text_stream = io.StringIO('to be or not to be that is the question')
print('in_memory_text_stream', in_memory_text_stream)
print(in_memory_text_stream.getvalue())
in_memory_text_stream.close()
 This generates:
in_memory_text_stream <_io.StringIO object at 0x10fdfaee8>
to be or not to be that is the question

(viii) Stream Properties
 It is possible to query a stream to determine what types of operations it
supports. This can be done using the readable(), seekable() and
writeable() methods. For example:
f = io.FileIO('myfile.txt')
text_stream = io.TextIOWrapper(br, encoding='utf-8')
print('text_stream', text_stream)
print('text_stream.readable():', text_stream.readable())
print('text_stream.seekable()', text_stream.seekable())
print('text_stream.writeable()', text_stream.writable())
text_stream.close()

 The output from this code snippet is:
text_stream <_io.TextIOWrapper name='myfile.txt' encoding='utf-8'>
text_stream.readable(): True
text_stream.seekable() True
text_stream.writeable() False

(ix) Closing Streams
 All opened streams must be closed. However, you can close the top-level
stream, and this will automatically close lower-level streams, for example:
f = io.FileIO('data.txt’)
text_stream = io.TextIOWrapper(br, 'utf-8')
print(text_stream.read())
text_stream.close()

(x) Returning to the open() Function
 If streams are so good then why don’t you use them all the time?
 Well actually in Python 3 you do! The core open function (and indeed the
io.open() function) both return a stream object.
 The actual type of object returned depends on the file mode specified, whether
buffering is being used etc. For example:

import io
# Text stream
f1 = open('myfile.txt', mode='r', encoding='utf-8')
print(f1)
# Binary IO aka Buffered IO
f2 = open('myfile.dat', mode='rb')
print(f2)
f3 = open('myfile.dat', mode='wb')
print(f3)
# Raw IO aka Unbufferedf IO
f4 = open('starship.png', mode='rb', buffering=0)
print(f4)

 When this short example is run the output is
<_io.TextIOWrapper name='myfile.txt' mode='r' encoding='utf-8'>
<_io.BufferedReader name='myfile.dat'>
<_io.BufferedWriter name='myfile.dat'>
<_io.FileIO name='starship.png' mode='rb' closefd=True>
 As you can see from the output, four different types of object have been returned from
the open() function. The first is a TextIOWrapper, the second a BufferedReader,
the third a BufferedWriter and the final one is a FileIO object. This reflects the
differences in the parameters passed into the open () function.
 For example, f1 references a io.TextIOWrapper because it must encode
(convert) the input text into Unicode using the UTF-8 encoding scheme. While
f2 holds a io.BufferedReader because the mode indicates that we want to
read binary data while f3 holds a io.BufferedWriter because the mode used
indicates we want to write binary data.

 The final call to open returns a FileIO because we have indicated that we do
not want to buffer the data and thus we can use the lowest level of stream
object.
 In general, the following rules are applied to determine the type of object
returned based on the modes and encoding specified:

4. Working with CSV Files
(i) Introduction
(ii) CSV Files
(a) The CSV Writer Class
(b) The CSV Reader Class
(c) The CSV DictWriter Class
(d) The CSV DictReader Class

(i) Introduction
 This chapter introduces a module that supports the generation of CSV (or
Comma Separated Values) files.

(ii) CSV Files
 The CSV (Comma Separated Values) format is the most common import and
export format for spreadsheets and databases.
 However, CSV is not a precise standard with multiple different applications
having different conventions and specific standards.
 The Python csv module implements classes to read and write tabular data in
CSV format.
 As part of this it supports the concept of a dialect which is a CSV format used
by a specific application or suite of programs, for example, it supports an
Excel dialect.

 This allows programmers to say, “write this data in the format preferred by
Excel,” or “read data from this file which was generated by Excel,” without
knowing the precise details of the CSV format used by Excel.
 Programmers can also describe the CSV dialects understood by other
applications or define their own special-purpose CSV dialects.
 The csv module provides a range of functions including:
 csv.reader (csvfile, dialect='excel', **fmtparams) Returns
a reader object which will iterate over lines in the given csvfile. An optional
dialect parameter can be given. This may be an instance of a subclass of the
Dialect class or one of the strings returned by the list_dialects()
function. The other optional fmtparams keyword arguments can be given to
override individual formatting parameters in the current dialect.

 csv.writer (csvfile, dialect='excel', **fmtparams)Returns a
writer object responsible for converting the user’s data into delimited strings
on the given csvfile. An optional dialect parameter provided. The
fmtparams keyword arguments can be given to override individual
formatting parameters in the current dialect.
 csv.list_dialects() Return the names of all registered dialects. For
example on a Mac OS X the default list of dialects is ['excel', 'excel-tab', 'unix'].

(a) The CSV Writer Class
 A CSV Writer is obtained from the csv.writer()function. The csvwriter
supports two methods used to write data to the CSV file:
 csvwriter.writerow(row) Write the row parameter to the writer’s file
object, formatted according to the current dialect.
 csvwriter.writerows(rows) Write all elements in rows (an iterable of
row objects as described above) to the writer’s file object, formatted
according to the current dialect.
 Writer objects also have the following public attribute:
csvwriter.dialect A read-only description of the dialect in use by the
writer.

 The following program illustrates a simple use of the csv module which creates
a file called sample.csv.
 As we have not specified a dialect, the default ‘excel’ dialect will be used. The
writerow()method is used to write each comma separate list of strings to the
CSV file.
 Refer FileWriting.py

(b) The CSV Reader Class
 A CSV Reader object is obtained from the csv.reader() function. It
implements the iteration protocol.
 If a csv reader object is used with a for loop, then each time round the loop it
supplies the next row from the CSV file as a list, parsed according to the
current CSV dialect.
 Reader objects also have the following public attributes:
 csvreader.dialect A read-only description of the dialect in use by the
parser.
 csvreader.line_num The number of lines read from the source iterator.
This is not the same as the number of records returned, as records can span
multiple lines.

 The following provides a very simple example of reading a CSV file using a csv
reader object:
 Refer FileReading.py

(c) The CSV DictWriter Class
 In many cases the first row of a CSV file contains a set of names (or keys) that
define the fields within the rest of the CSV.
 That is the first row gives meaning to the columns and the data held in the rest
of the CSV file.
 It is therefore very useful to capture this information and to structure the data
written to a CSV file or loaded from a CSV file based on the keys in the first
row.
 The csv.DictWriter returns an object that can be used to write values into
the CSV file based on the use of such named columns. The file to be used with
the DictWriter is provided when the class is instantiated.

 Refer DictWriting.py
 Note that when the DictWriter is created a list of the keys must be provided
that are used for the columns in the CSV file.
 The method writeheader() is then used to write the header row out to the
CSV file.
 The method writerow() takes a dictionary object that has keys based on the
keys defined for the DictWriter. These are then used to write data out to the
CSV (note the order of the keys in the dictionary is not important).
 In the above example code, the result of this is that a new file called names.csv
is created which can be opened in Excel

(d) The CSV DictReader Class
 As well as the csv.DictWriter there is a csv.DictReader. The file to be used with
the DictReader is provided when the class is instantiated.
 As with the DictReader the DictWriter class takes a list of keys used to define
the columns in the CSV file. If the headings to be used for the first row can be
provided although this is optional (if a set of keys are not provided, then the
values in the first row of the CSV file will be used as the fieldnames).
 The DictReader class provides several useful features including the fieldnames
property that contains a list of the keys/headings for the CSV file as defined by
the first row of the file.

 Refer DictReading.py
 The DictReader class also implements the iteration protocol and thus it can be
used in a for loop in which each row (after the first row) is returned in turn as a
dictionary.
 The dictionary object representing each row can then be used to access each
column value based on the keys defined in the first row.

5. Working with Excel Files
(i) Introduction
(ii) Excel Files
(iii) The Openpyxl. Workbook Class
(iv) The Openpyxl. WorkSheet Objects
(v) Working with Cells
(vi) Sample Excel File Creation Application
(vii) Loading a Workbook from an Excel File

(i) Introduction
 The openpyxl module can be used when working with Excel files.
 Excel is a software application developed by Microsoft that allows users to
work with spreadsheets.
 It is a very widely used tool and files using the Excel file format are commonly
encountered within many organizations.
 It is in effect the industry standard for spreadsheets and as such is a very
useful tool to have in the developers toolbox.

(ii) Excel Files
 Although CSV files are a convenient and simple way to handle data; it is very
common to need to be able to read or write Excel files directly.
 To this end there are several libraries available in Python for this purpose.
 One widely used library is the OpenPyXL library.
 This library was originally written to support access to Excel 2010 files. It is an
open-source project and is well documented.

 The OpenPyXL library provides facilities for
 reading and writing Excel workbooks,
 creating/accessing Excel worksheets,
 creating Excel formulas,
 creating graphs (with support from additional modules).
 As OpenPyXL is not part of the standard Python distribution you need to install
the library yourself using a tool such as Anaconda or pip (e.g. pip install
openpyxl).
 Alternatively, if you are using PyCharm you will be able to add the OpenPyXL
library to your project.

(iii) The Openpyxl. Workbook Class
 The key element in the OpenPyXL library is the Workbook class. This can be
imported from the module:
from openpyxl import Workbook
 A new instance of the (in memory) Workbook can be created using the
Workbook class (note at this point it is purely a structure within the Python
program and must be saved before an actual Excel file is created).
wb=workbook()

(iv) The Openpyxl. WorkSheet Objects
 A workbook is always created with at least one worksheet. You can get hold of
the currently active worksheet using the Workbook.active property:
ws = wb.active
 You can create additional worksheets using the workbooks’
create_sheet() method:
ws = wb.create_sheet('Mysheet’)
 You can access or update the title of the worksheet using the title property:
ws.title = 'New Title'

 The background colour of the tab holding this title is white by default. You can
change this providing an RRGGBB colour code to the worksheet.
sheet_properties.tabColor attribute, for example:
ws.sheet_properties.tabColor = "1072BA“

(v) Working with Cells
 It is possible to access the cells within a worksheet. A cell can be accessed
directly as keys on the worksheet, for example:
ws['A1'] = 42
or
cell = ws['A1’]
 This returns a cell object; you can obtain the value of the cell using the value
property, for example
print(cell.value)
 There is also the Worksheet.cell() method. This provides access to cells using
row and column notation:
d = ws.cell(row=4, column=2, value=10)

 A row of values can also be added at the current position within the Excel file
using append:
ws.append([1, 2, 3])
 This will add a row to the Excel file containing 1, 2, and 3.
 Ranges of cells can be accessed using slicing:
cell_range = ws['A1':'C2’]
 Ranges of rows or columns can also be obtained:
col = ws['C’]
col_range = ws['C:D’]
row10 = ws[10]
row_range = ws[5:10]

 The value of a cell can also be an Excel formula such as
ws['A3'] = '=SUM(A1, A2)’
 A workbook is actually only a structure in memory; it must be saved to a file for
permanent storage. These workbooks can be saved using the save() method.
This method takes a filename and writes the Workbook out in Excel format.
workbook = Workbook()
...
workbook.save('balances.xlsx’)

(vi) Sample Excel File Creation Application
 The following simple application creates a Workbook with two worksheets. It
also contains a simple Excel formula that sums the values held into other cells:
 Refer WorkBookCreation.py

(vii) Loading a Workbook from an Excel File
 Of course, in many cases it is necessary not just to create Excel files for data
export but also to import data from an existing Excel file. This can be done
using the OpenPyXL load_workbook() function.
 This function opens the specified Excel file (in read only mode by default) and
returns a Workbook object.
from openpyxl import load_workbook
workbook = load_workbook(filename='sample.xlsx’)
 You can now access a list of sheets, their names, obtain the currently active
sheet etc. using properties provided by the workbook object:

 workbook.active returns the active worksheet object.
 workbook.sheetnames returns the names (strings) of the worksheets in this
workbook.
 workbook.worksheets returns a list of worksheet objects.
 Refer LoadWorkBook.py

1. Introduction to Databases
2. Python DB-API
3. PyMySQL Module
DATABASE ACCESS

1. Introduction to Databases
(i) Introduction
(ii) What Is a Database?
(a) Data Relationships
(b) The Database Schema
(iii) SQL and Databases
(iv) Data Manipulation Language
(v) Transactions in Databases

(i) Introduction
 There are several different types of database system in common use today
including Object databases, NoSQL databases and (probably the most
common) Relational Databases.
 This chapter focusses on Relational Databases as typified by database systems
such as Oracle, Microsoft SQL Server and MySQL. The database we will use in
is MySQL.

(ii) What Is a Database?
 A database is essentially a way to store and retrieve data.
 Typically, there is some form of query language used with the database to help
select the information to retrieve such as SQL or Structured Query Language.
 In most cases there is a structure defined that is used to hold the data
(although this is not true of the newer NoSQL or non-relational unstructured
databases such as CouchDB or MongoDB).
 In a Relational Database the data is held in tables, where the columns define
the properties or attributes of the data and each row defines the actual values
being held, for example:

 In this diagram there is a table called students; it is being used to hold
information about students attending a meeting. The table has 5 attributes (or
columns) defined for id, name, surname, subject and email.
 In this case, the id is probably what is known as a primary key. The primary key
is a property that is used to uniquely identify the student row; it cannot be
omitted and must be unique (within the table).
 Obviously, names and subjects may well be duplicated as there may be more
than one student studying Animation or Games and students may have the
same first name or surname.
 It is probable that the email column is also unique as students probably don’t
share an email address but again this may not necessarily be the case.

 You might at this point wonder why the data in a Relational Database is called
relational and not tables or tabular?
 The reason is because of a topic known as relational algebra that underpins
Relational Database theory.
 Relational Algebra takes its name from the mathematical concept known as a
relation.
 However, for the purposes of this chapter you don’t need to worry about this
and just need to remember that data is held in tables.

(a) Data Relationships
 When the data held in one table has a link or relationship to data held in
another table then an index or key is used to link the values in one table to
another.
 This is illustrated below for a table of addresses and a table of people who live
in that address. This shows for example, that ‘Phoebe Gates’ lives at address
‘addr2’ which is 12 Queen Street, Bristol, BS42 6YY.

 This is an example of a many to one (often written as many:1) relationship; that
is there are many people who can live at one address (in the above Adam Smith
also lives at address ‘addr2’). In Relational Databases there can be several
different types of relationship such as:
 one:one where only one row in one table references one and only one row in
another table. An example of a one-to-one relationship might be from a
person to an order for a unique piece of jewellery.
 one:many this is the same as the above address example, however in this case
the direction of the relationship is reversed (that is to say that one address in
the addresses table can reference multiple persons in the people table).
 many:many This is where many rows in one table may reference many rows
in a second table. For example, many students may take a particular class and
a student may take many classes. This relationship usually involves an
intermediate (join) table to hold the associations between the rows.

(b) The Database Schema
 The structure of a Relational Database is defined using a Data Definition
Language or Data Description Language (a DDL).
 Typically, the syntax of such a language is limited to the semantics (meaning)
required to define the structure of the tables. This structure is known as the
database schema. Typically, the DDL has commands such as CREATE TABLE,
DROP TABLE (to delete a table) and ALTER TABLE (to modify the structure of
an existing table).
 Many tools provided with a database allow you to define the structure of the
database without getting too bound up in the syntax of the DDL; however, it is
useful to be aware of it and to understand that the database can be created in
this way.

 For example, we will use the MySQL database in this chapter. The MySQL
Workbench is a tool that allows you to work with MySQL databases to manage
and query the data held within a particular database instance.
 As an example, within the MySQL Workbench we can create a new table using a
menu option on a database:

 Using this we can interactively define the columns that will comprise the table:

 Here each column name, its type and whether it is the primary key (PK), not
empty (or Not Null NN) or unique (UQ) have been specified. When the changes
are applied, the tool also shows you the DDL that will be used to create the
database:

 When this is applied a new table is created in the database as shown below:

 The tool also allows us to populate data into the table; this is done by entering
data into a grid and hitting apply as shown below:

(iii) SQL and Databases
 We can now use query languages to identify and return data held in the
database often using specific criteria.
 For example, let us say we want to return all the people who have the surname
Jones from the following table:

 We can do this by specifying that data should be returned where the surname
equals ‘Jones’; in SQL this would look like:
SELECT * FROM students where surname='Jones’;
 The above SELECT statement states that all the properties (columns or
attributes) in a row in the table students are to be returned where the surname
equals ‘Jones’. The result is that two rows are returned:

 Note we need to specify the table we are interested in and what data we want to
return (the ‘*’ after the select indicated we want all the data). If we were only
interested in their first names then we could use:
SELECT name FROM students where surname='Jones’;
 This would return only the names of the students:

(iv) Data Manipulation Language
 Data can also be inserted into a table or existing data in a table can be updated.
This is done using the Data Manipulation Language (DML).
 For example, to insert data into a table we merely need to write an INSERT
SQL statement providing the values to be added and how they map to the
columns in the table:
INSERT INTO 'students' ('id', 'name', 'surname', 'subject','email’)
VALUES ('6', 'James', 'Andrews', 'Games','ja@my.com’);

 This would add the row 6 to the table students with the result that the table
would now have an additional row:
Updating an existing row is a little more complicated as it is first necessary to
identify the row to be updated and then the data to modify. Thus, an UPDATE
statement includes a where clause to ensure the correct row is modified:
UPDATE 'students' SET 'email'='grj@my.com' WHERE 'id'='2';

 The effect of this code is that the second row in the students table is modified
with the new email address:

(v) Transactions in Databases
 Another important concept within a database is that of a Transaction. A
Transaction represents a unit of work performed within a database
management system (or similar system) against a database instance, and is
independent of any other transaction.
 Transactions in a database environment have two main purposes
 To provide a unit of work that allows recovery from failures and keeps a
database consistent even in cases of system failure, when execution stops
(completely or partially). This is because either all the operations within a
transaction are performed or none of them are. Thus, if one operation
causes an error then all the changes being made by the transaction thus far
are rolled back and none of them will have been made.
 To provide isolation between programs accessing a database concurrently.
This means that the work being done by one program will not interact with
another programs work.

 A database transaction, by definition, must be atomic, consistent, isolated and
durable:
 Atomic This indicates that a transaction represents an atomic unit of work;
that is either all the operations in the transaction are performed or none of
them are performed.
 Consistent Once completed the transaction must leave the data in a
consistent state with any data constraints met (such as a row in one table
must not reference an non-existent row in another table in a one to many
relationship etc.).
 Isolated This relates to the changes being made by concurrent transactions;
these changes must be isolated from each other. That is, one transaction
cannot see the changes being made by another transaction until the second
transaction completes and all changes are permanently saved into the
database.
 Durable This means that once a transaction completes then the changes it has
made are permanently stored into the database (until some future transaction
modifies that data).

 Database practitioners often refer to these properties of database transactions
using the acronym ACID (for Atomic, Consistent, Isolated, Durable).
 Not all databases support transactions although all commercial, production
quality databases such as Oracle, Microsoft SQL Server and MySQL, do support
transactions.

2. Python DB-API
(i) Accessing a Database from Python
(ii) The DB-API
(a) The Connect Function
(b) The Connection Object
(c) The Cursor Object
(d) Mappings from Database Types to Python Types
(e) Generating Errors
(f) Row Descriptions
(iii) Transactions in PyMySQL

(i) Accessing a Database from Python
 The standard for accessing a database in Python is the Python DB-API.
 This specifies a set of standard interfaces for modules that wish to allow
Python to access a specific database.
 The standard is described in PEP 249 (https://www.python.org/dev/peps/pep-
0249)—a PEP is a Python Enhancement Proposal.
 Almost all Python database access modules adhere to this standard.

 This means that if you are moving from one database to another, or attempting
to port a Python program from one database to another, then the APIs you
encounter should be very similar (although the SQL processed by different
database can also differ).
 There are modules available for most common databases such as MySQL,
Oracle, Microsoft SQL Server etc.

(ii) The DB-API
 There are several key elements to the DB_API these are:
 The connect function. The connect() function that is used to connect to a database
and returns a Connection Object.
 Connection Objects. Within the DB-API access to a database is achieved through
connection objects. These connection objects provide access to cursor objects.
 Cursor objects are used to execute SQL statements on the database.
 The result of an execution. These are the results that can be fetched as a sequence
of sequences (such a tuple of tuples). The standard can thus be used to select, insert
or update information in the database.

 These elements are illustrated below:

 The standard specifies a set of functions and objects to be used to connect to a
database.
 These include the connection function, the Connection Object and the Cursor
object.

(a) The Connect Function
 The connection function is defined as:
connect(parameters...)
 It is used to make the initial connection to the database. The connection
returns a Connection Object.
 The parameters required by the connection function are database dependent.

(b) The Connection Object
 The Connection Object is returned by the connect() function. The
Connection object provides several methods including:
 close()used to close the connection once you no longer need it. The
connection will be unusable from this point onwards.
 commit() used to commit a pending transaction.
 rollback() used to rollback all the changes made to the database since
the last transaction commit (optional as not all databases provide
transaction support).
 cursor() returns a new Cursor object to use with the connection.

(c) The Cursor Object
 The Cursor object is returned from the connection.cusor()method. A
Cursor Object represents a database cursor, which is used to manage the
context of a fetch operation or the execution of a database command. Cursors
support a variety of attributes and methods:
 cursor.execute(operation, parameters) Prepare and execute a
database operation (such as a query statement or an update command).
Parameters may be provided as a sequence or mapping and will be bound
to variables in the operation. Variables are specified in a database-specific
notation.
 cursor.rowcount a read-only attribute providing the number of rows
that the last cursor.execute() call returned (for select style statements) or
affected (for update or insert style statements).

 cursor.description a read only attribute providing information on the
columns present in any results returned from a SELECT operation.
 cursor.close()closes the cursor. From this point on the cursor will not be
usable.
 In addition, the Cursor object also provides several fetch style methods.
 These methods are used to return the results of a database query.
 The data returned is made up of a sequence of sequences (such as a tuple of
tuples) where each inner sequence represents a single row returned by the
SELECT statement. The fetch methods defined by the standard are:

 cursor.fetchone()Fetch the next row of a query result set, returning a
single sequence, or None when no more data is available.
 cursor.fetchall()Fetch all (remaining) rows of a query result, returning
them as a sequence of sequences.
 cursor.fetchman(size)Fetch the next set of rows of a query result,
returning a sequence of sequences (e.g. a tuple of tuples). An empty sequence
is returned when no more rows are available. The number of rows to fetch
per call is specified by the parameter.

(d) Mappings from Database Types to Python Types
 The DB-API standard also specifies a set of mappings from the types used in a
database to the types used in Python. For a full listing see the DB-API
standard itself but the key mappings include:

(e) Generating Errors
 The standard also specifies a set of Exceptions that can be thrown in different
situations. These are presented below and in the following table:

 The above diagram illustrates the inheritance hierarchy for the errors and
warning associated with the standard.
 Note that the DB-API Warning and Error both extend the Exception class
from standard Python; however, depending on the specific implementation
there may be one or more additional classes in the hierarchy between these
classes.
 For example, in the PyMySQL module there is a MySQLError class that extends
Exception and is then extended by both Warning and Error.
 Also note that Warning and Error have no relationship with each other. This
is because Warnings are not considered Errors and thus have a separate class
hierarchies. However, the Error is the root class for all database Error classes.
 A description of each Warning or Error class is provided below.

(f) Row Descriptions
 The Cursor object has an attribute description that provides a sequence of
sequences; each sub sequence provides a description of one of the attributes of
the data returned by a SELECT statement.
 The sequence describing the attribute is made up of up to seven items, these
include:
 name representing the name of the attribute,
 type_code which indicates what Python type this attribute has been
mapped to,
 display_size the size used to display the attribute,
 internal_size the size used internally to represent the value,

 precision if a real numeric value the precision supported by the attribute,
 scale indicates the scale of the attribute,
 null_ok this indicates whether null values are acceptable for this attribute.
 The first two items (name and type_code) are mandatory, the other five are
optional and are set to None if no meaningful values can be provided.

(iii) Transactions in PyMySQL
 Transactions are managed in PyMySQL via the database connection object.
This object provides the following method:
 connection.commit() this causes the current transaction to commit all
the changes made permanently to the database. A new transaction is then
started.
 connection.rollback() this causes all changes that have been made
so far (but not permanently stored into the database i.e. Not committed) to
be removed. A new transaction is then started.
 The standard does not specify how a database interface should manage
turning on and off transaction (not least because not all databases support
transactions).

 However, MySQL does support transactions and can work in two modes; one
supports the use of transactions as already described; the other uses an auto
commit mode.
 In auto commit mode each command sent to the database (whether a SELECT
statement or an INSERT/UPDATE statement) is treated as an independent
transaction and any changes are automatically committed at the end of the
statement. This auto commit mode can be turned on in PyMySQL using:
 connection.autocommit(True)turn on autocommit (False to turn off
auto commit which is the default).
 Other associated methods include
 connection.get_autocommit()which returns a boolean indicating
whether auto commit is turned on or not.
 connection.begin()to explicitly begin a new transaction.

3. PyMySQL Module
(i) The PyMySQL Module
(ii) Working with the PyMySQL Module
(a) Importing the Module
(b) Connect to the Database
(c) Obtaining the Cursor Object
(d) Using the Cursor Object
(e) Obtaining Information About the Results
(f) Fetching Results
(g) Close the Connection

(iii) Complete PyMySQL Query Example
(iv) Inserting Data to the Database
(v) Updating Data in the Database
(vi) Deleting Data in the Database
(vii) Creating Tables

(i) PyMySQL Module
 The PyMySQL module provides access to a MySQL database from Python. It
implements the Python DB-API v 2.0.
 This module is a pure Python database interface implementation meaning that
it is portable across different operating systems; this is notable because some
database interface modules are merely wrappers around other (native)
implementations that may or may not be available on different operating
systems.
 For example, a native Linux based database interface module may not be
available for the Windows operating system. If you are never going to switch
between different operating systems, then this is not a problem of course.

 To use the PyMySQL module you will need to install it on your computer. This
will involve using a tool such as Anaconda or adding it to your PyCharm
project.
 You can also use pip to install it:
> pip install PyMySQL

(ii) Working with the PyMySQL Module
 To use the PyMySQL module to access a database you will need to follow these
steps.
1. Import the module.
2. Make a connection to the host machine running the database and to the
database you are using.
3. Obtain a cursor object from the connection object.
4. Execute some SQL using the cursor.execute() method.
5. Fetch the result(s) of the SQL using the cursor object (e.g. fetchall, fetchmany or
fetchone).
6. Close the database connection.
These steps are essentially boiler plate, code that is you will use them whenever
you access a database via PyMySQL (or indeed any DB-API compliant module).

(a) Importing the Module
 As the PyMySQL module is not one of the built-in modules provided by
default with Python you will need to import the module into your code, for
example using
import pymysql
 Be careful with the case used here as the module name is pymysql in the code
(if you try to import PyMySQL Python will not find it!).

(b) Connect to the Database
 Each database module will have their own specifics for connecting to the
database server; these usually involve specifying the machine that the
database is running on (as databases can be quiet resource intensive, they are
often run on a separate physical computer), the user to use for the connection
and any security information required such as a password and the database
instance to connect to.
 In most cases a database is looked after by a database management system (a
DBMS) that can manage multiple database instances and it is therefore
necessary to specify which database instance you are interested in.

 For MySQL, the MySQL database server is a DBMS that can indeed look after
multiple database instances. The pymysql.connect function thus requires
the following information when connecting to the database is:
 The name of the machine hosting the MySQL database server e.g.
dbserver. mydomain.com. If you want to connect to the same machine as
your Python program is running on, then you can use localhost. This is a
special name reserved for the local machine and avoids you needing to worry
about the name of your local computer.
 The user name to use for the connection. Most databases limit access to their
databases to named users. These are not necessary users such as humans that
log into a system but rather entities that are allowed to connect to the
database and perform certain operations. For example, one user may only be
able to read data in the database where as another user is allowed to insert
new data into the database. These users are authenticated by requiring them
to provide a password.

 The password for the user.
 The database instance to connect to. As mentioned in the previous chapter a
Database Management System (DMS) can manage multiple database
instances and thus it is necessary to say which database instance you are
interested in.
For example:
# Open database connection
connection = pymysql.connect('localhost','username','password','unidatabase’)
 In this case the machine we are connecting to is ‘localhost’ (that is the same
machine as the Python program itself is running on), the user is represented by
‘username’ and ‘password’ and the database instance of interest is called ‘uni-
database’.
 This returns a Connection object as per the DB-API standard.

(c) Obtaining the Cursor Object
 You can obtain the cursor object from the connection using the cursor()
method:
# prepare a cursor object using cursor() method
cursor = connection.cursor()

(d) Using the Cursor Object
 Once you have obtained the cursor object you can use it to execute an SQL
query or a DML insert, update or delete statement.
 The following example uses a simple select statement to select all the
attributes in the students table for all rows currently stored in the students
table:
# execute SQL query using execute() method.
cursor.execute('SELECT * FROM students')

 Note that this method executes the SELECT statement but does not return the
set of results directly.
 Instead, the execute method returns an integer indicating the number of rows
either affected by the modification or returned as part of the query.
 In the case of a SELECT statement the number returned can be used to
determine which type of fetch method to use.

(e) Obtaining Information About the Results
 The Cursor Object can also be used to obtain information about the results to
be fetched such as how many rows there are in the results and what the type is
of each attribute in the results:
 cusor.rowcount() this is a read-only property that indicates the
number of rows returned for a SELECT statement or rows affected for a
UPDATE or INSERT statement.
 cursor.description() this is a read-only property that provides a
description of each attribute in the results set. Each description provides the
name of the attribute and an indication of the type (via a type_code) as well
as further information on whether the value can be null or not and for
numbers scale, precision and size information.
 An example of using these two properties is given below:
print('cursor.rowcount', cursor.rowcount)
print('cursor.description', cursor.description)

(f) Fetching Results
 Now that a successful SELECT statement has been run against the database,
we can fetch the results.
 The results are returned as a tuple of tuples. As mentioned in the last chapter
there are several different fetch options available including
fetchone(),fetchmany(size) and fetchall().
 In the following example we use the fetchall() option as we know that
there are only up to six rows that can be returned.
# Fetch all the rows and then iterate over the data
data = cursor.fetchall()
for row in data:
print('row:', row)

 In this case we loop through each tuple within the data collection and print that
row out. However, we could just as easily have extracted the information in the
tuple into individual elements.
 These elements could then be used to construct an object that could then be
processed within an application, for example:
for row in data:
id, name, surname, subject, email = row
student = Student(id, name, surname, subject, email)
print(student)

(g) Close the Connection
 Once you have finished with the database connection it should be closed.
# disconnect from server
connection.close()

(iii) Complete PyMySQL Query Example
 A complete listing illustrating connecting up to the database, running a
SELECT statement and printing out the results using a Student class is given
below:
 Refer Student.py

(iv) Inserting Data to the Database
 As well as reading data from a database many applications also need to add
new data to the database. This is done via the DML (Data Manipulation
Language) INSERT statement.
 The process for this is very similar to running a query against the database
using a SELECT statement; that is, you need to make a connection, obtain a
cursor object and execute the statement.
 The one difference here is that you do not need to fetch the results.
 Refer Insert.py

 There are a couple of points to note about this code example. The first is that we
have used the double quotes around the string defining the INSERT
command— this is because a double quotes string allows us to include single
quotes within that string. This is necessary as we need to quote any string
values passed to the database (such as ‘Gryff’).
 The second thing to note is that by default the PyMySQL database interface
requires the programmer to decide when to commit or rollback a transaction. A
transaction was introduced in the last chapter as an atomic unit of work that
must either be completed or as a whole or rollback so that no changes are made.
 However, the way in which we indicate that a transaction is completed is by
calling the commit() method on the database connection. In turn we can indicate
that we want to rollback the current transaction by calling rollback(). In either
case, once the method has been invoked a new transaction is started for any
further database activity.

 In the above code we have used a try block to ensure that if everything
succeeds, we will commit the changes made, but if an exception is thrown (of
any kind) we will rollback the transaction—this is a common pattern.

(v) Updating Data in the Database
 If we are able to insert new data into the database, we may also want to
update the data in a database, for example to correct some information. This is
done using the UPDATE statement which must indicate which existing row is
being updated as well as what the new data should be.
 Refer Update.py

(vi) Deleting Data in the Database
 Finally, it is also possible to delete data from a database, for example if a
student leaves their course. This follows the same format as the previous two
examples with the difference that the DELETE statement is used instead:.
 Refer Delete.py

(vii) Creating Tables
 It is not just data that you can add to a database; if you wish you can
programmatically create new tables to be used with an application. This
process follows exactly the same pattern as those used for INSERT, UPDATE
and DELETE. The only difference is that the command sent to the database
contains a CREATE statement with a description of the table to be created.
This is illustrated below:
 Refer Create.py

1. The Data Structure of Pandas
(i) Series
(ii) DataFrame
(iii) Panel
2. Inserting and Exporting Data
(i) CSV
(ii) XLS
(iii) JSON
(iv) Database
DATA ANALYSIS WITH PANDAS

Empowering Data Analysis with Pandas
 The pandas library was developed by Wes McKinny when he was working at
AQR Capital Management. He wanted a tool that was flexible enough to
perform quantitative analysis on financial data. Later, Chang She joined him
and helped develop the package further.
 The pandas library is an open source Python library, specially designed for
data analysis. It has been built on NumPy and makes it easy to handle data.
NumPy is a fairly low-level tool that handles matrices really well.
 The pandas library brings the richness of R in the world of Python to handle
data. It’s has efficient data structures to process data, perform fast joins, and
read data from various sources, to name a few.

1. The Data Structure of Pandas
 The pandas library essentially has three data structures:
(i) Series
(ii) DataFrame
(iii) Panel

(i) Series
 Series is a one-dimensional array, which can hold any type of data, such as
integers, floats, strings, and Python objects too. A series can be created by
calling the following:
>>> import pandas as pd
>>> pd.Series(np.random.randn(5))
 The random.randn parameter is part of the NumPy package and it generates
random numbers.
 The series function creates a pandas series that consists of an index, which is
the first column, and the second column consists of random values. At the
bottom of the output is the datatype of the series.

 The index of the series can be customized by calling the following:
>>> pd.Series(np.random.randn(5),
index=['a', 'b', 'c', 'd', 'e'])
 A series can be derived from a Python dict too:
>>> d = {'A': 10, 'B': 20, 'C': 30}
>>> pd.Series(d)
 Refer SeriesEx.py

(ii) DataFrame
 DataFrame is a 2D data structure with columns that can be of different
datatypes. It can be seen as a table. A DataFrame can be formed from the
following data structures:
 A NumPy array
 Lists
 Dicts
 Series
 A 2D NumPy array

 A DataFrame can be created from a dict of series by calling the following
commands:
>>> d = {'c1': pd.Series(['A', 'B', 'C']),
'c2': pd.Series([1, 2., 3., 4.])}
>>> df = pd.DataFrame(d)
>>> df
 The DataFrame can be created using a dict of lists too:
>>> d = {'c1': ['A', 'B', 'C', 'D'],
'c2': [1, 2.0, 3.0, 4.0]}
>>> df = pd.DataFrame(d)
>>> print df
 Refer DataFrameEx.py

(iii) Panel
 A Panel is a data structure that handles 3D data. The following command is an
example of panel data:
>>> d = {'Item1': pd.DataFrame(np.random.randn(4, 3)),
'Item2': pd.DataFrame(np.random.randn(4, 2))}
>>> pd.Panel(d)

2. Inserting and exporting data
 The data is stored in various forms, such as CSV, TSV, databases, and so on.
 The pandas library makes it convenient to read data from these formats or to
export to these formats.

(i) CSV
 To read data from a .csv file, the following read_csv function can be used:
 Refer CSVEx.py
 The read_csv function takes the path of the .csv file to input the data.
 The command after this prints the first three rows of the first_name column in
the data.

 To write a data to the .csv file, the following to_csv function can be used
 The DataFrame is written to a .csv file by using the to_csv method. The
path and the filename where the file needs to be created should be mentioned.
 Refer CSVWriteEx.py

(ii) XLS
 In addition to the pandas package, the xlrd package needs to be installed for
pandas to read the data from an Excel file:
 Refer XLReadEx.py
 The preceding function is similar to the CSV reading command. To write to an
Excel file, the xlwt package needs to be installed.
 Refer XLWriteEx.py

(iii) JSON
 To read the data from a JSON file, Python's standard json package can be used.
The following commands help in reading the file:
 Refer JSONEx.py
 In the preceding command, the open()function opens a connection to the file.
The json.load()function loads the data into Python. The
json_data.close() function closes the connection to the file.
 The pandas library also provides a function to read the JSON file, which can
be accessed using pd.read_json().

(iv) Database
 To read data from a database, the following function can be used:
>>> pd.read_sql_table(table_name, con)
 The preceding command generates a DataFrame. If a table name and an
SQLAlchemy engine are given, they return a DataFrame. This function does
not support the DBAPI connection. The following are the description of the
parameters used:
 table_name: This refers to the name of the SQL table in a database
 con: This refers to the SQLAlchemy engine

 The following command reads SQL query into a DataFrame:
>>> pd.read_sql_query(sql, con)
 The following are the description of the parameters used:
 sql: This refers to the SQL query that is to be executed
 con: This refers to the SQLAlchemy engine

References
1. Hunt, J. (2019). Advanced Guide to Python 3 Programming. Springer.
2. Madhavan, S. (2015). Mastering Python for Data Science. Packt Publishing Ltd.

File Input/output, Database Access, Data Analysis with Pandas

More Related Content

Similar to File Input/output, Database Access, Data Analysis with Pandas (20)

More from Prabu U (20)

Recently uploaded (20)

File Input/output, Database Access, Data Analysis with Pandas