Database programming

Database
Programming
•Databases and Python,
•The Python DB-API,
•Python and ORMs,
•Non- Relational Databases

Introduction
 From a construction firm to a stock exchange, every organisation depends on large
databases. These are essentially collections of tables, and’ connected with each
other through columns.
 These database systems support SQL, the Structured Query Language, which is
used to create, access and manipulate the data.
 SQL is used to access data, and also to create and exploit the relationships between
the stored data.
 Additionally, these databases support database normalisation rules for avoiding
redundancy of data.
 The Python programming language has powerful features for database
programming.
 Python supports various databases like MySQL, Oracle, Sybase, PostgreSQL, etc.
 Python also supports Data Definition Language (DDL), Data Manipulation Language
(DML) and Data Query Statements.
 For database programming, the Python DB API is a widely used module that
provides a database application programming interface.

Benefits of Python for database
programming
 There are many good reasons to use Python for
programming database applications:
 Programming in Python is maybe more efficient and faster
compared to other languages.
 Python is famous for its portability.
 It is platform independent.
 Python supports SQL cursors.
 In many programming languages, the application developer needs
to take care of the open and closed connections of the database,
to avoid further exceptions and errors. In Python, these
connections are taken care of.
 Python supports relational database systems.
 Python database APIs are compatible with various databases, so it
is very easy to migrate and port database application interfaces.

DB-API (SQL-API) for Python
 Python DB-API is independent of any database engine, which
enables you to write Python scripts to access any
database engine.
 The Python DB API implementation for MySQL is MySQLdb.
 For PostgreSQL, it supports psycopg, PyGresQL and pyPgSQL
modules.
 DB-API implementations for Oracle are dc_oracle2 and
cx_oracle.
 Pydb2 is the DB-API implementation for DB2.
 Python’s DB-API consists of connection objects, cursor objects,
standard exceptions and some other module contents.

 The DB API provides a minimal standard for working
with databases using Python structures and syntax
wherever possible. This API includes the following:
 Importing the API module.
 Acquiring a connection with the database.
 Issuing SQL statements and stored procedures.
 Closing the connection

Functions and attributes
 connect(parameters...)Constructor for creating a
connection to the database.
 Returns a Connection Object.
 Parameters are the same as for the MySQL C API.
 In addition, there are a few additional keywords that
correspond to what you would pass mysql_options() before
connecting.
 Note that some parameters must be specified as keyword
arguments!
 The default value for each parameter is NULL or zero, as
appropriate. The important parameters are:

 Host: name of host to connect to. Default: use the local host via a
UNIX socket (where applicable)
 User: user to authenticate as. Default: current effective user.
 Passwd: password to authenticate with. Default: no password.
 Db: database to use. Default: no default database.
 Port: TCP port of MySQL server. Default: standard port (3306).
 unix_socket: location of UNIX socket. Default: use default
location or TCP for remote hosts.
 Conv: type conversion dictionary. Default: a copy of
MySQLdb.converters.conversions
 Compress :Enable protocol compression. Default: no
compression.

 connect_timeout: Abort if connect is not completed within given number of
seconds. Default: no timeout (?)
 named_pipe: Use a named pipe (Windows). Default: don't.
 init_command: Initial command to issue to server upon connection. Default:
Nothing.
 read_default_file: MySQL configuration file to read; see the MySQL
documentation for mysql_options().
 read_default_group: Default group to read; see the MySQL documentation for
mysql_options().
 Cursorclass: cursor class that cursor() uses, unless overridden. Default:
MySQLdb.cursors.Cursor. This must be a keyword parameter.
 Unicode: If set, CHAR and VARCHAR columns are returned as Unicode strings,
using the specified character set. None means to use a default encoding.

Connection Objects
 Connection objects are returned by the connect() function.
 commit() :If the database and the tables support transactions,
this commits the current transaction; otherwise this method
successfully does nothing.
 rollback() :If the database and tables support transactions, this
rolls back (cancels) the current transaction; otherwise a
NotSupportedError is raised.
 Compatibility note: The older MySQLmodule defines this
method, which sucessfully does nothing. This is dangerous
behavior, as a successful rollback indicates that the current
transaction was backed out, which is not true, and fails to notify
the programmer that the database now needs to be cleaned up
by other means.

 cursor([cursorclass]) : MySQL does not support cursors;
however, cursors are easily emulated.
 You can supply an alternative cursor class as an optional
parameter.
 If this is not present, it defaults to the value given when creating
the connection object, or the standard Cursor class. Also see the
additional supplied cursor classes in the usage section.
 begin() :Explicitly start a transaction.
 Normally you do not need to use this: Executing a query implicitly
starts a new transaction if one is not in progress.
 If AUTOCOMMIT is on, you can use begin() to temporarily turn
it off. AUTOCOMMIT will resume after the next commit() or
rollback.

Cursor Objects
 close() :Closes the cursor. Future operations raise ProgrammingError. If you are
using server-side cursors, it is very important to close the cursor when you are
done with it and before creating a new one.
 insert_id() : Returns the last AUTO_INCREMENT field value inserted into the
database. (Non-standard)
 info() : Returns some information about the last query. Normally you don't need
to check this. With the default cursor, any MySQL warnings cause Warning to be
raised. If you are using a cursor class without warnings, then you might want to use
this. See the MySQL docs for mysql_info(). (Non-standard)
 setinputsizes(): Does nothing, successfully.
 setoutputsizes(): Does nothing, successfully.
 nextset() :Advances the cursor to the next result set, discarding the remaining
rows in the current result set. If there are no additional result sets, it returns
None; otherwise it returns a true value.

Cursor Attributes
 Cursor.description details regarding the result columns
 This read-only attribute is a sequence of 7-item named tuples.
 Each of these named tuples contains information describing one result column:
 name
 type_code
 display_size
 internal_size
 precision
 scale
 null_ok
 The values for precision and scale are only set for numeric types. The values for
display_size and null_ok are always None.
 This attribute will be None for operations that do not return rows or if the
cursor has not had an operation invoked via the Cursor.execute() or
Cursor.executemany() method yet.

rowcount – number of rows of the
result
 Cursor.rowcount
 This read-only attribute specifies the number of rows
that the last Cursor.execute() or Cursor.executemany()
call produced (for DQL statements like SELECT) or
affected (for DML statements like UPDATE or INSERT).
 It is also set by the Cursor.copy_from() and
:meth’:Cursor.copy_to methods.
 The attribute is -1 in case no such method call has been
performed on the cursor or the rowcount of the last
operation cannot be determined by the interface.

Methods of Cursor
 close – close the cursor
 Close the cursor now (rather than whenever it is
deleted)
 Return type: None
 The cursor will be unusable from this point forward; an
Error (or subclass) exception will be raised if any
operation is attempted with the cursor.

execute – execute a database
operation
 Cursor.execute(operation[, parameters]) Prepare and
execute a database operation (query or command)
 Parameters: operation (str) – the database operation
 parameters – a sequence or mapping of
parameters
 Returns:the cursor, so you can chain commands

executemany – execute many similar
database operations
 Cursor.executemany(operation[, seq_of_parameters])
Prepare and execute many similar database operations
(queries or commands)
 Parameters: operation (str) – the database operation
 seq_of_parameters – a sequence or
mapping of parameter tuples or mappings
 Returns: the cursor, so you can chain commands
 Prepare a database operation (query or command) and
then execute it against all parameter tuples or mappings
found in the sequence seq_of_parameters.

data = [
('Jane', date(2005, 2, 12)),
('Joe', date(2006, 5, 23)),
('John', date(2010, 10, 3)),
]
 stmt = "INSERT INTO employees (first_name,
hire_date) VALUES (%s, %s)"
 cursor.executemany(stmt, data)

callproc – Call a stored procedure
 Cursor.callproc(self, procname, [parameters]): Call a stored
database procedure with the given name
 Parameters: procname (str) – the name of the database function
 parameters – a sequence of parameters (can be empty
or omitted)
 This method calls a stored procedure (function) in the PostgreSQL
database.
 The sequence of parameters must contain one entry for each input
argument that the function expects.
 The result of the call is the same as this input sequence; replacement
of output and input/output parameters in the return value is currently
not supported.
 The function may also provide a result set as output. These can be
requested through the standard fetch methods of the cursor.

fetchone – fetch next row of the
query result
 Cursor.fetchone() Fetch the next row of a query result set
 Returns: the next row of the query result set
 Return type: named tuple or None Fetch the next row of a
query result set, returning a single named tuple, or None
when no more data is available.
 The field names of the named tuple are the same as the
column names of the database query as long as they are valid
Python identifiers.
 An Error (or subclass) exception is raised if the previous call
to Cursor.execute() or Cursor.executemany() did not
produce any result set or no call was issued yet.

 # Using a while loop
cursor.execute("SELECT * FROM employees")
row = cursor.fetchone()
while row is not None:
print(row)
row = cursor.fetchone()
 # Using the cursor as iterator
cursor.execute("SELECT * FROM employees")
for row in cursor:
print(row)

fetchmany – fetch next set of rows of
the query result
 Cursor.fetchmany([size=None][, keep=False]) Fetch the
next set of rows of a query result
 Parameters: size (int or None) – the number of rows to be
fetched
 keep – if set to true, will keep the passed arraysize
 Tpye keep: bool
 Returns: the next set of rows of the query result
 Return type:list of named tuples
 Fetch the next set of rows of a query result, returning a list of
named tuples. An empty sequence is returned when no more
rows are available. The field names of the named tuple are the
same as the column names of the database query as long as they
are valid Python identifiers.

fetchall – fetch all rows of the query
result
 cursor.fetchall() Fetch all (remaining) rows of a query
result
 Returns: the set of all rows of the query result
 Return type: list of named tuples Fetch all (remaining)
rows of a query result, returning them as list of named
tuples.
 The field names of the named tuple are the same as the
column names of the database query as long as they are
valid Python identifiers.
 Note that the cursor’s arraysize attribute can affect the
performance of this operation.

 cursor.execute("SELECT * FROM employees ORDER BY
emp_no")
 head_rows = cursor.fetchmany(size=2)
 remaining_rows = cursor.fetchall()

arraysize - the number of rows to
fetch at a time
 Cursor.arraysize The number of rows to fetch at a time
 This read/write attribute specifies the number of rows to
fetch at a time with Cursor.fetchmany().
 It defaults to 1, meaning to fetch a single row at a time.

Example
import MySQLdb
# Open database connection
db =
MySQLdb.connect(host="localhost",port=3306,user="root",passwd="",db=
"test" )
# prepare a cursor object using cursor() method
cursor = db.cursor()
# execute SQL query using execute() method.
cursor.execute("SELECT VERSION()")
# Fetch a single row using fetchone() method.
data = cursor.fetchone()
print "Database version : %s " % data
# disconnect from server
db.close()

 If a connection is established with the datasource, then a
Connection Object is returned and saved into db for
further use, otherwise db is set to None.
 Next, db object is used to create a cursor object, which
in turn is used to execute SQL queries.
 Finally, before coming out, it ensures that database
connection is closed and resources are released.

Creating Database Table
 Once a database connection is established, we are ready to create tables
or records into the database tables using execute method of the created
cursor.

import MySQLdb
db =
MySQLdb.connect(host="localhost",port=3306,user="root",passwd="",db="test"
)
# Drop table if it already exist using execute() method.
cursor.execute("DROP TABLE IF EXISTS EMPLOYEE")
# Create table as per requirement
sql = """CREATE TABLE EMPLOYEE (
FIRST_NAME CHAR(20) NOT NULL,
LAST_NAME CHAR(20),
AGE INT,
SEX CHAR(1),
INCOME FLOAT )"""
cursor.execute(sql)
db.close()

INSERT Operation
import MySQLdb
db =
MySQLdb.connect(host="localhost",port=3306,user="root",passwd="",db="test" )
# Prepare SQL query to INSERT a record into the database.
sql = """INSERT INTO EMPLOYEE(FIRST_NAME,
LAST_NAME, AGE, GEN, INCOME)
VALUES ('Solanki', 'raviraj', 20, 'M', 2000)"""
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
db.close()

Insert 2nd
way
import MySQLdb
db = MySQLdb.connect(host="localhost",port=3306,user="root",passwd="",db="test" )
# Prepare SQL query to INSERT a record into the database.
sql = "INSERT INTO EMPLOYEE(FIRST_NAME,
LAST_NAME, AGE, GEN, INCOME)
VALUES ('%s', '%s', '%d', '%c', '%d' )" %
('Mac', 'Mohan', 20, 'M', 2000)
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
db.close()

READ Operation
 READ Operation on any database means to fetch some useful
information from the database.
 Once our database connection is established, you are ready to make
a query into this database.
 You can use either fetchone() method to fetch single record or
fetchall() method to fetch multiple values from a database table.
 fetchone(): It fetches the next row of a query result set.
 A result set is an object that is returned when a cursor object is used
to query a table.
 fetchall(): It fetches all the rows in a result set. If some rows have
already been extracted from the result set, then it retrieves the
remaining rows from the result set.
 rowcount: This is a read-only attribute and returns the number of
rows that were affected by an execute() method.

Types of Errors
 The following are the six different error types supported
by MySQL for Python.
 DataError
 IntegrityError
 InternalError
 NotSupportedError
 OperationalError
 ProgrammingError

 DataError
 This exception is raised due to problems with the processed data
(for example, numeric value out of range, division by zero,
and so on).
 IntegrityError
 If the relational integrity of the database is involved (for example a
foreign key check fails, duplicate key, and so on), this exception
is raised.
 InternalError
 This exception is raised when there is an internal error in the MySQL
database itself (for example, an invalid cursor, the transaction
is out of sync, and so on). This is usually an issue of timing out or
otherwise being perceived by MySQL as having lost connectivity with
a cursor.

 NotSupportedError
 MySQL for Python raises this exception when a method or database
API that is not supported is used (for example, requesting a
transaction-oriented function when transactions are not available.
They also can arise in conjunction with setting characters sets, SQL
modes, and when using MySQL in conjunction with Secure Socket
Layer (SSL).
 OperationalError
 Exception raised for operational errors that are not necessarily under
the control of the programmer (for example, an unexpected
disconnect, the data source name is not found, a transaction
could not be processed, a memory allocation error occurrs,
and so on.).

 ProgrammingError
 Exception raised for actual programming errors (for
example, a table is not found or already exists,
there is a syntax error in the MySQL statement, a
wrong number of parameters is specified, and so
on.).

PostgreSQL database
 PostgreSQL is a powerful, open source object-relational
database system.
 It is a multi-user database management system.
 It runs on multiple platforms including Linux, FreeBSD,
Solaris, Microsoft Windows and Mac OS X.
 PostgreSQL is developed by the PostgreSQL Global
Development Group.
 The PostgreSQL can be integrated with Python using
psycopg2 module. sycopg2 is a PostgreSQL database adapter
for the Python programming language.
 psycopg2 was written with the aim of being very small and
fast.

Sqlite3 module in python
 SQLite3 is a very easy to use database engine.
 It is self-contained, serverless, zero-configuration and
transactional.
 It is very fast and lightweight, and the entire database is
stored in a single disk file.
 It is used in a lot of applications as internal data storage.
 The Python Standard Library includes a module called
"sqlite3" intended for working with this database.
 This module is a SQL interface compliant with the DB-API
2.0 specification.
 You do not need to install this module separately because its
being shipped by default along with Python version 2.5.x
onwards.

 To use sqlite3 module, you must first create a connection
object that represents the database and then optionally
you can create cursor object which will help you in
executing all the SQL statements.

Using Python's SQLite Module
 To use the SQLite3 module we need to add an import statement to our python
script:
 import sqlite3
 Connecting SQLite to the Database
 We use the function sqlite3.connect to connect to the database. We
can use the argument ":memory:" to create a temporary DB in the
RAM or pass the name of a file to open or create it.
 # Create a database in RAM
 db = sqlite3.connect(':memory:')
 # Creates or opens a file called mydb with a SQLite3 DB
 db = sqlite3.connect('data/mydb')
 When we are done working with the DB we need to close the
connection:
 db.close()

Object-relational mappers (ORMs)
 Object relation mapping is a technique that lets you query and manipulate data
from a database using an oop.
 An object-relational mapper (ORM) is a code library that automates the
transfer of data stored in relational databases tables into objects that are more
commonly used in application code.

 An ORM is the software artefact who maps from
relational data base tables to object/class
 This means: maps the tables and columns in a relational
database directly to the object instance and wraps all
SQL/DDL functionality in his methods.

ORM Object Relational Mapping. We
communicate with the database using
the ORM and only use Python objects
and classes.

SQLAlchemy
 SQLAlchemy is the Python SQL toolkit and Object
Relational Mapper that gives application developers the
full power and flexibility of SQL.
 SQLAlchemy provides a full suite of well known
enterprise-level persistence patterns, designed for
efficient and high-performing database access, adapted
into a simple and Pythonic domain language.

Major SQLAlchemy features include:
 An industrial strength ORM, built from the core on the identity map, unit of work,
and data mapper patterns. These patterns allow transparent persistence of objects
using a declarative configuration system. Domain models can be constructed and
manipulated naturally, and changes are synchronized with the current transaction
automatically.
 A relationally-oriented query system, exposing the full range of SQL's capabilities
explicitly, including joins, subqueries, correlation, and most everything else, in
terms of the object model. Writing queries with the ORM uses the same
techniques of relational composition you use when writing SQL. While you can
drop into literal SQL at any time, it's virtually never needed.
 A comprehensive and flexible system of eager loading for related collections and
objects. Collections are cached within a session, and can be loaded on individual
access, all at once using joins, or by query per collection across the full result set.

 A Core SQL construction system and DBAPI interaction layer. The
SQLAlchemy Core is separate from the ORM and is a full database
abstraction layer in its own right, and includes an extensible Python-
based SQL expression language, schema metadata, connection
pooling, type coercion, and custom types.
 All primary and foreign key constraints are assumed to be
composite and natural. Surrogate integer primary keys are of course
still the norm, but SQLAlchemy never assumes or hardcodes to this
model.
 Database introspection and generation. Database schemas can be
"reflected" in one step into Python structures representing database
metadata; those same structures can then generate CREATE
statements right back out - all within the Core, independent of the
ORM.

Create engine
 create an “engine” which is basically an object that
knows how to communicate with the provided
database using the credentials you supply.
 In this case, we are using a Sqlite database that doesn’t
need credentials.

Echo-true
 set echo to True. This means that SqlAlchemy will output
all the SQL command it is executing to stdout.
 This is handy for debugging, but should be set to False
when you’re ready to put the code into production.

Metadata
 we create a MetaData object. This cool creation from the
SqlAlchemy team holds all the database metadata.
 It consists of Python objects that hold descriptions of the
tables and other schema-level objects of the database.
We can bind the metadata object to our database here
or in the create_all statement near the end of the code.

Table create
 The last section is how we create the tables programmatically. This is accomplished
by using SqlAlchemy’s Table and Column objects.
 we have various field types available to us, like String and Integer.
 There are many others too.
 For this example, we create a database and name it “users”, then pass in our
metadata object.
 Next, we put it in the Columns.
 The “id” column is set as our primary key.
 SqlAlchemy will magically increment this for us as we add users to the database.
 The “name” column is a String type and capped at 40 characters long.
 The “age” column is just a simple Integer and the “password” column is just set to
String.
 We didn’t set its length, but we probably should. The only major difference in the
addresses_table is how we set up the Foreign key attribute that connects the two
tables.
 Basically, we point it at the other table by passing the correct field name in a string
to the ForeignKey object.

Create_All()
 The final line of this snippet actually creates the database
and the table. You can call it as often as you like as it will
always check for the existence of the specified table
before trying to create it. That means you can create
additional tables and call create_all and SqlAlchemy will
only create the new table.

insert
 first, you need to create the Insert object by calling the
table’s insert method. Then you can use the Insert’s values
method to add the required values for the row. Next, we
create the Connection object via the engine’s connect
method. Finally, we call the Connection object’s execute
method on the Insert object.

Way of inserting
 In both cases, you will need to call the table object insert
method. Basically, you just take the engine out of the
picture in the second instance. The last insert method
we’ll look at is how to insert multiple rows:

select
 First we have to import the select method from
sqlalchemy.sql.
 Then we pass it the table as a one element list. Finally we
call the select object’s execute method and store the
returned data in the result variable.
 Now that we have all the results, we should probably see
if we got what we expected. Thus, we create a for loop
to iterate over the result.

 All we had to do was specify the column names in our
select statement. The little “c” basically means “column”,
so we do a select on column name and column age. If
you had multiple tables, then the select statement would
be something like this:
 select([tableOne, tableTwo])
 Of course, this will probably return duplicate results, so
you’ll want to do something like this to mitigate the
issue:
 s = select([tableOne, tableTwo],
tableOne.c.id==tableTwo.c.user_id)

Model
 declarative_base is a factory function, that returns a base class
(actually a metaclass), and the entities are going to inherit from
it.
 Once the definition of the class is done, the Table and mapper
will be generated automatically.
 There is some magic involved, but on the other hand
SQLAlchemy forces you to explicitly define things like the table
name, primary keys and relationships.
 First create the Base class:
 from sqlalchemy.ext.declarative import
declarative_base
 Base = declarative_base()

 The entities are classes, which derive from the Base class.
We are also using the relationship function to define the
relationships between the entities.
 The many-to-many relationship between tags and images
requires us to define an association table, which we'll be
joining over.
 When defining the images relationships, we are also using
the backref parameter, which adds the image properties
to the tags and comments entities.
 We want those references to be dynamically loaded,
because we probably don't want to load all images, when
accessing these entities.

Connecting and Creating the Schema
 First of all we need to create the engine, which is used to
connect to the database. This example uses SQLite3, which
should already be included in your Python installation.
 from sqlalchemy import create_engine
 engine =
create_engine(r'sqlite:///f:foo.db',echo=True)
 A call to the metadata of the Base class then generates the
Schema:
 Base.metadata.create_all(engine)
 Since we have set echo=True for the engine, we can see the
generated SQL:

Sessions
 A Session is a Python class, which handles the conversation
with the database for us.
 It implements the Unit of Work pattern for synchronizing
changes to the database.
 Basically it tracks all records you add or modify.
 We can acquire a Session class with the sessionmaker, which
simplifies the configuration, since we only bind our database
engine to it:
 from sqlalchemy.orm import sessionmaker
 Session = sessionmaker(bind=engine)
 And now whenever we want to talk to the database, we can
create a new Session:
 session = Session()

SQLObject
 SQLObject is a Python object-relational mapper
between a SQL database and Python objects.
 In SQLObject, database concepts are mapped into
Python in a way that's very similar to SQLAlchemy,
where tables are mapped as classes, rows as instances
and columns as attributes.
 It also provides a Python-object-based query language
that makes SQL more abstract.

Introduction to Nosql
 NoSQL is a non-relational database management systems,
different from traditional relational database management
systems in some significant ways.
 It is designed for distributed data stores where very large scale
of data storing needs (for example Google or Facebook which
collects terabits of data every day for their users).
 These type of data storing may not require fixed schema, avoid
join operations and typically scale horizontally.
 provides a mechanism for storage and retrieval of data other
than tabular relations model used in relational databases.
 NoSQL database doesn?t use tables for storing data.
 It is generally used to store big data and real-time web
applications.

MongoDB - Overview
 MongoDB is a cross-platform, document oriented database
that provides, high performance, high availability, and easy
scalability.
 MongoDB works on concept of collection and document.
 Database
 Database is a physical container for collections. Each
database gets its own set of files on the file system. A single
MongoDB server typically has multiple databases.

 Collection
 Collection is a group of MongoDB documents.
 It is the equivalent of an RDBMS table.
 A collection exists within a single database.
 Collections do not enforce a schema.
 Documents within a collection can have different
fields.
 Typically, all documents in a collection are of similar
or related purpose.

 Document
 A document is a set of key-value pairs.
 Documents have dynamic schema.
 Dynamic schema means that documents in the same
collection do not need to have the same set of fields or
structure, and common fields in a collection's documents
may hold different types of data.

Why Use MongoDB?
 Document Oriented Storage Data is stored in the form of JSON style−
documents.
 Index on any attribute
 Replication and high availability
 Auto-sharding
 Rich queries
 Fast in-place updates
 Professional support by MongoDB
 Where to Use MongoDB?
 Big Data
 Content Management and Delivery
 Mobile and Social Infrastructure
 User Data Management
 Data Hub

Database programming

More Related Content

What's hot (20)

Similar to Database programming (20)

More from Shree M.L.Kakadiya MCA mahila college, Amreli (20)

Recently uploaded (20)

Database programming