SlideShare a Scribd company logo
Relational Database Access
  with Python „sans‟ ORM
           Mark Rees
              CTO
 Century Software (M) Sdn. Bhd.
Your Current Relational Database Access Style?

# Django ORM
>>> from ip2country.models import Ip2Country

>>> Ip2Country.objects.all()
[<Ip2Country: Ip2Country object>, <Ip2Country: Ip2Country object>, '...(remaining
elements truncated)...']

>>> sgp = Ip2Country.objects.filter(assigned__year=2012)
... .filter(countrycode2='SG')

>>> sgp[0].ipfrom
1729580032.0
Your Current Relational Database Access Style?
# SQLAlchemy ORM
>>> from sqlalchemy import create_engine, extract
>>> from sqlalchemy.orm import sessionmaker
>>> from models import Ip2Country

>>> engine =
create_engine('postgresql://ip2country_rw:secret@localhost/ip2country')
>>> Session = sessionmaker(bind=engine)
>>> session = Session()

>>> all_data = session.query(Ip2Country).all()

>>> sgp = session.query(Ip2Country).
... filter(extract('year',Ip2Country.assigned) == 2012).
... filter(Ip2Country.countrycode2 == 'SG')

print sgp[0].ipfrom
1729580032.0
SQL Relational Database Access
SELECT * FROM ip2country;

"ipfrom";"ipto";"registry";"assigned";"countrycode2";"countrycode3";"countryname"
1729522688;1729523711;"apnic";"2011-08-05";"CN";"CHN";"China"
1729523712;1729524735;"apnic";"2011-08-05";"CN";"CHN";"China”
...

SELECT * FROM ip2country
WHERE date_part('year', assigned) = 2012
AND countrycode2 = 'SG';

"ipfrom";"ipto";"registry";"assigned";"countrycode2";"countrycode3";"countryname"
1729580032;1729581055;"apnic";"2012-01-16";"SG";"SGP";"Singapore"
1729941504;1729942527;"apnic";"2012-01-10";"SG";"SGP";"Singapore”
...

SELECT ipfrom FROM ip2country
WHERE date_part('year', assigned) = 2012
AND countrycode2 = 'SG';

"ipfrom"
1729580032
1729941504
...
Python + SQL == Python DB-API 2.0

• The Python standard for a consistent
  interface to relational databases is the
  Python DB-API (PEP 249)
• The majority of Python database interfaces
  adhere to this standard
Python DB-API UML Diagram
Python DB-API Connection Object
Access the database via the connection object
• Use connect constructor to create a
  connection with database
     conn = psycopg2.connect(parameters…)
• Create cursor via the connection
     cur = conn.cursor()
• Transaction management (implicit begin)
     conn.commit()
     conn.rollback()
• Close connection (will rollback current
  transaction)
     conn.close()
• Check module capabilities by globals
  psycopg2.apilevel psycopg2.threadsafety psycopg2.paramstyle
Python DB-API Cursor Object
A cursor object is used to represent a database
cursor, which is used to manage the context of
fetch operations.
• Cursors created from the same connection
   are not isolated
     cur = conn.cursor()
     cur2 = conn.cursor()
• Cursor methods
     cur.execute(operation, parameters)
     cur.executemany(op,seq_of_parameters)
     cur.fetchone()
     cur.fetchmany([size=cursor.arraysize])
     cur.fetchall()
     cur.close()
Python DB-API Cursor Object
• Optional cursor methods
     cur.scroll(value[,mode='relative']) cur.next()
     cur.callproc(procname[,parameters])
     cur.__iter__()
• Results of an operation
     cur.description
     cur.rowcount
     cur.lastrowid
• DB adaptor specific “proprietary” cursor
  methods
Python DB-API Parameter Styles
Allows you to keep SQL separate from parameters

Improves performance & security


Warning Never, never, NEVER use Python string
concatenation (+) or string parameters
interpolation (%) to pass variables to a SQL query
string. Not even at gunpoint.

From http://initd.org/psycopg/docs/usage.html#query-parameters
Python DB-API Parameter Styles
Global paramstyle gives supported style for the
adaptor

  qmark Question mark style
     WHERE countrycode2 = ?
  numeric Numeric positional style
     WHERE countrycode2 = :1
  named Named style
     WHERE countrycode2 = :code
  format ANSI C printf format style
     WHERE countrycode2 = %s
  pyformat Python format style
     WHERE countrycode2 = %(name)s
Python + SQL: INSERT
import csv, datetime, psycopg2
conn = psycopg2.connect("dbname=ip2country user=ip2country_rw password=secret”)
cur = conn.cursor()
with open("IpToCountry.csv", "rb") as f:
   reader = csv.reader(f)
   try:
       for row in reader:
          print row
          if row[0][0] != "#":
              row[3] = datetime.datetime.utcfromtimestamp(float(row[3]))
              cur.execute("""INSERT INTO ip2country(
                ipfrom, ipto, registry, assigned,
                countrycode2, countrycode3, countryname)
                VALUES (%s, %s, %s, %s, %s, %s, %s)""", row)
   except:
       conn.rollback()
   else:
       conn.commit()
   finally:
      cur.close()
      conn.close()
Python + SQL: SELECT
# Find ipv4 address ranges assigned to Singapore
import psycopg2, socket, struct

def num_to_dotted_quad(n):
  """convert long int to dotted quad string
     http://code.activestate.com/recipes/66517/"""
  return socket.inet_ntoa(struct.pack('!L',n))

conn = psycopg2.connect("dbname=ip2country user=ip2country_rw password=secret")

cur = conn.cursor()

cur.execute("""SELECT * FROM ip2country
          WHERE countrycode2 = 'SG'
          ORDER BY ipfrom""")

for row in cur:
   print "%s - %s" % (num_to_dotted_quad(int(row[0])),
                num_to_dotted_quad(int(row[1])))
SQLite

• sqlite3
  • CPython 2.5 & 3
  • DB-API 2.0
  • Part of CPython distribution since 2.5
PostgreSQL

• psycopg
  • CPython 2 & 3
  • DB-API 2.0, level 2 thread safe
  • Appears to be most popular
  • http://initd.org/psycopg/

• py-postgresql
  • CPython 3
  • DB-API 2.0
  • Written in Python with optional C
    optimizations
  • pg_python - console
  • http://python.projects.postgresql.org/
PostgreSQL

• PyGreSQL
  • CPython 2.3+
  • Classic & DB-API 2.0 interfaces
  • http://www.pygresql.org/
  • Last release 2009

• pyPgSQL
  • CPython 2
  • Classic & DB-API 2.0 interfaces
  • http://www.pygresql.org/
  • Last release 2006
PostgreSQL

• pypq
  • CPython 2.7 & pypy 1.7+
  • Uses ctypes
  • DB-API 2.0 interface
  • psycopg2-like extension API
  • https://bitbucket.org/descent/pypq

• psycopg2ct
  • CPython 2.6+ & pypy 1.6+
  • Uses ctypes
  • DB-API 2.0 interface
  • psycopg2 compat layer
  • http://github.com/mvantellingen/psyco
    pg2-ctypes
MySQL
• MySQL-python
  • CPython 2.3+
  • DB-API 2.0 interface
  • http://sourceforge.net/projects/mysql-
    python/
• PyMySQL
  • CPython 2.4+ & 3
  • Pure Python DB-API 2.0 interface
  • http://www.pymysql.org/
• MySQL-Connector
  • CPython 2.4+ & 3
  • Pure Python DB-API 2.0 interface
  • https://launchpad.net/myconnpy
Other “Enterprise” Databases

• cx_Oracle
  • CPython 2 & 3
  • DB-API 2.0 interface
  • http://cx-oracle.sourceforge.net/
• informixda
  • CPython 2
  • DB-API 2.0 interface
  • http://informixdb.sourceforge.net/
  • Last release 2007
• Ibm-db
  • CPython 2
  • DB-API 2.0 for DB2 & Informix
  • http://code.google.com/p/ibm-db/
ODBC

• mxODBC
  • CPython 2.3+
  • DB-API 2.0 interfaces
  • http://www.egenix.com/products/pytho
    n/mxODBC/doc
  • Commercial product

• PyODBC
  • CPython 2 & 3
  • DB-API 2.0 interfaces with extensions
  • http://code.google.com/p/pyodbc/

• ODBC interfaces not limited to Windows
  thanks to iODBC and unixODBC
Jython + SQL

• zxJDBC
  • DB-API 2.0 Written in Java using JDBC API
     so can utilize JDBC drivers
  • Support for connection pools and JNDI
     lookup
  • Included with standard Jython
     installation http://www.jython.org/
• jyjdbc
  • DB-API 2.0 compliant
  • Written in Python/Jython so can utilize
     JDBC drivers
  • Decimal data type support
  • http://code.google.com/p/jyjdbc/
IronPython + SQL

• adodbapi
  • IronPython 2+
  • Also works with CPython 2.3+ with
    pywin32
  • http://adodbapi.sourceforge.net/
Gerald, the half a schema

     • Database schema toolkit
       • via DB-API currently supports
          • PostgreSQL
          • MySQL
          • Oracle
     • http://halfcooked.com/code/gerald/
import gerald
s1 = gerald.PostgresSchema(’public',
       'postgres://ip2country_rw:secret@localhost/ip2country')
s2 = gerald.PostgresSchema(’public',
     'postgres://ip2country_rw:secret@localhost/ip2countryv4')

print s1.schema['ip2country'].compare(s2.schema['ip2country'])
DIFF: Definition of assigned is different
DIFF: Column countryname not in ip2country
DIFF: Definition of registry is different
DIFF: Column countrycode3 not in ip2country
DIFF: Definition of countrycode2 is different
SQLPython

      • A command-line interface to relational
        databases
        • via DB-API currently supports
           • PostgreSQL
           • MySQL
           • Oracle
      • http://packages.python.org/sqlpython/
$ sqlpython --postgresql ip2country ip2country_rw
Password:
0:ip2country_rw@ip2country> select * from ip2country where countrycode2='SG';
...
1728830464.0 1728830719.0 apnic 2011-11-02 SG             SGP         Singapore
551 rows selected.
0:ip2country_rw@ip2country> select * from ip2country where countrycode2='SG'j
[...
{"ipfrom": 1728830464.0, "ipto": 1728830719.0, "registry": "apnic”,"assigned": "2011-11-02",
"countrycode2": "SG", "countrycode3": "SGP", "countryname": "Singapore"}]
SQLPython, batteries included
0:ip2country_rw@ip2country> select * from ip2country where countrycode2 ='SG’;
...
1728830464.0 1728830719.0 apnic 2011-11-02 SG          SGP       Singapore
551 rows selected.
0:ip2country_rw@ip2country> py
Python 2.6.6 (r266:84292, May 20 2011, 16:42:25)
[GCC 4.4.5 20110214 (Red Hat 4.4.5-6)] on linux2

    py <command>: Executes a Python command.
    py: Enters interactive Python mode.
    End with `Ctrl-D` (Unix) / `Ctrl-Z` (Windows), `quit()`, 'exit()`.
    Past SELECT results are exposed as list `r`;
      most recent resultset is `r[-1]`.
    SQL bind, substitution variables are exposed as `binds`, `substs`.
    Run python code from external files with ``run("filename.py")``

>>> r[-1][-1]
(1728830464.0, 1728830719.0, 'apnic', datetime.date(2011, 11, 2), 'SG', 'SGP', 'Singapore')
>>> import socket, struct
>>> def num_to_dotted_quad(n):
... return socket.inet_ntoa(struct.pack('!L',n))
...
>>> num_to_dotted_quad(int(r[-1][-1].ipfrom))
'103.11.220.0'
SpringPython – Database Templates
# Find ipv4 address ranges assigned to Singapore
# using SpringPython DatabaseTemplate & DictionaryRowMapper

from springpython.database.core import *
from springpython.database.factory import *

conn_factory = PgdbConnectionFactory(
            user="ip2country_rw", password="secret",
            host="localhost", database="ip2country")
dt = DatabaseTemplate(conn_factory)


results = dt.query(
           "SELECT * FROM ip2country WHERE countrycode2=%s",
           ("SG",), DictionaryRowMapper())

for row in results:
   print "%s - %s" % (num_to_dotted_quad(int(row['ipfrom'])),
               num_to_dotted_quad(int(row['ipto'])))
Attributions

DB-API 2.0 PEP
 http://www.python.org/dev/peps/pep-0249/

Travis Spencer‟s DB-API UML Diagram
 http://travisspencer.com/

Andrew Kuchling's introduction to the DB-API
 http://www.amk.ca/python/writing/DB-API.html
Attributions

Andy Todd‟s OSDC paper

http://halfcooked.com/presentations/osdc2006/p
ython_databases.html

Source of csv data used in examples from
WebNet77 licensed under GPLv3
 http://software77.net/geo-ip/
Contact Details




             Mark Rees
mark at centurysoftware dot com dot my
              +Mark Rees
             @hexdump42
        hex-dump.blogspot.com

More Related Content

PPTX
Relational Database Access with Python
PPTX
Python database interfaces
PDF
No sql database and python -- Presentation done at Python Developer's meetup ...
PDF
Python in the database
PDF
Spark with Elasticsearch
PDF
Spark with Elasticsearch - umd version 2014
PDF
2014 holden - databricks umd scala crash course
PDF
HadoopCon 2016 - 用 Jupyter Notebook Hold 住一個上線 Spark Machine Learning 專案實戰
Relational Database Access with Python
Python database interfaces
No sql database and python -- Presentation done at Python Developer's meetup ...
Python in the database
Spark with Elasticsearch
Spark with Elasticsearch - umd version 2014
2014 holden - databricks umd scala crash course
HadoopCon 2016 - 用 Jupyter Notebook Hold 住一個上線 Spark Machine Learning 專案實戰

What's hot (20)

PDF
Building a near real time search engine & analytics for logs using solr
PPTX
Monitoring Spark Applications
PDF
Large Scale Log Analytics with Solr: Presented by Rafał Kuć & Radu Gheorghe, ...
PPTX
Beyond shuffling global big data tech conference 2015 sj
PDF
Why your Spark job is failing
PPTX
Beyond parallelize and collect - Spark Summit East 2016
PPTX
Mongo db
PDF
Effective testing for spark programs Strata NY 2015
PDF
Beyond Shuffling - Effective Tips and Tricks for Scaling Spark (Vancouver Sp...
PPTX
Algebird : Abstract Algebra for big data analytics. Devoxx 2014
PDF
Effective testing for spark programs scala bay preview (pre-strata ny 2015)
PDF
DataSource V2 and Cassandra – A Whole New World
PDF
Requery overview
PDF
Debugging PySpark: Spark Summit East talk by Holden Karau
PDF
Apache Spark Structured Streaming for Machine Learning - StrataConf 2016
PDF
A deeper-understanding-of-spark-internals
PDF
Norikra: SQL Stream Processing In Ruby
PDF
Kotlin @ Coupang Backed - JetBrains Day seoul 2018
PDF
Large Scale Log Analytics with Solr (from Lucene Revolution 2015)
PPTX
JS Fest 2019/Autumn. Maciej Treder. Angular Schematics - Develop for developers
Building a near real time search engine & analytics for logs using solr
Monitoring Spark Applications
Large Scale Log Analytics with Solr: Presented by Rafał Kuć & Radu Gheorghe, ...
Beyond shuffling global big data tech conference 2015 sj
Why your Spark job is failing
Beyond parallelize and collect - Spark Summit East 2016
Mongo db
Effective testing for spark programs Strata NY 2015
Beyond Shuffling - Effective Tips and Tricks for Scaling Spark (Vancouver Sp...
Algebird : Abstract Algebra for big data analytics. Devoxx 2014
Effective testing for spark programs scala bay preview (pre-strata ny 2015)
DataSource V2 and Cassandra – A Whole New World
Requery overview
Debugging PySpark: Spark Summit East talk by Holden Karau
Apache Spark Structured Streaming for Machine Learning - StrataConf 2016
A deeper-understanding-of-spark-internals
Norikra: SQL Stream Processing In Ruby
Kotlin @ Coupang Backed - JetBrains Day seoul 2018
Large Scale Log Analytics with Solr (from Lucene Revolution 2015)
JS Fest 2019/Autumn. Maciej Treder. Angular Schematics - Develop for developers
Ad

Viewers also liked (20)

PDF
Www Kitebird Com Articles Pydbapi Html Toc 1
ODP
Rethink db with Python
PPTX
Succumbing to the Python in Financial Markets
PDF
"PostgreSQL and Python" Lightning Talk @EuroPython2014
PDF
MySQL User Conference 2009: Python and MySQL
PDF
Test Tech 2011 Overview
PDF
User Manual Joomla2006
PPT
Database concepts
PPT
Presentation latin american_librarians
PDF
Scaling mysql with python (and Docker).
PDF
Python for Derivative Analytics
PPTX
Lesson Two Exploring An Access Database
PPT
MS Access Training
PDF
Programming with Python and PostgreSQL
PDF
Python Utilities for Managing MySQL Databases
PDF
PostgreSQLとPythonとSQL
PPTX
Python for Big Data Analytics
PPTX
Python for Big Data Analytics
DOC
Sample training manual
PDF
Ten Reasons Why You Should Prefer PostgreSQL to MySQL
Www Kitebird Com Articles Pydbapi Html Toc 1
Rethink db with Python
Succumbing to the Python in Financial Markets
"PostgreSQL and Python" Lightning Talk @EuroPython2014
MySQL User Conference 2009: Python and MySQL
Test Tech 2011 Overview
User Manual Joomla2006
Database concepts
Presentation latin american_librarians
Scaling mysql with python (and Docker).
Python for Derivative Analytics
Lesson Two Exploring An Access Database
MS Access Training
Programming with Python and PostgreSQL
Python Utilities for Managing MySQL Databases
PostgreSQLとPythonとSQL
Python for Big Data Analytics
Python for Big Data Analytics
Sample training manual
Ten Reasons Why You Should Prefer PostgreSQL to MySQL
Ad

Similar to Relational Database Access with Python ‘sans’ ORM (20)

PPTX
Angular2 inter3
PDF
Icinga 2010 at Nagios Workshop
PPTX
ql.io at NodePDX
PDF
Automating with NX-OS: Let's Get Started!
PPT
ELK stack at weibo.com
PPTX
ql.io: Consuming HTTP at Scale
PDF
Letswift19-clean-architecture
KEY
Introduction to cloudforecast
PPTX
Developing with the Go client for Apache Kafka
PDF
Icinga 2009 at OSMC
PDF
DBA だってもっと効率化したい!〜最近の自動化事情とOracle Database〜
PDF
Pycon - Python for ethical hackers
PDF
Python para equipos de ciberseguridad
PDF
2024 Dec 05 - PyData Global - Tutorial Its In The Air Tonight
PDF
Our Puppet Story (GUUG FFG 2015)
PDF
Serving Deep Learning Models At Scale With RedisAI: Luca Antiga
PDF
OSMC 2009 | Icinga by Icinga Team
PDF
ORM vs GraphQL - Python fwdays 2019
PDF
Oleksandr Tarasenko "ORM vs GraphQL"
PDF
OpenStack API's and WSGI
Angular2 inter3
Icinga 2010 at Nagios Workshop
ql.io at NodePDX
Automating with NX-OS: Let's Get Started!
ELK stack at weibo.com
ql.io: Consuming HTTP at Scale
Letswift19-clean-architecture
Introduction to cloudforecast
Developing with the Go client for Apache Kafka
Icinga 2009 at OSMC
DBA だってもっと効率化したい!〜最近の自動化事情とOracle Database〜
Pycon - Python for ethical hackers
Python para equipos de ciberseguridad
2024 Dec 05 - PyData Global - Tutorial Its In The Air Tonight
Our Puppet Story (GUUG FFG 2015)
Serving Deep Learning Models At Scale With RedisAI: Luca Antiga
OSMC 2009 | Icinga by Icinga Team
ORM vs GraphQL - Python fwdays 2019
Oleksandr Tarasenko "ORM vs GraphQL"
OpenStack API's and WSGI

More from Mark Rees (6)

PPTX
Porting a legacy app to python 3
PPTX
Seeing with Python presented at PyCon AU 2014
PPTX
Seeing with Python - Pycon SG 2014
PPTX
Pypy is-it-ready-for-production-the-sequel
PPTX
PyPy - is it ready for production
PPT
What do you mean it needs to be Java based? How jython saved the day.
Porting a legacy app to python 3
Seeing with Python presented at PyCon AU 2014
Seeing with Python - Pycon SG 2014
Pypy is-it-ready-for-production-the-sequel
PyPy - is it ready for production
What do you mean it needs to be Java based? How jython saved the day.

Recently uploaded (20)

PPTX
20250228 LYD VKU AI Blended-Learning.pptx
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
PDF
Unlocking AI with Model Context Protocol (MCP)
PDF
Chapter 3 Spatial Domain Image Processing.pdf
PDF
Spectral efficient network and resource selection model in 5G networks
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
PPTX
Cloud computing and distributed systems.
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
PDF
Network Security Unit 5.pdf for BCA BBA.
PDF
Building Integrated photovoltaic BIPV_UPV.pdf
PDF
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
PDF
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
PPTX
Understanding_Digital_Forensics_Presentation.pptx
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
PDF
Approach and Philosophy of On baking technology
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
PDF
Machine learning based COVID-19 study performance prediction
DOCX
The AUB Centre for AI in Media Proposal.docx
20250228 LYD VKU AI Blended-Learning.pptx
The Rise and Fall of 3GPP – Time for a Sabbatical?
Unlocking AI with Model Context Protocol (MCP)
Chapter 3 Spatial Domain Image Processing.pdf
Spectral efficient network and resource selection model in 5G networks
Agricultural_Statistics_at_a_Glance_2022_0.pdf
Cloud computing and distributed systems.
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
Network Security Unit 5.pdf for BCA BBA.
Building Integrated photovoltaic BIPV_UPV.pdf
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
Understanding_Digital_Forensics_Presentation.pptx
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
Approach and Philosophy of On baking technology
Advanced methodologies resolving dimensionality complications for autism neur...
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Machine learning based COVID-19 study performance prediction
The AUB Centre for AI in Media Proposal.docx

Relational Database Access with Python ‘sans’ ORM

  • 1. Relational Database Access with Python „sans‟ ORM Mark Rees CTO Century Software (M) Sdn. Bhd.
  • 2. Your Current Relational Database Access Style? # Django ORM >>> from ip2country.models import Ip2Country >>> Ip2Country.objects.all() [<Ip2Country: Ip2Country object>, <Ip2Country: Ip2Country object>, '...(remaining elements truncated)...'] >>> sgp = Ip2Country.objects.filter(assigned__year=2012) ... .filter(countrycode2='SG') >>> sgp[0].ipfrom 1729580032.0
  • 3. Your Current Relational Database Access Style? # SQLAlchemy ORM >>> from sqlalchemy import create_engine, extract >>> from sqlalchemy.orm import sessionmaker >>> from models import Ip2Country >>> engine = create_engine('postgresql://ip2country_rw:secret@localhost/ip2country') >>> Session = sessionmaker(bind=engine) >>> session = Session() >>> all_data = session.query(Ip2Country).all() >>> sgp = session.query(Ip2Country). ... filter(extract('year',Ip2Country.assigned) == 2012). ... filter(Ip2Country.countrycode2 == 'SG') print sgp[0].ipfrom 1729580032.0
  • 4. SQL Relational Database Access SELECT * FROM ip2country; "ipfrom";"ipto";"registry";"assigned";"countrycode2";"countrycode3";"countryname" 1729522688;1729523711;"apnic";"2011-08-05";"CN";"CHN";"China" 1729523712;1729524735;"apnic";"2011-08-05";"CN";"CHN";"China” ... SELECT * FROM ip2country WHERE date_part('year', assigned) = 2012 AND countrycode2 = 'SG'; "ipfrom";"ipto";"registry";"assigned";"countrycode2";"countrycode3";"countryname" 1729580032;1729581055;"apnic";"2012-01-16";"SG";"SGP";"Singapore" 1729941504;1729942527;"apnic";"2012-01-10";"SG";"SGP";"Singapore” ... SELECT ipfrom FROM ip2country WHERE date_part('year', assigned) = 2012 AND countrycode2 = 'SG'; "ipfrom" 1729580032 1729941504 ...
  • 5. Python + SQL == Python DB-API 2.0 • The Python standard for a consistent interface to relational databases is the Python DB-API (PEP 249) • The majority of Python database interfaces adhere to this standard
  • 7. Python DB-API Connection Object Access the database via the connection object • Use connect constructor to create a connection with database conn = psycopg2.connect(parameters…) • Create cursor via the connection cur = conn.cursor() • Transaction management (implicit begin) conn.commit() conn.rollback() • Close connection (will rollback current transaction) conn.close() • Check module capabilities by globals psycopg2.apilevel psycopg2.threadsafety psycopg2.paramstyle
  • 8. Python DB-API Cursor Object A cursor object is used to represent a database cursor, which is used to manage the context of fetch operations. • Cursors created from the same connection are not isolated cur = conn.cursor() cur2 = conn.cursor() • Cursor methods cur.execute(operation, parameters) cur.executemany(op,seq_of_parameters) cur.fetchone() cur.fetchmany([size=cursor.arraysize]) cur.fetchall() cur.close()
  • 9. Python DB-API Cursor Object • Optional cursor methods cur.scroll(value[,mode='relative']) cur.next() cur.callproc(procname[,parameters]) cur.__iter__() • Results of an operation cur.description cur.rowcount cur.lastrowid • DB adaptor specific “proprietary” cursor methods
  • 10. Python DB-API Parameter Styles Allows you to keep SQL separate from parameters Improves performance & security Warning Never, never, NEVER use Python string concatenation (+) or string parameters interpolation (%) to pass variables to a SQL query string. Not even at gunpoint. From http://initd.org/psycopg/docs/usage.html#query-parameters
  • 11. Python DB-API Parameter Styles Global paramstyle gives supported style for the adaptor qmark Question mark style WHERE countrycode2 = ? numeric Numeric positional style WHERE countrycode2 = :1 named Named style WHERE countrycode2 = :code format ANSI C printf format style WHERE countrycode2 = %s pyformat Python format style WHERE countrycode2 = %(name)s
  • 12. Python + SQL: INSERT import csv, datetime, psycopg2 conn = psycopg2.connect("dbname=ip2country user=ip2country_rw password=secret”) cur = conn.cursor() with open("IpToCountry.csv", "rb") as f: reader = csv.reader(f) try: for row in reader: print row if row[0][0] != "#": row[3] = datetime.datetime.utcfromtimestamp(float(row[3])) cur.execute("""INSERT INTO ip2country( ipfrom, ipto, registry, assigned, countrycode2, countrycode3, countryname) VALUES (%s, %s, %s, %s, %s, %s, %s)""", row) except: conn.rollback() else: conn.commit() finally: cur.close() conn.close()
  • 13. Python + SQL: SELECT # Find ipv4 address ranges assigned to Singapore import psycopg2, socket, struct def num_to_dotted_quad(n): """convert long int to dotted quad string http://code.activestate.com/recipes/66517/""" return socket.inet_ntoa(struct.pack('!L',n)) conn = psycopg2.connect("dbname=ip2country user=ip2country_rw password=secret") cur = conn.cursor() cur.execute("""SELECT * FROM ip2country WHERE countrycode2 = 'SG' ORDER BY ipfrom""") for row in cur: print "%s - %s" % (num_to_dotted_quad(int(row[0])), num_to_dotted_quad(int(row[1])))
  • 14. SQLite • sqlite3 • CPython 2.5 & 3 • DB-API 2.0 • Part of CPython distribution since 2.5
  • 15. PostgreSQL • psycopg • CPython 2 & 3 • DB-API 2.0, level 2 thread safe • Appears to be most popular • http://initd.org/psycopg/ • py-postgresql • CPython 3 • DB-API 2.0 • Written in Python with optional C optimizations • pg_python - console • http://python.projects.postgresql.org/
  • 16. PostgreSQL • PyGreSQL • CPython 2.3+ • Classic & DB-API 2.0 interfaces • http://www.pygresql.org/ • Last release 2009 • pyPgSQL • CPython 2 • Classic & DB-API 2.0 interfaces • http://www.pygresql.org/ • Last release 2006
  • 17. PostgreSQL • pypq • CPython 2.7 & pypy 1.7+ • Uses ctypes • DB-API 2.0 interface • psycopg2-like extension API • https://bitbucket.org/descent/pypq • psycopg2ct • CPython 2.6+ & pypy 1.6+ • Uses ctypes • DB-API 2.0 interface • psycopg2 compat layer • http://github.com/mvantellingen/psyco pg2-ctypes
  • 18. MySQL • MySQL-python • CPython 2.3+ • DB-API 2.0 interface • http://sourceforge.net/projects/mysql- python/ • PyMySQL • CPython 2.4+ & 3 • Pure Python DB-API 2.0 interface • http://www.pymysql.org/ • MySQL-Connector • CPython 2.4+ & 3 • Pure Python DB-API 2.0 interface • https://launchpad.net/myconnpy
  • 19. Other “Enterprise” Databases • cx_Oracle • CPython 2 & 3 • DB-API 2.0 interface • http://cx-oracle.sourceforge.net/ • informixda • CPython 2 • DB-API 2.0 interface • http://informixdb.sourceforge.net/ • Last release 2007 • Ibm-db • CPython 2 • DB-API 2.0 for DB2 & Informix • http://code.google.com/p/ibm-db/
  • 20. ODBC • mxODBC • CPython 2.3+ • DB-API 2.0 interfaces • http://www.egenix.com/products/pytho n/mxODBC/doc • Commercial product • PyODBC • CPython 2 & 3 • DB-API 2.0 interfaces with extensions • http://code.google.com/p/pyodbc/ • ODBC interfaces not limited to Windows thanks to iODBC and unixODBC
  • 21. Jython + SQL • zxJDBC • DB-API 2.0 Written in Java using JDBC API so can utilize JDBC drivers • Support for connection pools and JNDI lookup • Included with standard Jython installation http://www.jython.org/ • jyjdbc • DB-API 2.0 compliant • Written in Python/Jython so can utilize JDBC drivers • Decimal data type support • http://code.google.com/p/jyjdbc/
  • 22. IronPython + SQL • adodbapi • IronPython 2+ • Also works with CPython 2.3+ with pywin32 • http://adodbapi.sourceforge.net/
  • 23. Gerald, the half a schema • Database schema toolkit • via DB-API currently supports • PostgreSQL • MySQL • Oracle • http://halfcooked.com/code/gerald/ import gerald s1 = gerald.PostgresSchema(’public', 'postgres://ip2country_rw:secret@localhost/ip2country') s2 = gerald.PostgresSchema(’public', 'postgres://ip2country_rw:secret@localhost/ip2countryv4') print s1.schema['ip2country'].compare(s2.schema['ip2country']) DIFF: Definition of assigned is different DIFF: Column countryname not in ip2country DIFF: Definition of registry is different DIFF: Column countrycode3 not in ip2country DIFF: Definition of countrycode2 is different
  • 24. SQLPython • A command-line interface to relational databases • via DB-API currently supports • PostgreSQL • MySQL • Oracle • http://packages.python.org/sqlpython/ $ sqlpython --postgresql ip2country ip2country_rw Password: 0:ip2country_rw@ip2country> select * from ip2country where countrycode2='SG'; ... 1728830464.0 1728830719.0 apnic 2011-11-02 SG SGP Singapore 551 rows selected. 0:ip2country_rw@ip2country> select * from ip2country where countrycode2='SG'j [... {"ipfrom": 1728830464.0, "ipto": 1728830719.0, "registry": "apnic”,"assigned": "2011-11-02", "countrycode2": "SG", "countrycode3": "SGP", "countryname": "Singapore"}]
  • 25. SQLPython, batteries included 0:ip2country_rw@ip2country> select * from ip2country where countrycode2 ='SG’; ... 1728830464.0 1728830719.0 apnic 2011-11-02 SG SGP Singapore 551 rows selected. 0:ip2country_rw@ip2country> py Python 2.6.6 (r266:84292, May 20 2011, 16:42:25) [GCC 4.4.5 20110214 (Red Hat 4.4.5-6)] on linux2 py <command>: Executes a Python command. py: Enters interactive Python mode. End with `Ctrl-D` (Unix) / `Ctrl-Z` (Windows), `quit()`, 'exit()`. Past SELECT results are exposed as list `r`; most recent resultset is `r[-1]`. SQL bind, substitution variables are exposed as `binds`, `substs`. Run python code from external files with ``run("filename.py")`` >>> r[-1][-1] (1728830464.0, 1728830719.0, 'apnic', datetime.date(2011, 11, 2), 'SG', 'SGP', 'Singapore') >>> import socket, struct >>> def num_to_dotted_quad(n): ... return socket.inet_ntoa(struct.pack('!L',n)) ... >>> num_to_dotted_quad(int(r[-1][-1].ipfrom)) '103.11.220.0'
  • 26. SpringPython – Database Templates # Find ipv4 address ranges assigned to Singapore # using SpringPython DatabaseTemplate & DictionaryRowMapper from springpython.database.core import * from springpython.database.factory import * conn_factory = PgdbConnectionFactory( user="ip2country_rw", password="secret", host="localhost", database="ip2country") dt = DatabaseTemplate(conn_factory) results = dt.query( "SELECT * FROM ip2country WHERE countrycode2=%s", ("SG",), DictionaryRowMapper()) for row in results: print "%s - %s" % (num_to_dotted_quad(int(row['ipfrom'])), num_to_dotted_quad(int(row['ipto'])))
  • 27. Attributions DB-API 2.0 PEP http://www.python.org/dev/peps/pep-0249/ Travis Spencer‟s DB-API UML Diagram http://travisspencer.com/ Andrew Kuchling's introduction to the DB-API http://www.amk.ca/python/writing/DB-API.html
  • 28. Attributions Andy Todd‟s OSDC paper http://halfcooked.com/presentations/osdc2006/p ython_databases.html Source of csv data used in examples from WebNet77 licensed under GPLv3 http://software77.net/geo-ip/
  • 29. Contact Details Mark Rees mark at centurysoftware dot com dot my +Mark Rees @hexdump42 hex-dump.blogspot.com

Editor's Notes

  • #3: For some Python programmers, their only exposure to accessing relational data is via a object relational mapper (ORM). As powerful is the concept of mapping objects to data, sometimes it is much simpler to manipulate your relational data using SQL. This talk will be about using the DB-API, Python’s standard mechanism for accessing relational databases.
  • #4: Or maybe you prefer sqlalchemy to abstract away the database. This talk will be about using the DB-API, Python’s standard mechanism for accessing relational databases.
  • #5: SQL (Structured Query Language) is a DSL and we can achieve the same results as the previous two slides. This what DBA’s program in. 
  • #7: This diagram no longer seems toexist on Travis’s site
  • #11: Always use parameter binding. Why? * you normally get better performance from some database engines due to to SQL query caching * reduce the chance of SQL injection
  • #12: Always use parameter binding. Why? * you normally get better performance from some database engines due to to SQL query caching * reduce the chance of SQL injection
  • #24: Gerald is a general purpose database schema toolkit written in Python. It can be used for cataloguing, managing and deploying database schemas. It is designed to allow you to easily identify the differences between databases.
  • #25: SQLPython is a command-line interface to relational databases written in Python. It was created as an alternative to Oracle’s SQL\\*Plus, and can likewise be used instead of postgres’ psql or mysql’smysql text clients. In addition, it offers several extra features inspired by other command-line clients: Neatened output, smart prompt, tab completion, history, scripting, output to file, paste buffer &amp; os command, unix like commands – ls cat grep, data dictionary exploration. Another feature is special output formats. By replacing the ; that terminates a SELECT statement with a backslash-character sequence, you can get output in a number of useful formats like xml, json, csvetc
  • #26: One of the most powerful features is the py command. The py command allows the user to execute Python commands, either one-at-a-time (with py {command}) or in an interactive environment (beginning with a bare py statement, and continuing until Ctrl-D, quit(), or exit() is entered). A history of result sets from each query is exposed to the python session as the list r; the most recent result set is r[-1]. Each row can be references as a tuple, or as an object with an attribute for each column.
  • #27: Spring Python takes the concepts of the Spring Framework and Spring Security, and brings them to the world of Python. It isn&apos;t a simple line-by-line port of the code. Instead, it takes some powerful ideas that were discovered in the realm of Java, and pragmatically applies them in the world of Python.One of these paradigms is a Portable Service Abstraction called DatabaseTemplate. * It is portable because it uses Python&apos;s standardized API, not tying us to any database vendor. Instead, in our example, we injected in an instance of Sqlite3ConnectionFactory* It provides the useful service of easily accessing information stored in a relational database, but letting us focus on the query, not the plumbing code* It offers a nice abstraction over Python&apos;s low level database API with reduced code noise. This allows us to avoid the cost and risk of writing code to manage cursors and exception handlingDatabaseTemplate handles exceptions by catching and holding them, then properly closing the cursor. It then raises it wrapped inside a Spring Python DataAccessException. This way, database resources are properly disposed of without losing the exception stack trace.The Database Template can be used in isolation from the SpringPython framework.