2015 555 kharchenko_ppt

Maxym Kharchenko
Manage ORACLE databases
with Python

Whoami
■ Started as a database kernel developer with C and C++
■ ORACLE DBA for ~ 14 years: + SQL, PL/SQL and Perl
■ Persistence Engineer for the last 4: + Python, R and Scala
■ OCM, ORACLE Ace Associate
■ Blog: http://intermediatesql.com
■ Twitter: @maxymkh

Agenda
■ Talk about why Python is awesome
■ Design ORACLE multi db “ping” in Python
■ (hopefully) Demo it

The job of an engineer is
to make complex things simple

Why script outside the database ?

Do I have a good datafilebackup ?
SELECT …
FROM v$backup_datafile
 v$backup_datafile
 Backup files actually exist
 and can be validated
 Backup size is plausible
 On the end storage
 Network transfer as well
 No critical errors in logs
 Etc

Some things are
just difficult to do in a database

You will, generally, get better Zen
i := i + 1 i += 1

You will, generally, get better Zen

You, probably, have >1 database
ORACLE ORACLE
ORACLE ORACLE ORACLE
ORACLE ORACLE ORACLE
ORACLE MySql MySql
Postgres Cassandra
S3 S3

There should be one way to do it
And it should be obvious

Python enforces good coding practices

foreach my $p (keys %$p1) {if(exists($p1->{$p})
and exists($p2->{$p})) {
if(uc($p1->{$p}) eq uc($p2->{$p})) {$diff-
>{SAME}->{$p} = 1;
} else {$diff->{DIFF}->{$p} = 1;
}}} elsif(! exists($p2->{$p})) {$diff-
>{ONLY_IN_1}->{$p} = 1;}}

for p in db1_params:
if p in db2_params:
if db1_params[p] == db2_params[p]:
same_parameters.append(p)
else:
diff_parameters.append(p)
else:
only_in_1.append(p)

Python should come
with all the batteries included

Python interfaces to everything
import urllib
import json
url =
"http://api.openweathermap.org/data/2.5/weather?q=Seattle
,WA"
url_obj = urllib.urlopen(url)
data = url_obj.read().strip()
data = json.loads(data)
print(data['weather'])

import cx_Oracle
sql = "SELECT user FROM dual"
conn = cx_Oracle.connect('scott/tiger@orcl')
cursor = conn.cursor()
cursor.execute(sql)
db_user = cursor.fetchall()

import boto
from boto.s3.key import Key
s3 = boto.connect_s3()
bucket = s3.get_bucket('my_cats')
k.key = 'lazy_cat.jpg'
k.set_contents_from_filename('/tmp/lazy_cat.jpg')

If you know any scripting language
you (almost) know Python

Python is similar to other languages
def is_accessible(db_name):
""" Check if database is accessible """
ret = False
db_status = ping_db(db_name)
if "ACTIVE" == db_status:
ret = True
return ret

But: Spaces are first class citizens
def print_databases():
""" Print all databases from /etc/oratab """
with open("/etc/oratab") as file:
for line in file:
if line:
print line.strip().split(':')[0]

But: Spaces are first class citizens
""" Print all databases from /etc/oratab ""“
with open("/etc/oratab") as file:
for line in file:
if line:
File "./a.py", line 7
^
IndentationError:
expected an indented block

Compare to Perl
sub print_databases() {
open(my $f, "<", "/etc/oratab")
or die ("Unable to open: $!");
while(my $line = <$f>) {
if ($line =~ /S+/) {
my @aItems = split(':', $line);
print $aItems[0] . "n";
}
}
close($f);
}

Compare to Perl
sub print_databases(){open(my $f, "<", "/etc/oratab") or
die ("Unable to open: $!"); while(my $line = <$f>) {
if ($line =~ /S+/) {my @aItems = split(':', $line);
print $aItems[0] . "n";}} close($f);}

Functions are first class citizens too

Fun with functions
def outer_function(parameter_function):
""" This function accepts function as a parameter """
def inner_function(inner_parameter):
""" This is a nested function """
return inner_parameter
# This returns a function from a function
return inner_function(parameter_function)
# This "calls" function return value as a function
print outer_function(external_function)()

Fun with functions: Decorators
def do_stuff():
result = heavy_processing()
def do_stuff():
start = time()
result = heavy_processing()
end = time()
print "Elapsed: %f" % (end-start)
def do_stuff2():
…
def do_stuff3076():

def timeit(func):
""" Generic time profiling function """
def time_exec(*args, **kwargs):
start_time = time()
ret = func(*args, **kwargs)
end = time()
print "Elapsed: %f" % (end-start)
return ret
return time_exec

do_stuff = timeit(do_stuff)
@timeit
def do_stuff():
…
@timeit
def do_stuff2():
…
@timeit
def do_stuff3076():
…

Learn to think Pythonian. It helps!
file = open('/etc/oratab', 'r')
while True:
line = file.readline()
if len(line) == 0 and not line.endswith('n'):
break
file.close()
with open('/etc/oratab') as file:
for line in file:

Database “multi ping” tool
■ Ping ORACLE database
▪ Report AVAILABLE / NOT AVAILABLE status
■ Option to ping multiple databases
▪ Preferably in parallel
■ Bonus: track execution timing

cx_Oracle: Running SQL
import cx_Oracle
def is_db_alive(db_name):
is_alive = False
try:
conn = cx_Oracle.connect("user/password@%s" % db_name)
cursor = conn.cursor()
cursor.execute("SELECT user FROM dual")
except:
is_alive = False
else:
is_alive = True
return is_alive

Database ping
> dbping.py c15lv1
PING [c15lv1]: OK
> dbping.py c15lv2
PING [c15lv2]: UNAVAILABLE

Database “multi ping”
import dbping
def multi_dbping(db_list, ping_routine):
""" Ping all databases in a list """
for db in db_list:
ping_routine(db)
> dbping_list.py c15lv1 c15lv2 c15lv3
PING [c15lv1]: OK
PING [c15lv3]: OK

Parallel database “multi ping”
import multiprocessing
def parallel_ping(db_list, target=dbping.print_dbping):
""" Ping db_list databases in parallel """
jobs = []
for d in db_list:
p = multiprocessing.Process(
target=target, args=(d,)
)
jobs.append(p)
p.start()
for p in jobs:
p.join()

Parallel database “multi ping”
> dbping_parallel.py c15lv1 c15lv2 c15lv3 c15lv4
PING [c15lv1]: OK
PING [c15lv3]: OK
PING [c15lv4]: OK

Decorator: Execution Timing
def timeit(func):
""" Generic time profiling function """
def time_exec(*args, **kwargs):
start_time = time()
ret = func(*args, **kwargs)
ela = time() - start_time
print “tElapsed: %.3f seconds" % ela
return ret
return time_exec

Execution Timing:
> dbping_parallel_timing.py c15lv1 c15lv2 c15lv3
PING [c15lv3]: OK
Elapsed: 1.186 seconds
PING [c15lv1]: OK
@timeit
def print_dbping_with_timing(db_name):
return dbping.print_dbping(db_name)

How to start with Python
■ Lots of free resources on the web
▪ Tutorials
▪ Documentation
▪ Stackoverflow.com
▪ “Play” environments
▪ Even books
■ Python self documentation:
▪ dir()
▪ help()

2015 555 kharchenko_ppt

More Related Content

What's hot (20)

Similar to 2015 555 kharchenko_ppt (20)

Recently uploaded (20)

2015 555 kharchenko_ppt

Editor's Notes