Django as a Data Tool in the Enterprise - PyData New York 2015

Using Django as a Data Tool in the
Enterprise
Trent Oliphant
Continuum Analytics
PyData NYC – November 10, 2015
© 2015 Continuum Analytics- Confidential & Proprietary

NOT ALL DATA IS BIG

Enterprise Reporting
© 2015 Continuum Analytics- Confidential & Proprietary 3
Central Data
Store
Simple
Process
Clear
Results

BC
BC
BC
BC
BC
Complex
Processes Results
Extra

• Aggregated
• Multiple Business Centers
• Various Size Centers
• Different Data

Business Center Data
Multiple
Data Sources
Multiple
Processes
Results
Corporate

Business Center Reporting
• Needs to feed upstream
• Have their own needs
• Smaller Teams
• Smaller Budgets
• Smaller Data

OPEN SOURCE AS AN OPTION

Advantages
• Cost
• Ease of use
• Community Resources
– Github
– Stack Overflow
– Anaconda.org

Disadvantages
• Distribution and Installation
• Support
• Knowledge
– Lack of internal sharing
– No external sharing

Anaconda Enterprise
• Package Deployment
• Collaboration
• Support
• Indemnification

WHY DJANGO

What is django?
• http://djangoproject.com
• Web framework
• Written in python
• Model-View-Template model
• v. 1.8 or higher

Why Django?
• Easy Install and Setup
• Django ORM
• Built in Authentication
• Built in Admin Interface
• Talent Pool

Easy Install and Setup
• Using Anaconda
– conda install django
– django-admin startproject myproj
• Built in Development web server
– python manage.py runserver

Django ORM
• Create Models with fields
• DB Management Handled
• Work with Objects/Properties not SQL
• Can work with SQL directly

Built in Authentication
• django.contrib.auth
• Basic Permissions
• Groups
• Sessions

Built in Admin Interface
• django.contrib.admin
• Register model
• Basic data entry and editing

Talent Pool
• Large Community
• Active Community
• Available Developers

What about ______?
• SQLAlchemy
• Flask
• Turbo Gears

PROJECT SETUP

Requirements
• Automate forecasting
• Simple User Interface
• Regular Data Update
• Excel “integration”

Team Structure
• Four Groups
– Modeling
– Finance
– Data
– Development

Other influences
• Corporate Finance
• Corporate IT
• Internal Corporate Audit
• Regulations

Tools used
• SAS
• Oracle
• TeraData
• Excel
• Python

Environments
• Servers
– Production
– UAT (User Acceptance Testing)
– Development
• Workstations

Workstations
• Desktop/Laptops
• Windows 7 Enterprise
• Locked down

Servers
• Linux
• Apache
• Oracle

Data
• Aggregated from TeraData
• 115 Tables (including output tables)
• Each run generates ~30 MB of data
• “Future” data becomes real each month
• New future data sets created

SPECIFIC ISSUES

Data Governance and Controls
• Authentication (Single Sign On)
• Access Control
• Data Validation

Data Sharing
• Excel Files
– Multiple Copies
– Modifications
• Database
– Access Concerns

Data Sharing
• Specialization

Limited Machine Access
• No shell access

SPECIFIC SOLUTIONS

Request Flow
• Apache > SSO Agent > Django
• Request > Middleware > URL resolution >
View resolution > Template > Response
• Models can be used anywhere in the chain

Integrating with Authentication
• Create Custom Authentication
• Create Middleware Class
• Update settings.py file to recognize
– AUTHENICATION_BACKENDS
– MIDDLEWARE_CLASSES

Create Custom Authentication
class MyBackend(object):
def authenticate(self, username=None, password=None):
# Check the username/password and return a User.
return User.objects.get(username=username)
def get_user(self, user_id):
try:
return User.objects.get(pk=user_id)
except User.DoesNotExist:
return None

class IntegratedBackend(object):
def authenticate(self, **credentials):
username = credentials.get('STANDARDID')
first_name = credentials.get('FIRSTNAME')
last_name = credentials.get('LASTNAME')
email = credentials.get('EMAIL')
try:
user = User.objects.get(username=username)
except User.DoesNotExist:
user = User(username=username,
password='Using external login',
first_name=first_name,
last_name=last_name,
email=email,
is_active=False)
user.save()
if not user.is_active:
user = None
return user

Create Middleware Class
from django.contrib.auth import authenticate, login, logout
class SSOIntegrationMiddleware(object):
header_fields = ['STANDARDID','FIRSTNAME','LASTNAME','EMAIL']
def process_request(self, request):
headers = {x:request.META.get(x) for x in self.header_fields}
if not (request.user.username==request.META.get('STANDARDID')):
logout(request)
if not request.user.is_authenticated():
user = authenticate(**headers)
if user is not None:
login(request, user)
return None

Update settings.py file
AUTHENTICATION_BACKENDS = (
'auth.IntegratedBackend',
'django.contrib.auth.backends.ModelBackend'
)
MIDDLEWARE_CLASSES = (
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'middleware.SSOIntegrationMiddleware',
)

Mocking Integration
• Create yaml file
• Create mock function
• Update Middleware Class

Basic ssomock.yaml
active : trent
trent :
STANDARDID : 123456
FIRSTNAME : Trent
LASTNAME : Oliphant
EMAIL : trent.oliphant@continuum.io
bob :
STANDARDID : 987654
FIRSTNAME : Bob
LASTNAME : Rumsfield
EMAIL : bob@whitehouse.gov

Create mock function
import yaml
def _get_mocked_headers(self):
headers = None
with open('ssomock.yaml','r') as f:
raw = yaml.load(f)
active = raw.get('active')
if active:
headers = raw.get(active)
return headers

Update Middleware class
if not request.META.get('STANDARDID'):
headers = self._get_mocked_headers()
request.META.update(headers)
else:

Access Control
• Normally at the model level
– delete, change, add
• Uses django_content_type table
• Needed it to be at a view (page) level

Access Control
• Create Custom Content Type
• Custom model manager
• Create Custom Permission model
• Register admin interface
• Add decorator to views

Create Content Type
• Insert into django_content_type table
– app_label = ‘ui’
– model = ‘uipermission’
• Through admin interface or direct to DB

Custom Permission Manager
from django.db import Models
class UIPermissionManager(models.Manager):
def get_queryset(self):
return super(UIPermissionManager,
self).get_queryset().filter(
content_type__model='uipermission'
)

Custom Permission Model
from django.contrib.auth.models import Permission
from django.contrib.contenttypes.model import ContentType
class UIPermission(Permission):
objects = UIPermissionManager()
class Meta:
proxy = True
verbose_name = 'ui_permission'
def save(self, *args, **kwargs):
ct, create = ContentType.objects.get_or_create(
model=self._meta.model_name,
app_label=self._meta.app_label,
)
self.content_type = ct
super(UIPermission, self).save(*args)

Add permission to view
from django.contrib.auth.decorators import permission_required
@permission_required(‘permission_name’, login_url=‘/denied_page’)
def my_view(request):
…

Accessing Output
• Output written to database
• Create excel files
– email
– Download
• Download CSV and log files

Create Excel file
• Uses xlswriter
• Gets pandas dataframe from SQL query
• Each query written to own tab

Download File
import os
from django.http import HttpResponse
from django.core.servers.basehttp import FileWrapper
def download_file(request):
filepath = 'Newly created file'
wrapper = FileWrapper(open(filepath, 'rb'))
response = HttpResponse(wrapper, content_type='application/force-download')
response['Content-Length'] = os.path.getsize(filepath)
filename = os.path.basename(filepath)
response['Content-Disposition'] = 'attachment; filename={}'.format(filename)
return response

Uploading Data
• Simple form
• Tab names must match table/model names
• Column names must match
• Uses xlrd, pandas and cursor (not ORM)

Uploading Data
import xlrd
import pandas as pd
from django.shortcuts import render
from django.db import connection, IntegityError, DatabaseError
def upload_data(request):
if request.method == 'POST':
workbook = self.open_workbook(request.FILES['uploaded_file'])
for sheetname in workbook.sheet_name:
# Do some error checking
df = pd.read_excel(workbook, sheetname, engine='xlrd')
cols = ', '.join(df.columns)
# Django wrapper of the cx_oracle connector expects %s format
val_holder = ', '.join(['%s'])*len(df.columns)
stmt_text = "INSERT INTO {} ({}) VALUES {()}"
stmt = stmt_text.format(sheetname, cols, val_holder)
cursor = connection.cursor()
cursor.executemany(stmt, df.values.to_list())
return render(request, 'upload.html’)

Basic Admin Access
• __str__ representation of the object
• No data
from django.contrib import admin
from django.apps import apps
for model in apps.get_app_config('data').get_models():
admin.site.register(model)

Tabular view
• Use list_display as property of class
• Needs a ModelAdmin class
class ExampleModelAdmin(admin.ModelAdmin):
list_display('field1','field2','field3')
admin.site.register(ExampleModel, ExampleModelAdmin)

Tabular Admin View
for model in apps.get_app_config('data').get_models():
field_names = [f.name for f in model._meta.get_fields()
if f.concrete]
cls_nm = "{}_admin".format(model._meta.model_name)
options = {'list_display': field_names}
cls = type(cls_nm, (admin.ModelAdmin,), options)
admin.site.register(model, cls)

Using a Different Oracle Schema
• Runs check_migrate
– Reads USER_TABLES

Intercepting Django Logging
• Turn off default logging
– LOGGING_CONFIG = None
• Use ‘django’ as the name of logger

Overriding SETTINGS
• settings.py is just a python file
• Read yaml file
• Update globals() with those from file

Managed = False
• Different team deployed database schema
• No rights for Django to create schema
• manage.py sqlmigrate > output.sql

Things to watch out for
• Meta options
– table_name
– Managed
• Database Error, IntegrityError
– Django wraps the underlying cx_oracle

Django as a Data Tool in the Enterprise - PyData New York 2015

More Related Content

Viewers also liked (6)

Similar to Django as a Data Tool in the Enterprise - PyData New York 2015 (20)

Recently uploaded (20)

Django as a Data Tool in the Enterprise - PyData New York 2015

Editor's Notes