Applying the Unix Philosophy to Django projects: a report from the real world

Unix Philosophy & Django projects
Applying the Unix Philosophy to
Django projects
a report from the real world

Who am I?
Federico Capoano (A.K.A. Nemesis)
OpenWISP core developer
Working on public wif at Cineca since 2012

What is OpenWISP?
OpenWISP is a set of software modules that can be
used to deploy and manage wireless networks
(public wif, backbone, mesh networks)

OpenWISP 1
OpenWISP 1: started in 2008-2016
Focused on public wif for Italian cities
Fast growth of features to accommodate requirements

Problems of OpenWISP 1
● monolithic ruby on rails apps

● tight coupling

● tight coupling
● duplicated logic

● tight coupling
● policy not separated from mechanisms

● tight coupling
● policy not separated from mechanisms
● complexity

Negative consequences (OpenWISP 1)
These problems caused several
very negative consequences

1. "Giant ball of mud" problems
Many features in a single codebase
Hard to add new features in a clean way
Hard to maintain over time (eg: upgrade
dependencies, fx bugs)

2. Inflexibility
Hard to extend without changing the core
Hard to adapt it and reuse it in different contexts
Specialized features available in the
codebase are not easily reusable

3. Little or no contributions
Complexity scared away
potential external contributors
We only got low quality patches
which were never merged

4. Costs and risk of death
Increased costs to keep the project alive
Many years of work risked to fade into oblivion

5. Duplication of efforts
Little reusability means:
reinventing the wheel
no ecosystem of interoperable tools

This can't be...
There must be a better way!

The Unix Philosophy
What is the unix philosophy?

1. Approach to software development
● Bottom-up

● Bottom-up
● Pragmatic

● Bottom-up
● Pragmatic
● Grounded in experience

● Bottom-up
● Pragmatic
● Grounded in experience
● Founded in empirical skepticism

2. Mature
Evolved in the Unix community since 1970

3. Change tolerant
Born during the digital revolution 1970-2010
It kept pace with the dramatic
evolution of technology

The Unix Philosophy in short
What are the main concepts
of the Unix philosophy?

"Do one thing and do it well"
● Break down big problems in smaller problems

● Write simple programs that solve one problem well

● Write programs to work together

● Write programs to work together
● Value simplicity and maintainability over
performance and complexity

What advantages does it bring?
What are the main advantages
of applying the Unix Philosophy?

1. Saner development
Solving one problem at time is simpler

2. Maintainability
Maintaining a simple program is less expensive

3. Usability
A simple program is easier to use

4. Readability
A simple program is easier to read
and understand

5. Agility
A simple program will be more easily
readapted to new requirements

6. Reuse
Being easier to use, understand, read and maintain
it will be readapted to work in different contexts
and will attract more contributors

7. Ecosystem
Many such programs form an ecosystem of
interoperable tools
(like basic lego building blocks)
and
Allow to solve complex problems faster

The Unix Philosophy in detail
The 17 rules of the Unix Philosophy
From "The art of unix programming"
by Eric Raymond

1. Rule of Modularity
Write simple parts connected
by clean interfaces
source

2. Rule of Clarity
Clarity is better than cleverness
source

3. Rule of Composition
Design programs to be connected
to other programs
source

4. Rule of Separation
Separate policy from mechanism;
separate interfaces from engines
source

5. Rule of Simplicity
Design for simplicity;
add complexity only where you must
source

6. Rule of Parsimony
Write a big program only when it is clear by
demonstration that nothing else will do
source

7. Rule of Transparency
Design for visibility to make inspection
and debugging easier
source

8. Rule of Robustness
Software is said to be robust when it performs well
under unexpected conditions which stress the
designer's assumptions, as well as under normal
conditions
source

9. Rule of Representation
Fold knowledge into data so program logic
can be stupid and robust
source

10. Rule of Least Surprise
In interface design, always do
the least surprising thing
source

11. Rule of Silence
When a program has nothing surprising
to say, it should say nothing
source

12. Rule of Repair
When you must fail, fail noisily
and as soon as possible
source

13. Rule of Economy
Programmer time is expensive; conserve it
in preference to machine time
source

14. Rule of Generation
Avoid hand-hacking; write programs to
write programs when you can
source

15. Rule of Optimization
Prototype before polishing;
get it working before you optimize it
source

16. Rule of Diversity
Distrust all claims for “one true way”
source

17. Rule of Extensibility
Design for the future, because it will
be here sooner than you think
source

How to apply it to Django?
How to apply the Unix philosophy
to django projects?
Let's see some real world
examples from OpenWISP 2

● Develop main features as reusable django apps

● One django app for each group of related features

● Document public API

● Document its public API
● Include a license and a changelog

● Document its public API
● Include a license and a changelog
● Publish your app on pypi and djangopackages.org

1. Real world examples
● Confgurations of routers and VPNs:
django-netjsonconfg

django-netjsonconfg
● PKI management (x509 certifcates): django-x509

django-netjsonconfg
● PKI management (x509 certifcates): django-x509
● Multi-tenancy: contributed to django-organizations
reuse existing projects when possible!

2. Rule of Clarity
Explicit is better than implicit
This is already a widely accepted concept in
the python world

3. Rule of Composition
Combine, extend and customize django
reusable apps in your fnal django project

● openwisp-users: extends django-organizations

● openwisp-controller: depends on openwisp-users
and extends django-netjsonconfg and django-x509

● openwisp-controller: depends on openwisp-users
and extends django-netjsonconfg and django-x509
● Final result handled by ansible-openwisp2 in an
(almost) transparent manner

● implement mechanisms as libraries

● implement policy as a confguration

● make these libraries highly confgurable

● make these libraries work with data

● make these libraries work with data
● clearly defne input and output

● netjsonconfg: python library for generating
router & VPN confgurations from NetJSON
objects

● netjsonconfg: python library for generating
router & VPN confgurations from NetJSON
objects
● django-netjsonconfg: web interface that uses
netjsonconfg under the hood

>>> from netjsonconfig import OpenWrt
>>> router = OpenWrt({
... "general": {"hostname": "HomeRouter"}
... })
>>> print(router.render())
package system
config system 'system'
option hostname 'HomeRouter'
option timezone 'UTC'
option zonename 'UTC'

● start with very basic features

● release your project early
(even if you feel it's incomplete)

● add one feature at time as your understanding grows

● add one feature at time as your understanding grows
● add complexity only when necessary

as of April 2017:

as of April 2017:
● 24 releases of netjsonconfg

as of April 2017:
● 25 releases of django-netjsonconfg

as of April 2017:
● 5 releases of django-x509

as of April 2017:
● 5 releases of django-x509
OpenWISP 2 still lacks some features of OpenWISP 1

6. Rule of Parsimony
Prefer creating new reusable apps
when adding big features
unless doing this complicates things a lot
with no real advantages

PKI management in new app: django-x509
VPN confgurations added to existing app:
django-netjsonconfg

7. Rule of Transparency
Log unexpected events using
the python logging facility

Logging of bad requests in django-netjsonconfg
import logging
logger = logging.getLogger(__name__)
def invalid_response(request, error, status):
logger.warning(error, extra={'request':
request, 'stack': True})
return ControllerResponse(error,
status=status)

Provide good default logging
Provide support for sentry
Take a look at a real example

8. Rule of Robustness
Add constraints to your reusable django apps only
when necessary:
● avoid very strict validation rules
● provide confgurable settings

● settings in django-netjsonconfg
● settings in django-x509

9. Rule of Representation
Fold complex information in data structures
Process these data structures with algorithms
Try to make algorithms framework-agnostic
(if possible)

in OpenWISP 2, confguration of routers is
implemented as a single text feld formatted as
NetJSON
(in OpenWISP 1 each available confguration had its
own database table and model,
very hard to maintain and evolve)

# NetJSON DeviceConfiguration example
{
"general": {"hostname": "HomeRouter"},
"interfaces": [
{
"name": "eth0",
"type": "ethernet",
"addresses": [
{
"address": "192.168.1.1",
"mask": 24,
"proto": "static",
"family": "ipv4"
}
]
}
]
}

10. Rule of Least Surprise
This has become a widely accepted
concept in the IT industry

11. Rule of Silence
Just don't make your apps annoying
It's not that hard :-P

12. Rule of Repair
Errors should never pass silently
Unless explicitly silenced
(from the zen of python)
Let's add: fail fast, noisily and early

from django.core.exceptions import ImproperlyConfigured
from .settings import REGISTRATION_ENABLED, SHARED_SECRET
if REGISTRATION_ENABLED and not SHARED_SECRET:
msg = 'NETJSONCONFIG_SHARED_SECRET not set!'
raise ImproperlyConfigured(msg)
Full real world example available in
django-netjsonconfg

13. Rule of Economy
When Unix was born this was a radical idea:
assembler was the norm, C was considered a higher
level language
Python is a consequence of the success of that
radical idea
Embrace this concept

14. Rule of Generation
When you fnd yourself writing lots of boilerplate
code, try to use
meta-programming or code generators

14. meta-programmed urlpatterns
urls.py in openwisp-controller

14. Auto-generated UI from json-schema
JSON-Schema of this UI available on github

15. Rule of Optimization
● write a frst early version of your django app
● do not forget to write automated tests
● refactor and polish

16. Rule of Diversity
Maintain a skeptic attitude towards
dogmas and fads
Measure the effectiveness of an
approach before embracing it

17. Rule of Extensibility
Provide ways to extend and customize the
behaviour of modules without the
need of changing the core code

17. Extensibility: abstract models
● Provide abstract models in your
most important modules

● Provide abstract models in your
most important modules
● Store these in a python fle which does not import
concrete models
(otherwise other django apps won't be able to import them)

abstract models in django-netjsonconfg
example usage

abstract models in django-x509
example usage

17. Extensibility: base admin
● Provide base admin classes in your main modules
(avoid duplication)

● Provide base admin classes in your main modules
(avoid duplication)
● Store them in a python fle which does not import
concrete models
(otherwise other django-apps won't be able to import them)

base admin classes django-netjsonconfg
example usage

17. Extensibility: base views
● If your reusable django app has views, provide
generic views that can be extended
● Store them in a python fle which does not import
concrete models
(otherwise other django-apps won't be able to import them)

17. Extensibility: base views
base views in django-netjsonconfg
example usage

17. Extensibility: reusable urls
If your app provides views that can be extended,
third party apps will have to redefne their URLs
You may want to avoid this required duplication by
providing a mechanism to import urls

17. Extensibility: reusable urls
reusable urls in django-netjsonconfg
example usage:
from django_netjsonconfig.utils import get_controller_urls
from . import views # customized views
# creates new url patterns hooked to customized views
urlpatterns = get_controller_urls(views)

17. Extensibility: AppConfig
If your reusable django app relies on signal
connection for some features,
provide a base AppConfig class

17. Extensibility: AppConfig
base AppConfig class in django-netjsonconfg
from django_netjsonconfig.apps import OpenWispAppConfig
class MyOwnApp(OpenWispAppConfig):
name = 'yourapp.config'
label = 'config'
def __setmodels__(self):
# these are your custom models
from .models import Config, VpnClient
self.config_model = Config
self.vpnclient_model = VpnClient

Positive results
Positive results achieved in OpenWISP

1. Faster release cycle & evolution
We release new features more often
The project is evolving rapidly

2. Easier maintainance
Once a bug in a specifc module is replicated
fxing it is easier compared to the work needed to
fx bugs in OpenWISP 1

3. More derivative work
OpenWISP 2 was released officially
less than 1 year ago (as of April 2017)
notwithstanding that, there are already
a couple of derivative works

4. Growth
The user base has been growing rapidly
Users send feedback and patches

4. Growth: GSOC 2017
OpenWISP has also been accepted
as a mentoring organization
for the Google Summer of Code 2017

Disadvantages
Is not a bed of roses

1. Integration issues
Combining more django apps may
result in integration issues
Changes to project confguration can result
in bugs that are not caught by unit tests

1. Solution
Integration tests are important in this case
Shortcut: I was able to import tests of a base app
and repeat them in the extension app
This easy fx was good enough for my case

2. Big features
When I needed to introduce a new major feature I
needed to change at least a couple of modules

2. Solution
many times this will be necessary
If you have to change many modules every time,
refactor your code to be more loosely coupled

3. Repository management overhead
● working with many repositories can be overwhelming

● more git tags, pypi releases, versioning, changelogs

● more git tags, pypi releases, versioning, changelogs
● more announcements for new releases

3. Solutions
● import several projects in a single window of your editor

3. Solutions
● automate versioning and changelogs
(I haven't done this yet)

3. Solutions
● automate versioning and changelogs
(I haven't done this yet)
● do not send announcements for minor releases which
contain no real advantage for end users

4. Documentation fragmentation
We have many repositories with
their own READMEs and docs
We don't have a single documentation website yet

4. Solutions
I added links to specifc module docs from the website
I often have to send those links on the mailing list
But a central comprehensive documentation website
would be better (we don't have this yet)

5. Bug reports
Users don't know where to send bug reports
and they write to the mailing list

5. Solution
● For the moment I reply by pointing to them where
to open issues

5. Solution
● For the moment I reply by pointing to them where
to open issues
● But a single issue tracker for all the repositories
would be better
(even though it may not really solve the "which module?" problem)

6. Dispersion of popularity
We no longer have a single popular
repository with many github stars

6. Solution
Stop caring about this
Github stars are not an accurate indicator
of the success of a project

Conclusions
The creators of Unix are not newcomers
Following their heritage is a good idea

Delve deep
Further resources on the Unix Philosophy
and related concepts:
● the art of unix programming
● the pragmatic programmer

Practice
Start to gradually apply these concepts in your projects
EG: extract big features to separate projects

NO to dogmas
● Practice empirical (scientifc) skepticism

NO to dogmas
● Test your ideas

NO to dogmas
● Test your ideas
● Measure your results

Read existing code
Read the code of OpenWISP
@openwisp organization on github
(or other projects like @openstack)

Thank you
Ideas, critical feedback, suggestions?
Talk to me!
Find me also on:
● twitter (@nemesisdesign)
● github (@nemesisdesign)
● linkedin (Federico Capoano)
You can fnd the slides on slideshare

Applying the Unix Philosophy to Django projects: a report from the real world

More Related Content

What's hot (7)

Similar to Applying the Unix Philosophy to Django projects: a report from the real world (20)

Recently uploaded (20)

Applying the Unix Philosophy to Django projects: a report from the real world