Hacking ansible

Hacking Ansible
make it do more
ansiblefest 10/2014

Who am I?
programmer
DBA
system administrator
tinkerer
devops
tech-janitor
____________
< irc lurker >
------------
, ,
/( )`
___ / |
/- _ `-/ '
(// /
/ / | `
O O ) / |
`-^--'`< '
(_.) _ ) /
`.___/` /
`-----' /
<----. __ / __
<----|====O)))==) ) /====
<----' `--' `.__,'
| |
/
______( (_ / ______
,' ,-----' |
`--{__________) /

________________
/ It runs a TASK
on a HOST /
----------------
^__^
(oo)_______
(__) )/
||----w |
|| ||
What is ansible?
● Configuration management?
● Automation platform?
● Release management?
● Orchestration system?

The Guts: ansible internal objects
● inventory: defines my targets
● runner: actually does the work
● connection: how to get to my targets
● playbook: all plays in current invocation
○ play
■ host: where to do it
■ task: “it” aka “what to do”
● callback: show me what was done

Example hack - the core
--- a/lib/ansible/playbook/__init__.py
+++ b/lib/ansible/playbook/__init__.py
@@ -346,6 +346,8 @@ class PlayBook(object):
run_hosts=hosts
)
+ runner.module_vars.update({'play_hosts': hosts})
+
if task.async_seconds == 0:
results = runner.run()
else:

Example hack - the core
● 10 second patch to add play_hosts
● 3h to find correct spot to patch
● learned a lot about playbook/host iteration
● I did not take “serial:” into account
● runner rewrite in progress

PluginLoader
...
lookup_loader = PluginLoader(
'LookupModule',
'ansible.runner.lookup_plugins',
C.DEFAULT_LOOKUP_PLUGIN_PATH,
'lookup_plugins'
)
vars_loader = PluginLoader(
'VarsModule',
'ansible.inventory.vars_plugins',
C.DEFAULT_VARS_PLUGIN_PATH,
'vars_plugins'
)
lib/ansible/utils/plugins.py
__________________
/ all plugins
except inventory /
------------------
,__,
(oo)____
(__) )
||--|| *

Plugins
● Library: host tasks/actions/modules
● Action: master side tasks/actions/modules
● Cache: fact caching
● Callback: play output
● Connection: host connections

Plugins
● Shell: what shell to use to execute tasks
● Lookup: master side info lookup
● Vars: variable imports
● Inventory: aside from inventory scripts
● Filter: jinja2 filters for data modification
● Doc Fragment: shared docs for library

Library/Action - May 2013 (72)
add_host,debug,get_url,mount,postgresql_user,slurp,apt,
django_manage,git,mysql_db,rabbitmq_parameter,
subversion,apt_key,easy_install,group,mysql_user,
rabbitmq_plugin,supervisorctl,apt_repository,ec2,group_by,
nagios,rabbitmq_user,svr4pkg,assemble,ec2_facts,hg,ohai,
rabbitmq_vhost,sysctl,async_status,ec2_vol,ini_file,opkg,
raw,template,async_wrapper,facter,libr,pacman,script,uri,
authorized_key,fail,lineinfile,pause,seboolean,user,
cloudformation,fetch,lvol,ping,selinux,virt,command,file,
macports,pip,service,wait_for,copy,fireball,mail,pkgin,setup,
yum,cron,gem,mongodb_user,postgresql_db,shell,zfs

Library/Action - October 2014 (175)
a10_server,a10_service_group,a10_virtual_server,accelerate,acl,add_host,airbrake_deployment,alternatives,apache2_module,
apt,apt_key,apt_repository,apt_rpm,assemble,assert,async_status,at,authorized_key,azure,bigip_facts,bigip_monitor_http,
bigip_monitor_tcp,bigip_node,bigip_pool,bigip_pool_member,bigpanda,boundary_meter,bzr,campfire,capabilities,cloudformation,
command,composer,copy,cpanm,cron,datadog_event,debconf,debug,digital_ocean,digital_ocean_domain,digital_ocean_sshkey,
django_manage,dnsimple,dnsmadeeasy,docker,docker_image,easy_install,ec2,ec2_ami,ec2_ami_search,ec2_asg,ec2_eip,
ec2_elb,ec2_elb_lb,ec2_facts,ec2_group,ec2_key,ec2_lc,ec2_metric_alarm,ec2_scaling_policy,ec2_snapshot,ec2_tag,ec2_vol,
ec2_vpc,ejabberd_user,elasticache,facter,fail,fetch,file,filesystem,fireball,firewalld,flowdock,gc_storage,gce,gce_lb,gce_net,
gce_pd,gem,get_url,getent,git,github_hooks,glance_image,group,group_by,grove,hg,hipchat,homebrew,homebrew_cask,
homebrew_tap,hostname,htpasswd,include_vars,ini_file,irc,jabber,jboss,jira,kernel_blacklist,keystone_user,layman,
librato_annotation,lineinfile,linode,lldp,locale_gen,logentries,lvg,lvol,macports,mail,modprobe,mongodb_user,monit,mount,mqtt,
mysql_db,mysql_replication,mysql_user,mysql_variables,nagios,netscaler,newrelic_deployment,nexmo,nova_compute,
nova_keypair,npm,ohai,open_iscsi,openbsd_pkg,openvswitch_bridge,openvswitch_port,opkg,osx_say,ovirt,pacman,pagerduty,
pause,ping,pingdom,pip,pkgin,pkgng,pkgutil,portage,portinstall,postgresql_db,postgresql_privs,postgresql_user,
quantum_floating_ip,quantum_floating_ip_associate,quantum_network,quantum_router,quantum_router_gateway,
quantum_router_interface,quantum_subnet,rabbitmq_parameter,rabbitmq_plugin,rabbitmq_policy,rabbitmq_user,rabbitmq_vhost,
raw,rax,rax_cbs,rax_cbs_attachments,rax_cdb,rax_cdb_database,rax_cdb_user,rax_clb,rax_clb_nodes,rax_dns,rax_dns_record,
rax_facts,rax_files,rax_files_objects,rax_identity,rax_keypair,rax_meta,rax_network,rax_queue,rax_scaling_group,
rax_scaling_policy,rds,rds_param_group,rds_subnet_group,redhat_subscription,redis,replace,rhn_channel,rhn_register,riak,
rollbar_deployment,route53,rpm_key,s3,script,seboolean,selinux,service,set_fact,setup,shell,slack,slurp,sns,stackdriver,stat,
subversion,supervisorctl,svr4pkg,swdepot,synchronize,sysctl,template,twilio,typetalk,ufw,unarchive,uri,urpmi,user,virt,
vsphere_guest,wait_for,win_feature,win_get_url,win_group,win_msi,win_ping,win_service,win_stat,win_user,xattr,yum,
zabbix_maintenance,zfs,zypper,zypper_repository

Filters: Jinja2’ism
● best way to change data
● chainable pipes
● simple to expand
● can hide complexity
__________________________
< [hello, world]|join(‘ ‘) >
--------------------------
^__^
(oo)_______
(__) )/
||----w |
|| ||

Example hack - filter
import random
def randomize_list(mylist):
random.shuffle(mylist)
return mylist
...
def filters(self): # this maps filter names to functions
return {
‘shuffle’: randomize_list,
...
}
lib/ansible/runner/filter_plugins/core.py

- hosts: all
connection: local
gather_facts: false
vars:
- mylist: [1,2,3,4,5]
tasks:
- debug: msg=”{{item}}”
with_items: “{{mylist|shuffle}}”

#>ansible-playbook test.yml -i ‘localhost,’
__________________________
< TASK: debug msg={{item}} >
--------------------------
ok: [localhost] => (item=1) => {
"item": 1,
"msg": "1"
}
"item": 3,
"msg": "3"
}
"item": 4,
"msg": "4"
}
"item": 5,
"msg": "5"
}
"item": 2,
"msg": "2"
}
__________________________
--------------------------
"item": 3,
"msg": "3"
}
"item": 2,
"msg": "2"
}
"item": 5,
"msg": "5"
}
"item": 2,
"msg": "2"
}
"item": 1,
"msg": "1"
}
__________________________
--------------------------
"item": 2,
"msg": "2"
}
"item": 4,
"msg": "4"
}
"item": 1,
"msg": "1"
}
"item": 3,
"msg": "3"
}
"item": 5,
"msg": "5"
}

Lookups
● you are already using them: with_<lookup>
● they execute on the “master”
● a way to access external files and/or data
● normally returns a list

Example hack - lookup
lib/ansible/runner/lookup_plugins/etcd.py
class LookupModule(object):
def __init__(self, basedir=None, **kwargs):
self.basedir = basedir
self.etcd = etcd() # initializes etcd class
def run(self, terms, inject=None, **kwargs):
terms = utils.listify_lookup_plugin_terms(terms, self.basedir, inject)
if isinstance(terms, basestring):
terms = [ terms ]
ret = []
for term in terms:
key = term.split()[0]
value = self.etcd.get(key) # gets the data from etcd class
ret.append(value)
return ret

lib/ansible/runner/lookup_plugins/etcd.py
class etcd(): # simplified version
def __init__(self, url=ANSIBLE_ETCD_URL):
self.url = url
self.baseurl = '%s/v1/keys' % (self.url)
def get(self, key):
url = "%s/%s" % (self.baseurl, key)
r = urllib2.urlopen(url)
data = r.read()
item = json.loads(data)
value = item['value']
return value
_______________
/ written by
Jan-Piet Mens /
---------------
,__,
(oo)____
(__) )
||--|| *

#>ansible all -m debug -a ‘msg={{lookup(“etcd”,”key1”)}}’ -i ‘localhost,’
localhost | success >> {
"msg": "value1"
}
#>ansible all -m debug -a ‘msg={{lookup(“etcd”,”key2”)}}’ -i ‘localhost,’
localhost | success >> {
"msg": "value2"
}

Example hack - callback
callback_plugins/profile_tasks.py
class CallbackModule(object): # simplified example
def playbook_on_task_start(self, name, is_conditional):
_______________
/ written by
Jharrod LaFon /
---------------
,__,
(oo)____
if self.current is not None: # Record the total time of the previous
self.stats[self.current] = time.time() - self.stats[self.current]
self.current = name # Record the start time of the current task
self.stats[self.current] = time.time()
def playbook_on_stats(self, stats): # Sort the tasks by their running time
...
(__) )
||--|| *
results = sorted(self.stats.items(), key=lambda value: value[1], reverse=True)
for name, elapsed in results: # Print the timings
print "{0:-<70}{1:->9}".format('{0} '.format(name),'{0:.02f}s'.format(elapsed))

Example hack - callback
#>ansible-playbook test.yml -i ‘localhost,’
# re-running the previous filter/shuffle playbook
__________________________
--------------------------
"item": 2,
"msg": "2"
}
...
"item": 4,
"msg": "4"
}
PLAY RECAP ********************************************************************
debug msg={{item}} ------------------------------------------------------ 0.02s
localhost : ok=1 changed=0 unreachable=0 failed=0

Example hack - cache
import exceptions
class BaseCacheModule(object):
def get(self, key):
lib/ansible/cache/base.py
raise exceptions.NotImplementedError
def set(self, key, value):
def keys(self):
def contains(self, key):
def delete(self, key):
def flush(self):
def copy(self):
________________
/ I abandoned it
| and Josh Drake |
revived it /
----------------
/ ___ /
// / /
(( O O ))
/ //
/ | | /
| | | |
| | | |
| o |
| | | |
|m| |m|

from ansible.cache.base import BaseCacheModule
class CacheModule(BaseCacheModule):
def __init__(self, *args, **kwargs):
self._cache = {}
def get(self, key):
return self._cache.get(key)
self._cache[key] = value
def keys(self):
return self._cache.keys()
def contains(self, key):
return key in self._cache
def delete(self, key):
del self._cache[key]
def flush(self):
lib/ansible/cache/memory.py

from redis import StrictRedis
class CacheModule(BaseCacheModule): # simplified version
self._cache = StrictRedis(*connection)
...
def get(self, key):
value = self._cache.get(self._make_key(key))# make_key creates correct prefix
# guard against the key not being removed from the zset
if value is None:
self.delete(key)
raise KeyError
return json.loads(value)
value2 = json.dumps(value)
if self._timeout > 0: # a timeout of 0 is handled as meaning 'never expire'
self._cache.setex(self._make_key(key), int(self._timeout), value2)
else:
self._cache.set(self._make_key(key), value2)
self._cache.zadd(self._keys_set, time.time(), key)
...
lib/ansible/cache/redis.py

Tests: test/
● sadly the presenter sucks at test coverage
● no really, tests are good, regressions!
● assert: the test module
● destructive, non destructive, integration
● ‘make tests’, runs unit tests

Before you submit
● update documentation (it is also in the repo)
● create tests, assert! (also in the repo)
● prepare a clear usage/example to post in the PR
● its a dialog, be ready to make your ‘use case’
● patience … … … …
● not everything belongs in core

Hacking ansible

More Related Content

What's hot (20)

Viewers also liked (16)

Similar to Hacking ansible (20)

Recently uploaded (20)

Hacking ansible