Solving real world data problems with Jerakia
3 likes1,578 views
The document discusses the evolution and implementation of Jerakia, a data lookup tool designed to enhance configuration management in Puppet. It details the transition from complex code structures to a more flexible, hierarchical data separation approach, explaining how Jerakia allows for dynamic data sourcing and management. The presentation also highlights its capabilities, features, and integration with Puppet and offers examples of its use in real-world scenarios.
![An Example Jerakia Policy File
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end](https://image.slidesharecdn.com/jerakiamodded-160202165014/85/Solving-real-world-data-problems-with-Jerakia-27-320.jpg)
![An Example Jerakia Policy File
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end](https://image.slidesharecdn.com/jerakiamodded-160202165014/85/Solving-real-world-data-problems-with-Jerakia-28-320.jpg)
![An Example Jerakia Policy File
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end](https://image.slidesharecdn.com/jerakiamodded-160202165014/85/Solving-real-world-data-problems-with-Jerakia-29-320.jpg)
![An Example Jerakia Policy File
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end](https://image.slidesharecdn.com/jerakiamodded-160202165014/85/Solving-real-world-data-problems-with-Jerakia-30-320.jpg)
![confine / exclude
Invalidates a lookup unless/if the criteria is met
confine request.namespsace[0], "apache"
confine request.namespsace[0], [
/website_.*/,
"apache",
"php"
]](https://image.slidesharecdn.com/jerakiamodded-160202165014/85/Solving-real-world-data-problems-with-Jerakia-39-320.jpg)
![Stop
Do not proceed to the next lookup if this lookup is valid
lookup :special do
…
confine request.namespsace[0], "apache"
stop
end
lookup :main do
…](https://image.slidesharecdn.com/jerakiamodded-160202165014/85/Solving-real-world-data-problems-with-Jerakia-40-320.jpg)
![Datasource definition
lookup :main do
datasource :file, {
:format => :yaml,
:docroot => "/var/lib/jerakia",
:searchpath => [
"host/#{scope[:certname]}",
"env/#{scope[:environment]}",
"common",
]
}
end
/var/lib/jerakia/env/dev/apache.yaml](https://image.slidesharecdn.com/jerakiamodded-160202165014/85/Solving-real-world-data-problems-with-Jerakia-43-320.jpg)
![lookup :main do
datasource :file, {
:format => :yaml,
:docroot => "/var/lib/jerakia",
:searchpath => [
"host/#{scope[:certname]}",
"env/#{scope[:environment]}",
"common",
]
}
end
/var/lib/jerakia/env/dev/apache.yaml
Datasource definition](https://image.slidesharecdn.com/jerakiamodded-160202165014/85/Solving-real-world-data-problems-with-Jerakia-44-320.jpg)
![lookup :main do
datasource :file, {
:format => :yaml,
:docroot => "/var/lib/jerakia",
:searchpath => [
"host/#{scope[:certname]}",
"env/#{scope[:environment]}",
"common",
]
}
end
/var/lib/jerakia/env/dev/apache.yaml
Datasource definition](https://image.slidesharecdn.com/jerakiamodded-160202165014/85/Solving-real-world-data-problems-with-Jerakia-45-320.jpg)
![Data Layout
:searchpath => [
"host/#{scope[:certname]}",
"env/#{scope[:environment]}",
]
# cat /var/lib/jerakia/env/dev/apache.yaml
—-
port: 80
# cat /var/lib/jerakia/env/dev.yaml
—-
apache::port: 80
Hiera
Jerakia](https://image.slidesharecdn.com/jerakiamodded-160202165014/85/Solving-real-world-data-problems-with-Jerakia-47-320.jpg)
![lookup :main, :use => :mything do
plugin.mything.do_something
…
end
Using plugins
• Plugins are loaded into the lookup
• Referenced as plugin.name.method
lookup :main, :use => [ :mything, :foo ] do
…
end](https://image.slidesharecdn.com/jerakiamodded-160202165014/85/Solving-real-world-data-problems-with-Jerakia-50-320.jpg)
![lookup :main, :use => :hiera do
plugin.hiera.rewrite_lookup
datasource :file, {
:docroot => "/var/lib/jerakia",
:format => :yaml,
:searchpath => [
"env/#{scope[:environment]}",
"common",
]
end
The hiera plugin
• Provides compatibility to hiera filesystem layouts
• Shipped with Jerakia
# cat /var/lib/jerakia/env/dev.yaml
—-
apache::port: 80](https://image.slidesharecdn.com/jerakiamodded-160202165014/85/Solving-real-world-data-problems-with-Jerakia-51-320.jpg)
![policy :default do
lookup :main, do
datasource :file, {
:format => :yaml,
:docroot => "/var/lib/jerakia",
:searchpath => [
"hostname/#{scope[:fqdn]}",
"environment/#{scope[:environment]}",
"common"
],
}
end
end
Our main lookup is
responsible for the entire
infrastructure](https://image.slidesharecdn.com/jerakiamodded-160202165014/85/Solving-real-world-data-problems-with-Jerakia-57-320.jpg)
![policy :default do
lookup :ireland do
datasource :file, {
:format => :yaml,
:docroot => "/var/external/data/ie",
:searchpath => [
"project/#{scope[:project]}",
"common",
]
}
end
lookup :main, do
datasource :file, {
:format => :yaml,
:docroot => "/var/lib/jerakia",
:searchpath => [
"hostname/#{scope[:fqdn]}",
"environment/#{scope[:environment]}",
"common"
],
}
end
end
Lookup for the Ireland
team added above the
main lookup with
separate docroot and
searchpath](https://image.slidesharecdn.com/jerakiamodded-160202165014/85/Solving-real-world-data-problems-with-Jerakia-58-320.jpg)
![policy :default do
lookup :ireland do
datasource :file, {
:format => :yaml,
:docroot => "/var/external/data/ie",
:searchpath => [
"project/#{scope[:project]}",
"common",
]
}
confine scope[:location], "ie"
confine request.namespace[0], [
"apache",
"php",
]
end
lookup :main, do
datasource :file, {
:format => :yaml,
:docroot => "/var/lib/jerakia",
:searchpath => [
"hostname/#{scope[:fqdn]}",
"environment/#{scope[:environment]}",
"common"
],
}
end
end
Only use this lookup if the
requestor location is IE
and the namespace is
apache or php](https://image.slidesharecdn.com/jerakiamodded-160202165014/85/Solving-real-world-data-problems-with-Jerakia-59-320.jpg)
![policy :default do
lookup :ireland do
datasource :file, {
:format => :yaml,
:docroot => "/var/external/data/ie",
:searchpath => [
"project/#{scope[:project]}",
"common",
]
}
confine scope[:location], "ie"
confine request.namespace[0], [
"apache",
"php",
]
stop
end
lookup :main, do
datasource :file, {
:format => :yaml,
:docroot => "/var/lib/jerakia",
:searchpath => [
"hostname/#{scope[:fqdn]}",
"environment/#{scope[:environment]}",
"common"
],
}
end
end
If this lookup is valid then
do not proceed to the
main lookup, even if data
is not found.](https://image.slidesharecdn.com/jerakiamodded-160202165014/85/Solving-real-world-data-problems-with-Jerakia-60-320.jpg)
![Command line
$ jerakia lookup port —namespace apache
$ jerakia help lookup
Usage:
jerakia lookup [KEY]
Options:
c, [--config=CONFIG] # Configuration file
p, [--policy=POLICY] # Lookup policy
# Default: default
n, [--namespace=NAMESPACE] # Lookup namespace
t, [--type=TYPE] # Lookup type
# Default: first
s, [--scope=SCOPE] # Scope handler
# Default: metadata
[--scope-options=key:value] # Key/value pairs to be passed to the scope handler
m, [--merge-type=MERGE_TYPE] # Merge type
# Default: array
l, [--log-level=LOG_LEVEL] # Log level
v, [--verbose], [--no-verbose] # Print verbose information
D, [--debug], [--no-debug] # Debug information to console, implies --log-level debug
d, [--metadata=key:value] # Key/value pairs to be used as metadata for the lookup
Lookup [KEY] with Jerakia](https://image.slidesharecdn.com/jerakiamodded-160202165014/85/Solving-real-world-data-problems-with-Jerakia-61-320.jpg)
![Integration with Puppet
—-
:backends:
- jerakia
[master]
. . .
data_binding_terminus = jerakia](https://image.slidesharecdn.com/jerakiamodded-160202165014/85/Solving-real-world-data-problems-with-Jerakia-62-320.jpg)
Solving real world data problems with Jerakia
- 1. Solving real world data problems with Jerakia Craig Dunn, Config Management Camp, Ghent 2016
- 2. • Best practice • Code base design • Workflow mangement • Scaling Puppet • Installation and support • Module writing • Throughout Europe www.enviatics.com
- 3. • Puppet user since 2008 • IT consultant for 15+ years • Active community member • The “Roles and Profiles” guy • Problem solver • Lives in Málaga, Spain. • …. and hotels • Daddy! www.craigdunn.org Craig Dunn @crayfishx
- 4. A brief history of Puppet
- 5. In the beginning… • Over complex code • Unsharable modules • Making simple changes required alot of skill. The embedded data era
- 6. class ntp { if $env == ‘dev’ { $server = ‘dev.ntp.local’ } else { if $hostname == ‘gateway’ { $server = ‘pool.ntp.org’ } else { $server = ‘prod.ntp.local’ } } … }
- 7. And then…
- 8. Hiera The dawn of the data separation era
- 9. • Separation of data from code • Module authors could write sharable re-usable code • Code was less complex and more readable • The Forge became useful • Managing data became a lot easier
- 11. Pluggable • Pluggable interchangable backends • Data can be sourced from multiple formats • hiera-eyaml • hiera-mysql • hiera-http • hiera-redis • hiera-consul
- 13. Managing our data is now a critical part of configuration management
- 14. Infrastructure grows and requirements get more complex
- 16. • Different teams and customers require different hierarchies • A particular application needs to source data from a different place • Control access to sub-sets of data for teams within an organisation • Dynamically generate the lookup hierarchy at runtime • Group together application specific data into separate files • Manage encrypted data from any data source • Global hiera.yaml file creates restrictions
- 18. Jerakia • Data lookup tool • Open source • Extendable framework • Solving the most complex edge cases
- 19. Jerakia • Can be used as a Hiera backend • Can be wired directly into Puppet as a data binding terminus • Drop in replacement for Hiera, or not.
- 20. Why Jerakia?
- 21. One design goal…
- 22. Flexibility
- 23. • Lookup behaviour written in Ruby DSL • Almost everything is pluggable • Inter-changable data sources • Easy integration • Hiera compatible*
- 24. $ gem install jerakia
- 25. $ puppet module install crayfishx/jerakia
- 26. • A request is received containing a key and a namespace • A policy is chosen to perform the request • One or more lookups are called to act on the request • A response is sent back to the requestor • Container for lookups • Written in Ruby DSL • Different policies for different apps Policy File
- 27. An Example Jerakia Policy File policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end
- 28. An Example Jerakia Policy File policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end
- 29. An Example Jerakia Policy File policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end
- 30. An Example Jerakia Policy File policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end
- 31. • Lookups are contained within policies • A policy can contain multiple lookups • A lookup always contains at least a data source Lookups
- 32. Scope Handler Request Lookup Plugins Data Source Output Filter Response Data Anatomy of a Jerakia lookup
- 33. Scope Handler Request Lookup Plugins Data Source Output Filter Response Data Anatomy of a Jerakia lookup Request consists of a lookup key, a namespace and some metadata
- 34. Scope Handler Request Lookup Plugins Data Source Output Filter Response Data Anatomy of a Jerakia lookup Information to be used in determining how data is looked up
- 35. Scope Handler Request Lookup Plugins Data Source Output Filter Response Data Anatomy of a Jerakia lookup Lookup plugins can read and modify the scope and request objects
- 36. Scope Handler Request Lookup Plugins Data Source Output Filter Response Data Anatomy of a Jerakia lookup A pluggable data source is used to lookup data
- 37. Scope Handler Request Lookup Plugins Data Source Output Filter Response Data Anatomy of a Jerakia lookup Data returned from the datasource is passed to a pluggable output filter
- 38. Lookup methods
- 39. confine / exclude Invalidates a lookup unless/if the criteria is met confine request.namespsace[0], "apache" confine request.namespsace[0], [ /website_.*/, "apache", "php" ]
- 40. Stop Do not proceed to the next lookup if this lookup is valid lookup :special do … confine request.namespsace[0], "apache" stop end lookup :main do …
- 41. Datasources • Easily pluggable and extendable • File and HTTP datasources shipped out-of-the-box
- 42. Datasources datasource :name, { :option => “value”… }
- 43. Datasource definition lookup :main do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "host/#{scope[:certname]}", "env/#{scope[:environment]}", "common", ] } end /var/lib/jerakia/env/dev/apache.yaml
- 44. lookup :main do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "host/#{scope[:certname]}", "env/#{scope[:environment]}", "common", ] } end /var/lib/jerakia/env/dev/apache.yaml Datasource definition
- 45. lookup :main do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "host/#{scope[:certname]}", "env/#{scope[:environment]}", "common", ] } end /var/lib/jerakia/env/dev/apache.yaml Datasource definition
- 46. /var/lib/jerakia/env/dev/apache.yaml /var/lib/jerakia/env/dev/apache.d/www_corp_com.yaml /var/lib/jerakia/env/dev/apache.d/www_acme_net.yaml /var/lib/jerakia/env/dev/apache.d/www_fake_org.yaml Fragments • Introduced in 0.4 • If a .d directory is found, files within are concatenated • One document is returned
- 47. Data Layout :searchpath => [ "host/#{scope[:certname]}", "env/#{scope[:environment]}", ] # cat /var/lib/jerakia/env/dev/apache.yaml —- port: 80 # cat /var/lib/jerakia/env/dev.yaml —- apache::port: 80 Hiera Jerakia
- 48. Plugins • Access to request and scope • Can read or modify on-the-fly • Re-usable • Cleaner code in policy files
- 49. class Jerakia::Lookup::Plugin module Mything def do_something … end end end Writing plugins • Written as Ruby extensions • Can be placed in the plugin dir • Or shipped as rubygems
- 50. lookup :main, :use => :mything do plugin.mything.do_something … end Using plugins • Plugins are loaded into the lookup • Referenced as plugin.name.method lookup :main, :use => [ :mything, :foo ] do … end
- 51. lookup :main, :use => :hiera do plugin.hiera.rewrite_lookup datasource :file, { :docroot => "/var/lib/jerakia", :format => :yaml, :searchpath => [ "env/#{scope[:environment]}", "common", ] end The hiera plugin • Provides compatibility to hiera filesystem layouts • Shipped with Jerakia # cat /var/lib/jerakia/env/dev.yaml —- apache::port: 80
- 52. Output filters • Pluggable • Specified in the lookup • Parses data returned from the datasource
- 53. Output filters • Two are currently shipped • Encryption (provided by eyaml*) • Strsub *https://github.com/TomPoulton/hiera-eyaml
- 54. Output filters lookup :main do … output_handler :encryption end
- 55. Output filters lookup :main do … output_handler :encryption end
- 56. Example User Story • Team in Ireland manage PHP/Apache • Autonomous team that don’t manage infra • Their optimal hierarchy is different from “ours” • “We” need to service them from Puppet • They must not modify infra services • “We” also manage PHP/Apache for other clients
- 57. policy :default do lookup :main, do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "hostname/#{scope[:fqdn]}", "environment/#{scope[:environment]}", "common" ], } end end Our main lookup is responsible for the entire infrastructure
- 58. policy :default do lookup :ireland do datasource :file, { :format => :yaml, :docroot => "/var/external/data/ie", :searchpath => [ "project/#{scope[:project]}", "common", ] } end lookup :main, do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "hostname/#{scope[:fqdn]}", "environment/#{scope[:environment]}", "common" ], } end end Lookup for the Ireland team added above the main lookup with separate docroot and searchpath
- 59. policy :default do lookup :ireland do datasource :file, { :format => :yaml, :docroot => "/var/external/data/ie", :searchpath => [ "project/#{scope[:project]}", "common", ] } confine scope[:location], "ie" confine request.namespace[0], [ "apache", "php", ] end lookup :main, do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "hostname/#{scope[:fqdn]}", "environment/#{scope[:environment]}", "common" ], } end end Only use this lookup if the requestor location is IE and the namespace is apache or php
- 60. policy :default do lookup :ireland do datasource :file, { :format => :yaml, :docroot => "/var/external/data/ie", :searchpath => [ "project/#{scope[:project]}", "common", ] } confine scope[:location], "ie" confine request.namespace[0], [ "apache", "php", ] stop end lookup :main, do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "hostname/#{scope[:fqdn]}", "environment/#{scope[:environment]}", "common" ], } end end If this lookup is valid then do not proceed to the main lookup, even if data is not found.
- 61. Command line $ jerakia lookup port —namespace apache $ jerakia help lookup Usage: jerakia lookup [KEY] Options: c, [--config=CONFIG] # Configuration file p, [--policy=POLICY] # Lookup policy # Default: default n, [--namespace=NAMESPACE] # Lookup namespace t, [--type=TYPE] # Lookup type # Default: first s, [--scope=SCOPE] # Scope handler # Default: metadata [--scope-options=key:value] # Key/value pairs to be passed to the scope handler m, [--merge-type=MERGE_TYPE] # Merge type # Default: array l, [--log-level=LOG_LEVEL] # Log level v, [--verbose], [--no-verbose] # Print verbose information D, [--debug], [--no-debug] # Debug information to console, implies --log-level debug d, [--metadata=key:value] # Key/value pairs to be used as metadata for the lookup Lookup [KEY] with Jerakia
- 62. Integration with Puppet —- :backends: - jerakia [master] . . . data_binding_terminus = jerakia
- 64. Upcoming in 0.5 • Data Schemas • Better REST client/server • Deep merge behaviour • Lookup plugin “load method”
- 65. Contributions wanted • Code maturity • Caching • Features • Bugfixes • Documentation • #jerakia (freenode) Sponsored by
- 66. Jerakia 1.0