Understanding the Data Lookup Pattern
3 likes665 views
This document summarizes a presentation about the data lookup pattern and Jerakia, a tool for hierarchical data lookups. It discusses the problems of embedding data in code and managing data across diverse infrastructure. Jerakia provides a solution with hierarchical lookups of data stored separately from code and queried from various sources. It allows flexible definition of lookup policies and scopes to retrieve the appropriate data value.
![JERAKIA: POLICIES
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/lib/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end](https://image.slidesharecdn.com/jerakiacfgmgmt2pdffinal-180206155047/85/Understanding-the-Data-Lookup-Pattern-34-320.jpg)
![JERAKIA: POLICIES
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/lib/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end](https://image.slidesharecdn.com/jerakiacfgmgmt2pdffinal-180206155047/85/Understanding-the-Data-Lookup-Pattern-35-320.jpg)
![JERAKIA: POLICIES
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/lib/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end](https://image.slidesharecdn.com/jerakiacfgmgmt2pdffinal-180206155047/85/Understanding-the-Data-Lookup-Pattern-36-320.jpg)
![JERAKIA: POLICIES
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/lib/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end](https://image.slidesharecdn.com/jerakiacfgmgmt2pdffinal-180206155047/85/Understanding-the-Data-Lookup-Pattern-37-320.jpg)
![JERAKIA: POLICIES
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/lib/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end
/var/lib/jerakia/data/host/baz.foo.com/apache.yaml](https://image.slidesharecdn.com/jerakiacfgmgmt2pdffinal-180206155047/85/Understanding-the-Data-Lookup-Pattern-38-320.jpg)
![JERAKIA: POLICIES
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/lib/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end
/var/lib/jerakia/data/env/dev/apache.yaml](https://image.slidesharecdn.com/jerakiacfgmgmt2pdffinal-180206155047/85/Understanding-the-Data-Lookup-Pattern-39-320.jpg)
![JERAKIA: POLICIES
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/lib/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end
/var/lib/jerakia/data/common/apache.yaml](https://image.slidesharecdn.com/jerakiacfgmgmt2pdffinal-180206155047/85/Understanding-the-Data-Lookup-Pattern-40-320.jpg)
![ADVANCED FEATURES: CONFINE / EXCLUDE / STOP
policy :default do
lookup :ireland do
datasource :file, {
:format => :yaml,
:docroot => "/var/external/data/ie",
:searchpath => [
"project/#{scope[:project]}",
"common",
]
}
confine scope[:location], "ie"
confine request.namespace[0], "apache"
stop
end
lookup :main, do
datasource :file, {
:format => :yaml,
:docroot => "/var/lib/jerakia",
:searchpath => [
"hostname/#{scope[:fqdn]}",
"environment/#{scope[:environment]}",
"common"
],
}
end
end
➤ Isolate lookups](https://image.slidesharecdn.com/jerakiacfgmgmt2pdffinal-180206155047/85/Understanding-the-Data-Lookup-Pattern-42-320.jpg)
![➤ By default Jerakia will return the first value found while
traversing the hierarchy
➤ We can request for the entire hierarchy be searched and all
instances of a value returned in one array or hashmap
ADVANCED FEATURES: MERGE BEHAVIOUR
users:
- lucy
- bob
- mike
users:
- root
- craig
- jan
- max
- jessica
{
users: [
"lucy",
"bob",
"mike",
"root",
"craig",
"max",
"jessica"
]
}
CommonEnv (dev)](https://image.slidesharecdn.com/jerakiacfgmgmt2pdffinal-180206155047/85/Understanding-the-Data-Lookup-Pattern-44-320.jpg)
Understanding the Data Lookup Pattern
- 1. UNDERSTANDING THE DATA LOOKUP PATTERN Craig Dunn & Jan Ivar Beddari Config Management Camp, February 2018
- 2. CRAIG DUNN ➤ Freelance Consultant & Trainer ➤ Long time Puppet community member ➤ Problem Solver www.craigdunn.org @crayfishx
- 3. JAN IVAR BEDDARI ➤ Problem Solver ➤ Long time Jerakia advocate ➤ Works at Safespring ➤ Makes people happy.... @beddari
- 4. CONFIGURATION MANAGEMENT ➤ Automation ➤ Repeatability ➤ Many tool options available for configuring components ➤ Reducing the burden of managing infrastructure
- 5. ➤ Configuration management gives a consistent reliable method to configure a component of infrastructure ➤ Infrastructure configuration involves a lot of custom data... ➤ It's critical to maintaining our systems INFRASTRUCTURE DATA usernamesdomain_name firewall_rules apache_port gateway_ip dns_servers ntp_servers groups packages timezonelimitsversion
- 6. THE DATA PROBLEM ➤ Where do we store it? ➤ How do we query it? ➤ How do we avoid duplicating it? ➤ Whats the best way to express the data?
- 7. THE DATA PROBLEM ➤ Embedded data in code ➤ Data management as infrastructure gets more diverse
- 8. THE DATA PROBLEM ➤ Embedded data in code ➤ Code sharing becomes impossible ➤ Complexity increases ➤ High barrier to a simple task "The timezone should be set to UTC"
- 9. THE DATA PROBLEM ➤ Managing data ➤ Infrastructure is spread out across environments, regions and business streams ➤ Data will differ depending on a multitude of factors ➤ The edge cases will get you
- 10. THE DATA PROBLEM usernamesdomain_name firewall_rules apache_port gateway_ip dns_servers ntp_servers groups packages timezonelimitsversion
- 11. THE DATA PROBLEM DEV QA PROD DC1 DC2 DC3 Special Snowflake
- 12. THE DATA PROBLEM
- 13. ➤ Embedded code unmaintainable ➤ Complexity PUPPET: THE EARLY YEARS if $env == ‘dev’ { $server = ‘dev.ntp.local’ } else { if $hostname == ‘gateway’ { $server = ‘pool.ntp.org’ } else { $server = ‘prod.ntp.local’ } }
- 14. ➤ Hiera is the data lookup sub system for Puppet ➤ Data can be looked up from multiple sources ➤ Data separation ➤ Hierarchical lookups PUPPET: THE HIERA WAY
- 15. ➤ Remove complexity from the code ➤ Code becomes re-usable and easily shared ➤ Minimal expertise required to manage data values DATA SEPARATION ntp::server: pool.ntp.org
- 17. HIERARCHICAL LOOKUPS ➤ Sell globally in USD ➤ Bill in EUR for European countries ➤ GBP for the UK, CHF for Switzerland
- 18. HIERARCHICAL LOOKUPS: SCOPE ➤ Scope: Information used to determine what data to lookup ➤ The scope in this example is the country & the continent ➤ We look up the data using the scope data
- 19. HIERARCHICAL LOOKUPS "What is the value for currency in the context of this scope..."
- 20. HIERARCHICAL LOOKUPS ➤ Hierarchical lookups are suited to infrastructure ➤ We often express infrastructure from most granular to most encompassing factors
- 21. HIERARCHICAL LOOKUPS ➤ Hierarchical lookups are suited to infrastructure ➤ We often express infrastructure from most granular to most encompassing factors
- 22. HIERARCHICAL LOOKUPS ➤ Hierarchical lookups are suited to infrastructure ➤ We often express infrastructure from most granular to most encompassing factors
- 23. HIERARCHICAL LOOKUPS ➤ Hierarchical lookups are suited to infrastructure ➤ We often express infrastructure from most granular to most encompassing factors
- 24. HIERARCHICAL LOOKUPS ➤ Hierarchical lookups are suited to infrastructure ➤ We often express infrastructure from most granular to most encompassing factors
- 25. ➤ Hierarchical lookups express infrastructure data well ➤ There is a lack of generic tooling in this space ➤ The data was getting harder ➤ Then there was Baloise... WHAT I LEARNT
- 26. INTRODUCING JERAKIA ➤ Released in 2015 ➤ Hierarchical data lookup tool ➤ http://jerakia.io
- 28. INTRODUCING JERAKIA: GOALS ➤ De-coupled from any particular tool ➤ Pluggable and extendable ➤ Flexible configuration
- 29. INTRODUCING JERAKIA: FEATURES ➤ Standalone with a REST API ➤ Look up data from a variety of data sources ➤ Define your own hierarchical structure ➤ Flexibility around lookup behaviour
- 30. JERAKIA LOOKUPS ➤ A lookup request contains ➤ The lookup key ➤ A namespace ➤ The scope of the lookup
- 31. JERAKIA LOOKUPS ➤ A lookup is run against a Jerakia policy ➤ Written in Ruby ➤ A policy is a container for one or more lookups ➤ Lookups query a data source ➤ Results are returned to the requestor
- 32. JERAKIA DATASOURCES ➤ A pluggable source of data ➤ Each lookup invokes one data source ➤ Jerakia ships with data sources: ➤ File ➤ HTTP ➤ Consul ➤ Custom data sources are easy to write ➤ https://github.com/crayfishx/jerakia-datasource-template
- 33. ➤ Search through a filesystem hierarchy ➤ Data values are stored in YAML or JSON ➤ Typically maintained in and deployed from a VCS ➤ <dir>/<namespace>.yaml FILE DATASOURCE hostname/foo.enviatics.com/apache.yaml environment/production/apache.yaml location/london/apache.yaml common/apache.yaml
- 34. JERAKIA: POLICIES policy :main do lookup :default do datasource :file, { :docroot => "/var/lib/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end
- 35. JERAKIA: POLICIES policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end policy :main do lookup :default do datasource :file, { :docroot => "/var/lib/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end
- 36. JERAKIA: POLICIES policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end policy :main do lookup :default do datasource :file, { :docroot => "/var/lib/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end
- 37. JERAKIA: POLICIES policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end policy :main do lookup :default do datasource :file, { :docroot => "/var/lib/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end
- 38. JERAKIA: POLICIES policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end policy :main do lookup :default do datasource :file, { :docroot => "/var/lib/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end /var/lib/jerakia/data/host/baz.foo.com/apache.yaml
- 39. JERAKIA: POLICIES policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end policy :main do lookup :default do datasource :file, { :docroot => "/var/lib/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end /var/lib/jerakia/data/env/dev/apache.yaml
- 40. JERAKIA: POLICIES policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end policy :main do lookup :default do datasource :file, { :docroot => "/var/lib/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end /var/lib/jerakia/data/common/apache.yaml
- 41. JERAKIA: POLICIES Scope Handler Request Lookup Plugins Data Source Output Filter Response Data Request consists of a lookup key, a namespace and some metadata Information to be used in determining how data is looked up Lookup plugins can read and modify the scope and A pluggable data source is used to lookup data Data returned from the datasource is passed to a pluggable output filter
- 42. ADVANCED FEATURES: CONFINE / EXCLUDE / STOP policy :default do lookup :ireland do datasource :file, { :format => :yaml, :docroot => "/var/external/data/ie", :searchpath => [ "project/#{scope[:project]}", "common", ] } confine scope[:location], "ie" confine request.namespace[0], "apache" stop end lookup :main, do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "hostname/#{scope[:fqdn]}", "environment/#{scope[:environment]}", "common" ], } end end ➤ Isolate lookups
- 43. ADVANCED FEATURES: HANDLING SECRETS ➤ Jerakia supports decrypting values from any data source ➤ Uses a pluggable encryption provider ➤ Vault provider native ➤ Uses Vaults' transit feature as "cryptography as a service" ➤ Custom encryption providers possible https://www.craigdunn.org/2017/04/managing-puppet-secrets-with-jerakia-and-vault/
- 44. ➤ By default Jerakia will return the first value found while traversing the hierarchy ➤ We can request for the entire hierarchy be searched and all instances of a value returned in one array or hashmap ADVANCED FEATURES: MERGE BEHAVIOUR users: - lucy - bob - mike users: - root - craig - jan - max - jessica { users: [ "lucy", "bob", "mike", "root", "craig", "max", "jessica" ] } CommonEnv (dev)
- 45. ADVANCED FEATURES: READ MORE http://jerakia.io ➤ Plugins ➤ Schemas ➤ Datasources ➤ Output Filters ➤ API documentation
- 46. THE THIRD DATA PROBLEM Everyone Else When people starting saying "DevOps"...
- 47. THE THIRD DATA PROBLEM Everyone Else ...2018
- 48. THE THIRD DATA PROBLEM ➤ We use lots of tools to manage our infrastructure ➤ Provisioning ➤ Orchestration ➤ Configuration Management ➤ Monitoring ➤ Different tools have different data patterns ➤ The actual data is the same
- 49. JERAKIA: AIM ➤ A centralized source for all data lookups ➤ Powerful hierarchical lookups ➤ Current front integrations; ➤ Puppet (Hiera 5 backend) ➤ https://forge.puppet.com/crayfishx/jerakia ➤ Ansible (Lookup plugin) ➤ https://github.com/crayfishx/ansible-jerakia ➤ Potential for many more!
- 51. AND FINALLY... ➤ Features ➤ Ideas ➤ Fixes ➤ ... any other feedback https://github.com/crayfishx/jerakia
- 52. http://jerakia.io Craig Dunn @crayfishx Jan Ivar Beddari @beddari Thank you!