Puppet Camp Paris 2016 Data in Modules
2 likes1,440 views
The document presents an overview of data management in Puppet 4, emphasizing the separation of code and data using Hiera for configuration. It discusses explicit and implicit data lookups, configuration of data providers, and methods for organizing data in modules and environments. The content covers both advantages and disadvantages of these approaches, along with practical examples and references for further reading.
![Copyright example42 GmbH - 2016
Separation of
Code and Data
• data in code
class my_ntp {!
!if $::environment == ‘dev’ {!
$ntp_server = [‘pool.ntp.org’]!
} else {!
if $::facts[‘fqdn’] == ‘ntp1.example42.com’
{!
# switch back to ntp1 when issue is solved!
$ntp_server = [‘ntp2.example42.com’]!
} else {!
$ntp_server = [‘127.0.0.1’]!
}!
}!](https://image.slidesharecdn.com/puppetcampparis2016-datainmodules-160614170033/85/Puppet-Camp-Paris-2016-Data-in-Modules-5-320.jpg)
![Copyright example42 GmbH - 2016
Separation of
Code and Data
• explicit lookup
• hiera(‘key’[, ‘default’][, ‘override hierarchy’])](https://image.slidesharecdn.com/puppetcampparis2016-datainmodules-160614170033/85/Puppet-Camp-Paris-2016-Data-in-Modules-6-320.jpg)
![Copyright example42 GmbH - 2016
Separation of
Code and Data
• hiera.yaml
# version 1!
:backends:!
- yaml!
:yaml:!
:datadir: “/etc/puppetlabs/code/
environments/%{environment}/hieradata”!
:hierarchy:!
- “nodes/%{::trusted.certname}”!
- “os/%{::facts[‘os’][‘osfamily’]}”!
- common!](https://image.slidesharecdn.com/puppetcampparis2016-datainmodules-160614170033/85/Puppet-Camp-Paris-2016-Data-in-Modules-8-320.jpg)
![Copyright example42 GmbH - 2016
Data Provider
Function - Puppet
# ntp/functions/ntp/data.pp
function ntp::data() {!
$params = {!
‘ntp::ntpservers’ => [‘pool.ntp.org’],!
}!
$os_params = case $facts[‘os’][‘family’] {!
‘Debian’: {!
{ ‘ntp::ntpackage’ => ‘ntpd’, }!
},!
default: {!
{}!
}!
}!
$params + $os_params!
}](https://image.slidesharecdn.com/puppetcampparis2016-datainmodules-160614170033/85/Puppet-Camp-Paris-2016-Data-in-Modules-26-320.jpg)
![Copyright example42 GmbH - 2016
Data Provider
Function - Ruby
# ntp/lib/puppet/functions/ntp/data.rb
Puppet::Functions.create_function(:’ntp::data’) do!
def base_data()!
{ ‘ntp::ntpservers’ => [‘pool.ntp.org’], }!
end!
def os_data()!
case Facter.value(:os)[‘family’]!
when ‘Debian’!
{ ‘ntp::pkgname’ => ‘ntpd’, }!
else!
{}!
end!
def data()!
self.base_data.merge!(self.os_data)!
end!
end](https://image.slidesharecdn.com/puppetcampparis2016-datainmodules-160614170033/85/Puppet-Camp-Paris-2016-Data-in-Modules-28-320.jpg)
Puppet Camp Paris 2016 Data in Modules
- 1. Copyright example42 GmbH - 2016 Puppet 4 - Data in Modules PuppetCamp Paris 2016 Martin Alfke - ma@example42.com Image: Tatlin - tatlin.net Copyright example42 GmbH - 2016
- 2. Copyright example42 GmbH - 2016 Martin Alfke ! Berlin/Germany ! CEO example42 GmbH Freelance Puppet Expert Network ! Puppet since 2007 ! Puppet Trainer, Consultant ! Co-Author of “Puppet 4 Essentials”
- 3. Copyright example42 GmbH - 2016 Puppet 4 Data in Modules • Separation of Code and Data • Data in Modules • Lookup Priority • Data in Component Modules • Data in Environments
- 4. Copyright example42 GmbH - 2016 Separation of Code and Data Image: Tatlin - tatlin.net
- 5. Copyright example42 GmbH - 2016 Separation of Code and Data • data in code class my_ntp {! !if $::environment == ‘dev’ {! $ntp_server = [‘pool.ntp.org’]! } else {! if $::facts[‘fqdn’] == ‘ntp1.example42.com’ {! # switch back to ntp1 when issue is solved! $ntp_server = [‘ntp2.example42.com’]! } else {! $ntp_server = [‘127.0.0.1’]! }! }!
- 6. Copyright example42 GmbH - 2016 Separation of Code and Data • explicit lookup • hiera(‘key’[, ‘default’][, ‘override hierarchy’])
- 7. Copyright example42 GmbH - 2016 Separation of Code and Data • implicit lookup class my_ntp (! !Array $ntp_server,! ) {! !# …! }! ! contain my_ntp! ! # hiera data! my_ntp::ntp_server:! - ‘pool.ntp.org’!
- 8. Copyright example42 GmbH - 2016 Separation of Code and Data • hiera.yaml # version 1! :backends:! - yaml! :yaml:! :datadir: “/etc/puppetlabs/code/ environments/%{environment}/hieradata”! :hierarchy:! - “nodes/%{::trusted.certname}”! - “os/%{::facts[‘os’][‘osfamily’]}”! - common!
- 9. Copyright example42 GmbH - 2016 Separation of Code and Data • hieradata os/Debian.yaml apache::pkgname:! - ‘apache2’! - ‘apache2-ssl’! ! os/RedHat.yaml apache::pkgname:! - ‘httpd’! ! common.yaml apache::purge_configs: true!
- 10. Copyright example42 GmbH - 2016 Separation of Code and Data • hieradata os/FreeBSD.yaml apache::pkgname:! - ‘apache’!
- 11. Copyright example42 GmbH - 2016 Separation of Code and Data • puppet code # apache/manifests/params.pp class apache::params {! case $::operatingsystem {! ‘Debian’: { # … }! ‘RedHat’: { # … }! default: {! fail(‘OS not supported’)! }! }! }!
- 12. Copyright example42 GmbH - 2016 Data in Modules Image: Tatlin - tatlin.net
- 13. Copyright example42 GmbH - 2016 Lookup Priority Image: Tatlin - tatlin.net
- 14. Copyright example42 GmbH - 2016 Lookup Priority • Hiera -> Global Overrides ! ! !
- 15. Copyright example42 GmbH - 2016 Lookup Priority • Hiera -> Global Overrides ! • Environment Data -> Core Puppet Lookup !
- 16. Copyright example42 GmbH - 2016 Lookup Priority • Hiera -> Global Overrides ! • Environment Data -> Core Puppet Lookup ! • Module Data -> Defaults
- 17. Copyright example42 GmbH - 2016 Lookup Functions • Explicit lookup: lookup(‘key’)! ! • CLI lookup: puppet lookup ‘key’ ! • Automatic lookup: ‘<namespace>::<key>’
- 18. Copyright example42 GmbH - 2016 Lookup Functions • lookup(‘key’, <Type>, <merge_behavior>, <default>)! • e.g. lookup(‘ntp_servers’, Array) • Merge behavior: • first! • unique (array merge)! • hash! • deep!
- 19. Copyright example42 GmbH - 2016 Data Provider Configuration • Global Environment Provider: puppet.conf • environment_data_provider = <data provider>! ! ! !
- 20. Copyright example42 GmbH - 2016 Data Provider Configuration • Global Environment Provider: puppet.conf • environment_data_provider = <data provider>! • Environment Provider: environment.conf • environment_data_provider = <data provider>! !
- 21. Copyright example42 GmbH - 2016 Data Provider Configuration • Global Environment Provider: puppet.conf • environment_data_provider = <data provider>! • Environment Provider: environment.conf • environment_data_provider = <data provider>! • Module Provider: metadata.json • “data_provider”: “<data provider>”
- 22. Copyright example42 GmbH - 2016 Data Provider • none -> standard hiera lookup ! • hiera -> hiera lookup (hiera v4) ! • function -> data function lookup
- 23. Copyright example42 GmbH - 2016 Data Provider Hiera • replace hiera, hiera_array, hiera_hash with ‘lookup’ • needs hiera.yaml v4 configuration file • set data_provider to ‘hiera’ in puppet.conf, environment.conf or metadata.json • modify global hiera.yaml to use datadir outside environment
- 24. Copyright example42 GmbH - 2016 Data Provider Hiera # /etc/puppetlabs/code/environments/production/hiera.yaml # /etc/puppetlabs/code/environments/production/modules/<module>/ hiera.yaml # - - -! version: 4! datadir: hieradata! hierarchy:! - name: “Nodes”! backend: yaml! path: “nodes/%{trusted.certname}”! - name: “OS”! backend: json! path: “os/%{facts.os.family}”! - name: “common”! backend: yaml
- 25. Copyright example42 GmbH - 2016 Data Provider Function • write data function • Puppet 4 Function • <module>/functions/<module>/data.pp • <env>/functions/<env>/data.pp • set data_provider to function in puppet.conf, environment.conf or metadata.json
- 26. Copyright example42 GmbH - 2016 Data Provider Function - Puppet # ntp/functions/ntp/data.pp function ntp::data() {! $params = {! ‘ntp::ntpservers’ => [‘pool.ntp.org’],! }! $os_params = case $facts[‘os’][‘family’] {! ‘Debian’: {! { ‘ntp::ntpackage’ => ‘ntpd’, }! },! default: {! {}! }! }! $params + $os_params! }
- 27. Copyright example42 GmbH - 2016 Data Provider Function • write data function • Ruby Function (Puppet 4 function API) • <module>/lib/puppet/functions/<module>/ data.rb • <env>/lib/puppet/functions/<env>/data.rb • set data_provider to function in puppet.conf, environment.conf or metadata.json
- 28. Copyright example42 GmbH - 2016 Data Provider Function - Ruby # ntp/lib/puppet/functions/ntp/data.rb Puppet::Functions.create_function(:’ntp::data’) do! def base_data()! { ‘ntp::ntpservers’ => [‘pool.ntp.org’], }! end! def os_data()! case Facter.value(:os)[‘family’]! when ‘Debian’! { ‘ntp::pkgname’ => ‘ntpd’, }! else! {}! end! def data()! self.base_data.merge!(self.os_data)! end! end
- 29. Copyright example42 GmbH - 2016 Data in Component Modules Image: Tatlin - tatlin.net
- 30. Copyright example42 GmbH - 2016 Data in Component Modules • add data provider to metadata.json • provide OS defaults • remove params.pp / remove inheritance • allow users to overwrite any data
- 31. Copyright example42 GmbH - 2016 Data Provider Function - Ruby # my_ntp/manifests/init.pp class my_ntp (! $server = $my_ntp::params::server,! $pkgname = $my_ntp::params::pkgname,! $secure = $my_ntp::params::secure,! ) inherits my_ntp::params {! # ...! }!
- 32. Copyright example42 GmbH - 2016 Data in Environments Image: Tatlin - tatlin.net
- 33. Copyright example42 GmbH - 2016 Data in Environments • old hiera replacement • add hiera.yaml to environment base path • overwrite data from modules, roles & profiles
- 34. Copyright example42 GmbH - 2016 Summary Image: Tatlin - tatlin.net
- 35. Copyright example42 GmbH - 2016 Summary # /etc/puppetlabs/code/ ! ! hiera.yaml! ! ! hieradata/! ! ! environments/production/ ! ! ! ! ! ! environment.conf! ! ! ! ! ! ! hiera.yaml! ! ! ! ! ! ! hieradata/! modules/my_module/ ! ! ! ! ! ! ! ! ! ! metadata.json! ! ! ! ! ! ! ! ! ! ! hiera.yaml! ! ! ! ! ! ! ! ! ! ! hieradata/
- 36. Copyright example42 GmbH - 2016 Summary # /etc/puppetlabs/code/ ! ! hiera.yaml! ! ! hieradata/! ! ! environments/production/ ! ! ! ! ! environment.conf! ! ! ! ! ! lib/functions/data.pp! ! ! ! ! ! lib/puppet/functions/ ! ! ! ! ! ! ! ! ! production/data.rb! modules/my_module/ ! ! ! ! ! ! !! ! metadata.json! ! ! ! ! ! ! ! ! lib/functions/data.pp! ! ! ! ! ! ! ! ! lib/puppet/functions/ ! ! ! ! ! ! ! ! ! ! ! ! my_module/data.rb
- 37. Copyright example42 GmbH - 2016 Summary - Pro • Per hierarchy Hiera Data backend possible • Data Function lookups without need for hiera backend (e.g. Cloud Management API data) • No more inheritance required
- 38. Copyright example42 GmbH - 2016 Summary - Con • No single Source of Authority? • Debugging can be complex when iterating over many data providers and hierarchies
- 39. Copyright example42 GmbH - 2016 Module Developers • switch to data in modules • give users the possibility to provide own data • allow users to overwrite any data • allow users to know their data for missing OS support
- 40. Copyright example42 GmbH - 2016 Module Users • switch to hieradata in modules then • switch to data in environments • keep data simple and readable • don’t overcomplicate !
- 41. Copyright example42 GmbH - 2016 Module Users • hieradata common.yaml my_ntp: ‘pool.ntp.org’! apache::default_mods: false! apache::purge_configs: true! mysql::remove_default_accounts: true! mysql::root_password: ‘puppet’! oradb::database::version: ’12.1’! oradb::shout: ‘MISSING DATA’
- 42. Copyright example42 GmbH - 2016 References • http://docs.puppetlabs.com/puppet/4.3/reference/ lookup_quick.html • http://docs.puppetlabs.com/puppet/4.3/reference/ lookup_quick_module.html • http://puppet-on-the-edge.blogspot.de/2015/01/ puppet-40-data-in-modules-and.html
- 43. Copyright example42 GmbH - 2016 Puppet 4 - Data in modules PuppetCamp Paris 2016 Martin Alfke - ma@example42.com Image: Tatlin - tatlin.net Copyright example42 GmbH - 2016