The Grand Puppet Sub-Systems Tour - Nicholas Fagerlund, Puppet Labs
3 likes1,326 views
The document provides an overview of how Puppet agent runs a catalog by starting the Puppet daemon, which then runs the Puppet configurer. The configurer retrieves facts, prepares and retrieves a catalog from the Puppet master's compiler terminus via the catalog indirection and HTTP API. It then applies the catalog by creating a transaction to evaluate resources and report changes.
![Presented by
#run_command
#
lib/puppet/application/agent.rb
def
run_command
if
options[:fingerprint]
fingerprint
else
#
It'd
be
nice
to
daemonize
later,
but
we
have
to
daemonize
before
#
waiting
for
certificates
so
that
we
don't
block
daemon
=
daemonize_process_when(Puppet[:daemonize])
!
wait_for_certificates
!
if
Puppet[:onetime]
onetime(daemon)
else
main(daemon)
end](https://image.slidesharecdn.com/thegrandpuppetsub-systemstour-nicholasfagerlundpuppetlabs-140925200845-phpapp02/85/The-Grand-Puppet-Sub-Systems-Tour-Nicholas-Fagerlund-Puppet-Labs-8-320.jpg)
![Presented by
onetime() and main()
#
lib/puppet/application/agent.rb
begin
exitstatus
=
daemon.agent.run
rescue
=>
detail
Puppet.log_exception(detail)
end
!
daemon.stop(:exit
=>
false)
!
if
not
exitstatus
exit(1)
elsif
options[:detailed_exitcodes]
then
exit(exitstatus)
else
exit(0)
end
#
lib/puppet/application/agent.rb
def
main(daemon)
if
Puppet[:listen]
setup_listen(daemon)
end
Puppet.notice
"Starting
Puppet
client
version
#{Puppet.version}"
!
daemon.start
end](https://image.slidesharecdn.com/thegrandpuppetsub-systemstour-nicholasfagerlundpuppetlabs-140925200845-phpapp02/85/The-Grand-Puppet-Sub-Systems-Tour-Nicholas-Fagerlund-Puppet-Labs-9-320.jpg)
![Presented by
Obtaining the daemon
#
lib/puppet/application/agent.rb
def
daemonize_process_when(should_daemonize)
daemon
=
Puppet::Daemon.new(Puppet::Util::Pidlock.new(Puppet[:pidfile]))
daemon.argv
=
@argv
daemon.agent
=
@agent
!
daemon.daemonize
if
should_daemonize
!
daemon
end](https://image.slidesharecdn.com/thegrandpuppetsub-systemstour-nicholasfagerlundpuppetlabs-140925200845-phpapp02/85/The-Grand-Puppet-Sub-Systems-Tour-Nicholas-Fagerlund-Puppet-Labs-10-320.jpg)
![Puppet::Configurer#run_internal
Presented by
#
lib/puppet/configurer.rb
def
run_internal(options)
#
We
create
the
report
pre-‐populated
with
default
settings
for
#
environment
and
transaction_uuid
very
early,
this
is
to
ensure
#
they
are
sent
regardless
of
any
catalog
compilation
failures
or
#
exceptions.
options[:report]
||=
Puppet::Transaction::Report.new("apply",
nil,
@environment,
@transaction_uuid)
report
=
options[:report]
init_storage
!
Puppet::Util::Log.newdestination(report)
…
…
…](https://image.slidesharecdn.com/thegrandpuppetsub-systemstour-nicholasfagerlundpuppetlabs-140925200845-phpapp02/85/The-Grand-Puppet-Sub-Systems-Tour-Nicholas-Fagerlund-Puppet-Labs-18-320.jpg)
![Puppet::Configurer#run_internal
Presented by
#
lib/puppet/configurer/fact_handler.rb
…
facts
=
Puppet::Node::Facts.indirection.find(Puppet[:node_name_value],
:environment
=>
@environment)
…](https://image.slidesharecdn.com/thegrandpuppetsub-systemstour-nicholasfagerlundpuppetlabs-140925200845-phpapp02/85/The-Grand-Puppet-Sub-Systems-Tour-Nicholas-Fagerlund-Puppet-Labs-20-320.jpg)
![Presented by
RAL Catalog
#
lib/puppet/resource/catalog.rb
def
to_catalog(convert)
result
=
self.class.new(self.name,
self.environment_instance)
result.version
=
self.version
map
=
{}
resources.each
do
|resource|
next
if
virtual_not_exported?(resource)
next
if
block_given?
and
yield
resource
newres
=
resource.copy_as_resource
newres.catalog
=
result
if
convert
!=
:to_resource
newres
=
newres.to_ral
end
#
We
can't
guarantee
that
resources
don't
munge
their
names
#
(like
files
do
with
trailing
slashes),
so
we
have
to
keep
track
#
of
what
a
resource
got
converted
to.
map[resource.ref]
=
newres
result.add_resource
newres
end](https://image.slidesharecdn.com/thegrandpuppetsub-systemstour-nicholasfagerlundpuppetlabs-140925200845-phpapp02/85/The-Grand-Puppet-Sub-Systems-Tour-Nicholas-Fagerlund-Puppet-Labs-29-320.jpg)
![Presented by
RAL Catalog
#
lib/puppet/configurer.rb
def
apply_catalog(catalog,
options)
report
=
options[:report]
report.configuration_version
=
catalog.version
!
benchmark(:notice,
"Finished
catalog
run")
do
catalog.apply(options)
end
!
report.finalize_report
report
end](https://image.slidesharecdn.com/thegrandpuppetsub-systemstour-nicholasfagerlundpuppetlabs-140925200845-phpapp02/85/The-Grand-Puppet-Sub-Systems-Tour-Nicholas-Fagerlund-Puppet-Labs-32-320.jpg)
![Presented by
Back to Puppet::Configurer
#
lib/puppet/configurer.rb
result
=
Puppet::Resource::Catalog.indirection.find(
Puppet[:node_name_value],
query_options.merge(:ignore_cache
=>
true,
:environment
=>
@environment,
:fail_on_404
=>
true)
)](https://image.slidesharecdn.com/thegrandpuppetsub-systemstour-nicholasfagerlundpuppetlabs-140925200845-phpapp02/85/The-Grand-Puppet-Sub-Systems-Tour-Nicholas-Fagerlund-Puppet-Labs-41-320.jpg)
![Presented by
Avoiding the cache
#
lib/puppet/configurer.rb
result
=
Puppet::Resource::Catalog.indirection.find(
Puppet[:node_name_value],
query_options.merge(:ignore_cache
=>
true,
:environment
=>
@environment,
:fail_on_404
=>
true)
)](https://image.slidesharecdn.com/thegrandpuppetsub-systemstour-nicholasfagerlundpuppetlabs-140925200845-phpapp02/85/The-Grand-Puppet-Sub-Systems-Tour-Nicholas-Fagerlund-Puppet-Labs-50-320.jpg)
![Presented by
The app object
#
lib/puppet/network/http/rack.rb
class
Puppet::Network::HTTP::Rack
def
call(env)
request
=
Rack::Request.new(env)
response
=
Rack::Response.new
Puppet.debug
'Handling
request:
%s
%s'
%
[request.request_method,
request.fullpath]
!
begin
Puppet::Network::HTTP::RackREST.new.process(request,
response)
rescue
=>
detail
#
Send
a
Status
500
Error
on
unhandled
exceptions.
response.status
=
500
response['Content-‐Type']
=
'text/plain'
response.write
'Internal
Server
Error:
"%s"'
%
detail.message
#
log
what
happened
Puppet.log_exception(detail,
"Puppet
Server
(Rack):
Internal
Server
Error:
Unhandled
Exception:
"%s""
%
detail.message)
end
response.finish](https://image.slidesharecdn.com/thegrandpuppetsub-systemstour-nicholasfagerlundpuppetlabs-140925200845-phpapp02/85/The-Grand-Puppet-Sub-Systems-Tour-Nicholas-Fagerlund-Puppet-Labs-59-320.jpg)
The Grand Puppet Sub-Systems Tour - Nicholas Fagerlund, Puppet Labs
- 1. 2014 Presented by The Grand Puppet Sub-Systems Tour Nick Fagerlund Technical Writer / Weird Bugs | Puppet Labs @nfagerlund
- 3. Presented by What does Puppet DO? • Gather system information • Compile a catalog • Apply catalog • Report
- 4. Presented by What does Puppet DO? AGENT • Gather system information • Apply catalog • Report MASTER • Compile a catalog
- 5. STOP 1: Command Line and the Presented by
- 6. Presented by Someone ran puppet agent
- 7. Presented by ??? • /opt/puppet/bin/puppet • Puppet::Util::CommandLine.new.execute • Puppet::Application.find • Puppet::Application::Agent.new • Puppet::Application::Agent#setup • Puppet::Application::Agent#run_command
- 8. Presented by #run_command # lib/puppet/application/agent.rb def run_command if options[:fingerprint] fingerprint else # It'd be nice to daemonize later, but we have to daemonize before # waiting for certificates so that we don't block daemon = daemonize_process_when(Puppet[:daemonize]) ! wait_for_certificates ! if Puppet[:onetime] onetime(daemon) else main(daemon) end
- 9. Presented by onetime() and main() # lib/puppet/application/agent.rb begin exitstatus = daemon.agent.run rescue => detail Puppet.log_exception(detail) end ! daemon.stop(:exit => false) ! if not exitstatus exit(1) elsif options[:detailed_exitcodes] then exit(exitstatus) else exit(0) end # lib/puppet/application/agent.rb def main(daemon) if Puppet[:listen] setup_listen(daemon) end Puppet.notice "Starting Puppet client version #{Puppet.version}" ! daemon.start end
- 10. Presented by Obtaining the daemon # lib/puppet/application/agent.rb def daemonize_process_when(should_daemonize) daemon = Puppet::Daemon.new(Puppet::Util::Pidlock.new(Puppet[:pidfile])) daemon.argv = @argv daemon.agent = @agent ! daemon.daemonize if should_daemonize ! daemon end
- 11. STOP 2: The Daemon and the Presented by
- 12. Presented by DAEMON LAND?! Nope Bye
- 13. Presented by Puppet::Agent • Run a thing
- 14. Presented by Puppet::Daemon • Run a thing that runs the other thing
- 15. Lots of layers of abstraction?? • Historical reasons! Presented by
- 16. Presented by STOP 3: The Configurer
- 18. Puppet::Configurer#run_internal Presented by # lib/puppet/configurer.rb def run_internal(options) # We create the report pre-‐populated with default settings for # environment and transaction_uuid very early, this is to ensure # they are sent regardless of any catalog compilation failures or # exceptions. options[:report] ||= Puppet::Transaction::Report.new("apply", nil, @environment, @transaction_uuid) report = options[:report] init_storage ! Puppet::Util::Log.newdestination(report) … … …
- 19. Puppet::Configurer#run_internal Presented by # lib/puppet/configurer.rb … query_options = get_facts(options) unless query_options …
- 20. Puppet::Configurer#run_internal Presented by # lib/puppet/configurer/fact_handler.rb … facts = Puppet::Node::Facts.indirection.find(Puppet[:node_name_value], :environment => @environment) …
- 21. Puppet::Configurer#run_internal Presented by # lib/puppet/configurer.rb … unless catalog = prepare_and_retrieve_catalog(options, query_options) return nil end …
- 22. Puppet::Configurer#run_internal Presented by # lib/puppet/configurer.rb … apply_catalog(catalog, options) …
- 23. Presented by Where next? Retrieve Apply
- 24. STOP 4: The Transaction Presented by
- 26. Presented by Puppet::Resource::Catalog Catalog Resource Other Stuff
- 27. Presented by Puppet::Configurer (again) # lib/puppet/configurer.rb def convert_catalog(result, duration) catalog = result.to_ral catalog.finalize catalog.retrieval_duration = duration catalog.write_class_file catalog.write_resource_file catalog end
- 28. Presented by RAL Catalog
- 29. Presented by RAL Catalog # lib/puppet/resource/catalog.rb def to_catalog(convert) result = self.class.new(self.name, self.environment_instance) result.version = self.version map = {} resources.each do |resource| next if virtual_not_exported?(resource) next if block_given? and yield resource newres = resource.copy_as_resource newres.catalog = result if convert != :to_resource newres = newres.to_ral end # We can't guarantee that resources don't munge their names # (like files do with trailing slashes), so we have to keep track # of what a resource got converted to. map[resource.ref] = newres result.add_resource newres end
- 30. Presented by RAL Catalog Puppet::Resource ! ↓ ! (something) << Puppet::Type
- 31. Presented by RAL Catalog ???
- 32. Presented by RAL Catalog # lib/puppet/configurer.rb def apply_catalog(catalog, options) report = options[:report] report.configuration_version = catalog.version ! benchmark(:notice, "Finished catalog run") do catalog.apply(options) end ! report.finalize_report report end
- 33. Puppet::Resource::Catalog#apply Presented by # lib/puppet/resource/catalog.rb def apply(options = {}) Puppet::Util::Storage.load if host_config? ! transaction = create_transaction(options) ! begin transaction.report.as_logging_destination do transaction.evaluate end rescue Puppet::Error => detail !
- 34. Presented by Transaction ???
- 35. Presented by Transaction • Ral catalog • Prioritizer • Report
- 36. Presented by Transaction • Ral catalog • Prioritizer • Report • Event manager • Resource harness • Relationship graph
- 37. Presented by Transaction • Ral catalog • Prioritizer • Report • Event manager • Resource harness • Relationship graph • relationship_graph.traverse
- 38. Presented by Transaction • Ral catalog • Prioritizer • Report • Event manager • Resource harness • Relationship graph • relationship_graph.traverse • *ollies out*
- 39. Presented by STOP 5: The Indirector
- 40. Presented by Back to Puppet::Configurer • #prepare_and_retrieve_catalog
- 41. Presented by Back to Puppet::Configurer # lib/puppet/configurer.rb result = Puppet::Resource::Catalog.indirection.find( Puppet[:node_name_value], query_options.merge(:ignore_cache => true, :environment => @environment, :fail_on_404 => true) )
- 42. Presented by The Indirector ! Puppet::Resource::Catalog.indirection.find()
- 43. Presented by The Indirector ← ↑ ↓ ?
- 44. Presented by The Indirector • Globally configured • Abstract • Gets stuff for you
- 45. Presented by The Indirector # lib/puppet/resource/catalog.rb extend Puppet::Indirector indirects :catalog, :terminus_setting => :catalog_terminus
- 46. Presented by The Indirector • find • search • save • destroy
- 47. Presented by The Indirector • find • search • save • destroy • terminus_class • cache_class • lib/puppet/indirector/…
- 48. Presented by Puppet Agent • Catalog => REST • Report => REST • Facts => Facter
- 49. Presented by Puppet Apply • Catalog => Compiler • Report => Processor • Facts => Facter
- 50. Presented by Avoiding the cache # lib/puppet/configurer.rb result = Puppet::Resource::Catalog.indirection.find( Puppet[:node_name_value], query_options.merge(:ignore_cache => true, :environment => @environment, :fail_on_404 => true) )
- 51. Presented by The Indirector • Why?
- 52. Presented by The Indirector • Why? • Global? Rigid?
- 53. Presented by The Indirector • Why? • Global? Rigid? • IT MAKES SENSE IN CONTEXT
- 54. Presented by The Indirector • …Catalog.find( <FACTS> ) • REST terminus • HTTP POST request to master
- 55. STOP 6: The Puppet Master’s Presented by
- 56. Presented by Rack Web server ↓ Rack server ↓ “App” object that responds to #call() ↓ Actual application logic
- 57. Presented by config.ru $0 = "master" ! ARGV << "-‐-‐rack" ! ARGV << "-‐-‐confdir" << "/etc/puppet" ARGV << "-‐-‐vardir" << "/var/lib/puppet" ! require 'puppet/util/command_line' run Puppet::Util::CommandLine.new.execute !
- 58. Presented by Starting master w/ Rack # lib/puppet/application/master.rb def start_rack_master require 'puppet/network/http/rack' ! announce_start_of_master ! return Puppet::Network::HTTP::Rack.new() end
- 59. Presented by The app object # lib/puppet/network/http/rack.rb class Puppet::Network::HTTP::Rack def call(env) request = Rack::Request.new(env) response = Rack::Response.new Puppet.debug 'Handling request: %s %s' % [request.request_method, request.fullpath] ! begin Puppet::Network::HTTP::RackREST.new.process(request, response) rescue => detail # Send a Status 500 Error on unhandled exceptions. response.status = 500 response['Content-‐Type'] = 'text/plain' response.write 'Internal Server Error: "%s"' % detail.message # log what happened Puppet.log_exception(detail, "Puppet Server (Rack): Internal Server Error: Unhandled Exception: "%s"" % detail.message) end response.finish
- 60. Presented by #process # lib/puppet/network/http/handler.rb def process(request, response) new_response = Puppet::Network::HTTP::Response.new(self, response) ! request_headers = headers(request) request_params = params(request) request_method = http_method(request) request_path = path(request) … !
- 61. Presented by #process # lib/puppet/network/http/handler.rb if route = @routes.find { |route| route.matches?(new_request) } route.process(new_request, new_response)
- 63. lib/puppet/network/http/api/v1.rb • GET => indirection.find() • POST => indirection.find() (for catalogs) • (There was an issue with some servers rejecting GET parameters that were too long. Funny story about that.) • DELETE => indirection.destroy() • PUT => indirection.save() • GET to magical plural name => indirection.search() Presented by
- 64. Presented by BIG PICTURE Configurer calls catalog.indirection.find() ↓ HTTP POST request to master ↓ Request handler calls catalog.indirection.find() ↓ Compiler terminus handles it ↓
- 65. Presented by STOP 7: The Compiler
- 66. Presented by MOSTLY SKIPPING THIS Lexer ↓ Parser ↓ AST ↓ Evaluator ↓
- 67. Presented by THIS IS THE TEST “Compiler terminus to the catalog indirection” ! ???
- 68. Presented by THIS IS THE TEST GOOD WORK EVERYBODY!
- 70. Presented by THANKS!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! •Hope we’ll see you next year in Portland!