Puppet Availability and Performance at 100K Nodes - PuppetConf 2014
Download as PPTX, PDF13 likes8,830 views
This document discusses challenges encountered when scaling Puppet to manage over 100,000 nodes, and proposes an alternative approach called PVC (Puppet Variable Control) to help address those challenges. Some key issues identified include uneven and unpredictable Puppet run times that cause system thrashing. PVC aims to run Puppet reactively based on available capacity and local file changes, using frequent pings and fact collection to determine run timing in a way that achieves a flat and consistent service curve. It also aims to improve security by running Puppet immediately when monitored files change outside of Puppet runs.
Puppet Availability and Performance at 100K Nodes - PuppetConf 2014
- 1. puppet @ 100,000+ agents John Jawed (“JJ”) eBay/PayPal
- 2. but I don’t have 100,000 agents issues ahead encountered at <1000 agents
- 3. me responsible for Puppet/Foreman @ eBay how I got here: engineer -> engineer with root access -> system/infrastructure engineer
- 5. puppet @ eBay, quick facts -> perhaps the largest Puppet deployment -> more definitively the most diverse -> manages core security -> trying to solve the “p100k” problems
- 6. #’s • 100K+ agents – Solaris, Linux, and Windows – Production & QA – Cloud (openstack & VMware) + bare metal • 32 different OS versions, 43 hardware configurations – Over 300 permutations in production • Countless apps from C/C++ to Hadoop – Some applications over 15+ years old
- 7. currently • 3-4 puppet masters per data center • foreman for ENC, statistics, and fact collection • 150+ puppet runs per second • separate git repos per environment, common core modules – caching git daemon used by ppm’s
- 9. nodes growing, sometimes violently linear growth trendline
- 11. setup puppetmasters setup puppet master, it’s the CA too sign and run 400 agents concurrently, that’s less than half a percent of all the nodes you need to get through.
- 13. not exactly puppet issues entropy unavailable crypto is CPU heavy (heavier than you ever have and still believe) passenger children are all busy
- 14. OK, let’s setup separate hosts which only function as a CA
- 15. multiple dedicated CA’s much better, distributed the CPU I/O and helped the entropy problem. the PPM’s can handle actual puppet agent runs because they aren’t tied up signing. Great!
- 16. wait, how do the CA’s know about each others certs? some sort of network file system (NFS sounds okay).
- 17. shared storage for CA cluster -> Get a list of pending signing requests (should be small!) # puppet cert list … wait … wait …
- 19. optimize CA’s for large # of certs Traversing a large # of certs is too slow over NFS. -> Profile -> Implement optimization -> Get patch accepted (PUP-1665, 8x improvement)
- 21. optimizing foreman - read heavy is fine, DB’s do it well. - read heavy in a write heavy environment is more challenging. - foreman writes a lot of log, fact, and report data post puppet run. - majority of requests are to get ENC data - use makara with PG read slaves (https://github.com/taskrabbit/makara) to scale ENC requests - Needs updates to foreigner (gem) - If ENC requests areslow, puppetmasters fall over.
- 22. optimizing foreman ENC requests load balanced to read slaves fact/report/host info write requests sent to master makara knows how to arbitrate the connection (great job TaskRabbit team!)
- 23. more optimizations make sure RoR cache is set to use dalli (config.cache_store = :dalli_store), see foreman wiki fact collection optimization (already in upstream), without this reporting facts back to foreman can kill a busy puppetmaster! (if you care: https://github.com/theforeman/puppet-foreman/ pull/145)
- 24. <3 the foreman team
- 25. let’s add more nodes Adding another 30,000 nodes (that’s 30% coverage). Agent setup: pretty standard stuff, puppet agent as a service.
- 26. results average puppet run: 29 seconds. not horrible. but average latency is a lie because that usually represents the mean average (sum of N / N). the actual puppet run graph looks more like…
- 27. curve impossible No one in operations or infrastructure ever wants a service runtime graph like this. mean average
- 28. PPM running @ medium load PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16765 puppet 20 0 341m 76m 3828 S 53.0 0.1 67:14.92 ruby 17197 puppet 20 0 343m 75m 3828 S 40.7 0.1 62:50.01 ruby 17174 puppet 20 0 353m 78m 3996 S 38.7 0.1 70:07.44 ruby 16330 puppet 20 0 338m 74m 3828 S 33.8 0.1 66:08.81 ruby 17231 puppet 20 0 344m 75m 3820 S 29.8 0.1 70:00.47 ruby 17238 puppet 20 0 353m 76m 3996 S 29.8 0.1 69:11.94 ruby 17187 puppet 20 0 343m 76m 3820 S 26.2 0.1 70:48.66 ruby 17156 puppet 20 0 353m 75m 3984 S 25.8 0.1 64:44.62 ruby … system processes
- 29. 60 seconds later…idle PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 17343 puppet 20 0 344m 77m 3828 S 11.6 0.1 74:47.23 ruby 31152 puppet 20 0 203m 9048 2568 S 11.3 0.0 0:03.67 httpd 29435 puppet 20 0 203m 9208 2668 S 10.9 0.0 0:05.46 httpd 16220 puppet 20 0 337m 74m 3828 S 10.3 0.1 70:07.42 ruby 16354 puppet 20 0 339m 75m 3816 S 10.3 0.1 62:11.71 ruby … system processes
- 30. 120 seconds later…thrashing PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16765 puppet 20 0 341m 76m 3828 S 94.0 0.1 67:14.92 ruby 17197 puppet 20 0 343m 75m 3828 S 93.7 0.1 62:50.01 ruby 17174 puppet 20 0 353m 78m 3996 S 92.7 0.1 70:07.44 ruby 16330 puppet 20 0 338m 74m 3828 S 90.8 0.1 66:08.81 ruby 17231 puppet 20 0 344m 75m 3820 S 89.8 0.1 70:00.47 ruby 17238 puppet 20 0 353m 76m 3996 S 89.8 0.1 69:11.94 ruby 17187 puppet 20 0 343m 76m 3820 S 88.2 0.1 70:48.66 ruby 17156 puppet 20 0 353m 75m 3984 S 87.8 0.1 64:44.62 ruby 17152 puppet 20 0 353m 75m 3984 S 86.3 0.1 64:44.62 ruby 17153 puppet 20 0 353m 75m 3984 S 85.3 0.1 64:44.62 ruby 17151 puppet 20 0 353m 75m 3984 S 82.9 0.1 64:44.62 ruby … more ruby processes
- 32. what we really want A flat consistent runtime curve, this is important for any production service. Without predictability there is no reliability!
- 33. consistency @ medium load PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16765 puppet 20 0 341m 76m 3828 S 53.0 0.1 67:14.92 ruby 17197 puppet 20 0 343m 75m 3828 S 40.7 0.1 62:50.01 ruby 17174 puppet 20 0 353m 78m 3996 S 38.7 0.1 70:07.44 ruby 16330 puppet 20 0 338m 74m 3828 S 33.8 0.1 66:08.81 ruby 17231 puppet 20 0 344m 75m 3820 S 29.8 0.1 70:00.47 ruby 17238 puppet 20 0 353m 76m 3996 S 29.8 0.1 69:11.94 ruby 17187 puppet 20 0 343m 76m 3820 S 26.2 0.1 70:48.66 ruby 17156 puppet 20 0 353m 75m 3984 S 25.8 0.1 64:44.62 ruby … system processes
- 34. hurdle: runinterval near impossible to get a flat curve because of uneven and chaotic agent run distribution. runinterval is non-deterministic … even if you manage to sync up service times eventually it’s nebulous.
- 35. the puppet agent daemon approach is not going to work.
- 36. plan A: puppet via cron generate run time based some deterministic agent data point (IP, MAC address, hostname, etc.). IE, if you wanted a puppet run every 30 minutes, your crontab may look like: 08 * * * * puppet agent -t 38 * * * * puppet agent -t
- 37. plan A yields Fewer and predictable spikes
- 38. Improved. But does not scale because cronjobs help run times become deterministic but lack even distribution.
- 39. eliminate all masters? masterless puppet kicking the can down the road, somewhere infrastructure still has to serve the files and catalog to agents. masterless puppet creates a whole host of other issues (file transfer channels, catalog compiler host).
- 40. eliminate all masters? masterless puppet …and the same issues exists in albeit in different forms. shifts problems to “compile interval” and “manifest/module push interval”.
- 41. plan Z: increase your runinterval Z, the zombie apocalypse plan (do not do this!). delaying failure till you are no longer responsible for it (hopefully).
- 42. alternate setups SSL termination on load balancer – expensive - LB’s are difficult to deploy, cost more (you still need fail over otherwise it’s a SPoF!) caching – cache is meant to make things faster, not required to work. If cache is required to make services functional, solving the wrong problem.
- 43. zen moment maybe the issue isn’t about timing the agent from the host. maybe the issue is that the agent doesn’t know when there’s enough capacity to reliably and predictably run puppet.
- 44. enforcing states is delayed runinterval/cronjobs/masterless setups still render puppet as a suboptimal solution in a state sensitive environment (customer and financial data). the problem is not unique to puppet. salt, coreOS, et al. are susceptible.
- 45. security trivia web service REST3DotOh just got compromised and allows a sensitive file managed by puppet to be manipulated. Q: how/when does puppet set the proper state?
- 46. the how; sounds awesome A: every puppet runs ensures that a file is in its’ intended state and records the previous state if it was not.
- 47. the when; sounds far from awesome A: whenever puppet is scheduled to run next. up to runinterval minutes from the compromise, masterless push, or cronjob execution.
- 48. smaller intervals help but… all the strategies have one common issue: puppet masters do not scale with smaller intervals, exasperate spikes in the runtime curve.
- 49. this needs to change
- 50. pvc “pvc” – open source & lightweight process for a deterministic and evenly distributed puppet service curve… …and reactive state enforcement puppet runs.
- 51. pvc a different approach that executes puppet runs based on available capacity and local state changes. pings from an agent to check if its’ time to run puppet. file monitoring to force puppet runs when important files change outside of puppet (think /etc/shadow, /etc/sudoers).
- 52. pvc basic concepts: - Frequent pings to determine when to run puppet - Tied in to backend PPM health/capacity - Frequent fact collection without needing to run puppet - Sensitive files should be subject to monitoring - on change or updates outside of puppet, immediately run puppet! - efficiency an important factor.
- 53. pvc advantages -> variable puppet agent run timing - allows the flat and predictable service curve (what we want). - more frequent puppet runs when capacity is available, less frequent puppet runs less capacity is available.
- 54. pvc advantages -> improves security (kind of a big deal these days) - puppet runs when state changes rather than waiting to run. - efficient, uses inotify to monitor files. - if a file being monitored is changed, a puppet run is forced.
- 55. pvc advantages - orchestration between foreman & puppet - controlled rollout of changes - upload facts between puppet runs into foreman
- 56. pvc – backend 3 endpoints – all get the ?fqdn=<certname> parameter GET /host – should pvc run puppet or facter? POST /report – raw puppet run output, files monitored were changed POST /facts – facter output (puppet facts in JSON)
- 57. pvc – /host > curl http://hi.com./host?fqdn=jj.e.com < PVC_RETURN=0 < PVC_RUN=1 < PVC_PUPPET_MASTER=puppet.vip.e.com < PVC_FACT_RUN=0 < PVC_CHECK_INTERVAL=60 < PVC_FILES_MONITORED="/etc/security/access.conf /etc/passwd"
- 58. pvc – /facts allows collecting of facts outside of the normal puppet run, useful for monitoring. set PVC_FACT_RUN to report facts back to the pvc backend.
- 59. pvc – git for auditing push actual changes between runs into git - branch per host, parentless branches & commits are cheap. - easy to audit fact changes (fact blacklist to prevent spam) and changes between puppet runs. - keeping puppet reports between runs is not helpful.
- 60. pvc – incremental rollouts select candidate hosts based on your criteria and set an environment variable via the /host endpoint output: FACTER_UPDATE_FLAG=true in your manifest, check: if $::UPDATE_FLAG { … }
- 61. example pvc.conf host_endpoint=http://jj.e.com./host report_endpoint=http://jj.e.com./report facts_endpoint=http://jj.e.com./facts info=1 warnings=1
- 62. pvc – available on github $ git clone https://github.com/johnj/pvc make someone happy, achieve:
- 63. wishlist stuff pvc should probably have: • authentication of some sort • a more general backend, currently tightly integrated into internal PPM infrastructure health • whatever other users wish it had
- 64. misc. lessons learned your ENC has to be fast, or your puppetmasters fail without ever doing anything. upgrade ruby to 2.x for the performance improvements. serve static module files with a caching http server (nginx).
Editor's Notes
- #25: Greg, dominic, ohad