Puppet Troubleshooting

Editor's Notes

• #5: I started using puppet early on, 0.24 I know a guy that crushes coke cans for a living. It's soda pressing.
• #7: how do we get from this to this. These are the techniques I've used and what I've seen
• #8: so these errors can be broken into two groups, we'll talk about each separately. There's a third category, hopefully you don't fall into this camp, but if you aren't running the latest stuff you are missing out.
• #9: so this is network and certificate issues.
• #11: sometimes there's dragons between you and the server. sometimes the masterport isn't 8140.
• #12: you can tell mtr to use port 8140, we'll show each of these and why they are useful.
• #13: ping will show you if your client node can lookup the puppet server, you don't need ping to succeed to still be ok. It uses the gethostbyname system call, which will be the same what that puppet will lookup the host. if this fails, puppet will fail. check what your hostname is, hostname -f will see if your reverse lookup is working. you don't need to have your nodes resolve properly. only the master should resolve, locally is fine. mtr, my traceroute, originally matt's traceroute, it works different than traceroute, it uses icmp by default (traceroute uses udp) netcat - the swiss army knife of network tools.
• #14: this is not an x509 talk, but you should know how x509 works.
• #15: puppet uses the certname directive, that's how it knows what to do, it uses the hostname command to do that. ask facter what the hostname is, use config print to show the value basic unix permissions ← I hear Antoine Dodson in my head when it turns out to be basic unix permissions. it's not working, why can't you find that module, it's right there you idiot. it turns out someone changed perms manually or messed up with git. it could be selinux but please don't shut that off. it could be some other communication problem, so to fix it, you need to know how it works, puppet has a built in REST API
• #16: REST - Representational state transfer I say this in the book, but a lot of puppet is just https traffic ok, everyone hold hands, this might get rough
• #17: the node starts with a GET request to grab the CA. the server should respond with a certificate so how does the node construct this get request.
• #18: what resources are available? - next slide
• #20: verify you can download the CA, your own certificate check that it's the same. I kid you not, some companies actually interject themselves in any http traffic, you might not be getting the CA you asked for.
• #21: now that we have the ca cert, we can use that with curl to try and download our nodes certificate that should give you a cert, but if it doesn't, that's your problem. Ok, so we'll ask for cookbook and we get it, use openssl x509 to look at the cert. check the validity.
• #22: like anyone can even know that. well, there's more to it, but you will get a catalog back.
• #23: wget or curl will just grab whatever but obscure what's happening, if you are having trouble it's better to use an interactive client you're a sysadmin, you should know just enough about everything to get in trouble. so this is an http request, these tools take care of the ssl part of the communication. gnutls-cli can work with startssl type connections.
• #25: this is how I work, this is the first chapter of my book But we use apache or nginx or whatever and we use mod_proxy_balancer for instance in apache. we can look at the url's coming in, and based on the environment we redirect to a specific worker system that can compile the catalog for us. when you are trying to diagnose a problem it can be useful to just make a problem environment and go to another worker. you need to ability to create arbitrary test branches. so you need git in your workflow. workflow is something I talk about heavily in my LISA talk. next slide git
• #26: making a branch per user, ticket etc it very useful. ok, back to our problem worker.
• #27: you would not do this on a production server. this will generate a lot of logs, make sure you logrotate that stuff.
• #28: this is another way to run, the --trace option is the one that will get you a ton of data. but when you have a problem, that's the best way to find it.
• #29: this is a good technique, copy the part that you are having trouble with somewhere, run puppet apply on it with --trace you'll get to see how the exec is actually running.
• #30: so what is the catalog, it's yaml or json. yaml is easy to read json not so much, but jq works. what can you do with the catalog? -- next slide.
• #31: the catalog you compile should have the same classes as classes.txt if your last run was successful and you haven't changed the class list. errors can come from the class list changing
• #32: when the catalog fails to compile, debugging is usually a lot simpler either puppet found too many failing to apply is usually much harder to figure out. The first issue with variables is the hardest one to find i'll show examples of the bad exec and service.
• #34: so how do you fix duplicate resource? separate the thing you need multiple times into their own class. - that's the best option for things like httpd/apache virtual resources, works well for users, sometimes confusing if you can't find the module, maybe it's the modulepath, use config print to show that. it could be basic unix permissions, try going to that place as the puppet or pe-puppet user to make sure.
• #35: these problems are the stumpers, but after you fix them, they seem super simple
• #36: scripts that use environment variables, when the user runs it, it works when you sudo puppet agent it works too (unless envreset sure) confusing, but you can run with trace and see what's actually running. puppet agent --trace will show how the script was run.
• #37: puppet just runs these commands, if the return code is no good, puppet thinks the service is not operational puppet will try and start the service. puppet does a restart when it gets a notify, if restart is broken, your runs will fail
• #38: this one can stump you for a while. you have two resources with different names they point to the same thing but only after variable substitution. well actually that doesn't matter the main thing is that the name/title of the resource is different and the check to make sure they aren't working on the same thing is done after the catalog is compiled.
• #39: this one is even worse than the last one :-( the problem when you are debugging is that the error is in the defined type definition, the best thing I've found at this point is to look in the catalog and figure out where the define was called from.
• #40: anyone can do this, it's the echo "hello" thing
• #41: you need to use chaining to ensure that things happen near each other.
• #42: so this is a technique that I've used when I'm stumped. make a class for yourself that installs a script and does some debugging. we then execute that script in an exec, requiring the class with the script.
• #43: so here's an example, mine is actually a bit more verbose, but here's a start for y'all. explain the lsof and what's going on. and the output, you can see what I'm looking for..
• #44: ok break it down we use an inline template lets go through how that part works. we take that scope object, translate to hash use the reject function to remove the entries that are not strings place that all into the vars variable now take vars and sort it, go through each variable and print them one per line. so what does this look like ← next slide
• #45: this is very useful for knowing what the value of everything is at this moment. but when you are working with puppet you have to remember, even though you define something near something else, there's no guarantee that they are executed near each other ← next slide
• #46: i've seen this a few times, it stumps people.
• #49: here are the slides, again, please comment