Janice Singh - Writing Custom Nagios Plugins
2 likes1,243 views
The document is a presentation on writing custom Nagios plugins by Janice Singh, aimed at individuals with basic Nagios and coding knowledge. It covers the importance of custom plugins, guidelines for writing them, and practical examples using Perl, as well as troubleshooting common issues. Singh shares her experience at NASA Advanced Supercomputing and provides insights into the architecture of their Nagios setup.
![Putting it All Together –simplified
use Nagios::Plugin ;
my $p = Nagios::Plugin->new(
usage => "Usage: %s [ -f|--filesystem = <filesystem name> ]",
version => $VERSION,
blurb => 'This will test to see if the specified filesystem is mounted
or not. It will output OK if it is mounted, CRITICAL if it not.',
extra => "
-f, --filesystem is required.
"
);
janice.s.singh@nasa.gov 15](https://image.slidesharecdn.com/track2-29th-230pm-janice-singh-nagiospluginsfinal-151019210340-lva1-app6891/85/Janice-Singh-Writing-Custom-Nagios-Plugins-15-320.jpg)
Janice Singh - Writing Custom Nagios Plugins
- 1. Writing Custom Nagios Plugins Janice Singh Nagios Administrator
- 2. Introduction • About the Presentation – audience • anyone with basic nagios knowledge • anyone with basic scripting/coding knowledge – what a plugin is – how to write one – troubleshooting • About Me – work at NAS (NASA Advanced Supercomputing) – used Nagios for 5 years • started at Nagios 2.10 • written/maintain 25+ plugins janice.s.singh@nasa.gov 2
- 3. NASA Advanced Supercomputing • Pleiades – 11,312-node SGI ICE supercluster – 184,800 cores • Endeavour – 2 node SGI shared memory system – 1,536 cores • Merope – 1,152 node SGI cluster – 13,824 cores • Hyperwall visualization cluster – 128-screen LCD wall arranged in 8x16 configuration – measures 23-ft. wide by 10-ft. high – 2,560 processor cores • Tape Storage - pDMF cluster – 4 front ends – 47 PB of unique file data stored Ref: http://www.nas.nasa.gov/hecc/ janice.s.singh@nasa.gov 3
- 4. Nagios at NASA Advanced Supercomputing • one main Nagios server • systems behind firewall send data by nrdp • some clusters behind firewall – one cluster uses nrpe for gathering data – other clusters use ssh • Post processor prepares visualization (HUD) data – separate daemon – Nagios APIs provide configuration and status data – provides file read by HUD – general architecture adaptable for other uses janice.s.singh@nasa.gov 4
- 6. Plugins – Nagios extensions • Built-in plugins – Aren’t truly built-in, but they come standard when you install nagios-plugins • check_disk • check_ping • Custom plugins – Let you test anything – The sky’s the limit - if you can code it, you can test it janice.s.singh@nasa.gov 6
- 7. How to Use Plugins Nagios configuration to define a service that will use the plugin check_mydaemon.pl: define service { host linuxserver2 service_description Check MyDaemon check_command check_mydaemon!5!10 } define command { command_name check_mydaemon command_line check_mydaemon.pl –w $ARG1$ –c $ARG2$ } janice.s.singh@nasa.gov 7
- 8. Reasons to write your own plugin • There isn’t a plugin out there that tests what you want • You need to test it differently janice.s.singh@nasa.gov 8
- 9. Guidelines • Any Language you want • There is only one rule: it must return a nagios-accepted value ok (green) 0 warning (yellow) 1 critical (red) 2 unknown (orange) 3 janice.s.singh@nasa.gov 9
- 10. Plugin Psuedocode • General outline of what a plugin needs to do: – initialize object (if object oriented code) – read in the arguments – set variables – do the test – return results • This is just a suggestion janice.s.singh@nasa.gov 10
- 11. For Perl: Nagios::Plugin # Instantiate Nagios::Plugin object (the 'usage' parameter is mandatory) my $p = Nagios::Plugin->new( usage => ”usage_string", version => $version_number, blurb => ‘brief info on plugin', extra => ‘extended info on plugin’ ); janice.s.singh@nasa.gov 11
- 12. For Perl: Nagios::Plugin (cont). # adding an argument ex: check_mydaemon.pl -w # define help string neatly – use below instead of qq my $hlp_strg = ‘-w, --warning=INTEGER:INTEGERn’ . ‘ If omitted, warning is generated.’; $p->add_arg( spec => 'warning|w=s’, help => $hlp_strg required => 1, default => 10, ); #accessing the argument $p->getopts; $p->opts->warning janice.s.singh@nasa.gov 12
- 13. For Perl: Nagios::Plugin (cont). # finishing the script: $p->nagios_exit( return_code => $p->check_threshold($result), message => " info on what $result means" ); # if you are not using check_threshold use text for return code: return_code => ‘OK|WARNING|CRITICAL|UNKNOWN’ janice.s.singh@nasa.gov 13
- 14. For Perl: Nagios::Plugin (cont). • When you’ve done your code and have $result to compare to the thresholds: – $return_code = $p->check_threshold($result) – follows nagios convention of min:max • check_mydaemon.pl –w 5 will warn on > 5 • check_mydaemon.pl –w :5 will warn on > 5 • check_mydaemon.pl –w 5: will warn on < 5 • check_mydaemon.pl –w 5:7 will warn on <5 or >7 (outside the range) • check_mydaemon.pl –w @5:7 will warn on >=5 or <=7 (within the range) • if you overlap critical and warning, critical has precedent janice.s.singh@nasa.gov 14
- 15. Putting it All Together –simplified use Nagios::Plugin ; my $p = Nagios::Plugin->new( usage => "Usage: %s [ -f|--filesystem = <filesystem name> ]", version => $VERSION, blurb => 'This will test to see if the specified filesystem is mounted or not. It will output OK if it is mounted, CRITICAL if it not.', extra => " -f, --filesystem is required. " ); janice.s.singh@nasa.gov 15
- 16. Putting it All Together –simplified (cont). $p->add_arg( spec => 'filesystem|f=s', help => qq{-f, --filesystem=STRING The filesystem to check for.}, required => 1, ); $p->getopts; my $filesystem = $p->opts->filesystem; janice.s.singh@nasa.gov 16
- 17. Putting it All Together –simplified (cont). my $result = `mount | grep $filesystem`; if ($result ne "") { $p->nagios_exit( return_code => OK, message => "$filesystem is mounted" ); } else { $p->nagios_exit( return_code => UNKNOWN, message => "$filesystem is not mounted" ); } ./check_mounted.pl -f nobackupp8 MOUNTED OK - nobackupp8 is mounted janice.s.singh@nasa.gov 17
- 18. Timeouts • Many times you need to make sure a plugin ends within a specified time – in nagios.cfg: service_check_timeout=60 – in this case you’d want to specify a 60 second timeout • with Nagios::Plugin – ./mydaemon.pl –t 60 – alarm($p->opts->timeout); janice.s.singh@nasa.gov 18
- 19. Overcoming issues • Test needs elevated privilege • nagios can be run as root but is not secure – run the test as root via cronjob; write info to a flat file – use nagios plugin to read and process the file • Output of the test was too big – the resulting nrdp command hit a kernel limit – use scp to get the output to the main nagios server ex: scp command_output nagios_server: – use plugin on the main server to process it janice.s.singh@nasa.gov 19
- 20. Nagios perfdata • Nagios is designed to allow plugins to return optional performance data in addition to normal status data – in nagios.cfg enable the process_performance_data option. – Nagios collects this information to be displayed on the GUI – in the format “|key1=value1,key2=value2,…,keyN=valueN” – this can be anything that has a numerical value janice.s.singh@nasa.gov 20
- 21. Troubleshooting • The Nagios display says: return code XXX is out of bounds – your script returns anything other than 0,1,2,3 – otherwise it is a nagios error. • Google is your friend – ex: 13 usually means a permission error – sometimes all it tells you is “something went wrong” – these disappeared at our site when we switched to Nagios::Plugin • try running the plugin from the command line – verify who you are running as – verify the arguments passed in janice.s.singh@nasa.gov 21
- 22. Troubleshooting (cont). • Timing is everything! – files can get overwritten • by cron jobs • by multiple nagios processes – launching too many processes • if perfdata is enabled, the perfdata log is the most useful janice.s.singh@nasa.gov 22