Nagios Conference 2011 - Nathan Vonnahme - Integrating Nagios With Test Driven Development
Download as PPTX, PDF0 likes1,144 views
The document discusses the integration of Nagios with Test Driven Development (TDD), emphasizing the parallel between monitoring tools for system administrators and testing tools for developers. It covers various testing types, the importance of quality assurance, and suggests ways to utilize Nagios for unit, integration, and user acceptance monitoring through specific plugins. The article emphasizes the need for continuous testing and monitoring while proposing a hands-on approach to creating custom Nagios plugins.
Nagios Conference 2011 - Nathan Vonnahme - Integrating Nagios With Test Driven Development
- 1. Integrating Nagios withTest Driven Development Nathan VonnahmeNathan.Vonnahme@bannerhealth.comIntegrating Nagios withTest Driven Development
- 2. 20112Intro9 years of hospital IT
- 4. Sysadmin -> developer20113Central IdeaMonitoring tools are to the sysadmin what testing tools are to the developer.
- 6. OverviewTest-driven DevelopmentMonitoring and TDDSample toolsFurther ideas/implications2011
- 7. I.Test-driven Development (TDD)2011Nagios World Conference6
- 8. KwaliteeUhshuranse (QA)What is “quality”?“In its broadest sense, quality is a degree of excellence: the extent to which something is fit for its purpose. In the narrow sense, product or service quality is defined as conformance with requirement, freedom from defects or contamination, or simply a degree of customer satisfaction.”
- 9. Quality is a result“Quality is the result of a comparison between what was required and what was provided. It is judged not by the producer but by the receiver.”
- 10. How do you get quality?By, among other things:verifying before delivery that products and services possess the features required
- 11. preventing the supply of products and services which possess features which dissatisfy customersTestingEnsures essential function
- 12. Catches defectsSoftware Testing Types(Names/categories vary)UnitIntegrationUser Acceptance
- 13. 1. Unit testingDoes each individual piece do what we expect?
- 14. Yes or no?2. Integration testingDo the units work together?
- 15. We need a parallel universe!3. UserAcceptance testingAKA “Functional testing” or “Customer testing”
- 16. Does it work for the actual user?
- 17. Does it do what we promised?
- 18. “User stories”As a user closing the application, I want to be prompted to save anything that has changed since the last save so that I can preserve useful work and discard erroneous work.
- 19. Test AutomationThe robots will do our work for us!
- 21. Consistent
- 22. Fast
- 23. More likely to actually get run
- 25. “us 43 them 0”TDD (~1999)Testing is such a good idea, let’s do itEarly (test first, then implement)
- 28. Continuous IntegrationRun the test suite whenever source code changes!Alarm if any tests fail!2011
- 29. Behavior-driven development (~2003)BDD builds on TDD but especially emphasizes acceptance testing, and begins by writing user stories, not unit tests.2011
- 30. II. Monitoring and TDD2011Nagios World Conference20
- 31. Monitoring as TDDAt the highest level, monitoring is QA.2011
- 32. Translation neededHow do these types of testing translate for the sysadmin?Unit
- 33. Integration
- 34. Acceptance2011
- 37. CPU, memory properly utilized
- 38. OS functioning
- 40. Unit Monitoring with NagiosMany plugins for “unit” monitoring:check_ping
- 41. check_swap
- 42. check_load
- 43. check_disk
- 44. check_dhcp
- 45. check_http
- 46. check_ldap
- 47. check_mysql
- 48. check_nt
- 49. check_ntp_peer
- 50. check_pgsql
- 51. check_procs
- 52. check_radius
- 53. check_smtp
- 54. check_ssh2011
- 57. Data fresh
- 58. Expected scheduled processes have completed
- 59. Backups worked
- 61. Integration Monitoring with NagiosA few plugins for “integration” monitoring:check_mysql_query
- 62. check_hpjd ?Usually requires custom plugins. Some of mine:check_backup
- 63. check_printer
- 64. check_app_logins
- 65. check_chartserver*2011*See my other talk.
- 67. Acceptable speed
- 71. User can log in
- 72. Customer can buy stuff2011
- 73. User Acceptance Monitoring with NagiosMaybe some with check_http.2011
- 74. Growing Test CoverageMark Jason Dominus: Start by saying "I'll write one [Nagios check]".
- 75. You don't have to write all the [Nagios checks] before you start the project.2011
- 76. Test Bugs“When you fix (or find) a bug, add a [Nagios check] for it”You’ll fix it much faster next time
- 77. For serious repeating issues, add an event handler!2011
- 78. Test Features“When you add a feature, add a [Nagios check] for just that feature”A new service
- 79. A new piece of hardware2011
- 80. Test SysadminsThat thing you always forget to turn back on
- 81. That task you always procrastinate (SSL cert expiry?)
- 82. That filesystem that doesn’t always mount
- 83. That long process you forget to verify (backups, anyone?)
- 84. Random and senseless acts of system administration2011
- 85. Test Users“Login is slow!!!”
- 86. How slow?2011
- 87. III. Sample Tools for NagiosReusing software testing tools for monitoring2011Nagios World Conference34
- 88. 201135TAP – Test Anything ProtocolTestanything.org
- 89. TAP – Test Anything ProtocolWidespread for Perl, many others
- 90. Simple output (kind of like Nagios plugins)1..4 ok 1 - Input file opened not ok 2 - First line of the input valid ok 3 - Read the rest of the file not ok 4 - Summarized correctly2011
- 91. TAP is widely used for Perl unit testsThe core Perl language has over 92,000 tests There are over 142,000 tests for the libraries bundled with it.Whenever you install a CPAN module:t/check_stuff.t ................... okt/Nagios-Plugin-01.t .............. okt/Nagios-Plugin-02.t .............. okt/Nagios-Plugin-03.t .............. okt/Nagios-Plugin-04.t .............. Okt/Nagios-Plugin-Functions-01.t .... okt/Nagios-Plugin-Functions-02.t .... Ok . . . All tests successful.Files=16, Tests=971, 5 wallclocksecs ( 0.21 usr 0.07 sys + 4.27 cusr 0.42 csys = 4.97 CPU)Result: PASS2011
- 92. TAP producer exampleWriting test scripts is super easy.#!/usr/bin/perl -wuse Test::Simple tests =>1;ok(1+1==2);Other languages likewise.Start (with Perl) by reading Test::Tutorial.goo.gl/7C7pb 2011
- 93. check_tap.plExplained in blog. Available at goo.gl/j8Xt2check_tap.pl -s /full/path/to/testfoo.pl will run ‘testfoo.pl’ and return OK if 0 tests fail, but CRITICAL if any fail. check_tap.pl -s /full/path/to/testfoo.pl -c 2 will return OK if 0 tests fail, WARNING if more than 0 tests fail, and CRITICAL if more than 2 fail.2011
- 94. check_tap.plNon-Perl and remote test scriptscheck_tap.pl -e '/usr/bin/ruby -w' -s /full/path/to/testfoo.r will run ‘testfoo.r’ using Ruby with the -w flag.You can use any shell command and argument which produces TAP output, for example:check_tap.pl -e '/usr/bin/curl -sk' -s 'http://url/to/mytest.php'check_tap.pl -e '/usr/bin/cat' -s '/path/to/testoutput.tap'In fact, anything TAP::Harness or prove regards as a source or executable.2011
- 95. Ex. 1: firewall portsuse Test::Ping;# written by NJV 2005; not from CPANplan tests =>13;ping_ok(qw~localhost http ~);ping_ok(qw~localhost https ~);diag"app server internal_server";ping_ok('internal_server.my.com','http');service_ok('internal_server.my.com','http');ping_ok('internal_server.my.com','https');service_ok('internal_server.my.com','https');ping_ok('internal_server.my.com','1433',‘internal_serverpingable on MS SQL port 1433');service_ok('internal_server.my.com','1433‘, ‘mssql responds’);ping_ok('internal_server.my.com','3306',‘internal_serverpingable on MySQL port 3306');service_ok('internal_server.my.com','3306‘, ‘mysql responds’);diag"testing open ports to other inside hosts\n";service_ok('smtp.my.com','smtp',"can contact smtp (email) service on smtp.my.com");2011
- 96. Ex. 1: firewall ports – OUTPUT1..13ok 1 - this script is running on (xxx) the physician portal webserverok 2 - can connect to localhost:httpok 3 - can connect to localhost:https# app server internal_serverok 4 - can connect to internal_server.my.com:httpok 5 - service is answering on internal_server.my.com:httpok 6 - can connect to internal_server.my.com:httpsok 7 - service is answering on internal_server.my.com:httpsok 8 - can contact internal_server (web app server) on MS SQL port 1433ok 9 - service is answering on internal_server.my.com:1433ok 10 - can contact internal_server (web app server) on MySQL port 3306ok 11 - service is answering on internal_server.my.com:3306# testing open ports to other inside hostsok 12 - can contact smtp (email) service on smtp.my.comok 13 - Survived to endCheck command:check_tap.pl -v -e '/usr/bin/curl -sk' -s https://$HOSTADDRESS$/foo/porttest.pl2011
- 97. Ex. 2: farm plan tests =>29;# test thyself!my$me= hostname;like($me,qr/^portal/,"this script is running on ($me) a physician portal webserver");diag"checking citrix farm connectivity\n";my@stas=qw(vcitrixdc01.my.comvcitrixdc02.my.com);foreach(@stas){ping_ok($_,'http');service_ok($_,'http',"XML service on $_ is responding in some way");}my@citrix_farm=qw(citrixps01.my.com citrixps02.my.com# ... citrixps30.my.com);foreach(@stas,@citrix_farm){service_ok($_,'1494',"ICA service on $_ is responding in some way");}2011
- 98. Ex. 2 OUTPUT1..29ok 1 - this script is running on (xxx) a physician portal webserver# checking citrix farm connectivityok 2 - can connect to vcitrixdc01.my.com:httpok 3 - XML service on vcitrixdc01.my.com is responding in some wayok 4 - can connect to vcitrixdc02.my.com:httpok 5 - XML service on vcitrixdc02.my.com is responding in some wayok 6 - ICA service on vcitrixdc01.my.com is responding in some way# ...ok 13 - ICA service on citrixps06.my.com is responding in some waynot ok 14 - ICA service on citrixps07.my.com is responding in some way# Failed test (c:\inetpub\wwwroot\pp_nagios\check_citrix.pl at line 80)# ...# Looks like you failed 1 tests of 29.Check command:check_tap.pl -v -e '/usr/bin/curl -sk' -s https://$HOSTADDRESS$/foo/porttest.pl -w 3 -c 6TAP OK - FAIL. 1 of 29 failed. Warning/critical at 2/3. 1st='not ok 14 - ICA service on citrixps07.my.com is responding in some way' | total=29;; failures=1;; passed=28;;2011
- 99. Ex. 3 Symfony2 Functional testsSymfony2 == next-generation MVC framework for PHP apps
- 100. Symfony2 uses the PHPUnit framework.
- 101. PHPUnit supports TAP output.
- 102. Good developers are already writing these.2011
- 103. Ex. 3 Symfony2 functional testspublicfunctiontestSearch() {$client=$this->createClient();$crawler=$client->request('GET','/');$this->assertTrue($client->getResponse()->isSuccessful());$form=$crawler->selectButton('Search')->form();$form['q']='chest pain';$crawler=$client->submit($form);$this->assertTrue($client->getResponse()->isSuccessful());$this->assertTrue($crawler->filter('h3')->count()>0);}2011
- 104. Ex. 3 Symfony2 functional tests$ phpunit.bat --tap -c app src/BHS/.../DefaultControllerTest.phpTAP version 13ok 1 - BHS\...\DefaultControllerTest::testIndexok 2 - BHS\...\DefaultControllerTest::testSearchok 3 - BHS\...\DefaultControllerTest::testCareset1..32011
- 105. Ex. 3 Symfony2 functional tests$ perl check_tap.pl -e 'c:/xampp/php/phpunit.bat --tap-c c:/web/cpoe_ordersets/app' -s 'c:/web/cpoe_ordersets/src/BHS/CPOEBundle/Tests/Controller/DefaultControllerTest.php'TAP OK - PASS | total=3;; failures=0;; passed=3;;2011
- 107. cucumber-nagiosFrom the Ruby on Rails community.Thanks to RanjibDey for mentioning it to me!2011
- 108. cucumber-nagiosCucumber is for BDD in “natural” language:Feature: google.com It should be up And I should be able to search for things Scenario: Searching for things When I go to "http://www.google.com.au/" And I fill in "q" with "wikipedia" And I press "Google Search" Then I should see "www.wikipedia.org"2011
- 109. cucumber-nagioscucumber-nagios hooks Nagios up to these features:$ bin/cucumber-nagios features/ebay.com.au/bidding.feature CUCUMBER OK - Critical: 0, Warning: 0, 4 okay2011
- 111. 201154Test in productionYour software came with tests, right?Can you get them?Can you write them?
- 112. Monitor in developmentBuild confidence
- 114. Not a CI tool but:
- 115. Alerting rules
- 116. Escalation
- 117. Service handlers2011
- 118. Tolerate ambiguityHow much failure is worth being paged in the middle of the night?Example: Our Citrix farm2011
- 119. Check from both sidesI can use our web app; can the user?
- 120. Firewalls need asymmetrical testing2011
- 121. Monitoring at a higher levelMeasure and monitor the things the user sees/feelsMonitoring at a higher levelSelenium tests web apps using actual browsers
- 122. AutoIt, AppleScript for GUI apps?2011
- 123. Monitoring at a higher levelTesting mobile platforms is still hard2011
- 124. 201161CENTRAL IDEAMonitoring tools are to the sysadmin what testing tools are to the developer.
- 126. Learn from software engineering test practices
- 127. Monitor at multiple levels
- 128. Unit
- 129. Integration
- 130. Acceptance
- 131. Grow your monitoring suite gradually
- 132. Re-use testing tools for monitoring2011
- 133. Blatant plugCome to my other talk at 4:30!Writing Custom Nagios Plugins In PerlA hands-on workshop walking you through writing your first custom check script in Perl using the Nagios::Plugin toolkit. Basic familiarity with Perl, Ruby, PHP, or shell scripting would help, but the basic concepts will be transferable to other languages.2011
- 134. Questions? Comments?Comments optional at my blog post:goo.gl/Oc2rn 2011
- 135. Bonus Section
- 136. Bad restaurantCustomer drinks waterCustomer gets thirstyNotices water glass is emptyWaitsGets mad2011
- 137. Bad restaurantFlags waiter (waiter rolls eyes)Waiter opens a support ticket and assigns it to the busserBusser is on break
- 138. Bad restaurantCustomer waits, flags waiter againBusserrefills waterCustomer drinks and leaves a nice big tip
- 139. Nice restaurants employ ninja waitersThe customer never notices but the glass is always full.
Editor's Notes
- #6: Write this on a board?
- #20: E.g. not “Does the Adder object add numbers properly?” but “Can the cashier complete a sale?”
- #22: We’re all trying to deliver quality. XKCD mouseover text: “The weird sense of duty really good sysadmins have can border on the sociopathic, but it's nice to know that it stands between the forces of darkness and your cat blog's servers.”
- #24: We certainlyneed to monitor these things! They particularly speed problem resolution.
- #27: *See my other talk.
- #38: http://www.perl.org/about/whitepapers/perl-testing.html
- #65: (Plug my other talk)