Frank Migge It Security Patch Monitoring With Nagios 02

Are my systems alive secure?

IT SECURITY PATCH MONITORING WITH NAGIOS
Frank Migge, Manager Information Security Office

Agenda
1. Vulnerabilities 2. Improving Patch Management
Increasing Numbers IT Infrastructure Vendor Review
Enabling Factors Patch Monitoring for Windows
Focus on Operating Systems Patch Monitoring for AIX Unix
Mitigation and Elimination Patch Monitoring for Linux
Strategies Patch Monitoring for Cisco
Vendor Response
The Challenges of Patching 3. Experience and Future


1. Vulnerabilities

Definition:

A weakness in system security procedures, system design, implementation, or internal
controls that could be exploited impacting confidentiality, integrity or availability of the
system.

Vulnerable systems become exploitable for a malicious person or automated malware
(virus) using a variety of techniques like script code injection, SQL injection, buffer
overflow, etc.


Vulnerabilities: Increasing Numbers

Steep increase in recent vulnerabilities

Source: http://www.cert.org/stats/
CERT*, the Computer Emergency Readiness
Team, who coordinates communication
during security emergencies and helps to
prevent future incidents.

*CERT is one of the oldest institutions in IT Security, not
to confuse with the US‐CERT at http://www.us‐cert.gov/


Reasons:
increasing software complexity
faster time‐to‐market (sell first – update later)
Network connectivity built into everything
Internet everywhere on the planet, greater
pool of smart people on the “wrong” side

Million lines
Focus on Operating Systems

OS became the largest “piece” of SW:
Increased size due to progress in
GUI design, device support, “features”,
connectivity, integrated applications
Increased required disk space
But also increased vulnerability.

“complexity is the worst enemy of security”
Bruce Schneier, http://www.schneier.com/crypto‐gram‐0003.html#8


Network OS Vendor Cisco: Fighting with it’s IOS complexity
272722 different IOS Images known to the Cisco Feature Navigator (June 2009)
Source: “Router Exploitation” ‐ Felix ‘FX‘ Lindner, BlackHat 2009, P19: The IOS Image Hell ‐ http://www.blackhat.com/

CCO example: SOHO Router 1812 = 184 versions
Reasons: HW, IOS is still a single, large ELF binary
Which version is the latest? Which has bugs???

Each Operating System vendor has a different vulnerability and risk “profile”

Common myth based on past experience:
Windows has the highest risk.
As a target, yes, but not anymore by total
numbers of vulnerabilities.
Why does Linux look so “bad”?
Compared to a “barebone” OS, Linux
distributions contain large numbers of
applications in addition to the core OS

Source: http://secunia.com/advisories/vendor Secunia, established in 2002, is one of the leading vulnerability
intelligence provider and distributors. It’s freely available Security Advisories list is used by IT Security teams.


Other risk criteria:
Exposure, available exploits for vulnerabilities (virus)
Available protection for OS (hardening, access control)
Integration of additional HW and Devices
Deployed applications
Criticality for business

Vulnerability Mitigation and Elimination Strategies

OS Vendors: Secure Configuration Defaults
disable insecure services
enforce default password change
Secure Services
using encryption and authentication
Implementation of Mitigation Features
system firewalls or access control
enhanced privilege separation and definitions
Patches, patches, patches …


IT SW Industry: Add‐On Mitigation Software
Virus Scanner (Client, Server and Storage side)
Host‐Based IDS
Endpoint Security
Host‐Based Firewall
Device Control Wireless, USB
Configuration Control System scanner, Integrity Checker

The OS vendor patch response
New ways in patch provisioning, distribution, schedules and types:
manual online patch download built‐in, automatic online patch service
Microsoft: Windows Update Service and Windows Update Website (ActiveX)
Linux: Novell Update Service (SLES), Redhat RHN Update Service
IBM: Service Update Management Assistant (SUMA)

Simple vendor download site distributed, policy‐based patch‐server architecture
Microsoft: WSUS
Novell: ZENworks Patch Management Server
IBM: Tivoli® Provisioning Manager


New patch types: emergency (interim) patch, standard patch, service‐pack
ad‐hoc patch releases periodic patch days
Windows: monthly, first Tuesday in a month (Patch Tuesday, Black Tuesday)
Cisco: bi‐annual, fourth Wednesday of March and September
IBM: quarterly schedule for service packs

Challenges of Patching ‐ Why are systems unpatched?

Patching costs resources (= money), real risk is difficult to quantify
IT must balance operational costs vs. security risks
IT operations cost is under high pressure (Outsourcing, SAS, HW consolidation)
Patches need to be tested, any system change is a risk to current setup
Too many vulnerabilities (while patching is scheduled, new patches are released)
Vendors and security organizations announce ca 150 vulnerabilities/week
Patch notification and distribution is not standardized
Vulnerability and Patch management is central part of IT Security Programs


IT Security teams constantly re‐evaluate IT risk level based on new vulnerabilities, exploits,
current system and application patch level, estimate window between identification of
vulnerabilities and creation of exploits (shrinking). Among the common security tasks:
Execution of periodic Vulnerability Scans
Vulnerability Monitoring (time consuming, manual process)
Escalation of perceived “high‐risk” systems and situations

2. Improving IT Patch Management with Nagios
OS patch and version monitoring plug‐in’s for Windows, AIX, Linux and Cisco


Plugin descriptions and links also available via http://www.monitoringexchange.org “Articles”

2. Improving IT Patch Management
• Implementation of a vendor neutral patch status monitoring on all systems
• Implementation of immediate, standardized patch notification for all systems
• Leveraging existing systems inventory and monitoring escalation setup
• Real‐time view into the current systems patch status and software versions

Patch status becomes just another indicator for “system health”.

A task for

Benefits:
Faster, direct and standardized notification to the support engineers
Reduction of “human error” – missed systems / forgotten patches
Fast identification of vulnerable systems
Enforce and monitor patch policy compliance
Highly visible patch accountability

Today's typical IT Infrastructure and Vendors:

Traditional
Windows Network
UNIX Linux Servers Appliances
Servers Equipment
Servers
• Office • Database • Database • Switches • Storage
Backend • Application • Application • Routers • VOIP
• GroupWare • Web Servers • Web Servers • Firewalls • VMware
• App Servers hosts
Microsoft IBM, HP, SUN RedHat, SuSE Cisco Others

‐ Few network and server vendors, but each has it’s own distinctive patch management
‐ By implementing Nagios patch checks, we can cover almost all critical IT areas


OS distribution in IT Infrastructure: We
have a gap of 67 systems = 7.41%. Not
covered systems are appliances, i.e. PBX
and storage.

Nagios patch monitoring for Windows

• Windows update service
prerequisites: • SNMPtrapgen, [proxy]

• Windows update service via
data source Windows Scripting Host

• win_update_trapsend.vbs (client)
plugin script • send_trap_data.pl (Nagios server)

plugin • passive, scheduled once a day
execution • SNMP trap send to Nagios host


Leverages Microsoft built‐in scripting engine VBS for data collection
requires extra binary for sending SNMPtraps to minimize ‘footprint’
no monitoring daemon installation
Works well if update service is configured for Microsoft
Less efficient with WSUS systems due to limited patch visibility

Patch monitoring: Microsoft Windows – Configure The Monitored System
1. Install, configure and test the programs trapgen.exe and win_update_trapsend.vbs
C:update-monitor > cscript.exe -NoLogo C:update-monitorwin_update_trapsend.vbs > C:update-
monitor win_update_trapsend.log

2. Create batch file and schedule daily execution job through Windows Scheduler
C:update-monitor > edit win_update_trapsend.bat
cscript.exe -NoLogo C:update-monitorwin_update_trapsend.vbs > C:update-
monitorwin_update_trapsend.log


The batch needs local administrative rights to execute.

Patch monitoring: Microsoft Windows – Nagios Setup
1. Configure the SNMPtrap service and install/update the traphandler ‘send_trap_data.pl’
nagios ~ # cat /etc/snmp/snmptrapd.conf
###############################################################################
# snmptrapd.conf:
# configuration file for configuring the ucd-snmp snmptrapd agent.
###############################################################################
# first, we define the access control
authCommunity log,execute,net SECtrap
# Win update traphandler: SNMPv2-MIB::snmpTrapOID.0 = RFC1155-SMI::enterprises.2854.0.1
traphandle RFC1155-SMI::enterprises.2854.0.1 /srv/app/nagios/libexec/send_trap_data.pl

2. Verify passive data submission into Nagios through the named pipe nagios.cfg
# grep EXTERNAL /srv/app/nagios/var/nagios.log
[1251126027] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;jpnhoap016;check_trap_winpatch;0;No
critical updates. | Windows Version: Microsoft Windows Server 2003 5.2.3790, Update Service: WU
http://JPNHOMG029:8530

nagios ~ # vi /srv/app/nagios/etc/objects/patch-services-windows.cfg
############################################################################### 3. Configure the new
# Receive SNMP traps for Windows update status
###############################################################################
patch monitoring
define service {
use generic-patch-win
service
hostgroup 2-windows-servers
name check_trap_winpatch
service_description check_trap_winpatch
service_groups patch-checks-win, patch-compliance
}

Using external commands in Nagios http://linux.com/archive/feature/153285

Patch monitoring: Microsoft Windows – Nagios Views

Service View and
E‐Mail Notification


Nagios patch monitoring for IBM AIX 5.3

• IBM update service
prerequisites:
• SSH Service, [proxy]

data source • SUMA

• check‐rug‐update.pl (client)
plugin script
• check_snmp_extend.sh (svr)

plugin • active, scheduled once a day
execution • Passive, scheduled once a day


The most ‘conservatively’ patched systems
Least experienced staff needs the most help
AIX is more secure in terms of less SW vulnerabilities
AIX is weak in configuration due to the OS being quite ‘old’

Patch monitoring: IBM AIX 5.3 – Configuring the Monitored System
1. Configure and verify the “Service Update Management Assistant" SUMA
$ sudo suma -c -a HTTP_PROXY=http://192.168.100.184:80/
$ sudo suma -c -a DL_TIMEOUT_SEC=10
$ sudo suma -c |grep HTTP_PROXY
HTTP_PROXY=http://192.168.100.184:80/
$ sudo suma -c
…

2. Install the plugin script ‘check‐aix‐update.pl’ or ‘aix_update_trapsend.pl’
$ sudo /scripts/check-aix-update.pl
WARNING - 211 update(s) available: X11.Dt.lib Version 5.3.7.2 X11.Dt.rte Version 5.3.7.3 SLES10
X11.apps.rte Version 5.3.7.1 X11.base.lib Version 5.3.7.2 X11.base.rte Version 5.3.7.5
bos.64bit Version 5.3.7.7 bos.acct Version 5.3.7.8 bos.adt.base Version 5.3.7.3 bos.adt.debug
Version 5.3.7.3 bos.adt.include Version 5.3.7.7 bos.adt.insttools Version 5.3.7.2
... perfagent.tools Version 5.3.7.4 printers.rte Version 5.3.7.2|OS Version 5300-07-01-0748,
Proxy http://10.253.100.184:80/, Update-URL www14.software.ibm.com/webapp/set2/fixget

3. Decide the how to call and return the check result:

SSH SNMPtrap

check‐aix‐update.pl aix_update_trapsend.pl
ssh user@aixhost "sudo /scripts/check‐ cron‐scheduled once a day
aix‐update.pl"

Patch monitoring: IBM AIX 5.3 – Nagios Views

Service Views and


Patch monitoring: Novell Linux SLES10 ‐ Overview

• Novell update service
prerequisites:
• SNMP service, [proxy]

data source • ZENworks zmd service via rug

• check‐rug‐update.pl (client)
plugin script
• check_snmp_extend.sh (svr)

execution • SNMP request to SNMP extend


Depends on ‘rug’ and novell‐zmd service
zmd service ‘zombies’ experienced due to commit issues in sqlite backend
Due to high frequency of Linux patch releases (weekly), big benefit

Patch monitoring: Novell Linux SLES10 – Configuring the Monitored System
1. Configure and verify the SLES Zenworks update service, using the 'rug' command
# rug lu
S | Catalog | Bundle | Name | Version | Arch
--+-------------------+--------+----------+-----------+-------
| SLES10-SP2-Online | | Spident | 0.9-74.24 | noarch
# ./check-rug-update.pl
WARNING - 1 update(s) available: SPident Version 0.9-74.24

2. Install and test the plugin script ‘check‐rug‐update.pl’
# ./check-rug-update.pl --run-rug
OK - system is up to date SLES10
# cat ./test
S | Catalog | Bundle | Name | Version | Arch
--+-------------------+--------+----------+-----------+-------
| SLES10-SP2-Online | | Spident | 0.9-74.24 | noarch
# ./check-rug-update.pl --file=test WARNING - 1 update(s) available: SPident Version 0.9-74.24

3. Configure and test the remote plugin access through the UCD Net‐SNMP service
# echo "extend nagiosupdate /srv/app/nagios/libexec/check-rug-update.pl
--run-rug“ >> /etc/snmp/snmpd.conf
# /etc/init.d/snmpd restart
Shutting down snmpd: done
Starting snmpd
# snmpget -v 2c -c myread 127.0.0.1 NET-SNMP-EXTEND-MIB::nsExtendOutputFull.
“nagiosupdate"
NET-SNMP-EXTEND-MIB::nsExtendOutputFull.“nagiosupdate" = STRING: No updates
are available.

Patch monitoring: Novell Linux SLES10 – Nagios Setup
1. Get, install and test the ‘check_snmp_extend.sh’ script as a plugin
/srv/app/nagios/libexec # cp /tmp/check_snmp_extend.sh .
/srv/app/nagios/libexec # ls -l check_snmp_extend.sh
-rwxr-x--- 1 nagios nagios 1979 2008-10-02 16:50 check_snmp_extend.sh
/srv/app/nagios/libexec # ./check_snmp_extend.sh Syntax: check_snmp_extend.sh ipaddr community
/srv/app/nagios/libexec # ./check_snmp_extend.sh 192.168.11.22 myread nagiosupdate
OK - system is up to date

2. Configure the new plugin in the Nagios command.cfg section
/srv/app/nagios/etc/objects # vi commands.cfg
Nagios
# 'check_snmp_extend' command definition
# syntax: check_snmp_extend.sh host-ip snmp-community extend-name
define command{
command_name check_snmp_extend
command_line $USER1$/check_snmp_extend.sh $HOSTADDRESS$ $ARG1$ $ARG2$
}

/srv/app/nagios/etc/objects # vi sles10-patch-services.cfg
3. Configure the new
###############################################################################
# SLES10 OS Patch Update Check via SNMP extend scripts patch monitoring
###############################################################################
define service { service
use generic-patch
host_name ml08460
name check_snmp_extend
service_description check_patch_sles10
check_command check_snmp_extend!myread!nagiosupdate
}

Patch monitoring: Novell Linux SLES10 – Nagios Views

Service View and
(HTML formatted e‐mail body
with embedded service links,
send through /usr/bin/mutt)


Nagios patch monitoring for Cisco IOS, ASA, PIX

• SNMP service access
prerequisites:
• Cisco CCO account

data source • SNMPv2 MIB "sysDescr"

• check_snmp_patchlevel.pl
plugin script
• check_snmp_patchlevel.cfg

execution • SNMP request to SNMP MIB


Cisco is ‘conservatively’ patched due to risk and effort (reboot, cumbersome rollback)
Big benefit for standardizing OS versions and identifying ‘rogue’ devices
Network device numbers greatly surpass server numbers

Patch monitoring: Cisco IOS, ASA, PIX – Cisco Setup
1. Cisco SNMP service configuration eample
Router # conf t
Router(config)# snmp-server community SECro ro 192.168.103.34
Router(config)# snmp-server host 192.168.103.34 SECtrap
Router(config)# snmp-server enable traps snmp linkdown linkup coldstart warmstart

2. Test SNMP access to the Cisco device
# snmpget -v 1 -c SECro 192.168.203.4 SNMPv2-MIB::sysDescr.0
Cisco:
SNMPv2-MIB::sysDescr.0 = STRING: Cisco Internetwork Operating System Software IOS (tm) C2950
Software (C2950-I6Q4L2-M), Version 12.1(22)EA9, RELEASE SOFTWARE (fc1) Copyright (c) 1986-2006 Router
by cisco Systems, Inc. Compiled Fri 01-Dec-06 18:02 by weiliu
Switches
FW’s


Network devices are usually the best/most “monitored” systems for uptime/performance

They are already set up in Nagios, aren’t they?

Patch monitoring: Cisco IOS, ASA, PIX – Nagios Setup
1. Cisco plugin – version compliance check categories

Nagios

/srv/app/nagios/etc/objects # vi check_snmp_patchlevel.cfg
2. Cisco plugin –
# Below are the 'approved' versions we explicitly endorse for usage: #
###################################################################### compliance check
approved|ios|12.2(13)ZH2|not OK, but currently being actively upgraded
# Below are the 'obsolete' versions we explicitly disapprove of: # configuration file
######################################################################
obsolete|pix|7.2(2)|end-of-maintenance 2009-07-28
obsolete|ios|12.2(25)SEE4|end-of-maintenance date 2007-12-12
# Below are the 'med-vuln' versions with low to medium criticality #
######################################################################
med-vuln|ios|12.4(6)T8|multiple DOS confirmed (Voice, Stack)
######################################################################

Patch monitoring: Cisco IOS, ASA, PIX – Nagios Setup
1. Get, install and test the ‘check_snmp_patchlevel.pl’ script as a plugin
/srv/app/nagios/libexec # ./snmp_patchlevel.pl -H 192.168.203.4 -g ios -C SECro
IOS Version: 12.1(22)EA9 | Cisco Internetwork Operating System Software IOS (tm) C2950 Software
(C2950-I6Q4L2-M), Version 12.1(22)EA9, RELEASE SOFTWARE (fc1) Copyright (c) 1986-2006 by cisco
Systems, Inc. Compiled Fri 01-Dec-06 18:02 by weiliu

2. Configure the new plugin in the Nagios command.cfg section
/srv/app/nagios/etc/objects # vi commands.cfg
define command{
command_name check_snmp_cisco_ios
command_line $USER1$/check_snmp_patchlevel.pl -H $HOSTADDRESS$ -g ios -C $ARG1$ Nagios
}
define command{
command_name check_snmp_cisco_asa
command_line $USER1$/check_snmp_patchlevel.pl -H $HOSTADDRESS$ -g asa -C $ARG1$
}

/srv/app/nagios/etc/objects # vi sles10-patch-services.cfg
3. Configure the new
###############################################################################
# Check Cisco Router IOS version against a config file patch monitoring
###############################################################################
define service { service
use generic-patch
hostgroup cisco-routers
name check_snmp_ios_router
service_description check_snmp_ios_router
check_command check_snmp_cisco_ios!SECro
}

Patch monitoring: Cisco IOS, ASA, PIX – Nagios Views

Service View and


3. Experience and Future


Central patch
status view in
Nagios


Monitoring Patch Policy Compliance:
open, outstanding patches
time periods until patched
current OS versions and patch update settings

Patch monitoring Issues and Improvements

Windows: Improving patch identification for WSUS managed systems
Can we switch safely from WSUS to Windows Online and back to WSUS

Cisco: Automate the manual process to identify available updates
Investigate the Cisco IOS Auto‐Upgrade Manager, parse the Cisco Website with CCO credentials?

Expand patch and version monitoring into the applications space:

First target major DB vendors: IBM, Oracle, Microsoft
Combine the “DB up” check with a DB real login and return the DB version using JAVA thinclients


Thank you for listening.
Time for Questions?

Frank Migge It Security Patch Monitoring With Nagios 02

More Related Content

What's hot (20)

Viewers also liked (6)

Similar to Frank Migge It Security Patch Monitoring With Nagios 02 (20)

Recently uploaded (20)

Frank Migge It Security Patch Monitoring With Nagios 02