OSMC 2011 | Hardware Monitoring with the new IPMI Plugin v2 by Werner Fischer
0 likes53 views
The document outlines the capabilities and implementation of the Intelligent Platform Management Interface (IPMI) plugin for hardware monitoring. It covers IPMI's main features, including monitoring conditions of components like temperature and power supplies, and demonstrates its usage with tools like ipmitool and freeipmi. The document concludes with potential pitfalls and solutions regarding sensor state issues and unrecognized events.
![slide 19/37
Example: remote control with ipmitool
[user@adminpc ~]$ ipmitool I lan H 192.168.1.211
U admin power status
Password:
Chassis Power is off
[user@adminpc ~]$
[user@adminpc ~]$ ipmitool I lan H 192.168.1.211
U admin power status
Password:
Chassis Power is off
[user@adminpc ~]$
[user@adminpc ~]$ ipmitool I lan H 192.168.1.211
U admin power on
Password:
Chassis Power Control: Up/On
[user@adminpc ~]$
[user@adminpc ~]$ ipmitool I lan H 192.168.1.211
U admin power on
Password:
Chassis Power Control: Up/On
[user@adminpc ~]$
[user@adminpc ~]$ ipmitool I lan H 192.168.1.211
U admin power status
Password:
Chassis Power is on
[user@adminpc ~]$
[user@adminpc ~]$ ipmitool I lan H 192.168.1.211
U admin power status
Password:
Chassis Power is on
[user@adminpc ~]$](https://image.slidesharecdn.com/wernerfischerhardwaremonitoringwiththenewipmi-plugin-v2-190125101759/85/OSMC-2011-Hardware-Monitoring-with-the-new-IPMI-Plugin-v2-by-Werner-Fischer-19-320.jpg)
![slide 21/37
Discrete Threshold
IPMI Sensor Classes (2/2)
[root@test ~]# ipmitool sdr get "PS2 Status"
Sensor ID : PS2 Status (0x71)
Entity ID : 10.2 (Power Supply)
Sensor Type (Discrete): Power Supply
States Asserted : Power Supply
[Presence detected]
[Power Supply AC lost]
Assertion Events : Power Supply
[Presence detected]
[Power Supply AC lost]
Assertions Enabled : Power Supply
[Presence detected]
[Failure detected]
[Predictive failure]
[Power Supply AC lost]
[...]
Deassertions Enabled : Power Supply
[...]
[root@test ~]# ipmitool sdr get "PS2 Status"
Sensor ID : PS2 Status (0x71)
Entity ID : 10.2 (Power Supply)
Sensor Type (Discrete): Power Supply
States Asserted : Power Supply
[Presence detected]
[Power Supply AC lost]
Assertion Events : Power Supply
[Presence detected]
[Power Supply AC lost]
Assertions Enabled : Power Supply
[Presence detected]
[Failure detected]
[Predictive failure]
[Power Supply AC lost]
[...]
Deassertions Enabled : Power Supply
[...]
[root@test ~]# ipmitool sdr get "Fan 1"
Sensor ID : Fan 1 (0x50)
Entity ID : 29.1 (Fan Device)
Sensor Type (Analog) : Fan
Sensor Reading : 5719 (+/ 0) RPM
Status : ok
Nominal Reading : 6708.000
Normal Minimum : 2451.000
Normal Maximum : 10965.000
Lower critical : 1720.000
Lower noncritical : 1978.000
Positive Hysteresis : 86.000
Negative Hysteresis : 86.000
Minimum sensor range : Unspecified
Maximum sensor range : Unspecified
Event Message Control : Perthreshold
Readable Thresholds : lcr lnc
Settable Thresholds : lcr lnc
Threshold Read Mask : lcr lnc
Assertion Events :
Assertions Enabled : lnc lcr
Deassertions Enabled : lnc lcr
[root@test ~]# ipmitool sdr get "Fan 1"
Sensor ID : Fan 1 (0x50)
Entity ID : 29.1 (Fan Device)
Sensor Type (Analog) : Fan
Sensor Reading : 5719 (+/ 0) RPM
Status : ok
Nominal Reading : 6708.000
Normal Minimum : 2451.000
Normal Maximum : 10965.000
Lower critical : 1720.000
Lower noncritical : 1978.000
Positive Hysteresis : 86.000
Negative Hysteresis : 86.000
Minimum sensor range : Unspecified
Maximum sensor range : Unspecified
Event Message Control : Perthreshold
Readable Thresholds : lcr lnc
Settable Thresholds : lcr lnc
Threshold Read Mask : lcr lnc
Assertion Events :
Assertions Enabled : lnc lcr
Deassertions Enabled : lnc lcr](https://image.slidesharecdn.com/wernerfischerhardwaremonitoringwiththenewipmi-plugin-v2-190125101759/85/OSMC-2011-Hardware-Monitoring-with-the-new-IPMI-Plugin-v2-by-Werner-Fischer-21-320.jpg)
![slide 22/37
IPMI Sensor Types
root@test:~# ipmisensors L
Temperature
Voltage
Current
Fan
Physical_Security
Platform_Security_Violation_Attempt
Processor
Power_Supply
Power_Unit
Cooling_Device
[…]
root@test:~# ipmisensors L
Temperature
Voltage
Current
Fan
Physical_Security
Platform_Security_Violation_Attempt
Processor
Power_Supply
Power_Unit
Cooling_Device
[…]](https://image.slidesharecdn.com/wernerfischerhardwaremonitoringwiththenewipmi-plugin-v2-190125101759/85/OSMC-2011-Hardware-Monitoring-with-the-new-IPMI-Plugin-v2-by-Werner-Fischer-22-320.jpg)
![slide 23/37
Example: query sensors with FreeIPMI
[root@testserver ~]# ipmimonitoring
Record_ID | Sensor Name | Sensor Group | Monitoring Status|
Sensor Units | Sensor Reading
[...]
17 | Fan 5 | Fan | Nominal | RPM | 9052.000000
18 | Fan 6 | Fan | Nominal | RPM | 8060.000000
19 | PS1 AC Current | Current | Nominal | A | 0.124000
20 | PS2 AC Current | Current | Nominal | A | 0.992000
[...]
36 | Physical Scrty | Physical Security | Critical | N/A |
'General Chassis Intrusion'
[root@testserver ~]# ipmimonitoring
Record_ID | Sensor Name | Sensor Group | Monitoring Status|
Sensor Units | Sensor Reading
[...]
17 | Fan 5 | Fan | Nominal | RPM | 9052.000000
18 | Fan 6 | Fan | Nominal | RPM | 8060.000000
19 | PS1 AC Current | Current | Nominal | A | 0.124000
20 | PS2 AC Current | Current | Nominal | A | 0.992000
[...]
36 | Physical Scrty | Physical Security | Critical | N/A |
'General Chassis Intrusion'](https://image.slidesharecdn.com/wernerfischerhardwaremonitoringwiththenewipmi-plugin-v2-190125101759/85/OSMC-2011-Hardware-Monitoring-with-the-new-IPMI-Plugin-v2-by-Werner-Fischer-23-320.jpg)
![slide 24/37
Example: interpret discrete sensors( FreeIPMI)
root@test:~# cat /etc/freeipmi/freeipmi_interpret_sensor.conf
[…]
## IPMI_Physical_Security
#
# IPMI_Physical_Security_No_Event Nominal
# IPMI_Physical_Security_General_Chassis_Intrusion Critical
# IPMI_Physical_Security_Drive_Bay_Intrusion Critical
[…]
# IPMI_Power_Supply_No_Event Nominal
# IPMI_Power_Supply_Presence_Detected Nominal
# IPMI_Power_Supply_Power_Supply_Failure_Detected Critical
# IPMI_Power_Supply_Predictive_Failure Critical
# IPMI_Power_Supply_Power_Supply_Input_Lost_AC_DC Critical
[…]
## IPMI_Memory
#
# IPMI_Memory_No_Event Nominal
# IPMI_Memory_Correctable_Memory_Error Warning
# IPMI_Memory_Uncorrectable_Memory_Error Critical
root@test:~# cat /etc/freeipmi/freeipmi_interpret_sensor.conf
[…]
## IPMI_Physical_Security
#
# IPMI_Physical_Security_No_Event Nominal
# IPMI_Physical_Security_General_Chassis_Intrusion Critical
# IPMI_Physical_Security_Drive_Bay_Intrusion Critical
[…]
# IPMI_Power_Supply_No_Event Nominal
# IPMI_Power_Supply_Presence_Detected Nominal
# IPMI_Power_Supply_Power_Supply_Failure_Detected Critical
# IPMI_Power_Supply_Predictive_Failure Critical
# IPMI_Power_Supply_Power_Supply_Input_Lost_AC_DC Critical
[…]
## IPMI_Memory
#
# IPMI_Memory_No_Event Nominal
# IPMI_Memory_Correctable_Memory_Error Warning
# IPMI_Memory_Uncorrectable_Memory_Error Critical](https://image.slidesharecdn.com/wernerfischerhardwaremonitoringwiththenewipmi-plugin-v2-190125101759/85/OSMC-2011-Hardware-Monitoring-with-the-new-IPMI-Plugin-v2-by-Werner-Fischer-24-320.jpg)
![slide 25/37
IPMI System Event Log (SEL)
• stored in non-volatile storage
[root@testserver ~]# ipmitool sel elist
40 | 06/21/2010 | 14:29:29 | Power Supply PS1 Status | Power Supply AC lost | Asserted
54 | 06/21/2010 | 14:29:29 | Power Unit Power Redundancy | Fully Redundant
68 | 06/21/2010 | 14:29:29 | Power Unit Power Redundancy | Redundancy Lost
7c | 06/21/2010 | 14:29:29 | Power Unit Power Redundancy | NonRedundant: Sufficient from Redundant
[...]
2fc | 06/21/2010 | 15:20:32 | Physical Security Physical Scrty | General Chassis intrusion | Asserted
[root@testserver ~]# ipmitool sel elist
40 | 06/21/2010 | 14:29:29 | Power Supply PS1 Status | Power Supply AC lost | Asserted
54 | 06/21/2010 | 14:29:29 | Power Unit Power Redundancy | Fully Redundant
68 | 06/21/2010 | 14:29:29 | Power Unit Power Redundancy | Redundancy Lost
7c | 06/21/2010 | 14:29:29 | Power Unit Power Redundancy | NonRedundant: Sufficient from Redundant
[...]
2fc | 06/21/2010 | 15:20:32 | Physical Security Physical Scrty | General Chassis intrusion | Asserted
[root@testserver ~]# ipmitool sel elist
Power Supply PS1 Status | Power Supply AC lost | Asserted
Power Unit Power Redundancy | Fully Redundant
Power Unit Power Redundancy | Redundancy Lost
Power Unit Power Redundancy | NonRedundant: Sufficient from Redundant
[...]
Physical Security Physical Scrty | General Chassis intrusion | Asserted
[root@testserver ~]# ipmitool sel elist
Power Supply PS1 Status | Power Supply AC lost | Asserted
Power Unit Power Redundancy | Fully Redundant
Power Unit Power Redundancy | Redundancy Lost
Power Unit Power Redundancy | NonRedundant: Sufficient from Redundant
[...]
Physical Security Physical Scrty | General Chassis intrusion | Asserted](https://image.slidesharecdn.com/wernerfischerhardwaremonitoringwiththenewipmi-plugin-v2-190125101759/85/OSMC-2011-Hardware-Monitoring-with-the-new-IPMI-Plugin-v2-by-Werner-Fischer-25-320.jpg)
![slide 28/37
Plugin implementation
• Bash script
• uses FreeIPMI, gawk
# ./check_ipmi_sensor H 10.10.10.114 f /etc/ipmiconfig/ipmi.cfg v 2
IPMI Status: OK | 'System Temp'=29.000000 'FAN 1'=4320.000000 'FAN
[…]
System Temp = 29.000000 (Status: Nominal)
CPU Temp = 'Low' (Status: Nominal)
FAN 1 = 4320.000000 (Status: Nominal)
FAN 2 = 4320.000000 (Status: Nominal)
FAN 3 = 4590.000000 (Status: Nominal)
[…]
AVCC = 3.408000 (Status: Nominal)
Chassis Intru = 'OK' (Status: Nominal)
PS Status = 'Presence detected' (Status: Nominal)
# ./check_ipmi_sensor H 10.10.10.114 f /etc/ipmiconfig/ipmi.cfg v 2
IPMI Status: OK | 'System Temp'=29.000000 'FAN 1'=4320.000000 'FAN
[…]
System Temp = 29.000000 (Status: Nominal)
CPU Temp = 'Low' (Status: Nominal)
FAN 1 = 4320.000000 (Status: Nominal)
FAN 2 = 4320.000000 (Status: Nominal)
FAN 3 = 4590.000000 (Status: Nominal)
[…]
AVCC = 3.408000 (Status: Nominal)
Chassis Intru = 'OK' (Status: Nominal)
PS Status = 'Presence detected' (Status: Nominal)](https://image.slidesharecdn.com/wernerfischerhardwaremonitoringwiththenewipmi-plugin-v2-190125101759/85/OSMC-2011-Hardware-Monitoring-with-the-new-IPMI-Plugin-v2-by-Werner-Fischer-28-320.jpg)
![slide 32/37
Common pitfalls
• sensors with state N/A
• solution shortest-term: exclude (-x opt.)
• solution short-term: FreeIPMI update
tkwiki.cc/FreeIPMI-NA-Sensor
[…]
12 | CPU1 Temp | OEM Reserved | N/A | N/A | N/A | 'OEM Event = 0000h'
13 | CPU2 Temp | OEM Reserved | N/A | N/A | N/A | 'OEM Event = 0000h'
[…]
[…]
12 | CPU1 Temp | OEM Reserved | N/A | N/A | N/A | 'OEM Event = 0000h'
13 | CPU2 Temp | OEM Reserved | N/A | N/A | N/A | 'OEM Event = 0000h'
[…]](https://image.slidesharecdn.com/wernerfischerhardwaremonitoringwiththenewipmi-plugin-v2-190125101759/85/OSMC-2011-Hardware-Monitoring-with-the-new-IPMI-Plugin-v2-by-Werner-Fischer-32-320.jpg)
![slide 33/37
Common pitfalls
• unrecognized events
• solution shortest-term: ignore unrec. e.
tkwiki.cc/FreeIPMI-Unrec-Event
[…]
40 | Status | Cable/Interconnect | Nominal | N/A |
'Cable/Interconnect is connected'
41 | RAC Status | Module/Board | N/A | N/A | 'Unrecognized Event =
0001h' 'Unrecognized Event = 0002h' 'Unrecognized Event = 0004h'
42 | OS Watchdog | Watchdog 2 | Nominal | N/A | 'OK'
[…]
[…]
40 | Status | Cable/Interconnect | Nominal | N/A |
'Cable/Interconnect is connected'
41 | RAC Status | Module/Board | N/A | N/A | 'Unrecognized Event =
0001h' 'Unrecognized Event = 0002h' 'Unrecognized Event = 0004h'
42 | OS Watchdog | Watchdog 2 | Nominal | N/A | 'OK'
[…]](https://image.slidesharecdn.com/wernerfischerhardwaremonitoringwiththenewipmi-plugin-v2-190125101759/85/OSMC-2011-Hardware-Monitoring-with-the-new-IPMI-Plugin-v2-by-Werner-Fischer-33-320.jpg)
OSMC 2011 | Hardware Monitoring with the new IPMI Plugin v2 by Werner Fischer
- 1. Hardware Monitoring with the new IPMI Plugin v2 Werner Fischer, Technology Specialist Thomas-Krenn.AG 6. OSMC / Nuremberg / Germany 29th November 2011
- 2. slide 2/37 who I am not who I am Werner Fischer Introduction working for a Server vendor Linux user since 2001 Kernel or H/W developer
- 3. slide 3/37 who is Server & accessories "Made in Germany" Introduction based in Freyung, Bavaria serving all over Europe
- 4. slide 4/37 Some questions ... Should I use monitoring?
- 5. slide 5/37 Should I use monitoring? It depends on what you want to do in your free time... remember yesterday evening? Some questions ... ✘✔
- 6. slide 6/37 All drives of the RAID 6 O.K.? Some questions ... ✔✔ ✔ ✔
- 7. slide 7/37 All network connections O.K.? Some questions ... ✔ ✔
- 8. slide 8/37 All FANs O.K.? Some questions ... ?
- 9. slide 9/37 All power supplies O.K.? Some questions ... ?
- 10. slide 10/37 Can we monitor all these servers? Some questions ... any IPMI compatible server
- 11. slide 11/37 Can we monitor all these servers? Easily? Some questions ... any IPMI compatible server
- 12. slide 12/37 Can we monitor all these servers? Easily? With one single tool? Some questions ... any IPMI compatible server
- 13. slide 13/37 Can we monitor all these servers? Easily? With one single tool? Some questions ...
- 14. slide 14/37 Agenda 1) IPMI overview 2) Plugin implementation 3) Live demo 4) Common pitfalls
- 15. slide 15/37 Intelligent Platform Management Interface • IPMI developed by Intel, HP, NEC, Dell – 1998: IPMI v1.0 – 2001: IPMI v1.5 – 2004: IPMI v2.0
- 16. slide 16/37 IPMI main features Monitoring (temp, fans, ...) Recovery Control (power on/off/reset) Logging (System Event Log) Inventory (FRU information)
- 17. slide 17/37 IPMI overview Chassis board Motherboard Processor board Memory board Baseboard Management Controller (BMC) System bus NVS Storage SDR SEL FRU Chassis mgmt. (Satellite Controller) Sensors & Controls Fan sensor Temp. sensor Power control Reset control … FRU Temp. s. FRU private mgmt. busses IPMB M/B Serial Controller Serial Port Sharing BMC Serial Controller Serial/Modem interface LAN interface Serial Connector LAN Connector PCI mgmt. bus Network (LAN) Controller Remote Mmgt. Card (KVM over IP, ...) Auxillary IPMB Connector ICMB ICMB bridge System interface Redundant Power board FRU Temp. sensor … FRU access req. root privileges access req. username & password
- 18. slide 18/37 IPMI Channel Privilege Levels (LAN access) use privilege level 'User' for monitoring purposes Privilege Level Allows User ● query sensors Operator ● nearly all IPMI commands ● but no changing of out-of- band interfaces Administrator • all IPMI commands
- 19. slide 19/37 Example: remote control with ipmitool [user@adminpc ~]$ ipmitool I lan H 192.168.1.211 U admin power status Password: Chassis Power is off [user@adminpc ~]$ [user@adminpc ~]$ ipmitool I lan H 192.168.1.211 U admin power status Password: Chassis Power is off [user@adminpc ~]$ [user@adminpc ~]$ ipmitool I lan H 192.168.1.211 U admin power on Password: Chassis Power Control: Up/On [user@adminpc ~]$ [user@adminpc ~]$ ipmitool I lan H 192.168.1.211 U admin power on Password: Chassis Power Control: Up/On [user@adminpc ~]$ [user@adminpc ~]$ ipmitool I lan H 192.168.1.211 U admin power status Password: Chassis Power is on [user@adminpc ~]$ [user@adminpc ~]$ ipmitool I lan H 192.168.1.211 U admin power status Password: Chassis Power is on [user@adminpc ~]$
- 20. slide 20/37 Discrete Threshold multiple states possible: ● up to 15 states ● each state is reflected by a bit ● multiple state bits can active changes event status on: ● analog reading compared to threshold values can provide: ● generic states ● sensor-specific states provides: ● analog reading of the sensor ● discr. threshold comparison status bit other class similar to discrete: ● OEM: discrete sensor where the meaning of the states (offsets) are OEM defined IPMI Sensor Classes (1/2)
- 21. slide 21/37 Discrete Threshold IPMI Sensor Classes (2/2) [root@test ~]# ipmitool sdr get "PS2 Status" Sensor ID : PS2 Status (0x71) Entity ID : 10.2 (Power Supply) Sensor Type (Discrete): Power Supply States Asserted : Power Supply [Presence detected] [Power Supply AC lost] Assertion Events : Power Supply [Presence detected] [Power Supply AC lost] Assertions Enabled : Power Supply [Presence detected] [Failure detected] [Predictive failure] [Power Supply AC lost] [...] Deassertions Enabled : Power Supply [...] [root@test ~]# ipmitool sdr get "PS2 Status" Sensor ID : PS2 Status (0x71) Entity ID : 10.2 (Power Supply) Sensor Type (Discrete): Power Supply States Asserted : Power Supply [Presence detected] [Power Supply AC lost] Assertion Events : Power Supply [Presence detected] [Power Supply AC lost] Assertions Enabled : Power Supply [Presence detected] [Failure detected] [Predictive failure] [Power Supply AC lost] [...] Deassertions Enabled : Power Supply [...] [root@test ~]# ipmitool sdr get "Fan 1" Sensor ID : Fan 1 (0x50) Entity ID : 29.1 (Fan Device) Sensor Type (Analog) : Fan Sensor Reading : 5719 (+/ 0) RPM Status : ok Nominal Reading : 6708.000 Normal Minimum : 2451.000 Normal Maximum : 10965.000 Lower critical : 1720.000 Lower noncritical : 1978.000 Positive Hysteresis : 86.000 Negative Hysteresis : 86.000 Minimum sensor range : Unspecified Maximum sensor range : Unspecified Event Message Control : Perthreshold Readable Thresholds : lcr lnc Settable Thresholds : lcr lnc Threshold Read Mask : lcr lnc Assertion Events : Assertions Enabled : lnc lcr Deassertions Enabled : lnc lcr [root@test ~]# ipmitool sdr get "Fan 1" Sensor ID : Fan 1 (0x50) Entity ID : 29.1 (Fan Device) Sensor Type (Analog) : Fan Sensor Reading : 5719 (+/ 0) RPM Status : ok Nominal Reading : 6708.000 Normal Minimum : 2451.000 Normal Maximum : 10965.000 Lower critical : 1720.000 Lower noncritical : 1978.000 Positive Hysteresis : 86.000 Negative Hysteresis : 86.000 Minimum sensor range : Unspecified Maximum sensor range : Unspecified Event Message Control : Perthreshold Readable Thresholds : lcr lnc Settable Thresholds : lcr lnc Threshold Read Mask : lcr lnc Assertion Events : Assertions Enabled : lnc lcr Deassertions Enabled : lnc lcr
- 22. slide 22/37 IPMI Sensor Types root@test:~# ipmisensors L Temperature Voltage Current Fan Physical_Security Platform_Security_Violation_Attempt Processor Power_Supply Power_Unit Cooling_Device […] root@test:~# ipmisensors L Temperature Voltage Current Fan Physical_Security Platform_Security_Violation_Attempt Processor Power_Supply Power_Unit Cooling_Device […]
- 23. slide 23/37 Example: query sensors with FreeIPMI [root@testserver ~]# ipmimonitoring Record_ID | Sensor Name | Sensor Group | Monitoring Status| Sensor Units | Sensor Reading [...] 17 | Fan 5 | Fan | Nominal | RPM | 9052.000000 18 | Fan 6 | Fan | Nominal | RPM | 8060.000000 19 | PS1 AC Current | Current | Nominal | A | 0.124000 20 | PS2 AC Current | Current | Nominal | A | 0.992000 [...] 36 | Physical Scrty | Physical Security | Critical | N/A | 'General Chassis Intrusion' [root@testserver ~]# ipmimonitoring Record_ID | Sensor Name | Sensor Group | Monitoring Status| Sensor Units | Sensor Reading [...] 17 | Fan 5 | Fan | Nominal | RPM | 9052.000000 18 | Fan 6 | Fan | Nominal | RPM | 8060.000000 19 | PS1 AC Current | Current | Nominal | A | 0.124000 20 | PS2 AC Current | Current | Nominal | A | 0.992000 [...] 36 | Physical Scrty | Physical Security | Critical | N/A | 'General Chassis Intrusion'
- 24. slide 24/37 Example: interpret discrete sensors( FreeIPMI) root@test:~# cat /etc/freeipmi/freeipmi_interpret_sensor.conf […] ## IPMI_Physical_Security # # IPMI_Physical_Security_No_Event Nominal # IPMI_Physical_Security_General_Chassis_Intrusion Critical # IPMI_Physical_Security_Drive_Bay_Intrusion Critical […] # IPMI_Power_Supply_No_Event Nominal # IPMI_Power_Supply_Presence_Detected Nominal # IPMI_Power_Supply_Power_Supply_Failure_Detected Critical # IPMI_Power_Supply_Predictive_Failure Critical # IPMI_Power_Supply_Power_Supply_Input_Lost_AC_DC Critical […] ## IPMI_Memory # # IPMI_Memory_No_Event Nominal # IPMI_Memory_Correctable_Memory_Error Warning # IPMI_Memory_Uncorrectable_Memory_Error Critical root@test:~# cat /etc/freeipmi/freeipmi_interpret_sensor.conf […] ## IPMI_Physical_Security # # IPMI_Physical_Security_No_Event Nominal # IPMI_Physical_Security_General_Chassis_Intrusion Critical # IPMI_Physical_Security_Drive_Bay_Intrusion Critical […] # IPMI_Power_Supply_No_Event Nominal # IPMI_Power_Supply_Presence_Detected Nominal # IPMI_Power_Supply_Power_Supply_Failure_Detected Critical # IPMI_Power_Supply_Predictive_Failure Critical # IPMI_Power_Supply_Power_Supply_Input_Lost_AC_DC Critical […] ## IPMI_Memory # # IPMI_Memory_No_Event Nominal # IPMI_Memory_Correctable_Memory_Error Warning # IPMI_Memory_Uncorrectable_Memory_Error Critical
- 25. slide 25/37 IPMI System Event Log (SEL) • stored in non-volatile storage [root@testserver ~]# ipmitool sel elist 40 | 06/21/2010 | 14:29:29 | Power Supply PS1 Status | Power Supply AC lost | Asserted 54 | 06/21/2010 | 14:29:29 | Power Unit Power Redundancy | Fully Redundant 68 | 06/21/2010 | 14:29:29 | Power Unit Power Redundancy | Redundancy Lost 7c | 06/21/2010 | 14:29:29 | Power Unit Power Redundancy | NonRedundant: Sufficient from Redundant [...] 2fc | 06/21/2010 | 15:20:32 | Physical Security Physical Scrty | General Chassis intrusion | Asserted [root@testserver ~]# ipmitool sel elist 40 | 06/21/2010 | 14:29:29 | Power Supply PS1 Status | Power Supply AC lost | Asserted 54 | 06/21/2010 | 14:29:29 | Power Unit Power Redundancy | Fully Redundant 68 | 06/21/2010 | 14:29:29 | Power Unit Power Redundancy | Redundancy Lost 7c | 06/21/2010 | 14:29:29 | Power Unit Power Redundancy | NonRedundant: Sufficient from Redundant [...] 2fc | 06/21/2010 | 15:20:32 | Physical Security Physical Scrty | General Chassis intrusion | Asserted [root@testserver ~]# ipmitool sel elist Power Supply PS1 Status | Power Supply AC lost | Asserted Power Unit Power Redundancy | Fully Redundant Power Unit Power Redundancy | Redundancy Lost Power Unit Power Redundancy | NonRedundant: Sufficient from Redundant [...] Physical Security Physical Scrty | General Chassis intrusion | Asserted [root@testserver ~]# ipmitool sel elist Power Supply PS1 Status | Power Supply AC lost | Asserted Power Unit Power Redundancy | Fully Redundant Power Unit Power Redundancy | Redundancy Lost Power Unit Power Redundancy | NonRedundant: Sufficient from Redundant [...] Physical Security Physical Scrty | General Chassis intrusion | Asserted
- 26. slide 26/37 Agenda 1) IPMI overview 2) Plugin implementation 3) Live demo 4) Common pitfalls
- 27. slide 27/37 Plugin implementation • Bash script • uses FreeIPMI, gawk # ./check_ipmi_sensor H 10.10.10.114 f /etc/ipmiconfig/ipmi.cfg IPMI Status: OK | 'System Temp'=29.000000 'FAN 1'=4185.000000 'FAN 2'=4320.000000 'FAN 3'=4590.000000 'FAN 4'=4320.000000 'FAN A'=4590.000000 'Vcore'=0.712000 '3.3VCC'=3.392000 '12V'=12.190000 'VDIMM'=1.528000 '5VCC'=5.088000 '12V'=11.681000 'VBAT'=3.024000 'VSB'=3.344000 'AVCC'=3.408000 # ./check_ipmi_sensor H 10.10.10.114 f /etc/ipmiconfig/ipmi.cfg IPMI Status: OK | 'System Temp'=29.000000 'FAN 1'=4185.000000 'FAN 2'=4320.000000 'FAN 3'=4590.000000 'FAN 4'=4320.000000 'FAN A'=4590.000000 'Vcore'=0.712000 '3.3VCC'=3.392000 '12V'=12.190000 'VDIMM'=1.528000 '5VCC'=5.088000 '12V'=11.681000 'VBAT'=3.024000 'VSB'=3.344000 'AVCC'=3.408000
- 28. slide 28/37 Plugin implementation • Bash script • uses FreeIPMI, gawk # ./check_ipmi_sensor H 10.10.10.114 f /etc/ipmiconfig/ipmi.cfg v 2 IPMI Status: OK | 'System Temp'=29.000000 'FAN 1'=4320.000000 'FAN […] System Temp = 29.000000 (Status: Nominal) CPU Temp = 'Low' (Status: Nominal) FAN 1 = 4320.000000 (Status: Nominal) FAN 2 = 4320.000000 (Status: Nominal) FAN 3 = 4590.000000 (Status: Nominal) […] AVCC = 3.408000 (Status: Nominal) Chassis Intru = 'OK' (Status: Nominal) PS Status = 'Presence detected' (Status: Nominal) # ./check_ipmi_sensor H 10.10.10.114 f /etc/ipmiconfig/ipmi.cfg v 2 IPMI Status: OK | 'System Temp'=29.000000 'FAN 1'=4320.000000 'FAN […] System Temp = 29.000000 (Status: Nominal) CPU Temp = 'Low' (Status: Nominal) FAN 1 = 4320.000000 (Status: Nominal) FAN 2 = 4320.000000 (Status: Nominal) FAN 3 = 4590.000000 (Status: Nominal) […] AVCC = 3.408000 (Status: Nominal) Chassis Intru = 'OK' (Status: Nominal) PS Status = 'Presence detected' (Status: Nominal)
- 29. slide 29/37 Plugin implementation • clear illustration in webinterfaces
- 30. slide 30/37 Agenda 1) IPMI overview 2) Plugin implementation 3) Live demo 4) Common pitfalls
- 31. slide 31/37 Agenda 1) IPMI overview 2) Plugin implementation 3) Live demo 4) Common pitfalls
- 32. slide 32/37 Common pitfalls • sensors with state N/A • solution shortest-term: exclude (-x opt.) • solution short-term: FreeIPMI update tkwiki.cc/FreeIPMI-NA-Sensor […] 12 | CPU1 Temp | OEM Reserved | N/A | N/A | N/A | 'OEM Event = 0000h' 13 | CPU2 Temp | OEM Reserved | N/A | N/A | N/A | 'OEM Event = 0000h' […] […] 12 | CPU1 Temp | OEM Reserved | N/A | N/A | N/A | 'OEM Event = 0000h' 13 | CPU2 Temp | OEM Reserved | N/A | N/A | N/A | 'OEM Event = 0000h' […]
- 33. slide 33/37 Common pitfalls • unrecognized events • solution shortest-term: ignore unrec. e. tkwiki.cc/FreeIPMI-Unrec-Event […] 40 | Status | Cable/Interconnect | Nominal | N/A | 'Cable/Interconnect is connected' 41 | RAC Status | Module/Board | N/A | N/A | 'Unrecognized Event = 0001h' 'Unrecognized Event = 0002h' 'Unrecognized Event = 0004h' 42 | OS Watchdog | Watchdog 2 | Nominal | N/A | 'OK' […] […] 40 | Status | Cable/Interconnect | Nominal | N/A | 'Cable/Interconnect is connected' 41 | RAC Status | Module/Board | N/A | N/A | 'Unrecognized Event = 0001h' 'Unrecognized Event = 0002h' 'Unrecognized Event = 0004h' 42 | OS Watchdog | Watchdog 2 | Nominal | N/A | 'OK' […]
- 34. slide 34/37 Agenda 1) IPMI overview 2) Plugin implementation 3) Live demo 4) Common pitfalls some conclusions …
- 35. slide 35/37 Conclusions (1/2) • Download: www.thomas-krenn.com/en/oss • Mailing List: lists.thomas-krenn.com • Thanks for your contribution: Nikolaus Filus, Timme Katz, Lars Meuser, Sebastian Mörchen, Gustav Olsson, Holger Paschke, Andy Spiegl, Ulrich Zehl
- 36. slide 36/37 Conclusions (2/2) Monitor hardware with Icinga & IPMI Problems? They will tell you! It'll save you time & money
- 37. Get German article on the plugin for free at tkwiki.cc/ipmi-plugin Thanks for your time!