Crash Dump Analysis 101

CRASH DUMP
ANALYSIS 101
JOHN S. HOWARD
JOHN.HOWARD@NEXENTA.COM

1 © Copyright Nexenta 2012

AGENDA

!  
Terminology
!  
Core Dumps and Crash Dumps
!  
C Language Basics
!  
The Mechanism of a Panic
! mdb Overview
!  
Basic Crash Dump Analysis


PROCESS, THREAD, LWP

! Process
 
!  A program in execution
!  May be comprised of threads or LWPs
!  Thread
!  The smallest unit of scheduling
!  Shared address space and resources
!  Light Weight Process (LWP)
!  A many-to-1 mapping of user threads to a kernel thread
!  Provides user-level multitasking


INTERRUPTS AND TRAPS

! I nterrupts are asynchronous messages notifying the kernel of
external device events
!  Some interrupts are handled as traps
!  Traps are synchronous messages, essentially a software
interrupt
!  Bus errors are issued to a processor when referencing a
location that can’t be resolved or located


HANGS, CRASHES, AND PANICS

! Hang
 
!  Potentially limited or no forensic information
!  System up, but unresponsive
!  Crash
!  Potentially limited forensic information
!  System down or rebooted
!  Panic
!  Maximum potential forensic information
!  System down or rebooted


FORENSIC INFORMATION SOURCES

! Forensic Information Sources
 
!  Console
!  syslog, typically logged to
/var/adm/messages
!  Core ﬁle or crash dump


CORE FILE

!  
A dump of the contents of all memory allocated to the
process
!  
Inert and static record of state
!  
Process core files are dumped to the working directory by
default
!  
Core file properties managed via coreadm
!  
Requires the same libraries to be read


CRASH DUMP

! A dump of the contents of all memory allocated to the kernel
 
!  Inert and static record of state
!  Written to the pre-specified dump device or swap partition
!  Written “backwards”
!  Reading requires the same OS version
!  Kernel core file facility managed via dumpadm


DUMPADM

! dumpadm with no options shows current settings
# dumpadm
!Dump content: kernel pages!
!Dump device: /dev/zvol/dsk/rpool/dump (dedicated)!
!Savecore directory: /var/crash/myhost!
!Savecore enabled: yes!
!  To force a crash dump:
# savecore -L

!  Note that savecore does not quiesce system, so memory contents
are changing
# uadmin 5 0

# reboot -dn

PANIC

! Kernel detected inconsistency
 
!  Protect by exiting
!  Three major tasks to be performed in a system panic:
!  record information about the panic in memory (making it
part of the crash dump)
!  synchronize the file systems to preserve user file data
!  generate the crash dump


C PROGRAMMING LANGUAGE DATATYPES

! Built-ins
 
! int, float,char
! struct
!  A grouping of data
!  union
!  variant records
!  All constituent data items are overlaid
! typedef
!  Pointers
!  A reference to a memory location


C DATATYPES EXAMPLES

int ap;!
char buf[128];!
int *user = sr;!
typedef struct smb_mtype {!
! !char! !*mt_name;!
! !int ! !mt_namelen;!
! !int ! !mt_flags;!
} smb_mtype_t


C FUNCTIONS

! Declaration
 
!  Definition
!  Parameters are pass by value


C FUNCTION EXAMPLES

Declaration
static void smb_tree_log(smb_request_t *, const char *, !
const char *, ...);!
Deﬁnition 
smb_tree_log(smb_request_t *sr, const char *sharename,!
const char *fmt, ...) 
{ 
. 
. 
. 
}!


PANIC()

! panic(),
  cmn_err()
!  Common entry points for vpanic()
!  Responsible for providing panic information
!  die()
! vpanic()
!  Assembly language function for saving register state
!  ASSERT(condition)
!  Halts execution of the kernel if condition is false
!  Evaluated and executed only when the DEBUG compilation
symbol is deﬁned
!  VERIFY(condition)
!  Similar to ASSERT, but active even when DEBUG isn’t deﬁned
!  Stack will contain assfail() near top


EXAMPLE 1: PANIC STRING

panic[cpu1]/thread=ffffff000e4e7c60:
BAD TRAP: type=e (#pf Page fault)
rp=ffffff000e4e77c0 addr=0 occurred in module
"unix" due to a NULL pointer dereference


EXAMPLE 1: STACK TRACE

ffffff000e4e76a0 unix:die+dd ()
ffffff000e4e77b0 unix:trap+177b ()
ffffff000e4e77c0 unix:cmntrap+e6 ()
ffffff000e4e78c0 unix:strcasecmp+16 ()
ffffff000e4e7a50 smbsrv:smb_tree_log+b3 ()
ffffff000e4e7a90 smbsrv:smb_tree_connect_core+14a ()
ffffff000e4e7ac0 smbsrv:smb_tree_connect+35 ()
ffffff000e4e7ae0 smbsrv:smb_com_tree_connect_andx+16 ()
ffffff000e4e7b80 smbsrv:smb_dispatch_request+4a9 ()
ffffff000e4e7bb0 smbsrv:smb_session_worker+6c ()
ffffff000e4e7c40 genunix:taskq_d_thread+b1 ()
ffffff000e4e7c50 unix:thread_start+8 ()


MDB – MODULAR DEBUGGER

! Extensible utility for low-level debugging and editing
 
!  On live kernel:
# mdb -k
# mdb -kw to edit (VERY
DANGEROUS)
!  On a core file:
mdb syseventd.core.125
!  On a crash dump:
# mdb -k unix.3 vmcore.3


ANALYZE-CRASH.SH

! Extracts the crash dump from the dump device
 
(savecore -vf filename) if necessary
!  Scripted mdb commands for basic crash information:
!  Panic string and registers
! dmesg buffer
!  Stack
!  Thread list
!  Executed automatically by the NMC `support` command
(NS 3.1.2 and later)


HAVE I SEEN THIS BEFORE?

! Footprints
 
!  Known problem or new?
! Redmine
!  Search illumos Hg issues
https://www.illumos.org/issues/
! SunSolve is gone, however “We Sun Solve” is rescuing
the data from SunSolve.Sun.COM
http://wesunsolve.net/bsearch
! illumos Source browser
http://src.illumos.org/source/



panic[cpu1]/thread=ffffff000e4e7c60:
BAD TRAP: type=e (#pf Page fault)
rp=ffffff000e4e77c0 addr=0 occurred in module
"unix" due to a NULL pointer dereference


EXAMPLE 1: STACK TRACE

ffffff000e4e76a0 unix:die+dd ()
ffffff000e4e77b0 unix:trap+177b ()
ffffff000e4e77c0 unix:cmntrap+e6 ()
ffffff000e4e78c0 unix:strcasecmp+16 ()
ffffff000e4e7a50 smbsrv:smb_tree_log+b3 ()
ffffff000e4e7a90 smbsrv:smb_tree_connect_core+14a ()
ffffff000e4e7ac0 smbsrv:smb_tree_connect+35 ()
ffffff000e4e7ae0 smbsrv:smb_com_tree_connect_andx+16 ()
ffffff000e4e7b80 smbsrv:smb_dispatch_request+4a9 ()
ffffff000e4e7bb0 smbsrv:smb_session_worker+6c ()
ffffff000e4e7c40 genunix:taskq_d_thread+b1 ()
ffffff000e4e7c50 unix:thread_start+8 ()


EXAMPLE 2: PANIC INFO

panic[cpu5]/thread=ffffff000fd72c60:
BAD TRAP: type=0 (#de Divide error) rp=ffffff000fd72a40 addr=ffffff02da92e900

sched:
#de Divide error
addr=0xffffff02da92e900
pid=0, pc=0xfffffffff7ad977b, sp=0xffffff000fd72b30, eflags=0x10246
cr0: 8005003b<pg,wp,ne,et,ts,mp,pe> cr4: 6f8<xmme,fxsr,pge,mce,pae,pse,de>
cr2: fffffd7fff2a60c8
cr3: 5000000
cr8: c
        rdi: ffffff02d282e840 rsi:                0 rdx:                0
        rcx:               64 r8: ffffff000fd72c60 r9:                0
        rax:                0 rbx:                0 rbp: ffffff000fd72b90
        r10:                0 r11: ffffff02f46e8264 r12: ffffff02da316338
        r13: ffffff02da3163d0 r14: ffffff02d5061a50 r15: ffffff02da92e900
        fsb:                0 gsb: ffffff02da9a1540 ds:               4b
         es:               4b fs:                0 gs:              1c3
        trp:                0 err:                0 rip: fffffffff7ad977b
         cs:               30 rfl:            10246 rsp: ffffff000fd72b30
         ss:               38


EXAMPLE 2: STACK

ffffff000fd72920 unix:die+10f ()
ffffff000fd72a30 unix:trap+1555 ()
ffffff000fd72a40 unix:cmntrap+e6 ()
ffffff000fd72b90 cpudrv:cpudrv_monitor+1cb ()
ffffff000fd72c40 genunix:taskq_thread+285 ()
ffffff000fd72c50 unix:thread_start+8 ()
syncing file systems...
done
dumping to /dev/zvol/dsk/syspool/dump, offset 65536, content: kernel + curproc

STACK
---
ffffff000fd72b90 cpudrv_monitor+0x1cb(ffffff02da316338)
ffffff000fd72c40 taskq_thread+0x285(ffffff02da859140)
ffffff000fd72c50 thread_start+8()


EXAMPLE 2: THREAD LIST

ffffff000fd72c60 fffffffffbc2dbf0                0   0 60                0
PC: panicsys+0x9b    TASKQ: cpudrv_cpudrv_monitor
stack pointer for thread ffffff000fd72c60: ffffff000fd726e0
    xc_insert+0x36()
    0xffffff0200000000()
    cpudrv_monitor+0x1cb()
    taskq_thread+0x285()
    thread_start+8()


EXAMPLE 2: SOURCE
CODE

From cpudrv_monitor()
1109      /*
   1110       * Adjust counts based on the delay added by timeout and taskq.
   1111       */
   1112      idle_cnt = (idle_cnt * cur_spd->quant_cnt) / tick_cnt;
   1113      user_cnt = (user_cnt * cur_spd->quant_cnt) / tick_cnt;
   1114


HARDWARE, FIRMWARE, OR SOFTWARE?

!  
Crash dumps are inconclusive on hardware errors
!  
Correlate to fmdump output
!  
PCI-X panics are the most common hardware caused panic
!  
PCI Vendor Database http://pcidatabase.com
!  
KB Article: “Understanding and decoding PCI(-X) Express
Fatal Error panics”


AND STACK TRACE

panic[cpu7]/thread=ffffff005cbdbc60:
pcieb-3: PCI(-X) Express Fatal Error. (0x101)

ffffff005cbdbbb0 pcieb:pcieb_intr_handler+228 ()
ffffff005cbdbc00 unix:av_dispatch_autovect+7c ()
ffffff005cbdbc40 unix:dispatch_hardint+33 ()
ffffff005cbaba80 unix:switch_sp_and_call+13 ()
ffffff005cbabad0 unix:do_interrupt+b8 ()
ffffff005cbabae0 unix:_interrupt+b8 ()
ffffff005cbabbd0 unix:i86_mwait+d ()
ffffff005cbabc20 unix:cpu_idle_mwait+f1 ()
ffffff005cbabc40 unix:idle+114 ()
ffffff005cbabc50 unix:thread_start+8 ()


IDENTIFYING THE PCI-X
COMPONENT

Mar 30 2011 00:53:53.606674454 ereport.io.pci.fabric
nvlist version: 0
class = ereport.io.pci.fabric
ena = 0xbcd565541a801401
detector = (embedded nvlist)
nvlist version: 0
version = 0x0
scheme = dev
device-path = /pci@0,0/pci8086,3408@1
(end detector)

bdf = 0x8
device_id = 0x3408
vendor_id = 0x8086


IDENTIFYING THE VENDOR

Device ID Chip Description Vendor ID Vendor Name
0x3408 Intel 7500 Chipset PCIe Root Port 0x8086 Intel Corporation

device-path = /pci@0,0/pci8086,3408@1
device-path = /pci@0,0/pci8086,3408@1/pci108e,484c@0
device-path = /pci@0,0/pci8086,3408@1/pci108e,484c@0,1

If no entries in neither the PCI vendor database nor
`/usr/share/hwdata/pci.ids` then grep
`/etc/path_to_inst`:

"/pci@0,0/pci8086,3408@1" 0 "pcie_pci"
"/pci@0,0/pci8086,3408@1/pci108e,484c@0" 0 "igb"
"/pci@0,0/pci8086,3408@1/pci108e,484c@0,1" 1 "igb“
igb is the intel Gigabit NIC driver

DETERMINE DRIVER AND
PACKAGE DETAILS

# dpkg -S igb | grep '/kernel’
sunwigb: /var/lib/dpkg/alien/sunwigb/reloc/kernel/drv/igb.conf
sunwigb: /kernel/drv/amd64/igb
sunwigb: /var/lib/dpkg/alien/sunwigb/reloc/kernel/drv
sunwigb: /kernel/drv/igb
sunwigb: /var/lib/dpkg/alien/sunwigb/reloc/kernel

Examine the package details:

# dpkg -l sunwigb
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Installed/Config-f/Unpacked/Failed-cfg/Half-inst/t-aWait/T-pend
|/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad)
||/ Name Version Description
+++-=======================-======================-======================================
ii sunwigb 5.11.134-31-8234-1 Intel 82575 1Gb PCI Express NIC
Driver


A PCI-X CONCLUSION, OF SORTS

!  Searching redmine for “igb driver” will find a bug, but also
check for any Intel 82575 gigabit issues
!  Next, determine:
!  Is the driver is down revision?
!  Is the firmware is down revision?
!  If the driver and firmware are current, then this is most likely
a hardware problem
!  CDA is inconclusive for proving hardware failures


Crash Dump Analysis 101

More Related Content

What's hot (20)

Similar to Crash Dump Analysis 101 (20)

Recently uploaded (20)

Crash Dump Analysis 101