Concurrency bug identification through kernel panic log (english)
Download as PPT, PDF0 likes605 views
The document outlines the analysis of a kernel panic log to identify a concurrency bug in a driver. The analysis included: 1) Log analysis found a buggy memory write, 2) Dump file analysis traced execution to this write, 3) Further log analysis found an interrupted context was involved, suggesting a concurrency bug. Tracing ARM instructions in the dump file around the interruption point helped identify that a variable was being incorrectly accessed.
![Experiment environment• Kernel panic scenario:
– Unable to handle kernel paging request at virtual address [bdf046ab]
• Environment:
– Hardware : Tiny6410
– Kernel : 2.6.36
– Driver version : before the ping failed bug fix(for panda)
• CheckResponseTimeout() in driver run per 1ms
– CM : query thread run ioctl per 1ms
• Analysis tool:
– Source code
– Error log
– Objdump
• ex: objdump wimaxsdio.ko -S
2](https://image.slidesharecdn.com/8-150106113332-conversion-gate01/85/Concurrency-bug-identification-through-kernel-panic-log-english-2-320.jpg)
Concurrency bug identification through kernel panic log (english)
- 1. Concurrency bug indentification through kernel panic log - Outline • Experiment • Log analysis and dump file tracing • Fault identify • Log analysis and dump file tracing – part 2 • Tracing ARM instruction • Conclusion 1
- 2. Experiment environment• Kernel panic scenario: – Unable to handle kernel paging request at virtual address [bdf046ab] • Environment: – Hardware : Tiny6410 – Kernel : 2.6.36 – Driver version : before the ping failed bug fix(for panda) • CheckResponseTimeout() in driver run per 1ms – CM : query thread run ioctl per 1ms • Analysis tool: – Source code – Error log – Objdump • ex: objdump wimaxsdio.ko -S 2
- 3. Log analysis • Error log – part1 – Buggy memory write at CheckResponseTimeout at 0x78/0x19c 3 This is our suspect.
- 4. Dump analysis • What happens at that memory write – We can trace our dump file: – 0x303c(start of CheckResponseTimeout) + 0x78 = 30B4 4
- 5. Fault identify • It seems error happens at “reading the pLocalDeviceData->status variable”, but why? • that does not happen all the time!!! – It seems not an usual normal software bug. – It happens just for one time among thousands experiment runs! (shall we take this bug as a concurrency bug) • Luckily we still have other information in error log. 5
- 6. Log analysis part 2• If we are considering a concurrency bug, those followed additional log would just be the perfect clue. – Because it really shows the interrupted context “wimax_rx_thread” get involved. 6
- 7. Dump analysis part 2 • 0x37c0(start of ReceiveResponse)+0x390(offset indicated by log) = 0x4000. • 0x4000 is the point interrupted. 7
- 8. Trace ARM instruction• Lsr : logical shift right • Strb: store 1 byte to memory address • So 0xfff3’s will be the graph in bottom right (pLddbChain->head) 8 Low memory Low memory a b c dR4 register F F F 3R3 register R1 register Low bitsHigh bits d c b a 0xfff4 0xfff5 0xfff6 0xfff3 abc ab a 1 23 4 1 2 3 4 5 6 7 5 6 7
- 9. Trace ARM instruction• Lsr : logical shift right • Strb: store 1 byte to memory address • Program is interrupted just after lsr r1,r4,#16 • So 0xfff3’s will be the graph in bottom right (pLddbChain->head) 9 Low memory Low memory a b c dR4 register F F F 3R3 register R1 register Low bitsHigh bits undefine c undefine undefine 0xfff4 0xfff5 0xfff6 0xfff3 abc ab 1 23 1 2 3
- 10. Conclusion • It’s an option that identify symptom is caused by concurrency bug, when symptom doesn’t happen all the time. • Don’t get panic when driver panic. – Calm down and Collecting enough log information as much as possible, maybe next time we won’t be as lucky as we are now. • Exploring objdump tool to get advanced debug information. • Exploring kernel config option to get advanced debug information . 10