![Restricted © 2017 Mentor Graphics Corporation
Array Bound Violations
No checking in C
— Considered a runtime overhead
— Could add in C++
– Overload the [ ] operator
— Pointers can get around bounding
Using guard words makes sense
— Most programming errors result in writing
to memory immediately adjacent to the
end of the array](https://image.slidesharecdn.com/4-colinwalls-self-testinginembeddedsystems-180426152723/85/4-colin-walls-self-testing-in-embedded-systems-25-320.jpg)
4 colin walls - self-testing in embedded systems
- 1. Colin Walls Self-Testing in Embedded Systems colin_walls@mentor.com http://blogs.mentor.com/colinwalls
- 2. Restricted © 2017 Mentor Graphics Corporation Agenda Introduction CPU Failure Peripheral Failure Memory Failure Software Error Conditions Failure Recovery and Reporting Conclusions
- 3. Restricted © 2017 Mentor Graphics Corporation Agenda Introduction CPU Failure Peripheral Failure Memory Failure Software Error Conditions Failure Recovery and Reporting Conclusions
- 4. Restricted © 2017 Mentor Graphics Corporation Introduction Failure is almost inevitable Important to accept this fact Key issues: — How to reduce likelihood of failure — How to handle impending failure — How to recover from failure conditions
- 5. Restricted © 2017 Mentor Graphics Corporation Failure and Testing What can fail in an embedded system? — CPU — Peripherals — Memory — Software Start-up testing Background testing Watchdog
- 6. Restricted © 2017 Mentor Graphics Corporation Agenda Introduction CPU Failure Peripheral Failure Memory Failure Software Error Conditions Failure Recovery and Reporting Conclusions
- 7. Restricted © 2017 Mentor Graphics Corporation CPU Failure Quite unlikely for just processor to fail Possibly no hope of recovery Failure most likely at power up Partial failure very rare Multicore designs offer options
- 8. Restricted © 2017 Mentor Graphics Corporation CPU Failure Quite unlikely for just processor to fail Possibly no hope of recovery Failure most likely at power up Partial failure very rare Multicore designs offer options
- 9. Restricted © 2017 Mentor Graphics Corporation Agenda Introduction CPU Failure Peripheral Failure Memory Failure Software Error Conditions Failure Recovery and Reporting Conclusions
- 10. Restricted © 2017 Mentor Graphics Corporation Peripheral Failure Total failure: not responding to address — Trap handler Other failures/tests are device dependent Loop back testing is common
- 11. Restricted © 2017 Mentor Graphics Corporation Agenda Introduction CPU Failure Peripheral Failure Memory Failure Software Error Conditions Failure Recovery and Reporting Conclusions
- 12. Restricted © 2017 Mentor Graphics Corporation Memory Failure Systems have a lot of memory — Surprising that failure is not more common Failure most likely at power up — That is the time for comprehensive testing 3 failure modes: — Not responding — Stuck bits — Cross-talk
- 13. Restricted © 2017 Mentor Graphics Corporation Memory Testing “Moving Ones” test — Looks for stuck bits and cross talk with fine grain resolution — Perform on start up — Code to only use registers — “Moving Zeros” is the same idea Pattern test — Also looks for stuck bits and cross talk — Perform in background task All tests can be optimized if the memory architecture is known
- 14. Restricted © 2017 Mentor Graphics Corporation Moving Ones Test set every bit of memory to 0 for each bit of memory { verify that all bits are 0 set the bit under test to 1 verify that it is 1 verify all other bits are 0 set the bit under test to 0 }
- 15. Restricted © 2017 Mentor Graphics Corporation Memory Testing “Moving Ones” test — Looks for stuck bits and cross talk with fine grain resolution — Perform on start up — Code to only use registers — “Moving Zeros” is the same idea Pattern test — Also looks for stuck bits and cross talk — Perform in background task All tests can be optimized if the memory architecture is known
- 16. Restricted © 2017 Mentor Graphics Corporation Pattern Test for each byte of memory { turn off interrupts save memory byte contents for values 0x00, 0xff, 0xaa, 0x55 { write value to byte under test verify value of byte } restore byte data turn on interrupts }
- 17. Restricted © 2017 Mentor Graphics Corporation Memory Testing “Moving Ones” test — Looks for stuck bits and cross talk with fine grain resolution — Perform on start up — Code to only use registers — “Moving Zeros” is the same idea Pattern test — Also looks for stuck bits and cross talk — Perform in background task All tests can be optimized if the memory architecture is known
- 18. Restricted © 2017 Mentor Graphics Corporation Agenda Introduction CPU Failure Peripheral Failure Memory Failure Software Error Conditions Failure Recovery and Reporting Conclusions
- 19. Restricted © 2017 Mentor Graphics Corporation Software Error Conditions Bugs can lead to unpredictable failure Defensive code can anticipate problems 2 key failure modes: — Data corruption — Code looping
- 20. Restricted © 2017 Mentor Graphics Corporation Data Corruption Pointers! Null pointer — Trap handler Incorrect pointer — May point anywhere — Leads to random corruption — MMU may help — Special cases – Stack overflow/underflow – Array bound violations
- 21. Restricted © 2017 Mentor Graphics Corporation Stack Overflow/Underflow Avoid by careful testing — Use memory access breakpoints during debug — Unexpected recursion depth is hard to predict Use guard words — Test periodically – Background task? — Choose odd value to avoid addresses — Use unlikely value – Not 0, 1, 0xffffffff — 4 billion to 1 chance of a false alarm — Might use MMU
- 22. Restricted © 2017 Mentor Graphics Corporation Stack Overflow/Underflow 0x16951695 0x16951695 Avoid by careful testing — Use memory access breakpoints during debug — Unexpected recursion depth is hard to predict Use guard words — Test periodically – Background task? — Choose odd value to avoid addresses — Use unlikely value – Not 0, 1, 0xffffffff — 4 billion to 1 chance of a false alarm — Might use MMU
- 23. Restricted © 2017 Mentor Graphics Corporation Stack Overflow/Underflow 0x00000000 0xffffffff 0x00000000 0x00000000 0x99000000 0x00000000 0x00000000 0x00000000 0x12341234 0x16951695 0x16951695 Avoid by careful testing — Use memory access breakpoints during debug — Unexpected recursion depth is hard to predict Use guard words — Test periodically – Background task? — Choose odd value to avoid addresses — Use unlikely value – Not 0, 1, 0xffffffff — 4 billion to 1 chance of a false alarm — Might use MMU
- 24. Restricted © 2017 Mentor Graphics Corporation Stack Overflow/Underflow 0x16951695 0x16951695 0x77770000 0x00000000 0xffffffff 0x00000000 0x00000000 0x99000000 0x00000000 0x00000000 0x00000000 0x12341234 Avoid by careful testing — Use memory access breakpoints during debug — Unexpected recursion depth is hard to predict Use guard words — Test periodically – Background task? — Choose odd value to avoid addresses — Use unlikely value – Not 0, 1, 0xffffffff — 4 billion to 1 chance of a false alarm — Might use MMU
- 25. Restricted © 2017 Mentor Graphics Corporation Array Bound Violations No checking in C — Considered a runtime overhead — Could add in C++ – Overload the [ ] operator — Pointers can get around bounding Using guard words makes sense — Most programming errors result in writing to memory immediately adjacent to the end of the array
- 26. Restricted © 2017 Mentor Graphics Corporation Code Looping Infinite loops should not occur May be a programming error May be a device failing to respond — Software should have timeout Hardware watchdog helpful In multi-threaded environment, use a watchdog task — Event flag for each task — Watchdog sets flags to 1 and goes to sleep — Other tasks periodically set their flag to 0 — When watchdog wakes, any 1 flags result in alarm
- 27. Restricted © 2017 Mentor Graphics Corporation Using a Watchdog Task Task 1 Task 2 Task 3 Watchdog Task 1 1 1
- 28. Restricted © 2017 Mentor Graphics Corporation Using a Watchdog Task Task 1 Task 2 Task 3 Watchdog Task 1 1 10 0
- 29. Restricted © 2017 Mentor Graphics Corporation Using a Watchdog Task Task 1 Task 2 Task 3 Watchdog Task 1 1 10 0 Watchdog Task
- 30. Restricted © 2017 Mentor Graphics Corporation Agenda Introduction CPU Failure Peripheral Failure Memory Failure Software Error Conditions Failure Recovery and Reporting Conclusions
- 31. Restricted © 2017 Mentor Graphics Corporation Failure Reporting and Recovery Action on finding a fault is very system dependent If there is a user interface, an alarm may be activated A deeply embedded system may have no option other than a reset Example: heart pacemaker
- 32. Restricted © 2017 Mentor Graphics Corporation Sounding the Alarm Display — Text/graphics Sound Network — Send email — Web page LEDs — On/Off — Color — Flashing
- 33. Restricted © 2017 Mentor Graphics Corporation Flashing LEDs Speed — Slow – “heartbeat” — Fast – “error” Duty cycle — Morse code? LONG = 500 SHORT = 50 flash_delay = LONG LED_state = 0 loop-forever { flags = 0xff sleep(flash_delay) set_LED(LED_state) if LED_state = 0 LED_state = 1 else LED_state = 0 if flags <> 0 flash_delay = SHORT }
- 34. Restricted © 2017 Mentor Graphics Corporation Reset Automatic reset is sometimes the only option — Deeply embedded systems with no UI — Maybe log for later reference User input — Reset button may be reassuring — Multi-key sequences not intuitive or reliable
- 35. Restricted © 2017 Mentor Graphics Corporation Agenda Introduction CPU Failure Peripheral Failure Memory Failure Software Error Conditions Failure Recovery and Reporting Conclusions
- 36. Restricted © 2017 Mentor Graphics Corporation Conclusions First rule is accept that failure is possible Consider all possible failure modes Add code to monitor system “health” Consider action on failure — Warn — Fix