![What if Linux Kernel Panics
● Kexec: system call to load and boot into another kernel from the currently
running kernel (4.9.74).
○ crashkernel=128M [normal kernel cmdline]
○ irqpoll, nosmp, reset_devices [crash kernel
cmdline]
○ --load-panic option
● Kdump: Linux mechanism to dump machine memory content on kernel
panic.
● Kexec/Kdump support on ARM platforms is still experimental](https://image.slidesharecdn.com/amp-kyneticselc2018portland-180313140527/85/Asymmetric-Multiprocessing-Kynetics-ELC-2018-portland-23-320.jpg)
Asymmetric Multiprocessing - Kynetics ELC 2018 portland
- 1. An Introduction to Asymmetric Multiprocessing: when this architecture can be a game changer and how to survive it. Nicola La Gloria, Laura Nao Kynetics LLC Santa Clara, California
- 2. Hybrid Architecture: warpx.io The Hybrid Design Architecture (HDA) combines the power of an application processor with the ease-of-use of micro-controllers.
- 3. Software on warpx HDA
- 4. SMP vs AMP SMP on homogeneous architectures: ● Single OS controlling two or more identical cores sharing system resources ● Dynamic scheduling and load balancing Core1 CoreN Kernel SMP OS App App . . .
- 5. SMP vs AMP Core1 CoreN MCAPI OS OS/RTOS App App Task Task AMP on heterogeneous architectures: ● Different OS on each core --> full-featured OS alongside a real-time kernel ● Inter processor communication protocol ● Efficient when the application can be statically partitioned across cores - high performance is achieved locally . . .
- 6. Supervised vs Not Supervised ● Strong isolation ● Hides non-trivial AMP details (e.g. resource assignment, inter-core communication) ● Security and robustness ● Overhead of a software layer ● Achieve best performances by running natively ● Boot sequence complexity ● Harder to debug
- 8. NXP i.MX7 overview ● Cortex-A7 core + Cortex-M4 core ● Master - Slave architecture ○ A7 is the master ○ M4 is the slave ● Inter processor communication ○ MU - Messaging Unit ○ RPMsg component (OpenAMP framework) ● Safe sharing of resources ○ RDC - Resource Domain Controller
- 9. NXP i.MX7 - RDC
- 10. NXP i.MX7 IPC - MU
- 11. MAC (VirtIO)
- 12. The OpenAMP framework - RPMsg
- 13. RPMsg on Linux
- 14. Hybrid Linux/FreeRTOS Demo Demo Goal: ● IMU sensor (I2C) read by MCU task ● Calculate objective function (module of acc, mag, gyro vectors) ● Log/plot sensor samples on MPU ● Safely recover from a kernel panic Hardware Setup ● Boundary Devices Nitrogen 7, Toradex Colibri i.MX7 SOM ○ NXP i.MX7D processor - ARM dual Cortex-A7 + ARM Cortex-M4 ○ Segger J-Link Probe
- 15. Cortex M4 Bring Up (1) Environment setup: ● Download FreeRTOS sources https://github.com/boundarydevices/freertos-boundary.git ● Download GNU ARM Embedded Toolchain https://developer.arm.com/open-source/gnu-toolchain/gnu-rm/download ● Example applications for Cortex-M4 are located in the examples/imx7d_nitrogen7_m4/ folder ● Scripts for building both debug and release binaries are available in the armgcc subfolder
- 16. Cortex M4 Bring Up (2) M4 Binary application can be loaded on the Cortex-M4 in different ways: ● U-Boot - ums gadget + m4update ● using remoteproc framework (linux userspace) ● using imx-m4fwloader from NXP (linux userspace): https://github.com/codeauroraforum/imx-m4fwloader M4 code can be linked and loaded to one of the following: ● TCM - 32KB (preferred) ● OCRAM - 32KB ● DDR - up to 1MB ● QSPI Flash - 128KB
- 17. IDE Setup ● Eclipse for C/C+ ○ GNU MCU Eclipse : plugins and tools for embedded ARM development - https://marketplace.eclipse.org/content/gnu-mcu-eclipse ● GDB ● J-Link scripts for iMX7D for debugging both Cortex-A7 cores and Cortex-M4 - https://wiki.segger.com/IMX7D ● FreeRTOS Kernel Awareness plugin from NXP http://freescale.com/lgfiles/updates/Eclipse/KDS ● ARM DS-5 (not free) ● Sourcery Codebench (not free)
- 19. Demo Parameters Remote core: ● Sample IMUs every 10ms ● Calculate the objective function on MCU (module of vectors) ● Buffer of 300 elements = 3Kb (stored TCM Memory only 32 Kb) ● Items (12 byte each) are dequeued and sent to master 10 at a time every 100 ms Master core: ● Master reads incoming samples by polling the character device
- 20. Architecture Overview start_cmd, stop_cmd, heartbeatLinux 4.9 FreeRTOS 1.0.1
- 21. The OpenAMP framework - RPMsg
- 22. Control Flow (2 cores) ● S0 RPMsg channel is down ● S1 RPMsg channel is up, /dev/rpmsg0 is created ● S2 RPMsg channel is up, endpoint created, data is dumped into a log file ● S0 RPMsg channel is down ● S1 RPMsg channel is up (sampling IMU sensor, buffering data) ● S2 RPMsg channel is up, sending data to master core, (sampling IMU sensor, buffering data), Register rpmsg char driver + probe open /dev/rpmsg0 read /dev/rpmsg0 Channel created stop_cmd Master is dead start_cmd Master heartbeat rpmsg_char_client Data sender task close /dev/rpmsg0
- 23. What if Linux Kernel Panics ● Kexec: system call to load and boot into another kernel from the currently running kernel (4.9.74). ○ crashkernel=128M [normal kernel cmdline] ○ irqpoll, nosmp, reset_devices [crash kernel cmdline] ○ --load-panic option ● Kdump: Linux mechanism to dump machine memory content on kernel panic. ● Kexec/Kdump support on ARM platforms is still experimental
- 24. Video of the Demo VIDEO
- 25. Pitfalls ● Before announcing the remote service, MCU checks whether master is up. If notification arrives too early (virtqueue kick function call) when booting crash kernel the system might hang ● Sometimes kexec still hangs and fails to soft-reboot - more frequent when streaming continuously instead of sending data bursts (but we don’t know why)
- 26. References ● OpenAMP project page: https://github.com/OpenAMP/ ● Asymmetric multiprocessing and embedded linux (ELC 2017): https://elinux.org/images/3/3b/NOVAK_CERVENKA.pdf ● Mantainers: ○ Open-amp: ■ Wendy Liang ○ RPMsg (Linux) ■ Ohad Ben-Cohen ■ Bjorn Andersson ○ Kexec (Linux) ■ Eric Biederman ○ Kdump (Linux) ■ Dave Young ■ Baoquan He
- 27. Q/A