TSN apps running on VxWorks RTOS as a KVM guest at the intelligent edge
0 likes499 views
The document discusses the implementation of applications on VxWorks RTOS within a KVM guest environment for intelligent edge computing, emphasizing the separation of control from sensing and actuation. It details the configuration of Time-Sensitive Networking (TSN), Linux, and VxWorks, and presents various demo configurations for real-time control and virtualization. The conclusion notes the importance of determinism, real-time capabilities, and proper system configuration for successful edge deployments using VxWorks.
![2
0
2
1
W
I
N
D
R
I
V
E
R
,
A
L
L
R
I
G
H
T
S
R
E
S
E
R
V
E
D
-> ptpdAdd "gei0", 2, 2, 1, 0, 254, 0, 1
-> ptpdStart
-> ptpdStatusShow "gei0"
Interface name : gei0
Now in state : PTP_SLAVE
Demo Configuration: PTP
QEMU VM
Core 0
OpenIL Linux RT KVM
PTP slave
VxWorks
Unikernel
Core 1 Core 2 Core 3
LS1046ARDB
i210
Core 0
PTP master
OpenIL Linux
Core 1
LS1028ARDB
eno0
PTP
https://github.com/open62541/open62541/tree/master/examples/pubsub_realtime/vxworks
Edge Node
PCI passthrough
192.168.10.46 192.168.10.28
Edge Device
sudo daemonize -E BUILD_ID=dontKillMe -o /var/log/ptp4l.log -e
/var/log/ptp4l.err.log /usr/bin/taskset -c 1 chrt 90 /usr/sbin/ptp4l -i eno0 -2
-mq -f /etc/ptp4l_cfg/gPTP.cfg --step_threshold=1 --fault_reset_interval=0
--announceReceiptTimeout=10 --transportSpecific=1
root@LS1028ARDB-Ubuntu:~# tail -f /var/log/ptp.log
ptp4l[186906.248]: selected local clock 5ecf89.fffe.ebf722 as best master
ptp4l[186906.248]: port 1: assuming the grand master role](https://image.slidesharecdn.com/tsnappsrunningonvxworksrtosasakvmguestattheintelligentedge-210525103336/85/TSN-apps-running-on-VxWorks-RTOS-as-a-KVM-guest-at-the-intelligent-edge-11-320.jpg)
TSN apps running on VxWorks RTOS as a KVM guest at the intelligent edge
- 1. 2 0 2 1 W I N D R I V E R , A L L R I G H T S R E S E R V E D Andrei Kholodnyi https://github.com/razr https://discourse.ros.org/u/razr https://www.linkedin.com/in/akholodnyi/ T e c h n o l o g y O f f i c e TSN apps running on VxWorks RTOS as a KVM guest at the intelligent edge
- 2. Agenda 2 0 2 1 W I N D R I V E R , A L L R I G H T S R E S E R V E D Why a separation of control from sensor and actuation functionality is important for the intelligent edge solution 1 Why TSN and RTOS are essential parts of the intelligent edge story. 2 How to properly configure TSN, Linux and VxWorks KVM guest to run a OPC/UA TSN example 3
- 3. 2 0 2 1 W I N D R I V E R , A L L R I G H T S R E S E R V E D EDGE – WHY DO WE NEED TSN AND REAL-TIME? Edge Cloud Devices Customer premises Access network ~2 ms <5 ms 1-3 ms Edge compute ~5-20 ms 0-25 km 25-100 km Centralized Data Center 100s-1000s km
- 4. 2 0 2 1 W I N D R I V E R , A L L R I G H T S R E S E R V E D Edge use case : decoupling control from sensing and actuation VM Real-time Control VxWorks OPC/UA Sensing Control Actuation Edge Real-time Control VxWorks TSN OPC/UA VM Linux OPC/UA ML MW Edge Virtualization VxWorks OPC/UA Sensing Actuation TSN Control ML Apps Real-time device TSN
- 5. 2 0 2 1 W I N D R I V E R , A L L R I G H T S R E S E R V E D Demo Configuration: Edge real time control QEMU VM Core 0 OpenIL Linux RT KVM OPC/UA TSN VxWorks Unikernel PubSub Tx Core 1 Core 2 Core 3 LS1046ARDB i210 Core 0 OPC/UA TSN OpenIL Linux PubSub Rx Core 1 LS1028ARDB eno0 TSN Linux Apps https://github.com/open62541/open62541/tree/master/examples/pubsub_realtime/vxworks Edge Node Edge Device PCI passthrough 192.168.10.46 192.168.10.28
- 6. 2 0 2 1 W I N D R I V E R , A L L R I G H T S R E S E R V E D • Virtualization • Linux host • QEMU • Real-time guest OS • PCI pass-through • i210 card passed through to the guest DEMO – Virtualization QEMU VM Core 0 OpenIL Linux RT KVM OPC/UA TSN VxWorks Unikernel PubSub Tx Core 1 Core 2 Core 3 LS1046ARDB i210 Linux Apps Edge Node dev=0001:01:00.0 echo $dev > /sys/bus/pci/devices/$dev/driver/unbind echo vfio-pci > /sys/bus/pci/devices/$dev/driver_override echo $dev > /sys/bus/pci/drivers/vfio-pci/bind ~# lspci -k 0001:01:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) Subsystem: Intel Corporation Ethernet Server Adapter I210-T1 Kernel driver in use: vfio-pci
- 7. 2 0 2 1 W I N D R I V E R , A L L R I G H T S R E S E R V E D • VxWorks kernel library • librsl.a • UP mode • Real-time kernel application • tsnperf DEMO – Real-time unikernel Core 0 OPC/UA TSN VxWorks Unikernel PubSub Tx Core 1 Core 2 Core 3 LS1046ARDB i210 Linux Apps Edge Node $ cat ~/stash/projects/TO/rsl_apps/tsnperf/Makefile APP = tsnPerf all: $(APP) $(APP): $(APP).o main.o Makefile $(RSL_SDK)/krnl/librsl.a rsl build $(APP).o main.o -o $(APP) What is VxWorks? • it is a proprietary hard real-time OS, POSIX PSE52 • 64 bits on ARM, Intel, MIPS, PowerPC, and RISC-V • Kernel and user space separation, user space optional • C/C++17, possible to develop kernel C++ modules and user apps, and recently Rust and Python • Safety certifiable: DO-178, ISO 26262, IEC 61508 • Toolchain LLVM 11, Dinkumware C/C++ libs • Proprietary build system. CMake, autotools support for apps, e.g. ROS2 • Kernel shell • Eclipse, and Visual Studio Code - modern IDEs, Windows/Linux hosts
- 8. 2 0 2 1 W I N D R I V E R , A L L R I G H T S R E S E R V E D ▪ Isolate CPU 2 and 3 to run QEMU rcu_nocbs=2,3 nohz_full=2,3 isolcpus=2,3 irqaffinity=0,1 audit=0 watchdog=0 skew_tick=1 ▪ No CPU frequency scaling ▪ Other settings DEMO – RT_PREEMPT QEMU VM Core 0 OpenIL Linux RT KVM OPC/UA TSN VxWorks Unikernel PubSub Tx Core 1 Core 2 Core 3 LS1046ARDB i210 Linux Apps Edge Node CONFIG_NO_HZ_FULL=y CONFIG_HIGH_RES_TIMERS=y CONFIG_PREEMPT_RT=y CONFIG_HZ_1000=y # CONFIG_CPU_FREQ is not set
- 9. 2 0 2 1 W I N D R I V E R , A L L R I G H T S R E S E R V E D ▪ taskset -c 2 /* run on CPU2 */ ▪ -enable-kvm /* no emulation */ ▪ -smp 1 /* one CPU only */ ▪ -nic none /* no nic */ ▪ -name debug-threads=on /* name threads */ ▪ -device vfio-pci,host=$dev /* passthrough */ DEMO – QEMU QEMU VM Core 0 OpenIL Linux RT KVM OPC/UA TSN VxWorks Unikernel PubSub Tx Core 1 Core 2 Core 3 LS1046ARDB i210 Edge Node taskset -c 2 /work/qemu/installArm/bin/qemu-system-aarch64 -cpu host -enable-kvm -smp 1 -name debug-threads=on -nic none -machine type=virt,gic-version=host -nographic -m 256 -overcommit mem- lock=on -append "fs(0,0)host:vxWorks h=192.168.10.28 e=192.168.10.46 u=user pw=invalid f=0x01 o=gei0" -kernel /root/tsnPerf -device vfio-pci,host=$dev
- 10. 2 0 2 1 W I N D R I V E R , A L L R I G H T S R E S E R V E D ▪ Move vCore process to the CPU 3 ▪ Let other QEMU threads on the CPU 2 DEMO – pin vCore to pCore QEMU VM Core 0 OpenIL Linux RT KVM OPC/UA TSN VxWorks Unikernel PubSub Tx Core 1 Core 2 Core 3 LS1046ARDB i210 Edge Node QEMU_PID=`pidof qemu-system-aarch64` user@LS1046ARDB-Ubuntu:~$ pstree -t $QEMU_PID qemu-system-aar─┬─{CPU 0/KVM} ├─{qemu-system-aar} └─{signalfd_compat} VCPU_PID=`ps -aL |grep -E "(kvm-vcpu|CPU.*KVM)"|awk '{print $2}’ VCPU_CORE=3 taskset -c -p $VCPU_CORE $VCPU_PID QEMU thread #1 QEMU thread #n
- 11. 2 0 2 1 W I N D R I V E R , A L L R I G H T S R E S E R V E D -> ptpdAdd "gei0", 2, 2, 1, 0, 254, 0, 1 -> ptpdStart -> ptpdStatusShow "gei0" Interface name : gei0 Now in state : PTP_SLAVE Demo Configuration: PTP QEMU VM Core 0 OpenIL Linux RT KVM PTP slave VxWorks Unikernel Core 1 Core 2 Core 3 LS1046ARDB i210 Core 0 PTP master OpenIL Linux Core 1 LS1028ARDB eno0 PTP https://github.com/open62541/open62541/tree/master/examples/pubsub_realtime/vxworks Edge Node PCI passthrough 192.168.10.46 192.168.10.28 Edge Device sudo daemonize -E BUILD_ID=dontKillMe -o /var/log/ptp4l.log -e /var/log/ptp4l.err.log /usr/bin/taskset -c 1 chrt 90 /usr/sbin/ptp4l -i eno0 -2 -mq -f /etc/ptp4l_cfg/gPTP.cfg --step_threshold=1 --fault_reset_interval=0 --announceReceiptTimeout=10 --transportSpecific=1 root@LS1028ARDB-Ubuntu:~# tail -f /var/log/ptp.log ptp4l[186906.248]: selected local clock 5ecf89.fffe.ebf722 as best master ptp4l[186906.248]: port 1: assuming the grand master role
- 12. 2 0 2 1 W I N D R I V E R , A L L R I G H T S R E S E R V E D Demo Configuration: TSN (finally) QEMU VM Core 0 OpenIL Linux RT KVM OPC/UA TSN VxWorks Unikernel PubSub Tx Core 1 Core 2 Core 3 LS1046ARDB i210 Core 0 OPC/UA TSN OpenIL Linux PubSub Rx Core 1 LS1028ARDB eno0 TSN https://github.com/open62541/open62541/tree/master/examples/pubsub_realtime/vxworks Edge Node Edge Device PCI passthrough 192.168.10.46 192.168.10.28 QEMU thread #1 QEMU thread #n -> tsnConfig("gei", 4, 0, 0, "/romfs/62541.json") value = 0 = 0x0 -> open62541PubTSNStart("gei", 4, 0, 0, 1) ./tsnPerfLxRx -o "01:00:5E:00:00:01" -I "eno0" -v 3000 -n 1 2>&1 | tee tsnPerfLxRx.log
- 14. 2 0 2 1 W I N D R I V E R , A L L R I G H T S R E S E R V E D ▪ The Edge needs determinism, TSN, WTSN, and 5G ▪ The Edge requires real-time and virtualization ▪ A careful configuration of the edge system including TSN is needed to fulfill real-time requirements ▪ VxWorks is a good candidate as a guest OS at the edge, providing necessary real-time features, unikernel, and TSN protocols support DEMO – Conclusion VM Edge Real-time Control VxWorks TSN OPC/UA VM Linux OPC/UA ML MW Edge Virtualization VxWorks OPC/UA Sensing Actuation TSN Control ML Apps Real-time device TSN