SlideShare a Scribd company logo

Making Linux do Hard Real-time

National Cheng Kung University, profile picture
National Cheng Kung University

The document discusses making Linux capable of hard real-time performance, explaining concepts such as the differences between hard and soft real-time systems, the limitations of standard Linux scheduling and interrupt handling, and introduces extensions like RT-Preempt and Xenomai. It emphasizes the need for deterministic response times in critical applications, such as aviation and medical devices, and explores the improvements introduced in Linux 2.6, including priority inheritance and threaded interrupts. The overall aim is to enhance latency management and ensure predictable execution in real-time environments.

Technology
1 of 136
Downloaded 2,178 times
1
2
3
4
5
6
Most read
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
Most read
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
Most read
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
Making Linux do Hard Real-time Jim Huang ( 黃敬群 ) <jserv.tw@gmail.com> Jul 26, 2014
Rights to copy Attribution – ShareAlike 3.0 You are free to copy, distribute, display, and perform the work to make derivative works to make commercial use of the work Under the following conditions Attribution. You must give the original author credit. Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one. For any reuse or distribution, you must make clear to others the license terms of this work. Any of these conditions can be waived if you get permission from the copyright holder. Your fair use and other rights are in no way affected by the above. License text: http://creativecommons.org/licenses/by-sa/3.0/legalcode Based on a work of Free Electrons © Copyright 2004-2009 © Copyright 2014 Jim Huang © Copyright 2004-2010 Free Electrons
Agenda General Concepts Linux scheduler Interrupt Handling Latency in Linux Real-time Features in Linux RT-Preempt Linux Realtime Extension: Xenomai
Making Linux do Hard Real-time General Concepts
Real Time Real Time != Real Fast is determinism RTOS will deliver decent overall throughput (performance) but can sacrifice throughput for being deterministic (or predictable) Hard real-time Missing a deadline is a total system failure Soft real-time The usefulness of a result degrades after its deadline, thereby degrading the system's quality of service
Hard Real Time A system is considered as a hard real time if it can answer to an internal or external stimulus within a given maximum amount of time. “Guaranteed worst case” Hard real time systems are used wherever failing to react in time can cause a system failure or damage, or put its users in danger. Typical examples: air traffic control vehicle subsystems control Medicine (pacemakers, etc.)
Soft Real Time A system is considered as soft real time if it is built to react to stimuli as quickly as it can: “best effort”. However, if the system loses events or fails to process them in time, there is no catastrophic consequence on its operation. There is just a degradation in quality. Typical examples Voice over IP multimedia streaming computer games
Standard Linux and Real Time The vanilla Linux kernel was not designed as a hard real time system: Like Unix, it is a time sharing operating system designed to maximize throughput and give a fair share of the CPU in a multi-user environment. scheduler avoids process starvation Non deterministic timing behavior of some kernel services: memory allocation, system calls... By default, processes are not preemptible when they execute system calls.
Throughput vs. Determinism Linux as optimized for throughput rather than determinism. Linux CPUs typically utilize large, multi-layer memory caches Caches make CPUs run like a hare but, in real-time systems, the tortoise wins! CPU memory caching prevents Hard RT processes from safely utilizing more than a small fraction of the available time.
Realtime Performance (Standard) LinuxReal-time LinuxCustomized Realtime kernel One (physical) address space with n-tasks Application program 100% source compatibility better binary compatibility API Response time: < 10 µs Response time: ~1ms Response time: < 10 µs same homogeneity One process address space with n-threads All tasks have one common API implemented by one common library All threads of a process have one common API implemented by one common library
Jitter It means the scatter of the control period of all the tasks defined in the system The total jitter is the time difference between the minimal response time and the maximal response time
Real-time Approaches Two major approaches real time Linux rt-preempt (PREEMPT_RT patch) Allows preemption, so minimize latencies Execute all activities (including IRQ) in “schedulable/thread” context Many of the RT patch have been merged Linux (realtime) extensions Add extra layer between hardware and the Linux kernel to manage real-time tasks separately
Making Linux do Hard Real-time Linux Scheduler
Linux Scheduler Decides which process to run next Allocates processor(s)’ time among run-able processes Realizes multi-tasking in a single processor machine Schedules SMP as well Affects how the system behaves, responsiveness
Linux Scheduler Framework Kernel supports various scheduling policies by plug-in Scheduling classes contains details of the scheduling policy Each operation can be requested by the global scheduler; Allows for creating of the generic scheduler without any knowledge about different scheduler classes
Linux Scheduler Classes SCHED_OTHER (default, non-RT) SCHED_FIFO Use FIFO real-time scheduling SCHED_RR Use round-robin real-time scheduling
Linux Scheduler Priority Two separate priority ranges Nice value: -20 ... +19 (19 being the lowest) Real-time priority: 0 ... 99 (higher value is higher prio) ps priority: 0 ... 139 (0: lowest and 139: highest)
Linux Scheduler System Calls
Linux Scheduler Utilities How to configure priority chrt [option] ­p [prio] pid -r: SCHED_RR; -f: SCHED_FIFO sched_setscheduler() Checking priority, nice, rt-priority ps -eo pid,class,pri,nice,rtprio,comm
Making Linux do Hard Real-time Interrupt Handling
Interrupts (1) I/O devices can be checked if ready to transfer data by pooling (ie. continuously checking device registers) while (<Device is NOT ready for transfer data>)     do_wait(); Transfer_Data(); This operation has a busy-waiting part which consumes CPU To minimize busy-waiting, checking registers can be done by a periodic task If no data is ready to transfer the periodic task sleeps until the next period The busy waiting condition can be completely removed using interrupts
Interrupts (2) When an I/O device is ready to transfer data it signals the CPU (issues an interrupt) The CPU interrupts the current task an calls an Interrupt Service Routine (ISR) to handle the interrupt
Interrupts (3) The length of Linux interrupt vector is 256 32 entries are reserved to handle CPU exceptions The remaining 224 entries can point: software interrupts' handlers (like int 0x80: system calls) hardware interrupts' handlers (assigned to a IRQ line) On x86 architecture normally only 16 IRQ lines are available
Interrupts (4) Devices are connected to two Programmable Interrupt Controllers (intel 8259x) : A maximum of 8 8259s can be connected in cascade An ISR will be executed every time an interrupt occur in the IRQ line assigned The RTC can be used to generate interrupts on IRQ 8.
Interrupts (5) PICs can be accessed through I/O ports see how many PICs are available; IRQ lines reserved: cat /proc/ioports  cat /proc/interrupts
Context Switching (1) The context of the interrupted task (at least the values of the CPU registers at the time of the interrupt) must be preserved, so that it can be resumed with the same state Task context can be stored in: Interrupted task local stack system stack in a set of specialized CPU registers Probably the most efficient is the last one If the CPU itself has some registers to save context, it can be done in a few cycles (or just one) Some DSPs and PowerPCs have a set of these registers
Context Switching (2) If the ISR itself can not be interrupted any one of those 3 strategies can be used to save task context However sometimes it is needed to interrupt an ISR (to handle a fast I/O device which issues an high priority interrupt) A Programmable interrupt controller PIC can have some levels (priorities of interrupts)
Context Switching (3) It is possible to store task context in any stack, even if several ISRs are nested is just another set of data to put on top of the stack stacks have limited capacity However CPUs have a restricted set of registers to store context, making impossible to store context of several ISRs … or even one, it depends on the CPU
Clock Interrupts (1) In a scheduling mechanism, such as round-robin, after a time quantum or time-slice: Dispatcher removes the task currently assigned to the CPU and assigns the CPU to the first task in the ready tasks queue (switching task context)
Clock Interrupts (2) The dispatcher can be an ISR, awaken by a periodic interrupt, generated by a timer, in each time quantum Assuming that the time-slice is fixed, the timer must generate an interrupt after a fixed number of cycles or ticks These interrupts are referred as tick-based clock interrupts
Clock Interrupts (3) If time-slice may be different for different tasks, the timer must be programmable When a new task is assigned to the CPU, the timer is programmed with the time-slice of this task These variable time interrupts are referred as event-based clock interrupts
Making Linux do Hard Real-time Latency in Linux
Latency in Kernel Real time means external event should be handled within the bounded time Interrupt handler responds to the event and inform user-space process Latency Time taken from external interrupt till a user-space process to react to the interrupt
Linux kernel latency components Typical scenario: process waiting for the completion of device I/O (signaled by an interrupt) to resume execution. Waiting task interrupt latency Interrupt handler Scheduler Running task Interrupt handler duration scheduler latency scheduler duration Process context Interrupt context Makes the task runnable kernel latency = interrupt latency + handler duration + scheduler latency + scheduler duration
Interrupt latency Waiting task interrupt latency Interrupt handler Scheduler Running task Interrupt handler duration scheduler latency scheduler duration Makes the task runnable Time elapsed before executing the interrupt handler
Sources of interrupt latency Time between an event happens (raising an interrupt) and the handler is called. Sources of latency: Interrupts disabled by kernel code: spinlocks, evil drivers masking out interrupts. Other interrupts processed before: Shared interrupt line: all handlers are executed (don't use shared IRQ lines for critical events) Interrupts with higher priority can preempt lower priority ones (managed by the CPU, not by the scheduler). Not a problem in a correctly designed product: you use the top priority interrupt source. Summary: the only real problem is disabled interrupts. interrupt latency Interrupt handler Interrupt handler with greater priority Linux processes handler
Interrupt handler duration Waiting task interrupt latency Interrupt handler Scheduler Running task Interrupt handler duration scheduler latency scheduler duration Makes the task runnable Time taken to execute the interrupt handler
Interrupt Handler Interrupt handlers are split into two parts Top-half: Process, as quickly as possible, the work during interrupt disabled, such as queue the information for the bottom- half Schedule the bottom-half bottom-half Process the required tasks from the triggered interrupt
Kernel preemption (1) Linux kernel is a preemptive operating system When a task runs in user-space mode and gets interrupted by an interruption, if the interrupt handler wakes up another task, this task can be scheduled as soon as we return from the interrupt handler. Task A (running in user mode) Interrupt handler Wakes up Task B Task B (running in user mode) Interrupt
Kernel preemption (2) However, when the interrupt comes while the task is executing a system call, this system call has to finish before another task can be scheduled. By default, the Linux kernel does not do kernel preemption. This means that the time before which the scheduler will be called to schedule another task is unbounded. Task A (user mode) Interrupt handler Wakes up Task B Task B (user mode) System call Task A (kernel mode) Task A (kernel mode) Interrupt ? Return from syscall
Preemptive Kernel non-preemptive system preemptive system A concept linked to that of real time is preemption: the ability of a system to interrupt tasks at many “preemption points”.The longer the non-interruptible program units are, the longer is the waiting time (‘latency’) of a higher priority task before it can be started or resumed. GNU/Linux is “user-space preemptible”: it allows user tasks to be interrupted at any point. The job of real- time extensions is to make system calls preemptible as well.
Interrupt handler implementation Run with interrupts disabled (at least on the current line). Hence, they need to complete and restore interrupts as soon as possible. To satisfy this requirement, interrupt handlers are often implemented in 2 parts: top half and bottom half (soft IRQ) Top half Interrupt ACK Exit Bottom half Schedule bottom half other interrupt handlers... Handle device data... Wake up waiting tasks User space... Soft IRQs are handled in interrupt context (not by the scheduler)
Sources of interrupt handler execution time Time taken to execute your interrupt handler. Causes which can make this duration longer: Preemption by other interrupts with a greater priority. Again, not a problem if priorities are configured properly. Interrupt handler with a softirq (“bottom half”) component: run after all interrupts are serviced. => For critical processing, better not have a softirq (may require rewriting the driver if reusing an existing one). handler duration Interrupt handler Interrupt interrupt latency
Scheduler latency Waiting task interrupt latency Interrupt handler Scheduler Running task Interrupt handler duration scheduler latency scheduler duration Makes the task runnable Time elapsed before executing the scheduler
When is the scheduler run next? In the general case (still after a task is woken up) Upon return from interrupt context (our particular case), unless the current process is running a system call (system calls are not preemptible by default). After returning from system calls. When the current process runs an action which sleeps / waits (voluntarily or not, in kernel or user mode), or explicitly calls the scheduler.
Sources of scheduler latency Possible sources of latency: Other interrupts coming in (even with a lower priority). Back to our case We are returning from interrupt context. The scheduler is run immediately if the current task was running in userspace. Otherwise, we have to wait for the end of the current system call. Interrupt handler Scheduler scheduler latency
Scheduler duration Waiting task interrupt latency Interrupt handler Scheduler Running task Interrupt handler duration scheduler latency scheduler duration Makes the task runnable Time taken to execute the scheduler and switch to the new task.
Sources of scheduler execution time Time to execute the scheduler and switch to a new task Time to execute the scheduler: depended on the number of running processes. Context switching time: time to save the state of the current process (CPU registers) and restore the new process to run. This time is constant, so if not an issue for real-time. SCHED_FIFO & SCHED_RR use bit map to find out the next task CFS (Completely Fair Scheduler) uses Red-black tree as a sorted queue Scheduler New task scheduler duration
Other sources of latency (1) Demand paging When it starts a process, the Linux kernel first sets up the virtual address space of the program (code, stack and data). However, to improve performance and reduce memory consumption, Linux loads the corresponding pages in RAM only when they are needed. When a part of the address space is accessed (like a particular part of the code) and is not in RAM yet, a page fault is raised by the MMU. This triggers the loading of the page in RAM. This usually involves reading from disk: unexpected latencies!
Address Translation
Other sources of latency (2) Linux is highly based on virtual memory provided by MMU so that memory is allocated on demand. Whenever an application accesses code or data for the first time, it is loaded on demand, which can creates huge delays. Memory allocation The Linux allocator is fast, but does not guarantee a maximum allocation time.
Other sources of latency (3) System calls Similarly, no guaranteed response time. Device drivers System calls handlers or interrupt handlers usually not implemented with care for real-time requirements. Expect uncertainties unless you implemented all kernel support code by yourself.
Issue: priority inversion 99 50 3 1. Low Priority task takes mutex 3. Hi Priority task block on mutex 4. Medium Priority task preempts low priority task and high priority task 2. High Priority task preempts low priority task Time TaskPriority
Issue: priority inversion Issue: a process with more priority can preempt a process holding the lock. The top priority process could wait for a very long time. This happened with standard Linux before version 2.6.18. Acquires a lock Priority Time preempted Tries to get the same lock waits
Solution: priority inheritance Solution: priority inheritance. The process holding the lock inherits the priority of the process waiting for the lock with the greatest priority. Acquires a lock Priority Time preempted Tries to get the lock waits Executes with the priority of the waiting process Releases the lock Acquires the lock preempted
Issue: interrupt inversion Issue: even your top priority task can be “preempted” by any interrupt handler, even for interrupts feeding tasks with lower priority. Solution: threaded interrupts (scheduled for later execution). top priority task Any interrupt top priority task Any interrupt...
Making Linux do Hard Real-time Linux 2.6 features for real­time Linux features for real-time
Linux 2.6 improvements (1) Scheduler: starting from 2.6.23, Linux implements CFS. complexity of O(log N), where N is the number of tasks in the runqueue. Choosing a task can be done in constant time O(1), but reinserting a task after it has run requires O(log N) operations, because the runqueue is implemented as a red-black tree. May be built with no virtual memory support (on supported platforms). Benefits: reduced context switching time (address space shared by all processes), better performance (CPU cache still valid after switching). Full POSIX real-time API support. the standard Unix API to implement real-time applications.
POSIX 1003.1b Real-time extensions Priority Scheduling Real-Time Signals Clocks and Timers Semaphores Message Passing Shared Memory Asynchronous and Synchronous I/O Memory Locking
Linux 2.6 improvements (2) Threaded Interrupt Interrupt can be executed by a kernel thread, we can assign a priority to each thread RT-Mutexes RT-mutexes extend the semantics of simple mutexes by the priority inheritance protocol. BKL-free Since 2.6.37 kernel can be built completely without the use of Big Kernel Lock.
Linux 2.6 improvements (3) Increased determinism with improved POSIX RT API support. By default, system calls still not preemptible. New CONFIG_PREEMPT option making most of the kernel code preemptible. However, even CONFIG_PREEMPT was not enough for audio users (need guaranteed latency in the milliseconds range), and of course for real-time users with stronger requirements.
New preemption options in Linux 2.6 2 new preemption models offered by standard Linux 2.6:
1st option: no forced preemption CONFIG_PREEMPT_NONE Kernel code (system calls) never preempted. Default behavior in standard kernels. Best for systems making intense computations, on which overall throughput is key. Best to reduce task switching to maximize CPU and cache usage (by reducing context switching). Can also benefit from a lower timer frequency (100 Hz instead of 250 or 1000).
2nd option: voluntary kernel preemption CONFIG_PREEMPT_VOLUNTARY Kernel code can preempt itself Typically for desktop systems, for quicker application reaction to user input. Adds explicit rescheduling points throughout kernel code. Minor impact on throughput.
3rd option: preemptible kernel CONFIG_PREEMPT Most kernel code can be involuntarily preempted at any time. When a process becomes runnable, no more need to wait for kernel code (typically a system call) to return before running the scheduler. Exception: kernel critical sections (holding spinlocks). Typically for desktop or embedded systems with latency requirements in the milliseconds range. Still a relatively minor impact on throughput.
Priority inheritance support Available since Linux 2.6.18 Implemented in kernel space locking Available in user space locking too, through fast user-space mutexes (“futex”). Priority inheritance needs to be explicited for each mutex though. See the kernel documentation for details: Documentation/pi­futex.txt Documentation/rt­mutex­design.txt
High-resolution timers Make it possible for POSIX timers and nanosleep() to be as accurate as allowed by the hardware (typically 1 us). Together with tickless support, allow for “on-demand” timer interrupts. Available in the vanilla kernel since 2.6.21.
Making Linux do Hard Real-time Linux rt-preempt
Linux real-time preemption http://www.kernel.org/pub/linux/kernel/projects/rt/ led by kernel developers including Ingo Molnar, Thomas Gleixner, and Steven Rostedt Large testing efforts at RedHat, IBM, OSADL, Linutronix Goal is to improve real time performance Configurable in the Processor type and features (x86), Kernel  Features (arm) or Platform options (ppc)...
Wrong ideas about real-time preemption Myth (1): It will improve throughput and overall performance Wrong: it will degrade overall performance. Myth (2): It will reduce latency Often wrong. The maximum latency will be reduced. The primary goal is to make the system predictable and deterministic.
Current issues and limitations Linux RT patches not mainstream yet. Facing resistance to retain the general purpose nature of Linux. Though they are preemptible, kernel services (such as memory allocation) do not have a guaranteed latency yet! However, not really a problem: real-time applications should allocate all their resources ahead of time. Kernel drivers are not developed for real-time constraints yet (e.g. USB or PCI bus stacks) Binary-only drivers have to be recompiled for RT preempt Memory management (SLOB/SLUB allocator) still runs with preemption disabled for long periods of time Lock around per-CPU data accesses, eliminating the preemption problem but wrecking scalability
PREEMPT_RT: complete RT preemption Replace non-preemptible constructs with preemptible ones Make OS preemptible as much as possible except preempt_disable and interrupt disable Make Threaded (schedulable) IRQs so that it can be scheduled spinlocks converted to mutexes (a.k.a. sleeping spinlocks) Not disabling interrupt and allows preemption Works well with threaded interrupts
Toward complete RT preemption Most important aspects of Real-time Controlling latency by allowing kernel to be preemptible everywhere
original Linux Kernel User Space User Context Interrupt Handlers Kernel Space Interrupt Context SoftIRQs Hiprio tasklets Network Stack Timers Regular tasklets ... Scheduling Points Process Thread Kernel Thread
PREEMPT_RT User Space SO_NODELAY Interrupt Handlers Kernel Space NetworkStack Timers Tasklets Scheduling Points Process Thread Kernel Threads
Interface changed by PREEMPT_RT spinlocks and local_irq_save() no longer disable hardware interrupts. spinlocks no longer disable preemption. Raw_ variants for spinlocks and local_irq_save() preserve original meaning for SO_NODELAY interrupts. Semaphores and spinlocks employ priority inheritance
Threaded Interrupts Interrupt handlers run in normal kernel threads Priorities can be configured Main interrupt handler Do minimal work and wake-up the corresponding thread Thread interrupts allows to use sleeping spinlocks in PREEMPT_RT, all interrupt handlers are switched to threaded interrupt drivers in mainline get gradually converted to the new threaded interrupt API that has been merged in Linux 2.6.30.
Threaded Interrupts The vanilla kernel Interrupts as threads Real world behavior
Threaded Interrupt handler Almost all kernel space is now preemptible An interrupt can occur at any time The woken up processes by interrupt can start immediately Treaded IRQs: Kernel thread per ISR Priority must be set Interrupt handler threads Softirq threads Other kernel threads Real time application processes/threads
Timer Frequency Linux timer interrupts are raised every HZ of second. Resolution of the system timer (HZ) 100 HZ to 1000 HZ depends on arch and configuration Resolution of 10 ms to 1 ms Adds overheads → Compromise between system responsiveness and global throughput.
Effect of Timer Frequency 10 20 10 20 15 Requested sleep time, in ms HZ=100 Real sleep time, in ms
Effect of Timer Frequency 10 20 10 20 15 Requested sleep time, in ms Real sleep time, in ms HZ=1000
High Resolution Timers Use non-RTC interrupt timer sources to deliver timer interrupt between ticks. Allows POSIX timers and nanosleep() to be as accurate as the hardware allows This feature is transparent. When enabled it makes these timers much more accurate than the current HZ resolution. Around 1usec on typical hardware.
Use POSIX Timer for Periodical Task const int NSEC_IN_SEC = 1000000000l, INTERVAL = 1000000l; struct timespec timeout; clock_gettime(CLOCK_MONOTONIC, &timeout); while (1) {     do_some_work(&some_data);     timeout.tv_nsec += INTERVAL;     if (timeout.tv_nsec >= NSEC_IN_SEC) {         timeout.tv_nsec ­= NSEC_IN_SEC;         timeout.tv_sec++;     }     clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &timeout, NULL); }
Making Linux do Hard Real-time Implementing real-time constraints
Process vs. Thread In Unix, a process is created using fork() and is composed of An address space, which contains the program code, data, stack, shared libraries, etc. One thread, that starts executing the main() function. Upon creation, a process contains one thread Additional threads can be created inside an existing process, using pthread_create() They run in the same address space as the initial thread of the process They start executing a function passed as argument to pthread_create()
Process vs. Thread: Kernel point of view Kernel represents each thread running in the system by a structure of type task_struct From a scheduling point of view, it makes no difference between the initial thread of a process and all additional threads created dynamically using pthread_create() Address space Thread A Process after fork() Address space Thread A Thread B Same process after pthread_create()
Creating threads Linux support the POSIX thread API To create a new thread pthread_create(pthread_t *thread, pthread_attr_t *attr, void *(*routine)(*void*), void *arg); The new thread will run in the same address space, but will be scheduled independently Exiting from a thread pthread_exit(void *value_ptr); Waiting for the termination of a thread pthread_join(pthread_t *thread, void **value_ptr);
Scheduler policy SCHED_FIFO: the process runs until completion unless it is blocked by an I/O, voluntarily relinquishes the CPU, or is preempted by a higher priority process. SCHED_RR: the processes are scheduled in a Round Robin way. Each process is run until it exhausts a max time quantum. Then other processes with the same priority are run, and so and so... SCHED_BATCH: Does not preempt nearly as often as regular tasks would, thereby allowing tasks to run longer and make better use of caches but at the cost of interactivity. This is well suited for batch jobs. SCHED_IDLE: Lowest priority in the system SCHED_NORMAL: Used for regular processes with non-RT priority.
Scheduling classes (1) sched_setscheduler() API can be used to change the scheduling class and priority of a process int sched_setscheduler(pid_t pid, int  policy, const struct sched_param *param); policy can be SCHED_OTHER, SCHED_FIFO,  SCHED_RR, etc. param is a structure containing the priority
Scheduling classes (2) Priority can be set on a per-thread basis when its creation: Then the thread can be created using pthread_create(), passing the attr structure. Several other attributes can be defined this way: stack size struct sched_param parm; pthread_attr_t attr; pthread_attr_init(&attr); pthread_attr_setinheritsched(&attr,                              PTHREAD_EXPLICIT_SCHED); pthread_attr_setschedpolicy(&attr, SCHED_FIFO); parm.sched_priority = 42; pthread_attr_setschedparam(&attr, &parm);
How fork() seems to work Parent Process Parent Process Child Process Copy A complete copy is created of the parent. Both parent and child continue execution from the same spot.
How fork() really works Parent Process Parent Process Child Process Memory RW Memory RO RO Child process gets a copy of stack and file descriptors. Copy On Write reference is used for the memory.
What happens during write? Parent Process Child Process Original Memory RO RO When write is attempted, a single memory page is copied and references updated. This is called: “breaking the CoW”. RW
Locking pages in RAM A solution to latency issues with demand paging! Available through <sys/mman.h> (see man mman.h) mlock Lock a given region of process address space in RAM. Makes sure that this region is always loaded in RAM. mlockall Lock the whole process address space. munlock, munlockall Unlock a part or all of the process address space.
Realtime Hello World
Mutexes Allows mutual exclusion between two threads in the same address space Initialization/destruction pthread_mutex_init(pthread_mutex_t *mutex,  const pthread_mutexattr_t *mutexattr); pthread_mutex_destroy(pthread_mutex_t *mutex); Lock/unlock pthread_mutex_lock(pthread_mutex_t *mutex); pthread_mutex_unlock(pthread_mutex_t *mutex); Priority inheritance must be activated explicitly pthread_mutexattr_t attr; pthread_mutexattr_init (&attr); pthread_mutexattr_getprotocol(&attr,PTHREAD_PRIO_INHERIT);
Timers timer_create(clockid_t clockid, struct sigevent *evp, timer_t *timerid) Create a timer. clockid is usually CLOCK_MONOTONIC. sigevent defines what happens upon timer expiration: send a signal or start a function in a new thread. timerid is the returned timer identifier. timer_settime(timer_t timerid, int flags, struct itimerspec *newvalue, struct itimerspec *oldvalue) Configures the timer for expiration at a given time. timer_delete(timer_t timerid), delete a timer clock_getres(), get the resolution of a clock Other functions: timer_getoverrun(), timer_gettime()
Signals Signals are asynchronous notification mechanisms Notification occurs either By the call of a signal handler. Be careful with the limitations of signal handlers! By being unblocked from the sigwait(), sigtimedwait() or sigwaitinfo() functions. Usually better. Signal behaviour can be configured using sigaction() The mask of blocked signals can be changed with pthread_sigmask() Delivery of a signal using pthread_kill() or tgkill()
Benchmarking (1) cyclictest measuring accuracy of sleep and wake operations of highly prioritized realtime threads https://rt.wiki.kernel.org/index.php/Cyclictest vanilla kernel PREEMPT_RT
Benchmarking (2) QA farm running at OSADL.ORG worst case latency comparison between 2.6.32 and 2.6.33-rt https://www.osadl.org/Compare-systems.qa-farm-compare-latency.0.html Worst case latency in non-rt is over 4,000 usec, in rt around 100 usec
Again, Response time vs. Throughput Overhead for real-time preemption Mutex instead of spin lock Priority inheritance Threaded interrupt handler Due to overhead of real-time preemption Throughput is reduced Due to the flexibility of preemption Better worst case latency
Making Linux do Hard Real-time Linux hard real-time extensions
Linux hard real-time extensions Three generations RTLinux RTAI Xenomai A common principle Add a extra layer between the hardware and the Linux kernel, to manage real-time tasks separately. Hardware Micro/Nano-kernel Linux kernel real­time tasks real-time tasks
Interrupt Response Time ConfigurationConfiguration AvgAvg MaxMax MinMin XENOMAIXENOMAI 4343 5858 22 PREEMPT 1st run PREEMPT 1st run 8888 415415 2727 PREEMPT 2nd run PREEMPT 2nd run 8181 18291829 1313 Hardware: Freescale i.MX53 ARM Cortex-A8 processor operating at 1GHz. Time in micro second. PREEMPT: standard kernel with CONFIG_PREEMPT (“Preemptible Kernel (Low-Latency Desktop)) enabled cyclictest –m ­n ­p99 ­t1 ­i10000  ­1360000 XENOMAI: Kernel + Xenomai 2.6.0-rc4 + I-Pipe 1.18-03 cyclictest ­n ­p99 ­t1 ­i10000  ­1360000
Xenomai project http://www.xenomai.org/ Started in the RTAI project (called RTAI / fusion). Skins mimicking the APIs of traditional RTOS such as VxWorks, pSOS+, and VRTXsa. Initial goals: facilitate the porting of programs from traditional RTOS to RTAI on GNU / Linux. Now an independent project and an alternative to RTAI. Many contributors left RTAI for Xenomai, frustrated by its goals and development style.
Xenomai architecture Adeos i-pipe Xenomai RTOS (nucleus) VxWorks application glibc Xenomai libvxworks POSIX application glibc Xenomai libpthread_rt Linux application glibc VFS Network Memory ... System calls Linux kernel space Pieces added by Xenomai Xenomai skins ipipe = interrupt pipeline
Linux vs. Xenomai A special way of passing of the interrupts between real-time kernel and the Linux kernel is needed. Each flavor of RT Linux does this is in its own way. Xenomai uses an interrupt pipeline from the Adeos project.
From the Adeos point of view, guest OSes are prioritized domains. For each event (interrupts, exceptions, syscalls, ...), the various domains may handle the event or pass it down the pipeline. Interrupt pipeline abstraction
Main idea support interrupt priority ensure that domains support segments of code that can not be interrupted (like critical sections) Adeos implements the Optimistic protection scheme (Stodolsky, 1993): When a segment of code is executed in a domain with interrupts disabled, that stage of the pipeline stalls, so interrupts can not be serviced nor propagated further in the pipeline i-pipe: Optimistic protection scheme (1)
If a real time domain (like Xenomai) has higher priority it is the first in the pipeline It will receive interrupt notification first without delay (or at least with predictable latency) Then it can be decided if interrupts are propagated to low priority domains (like Linux) or not i-pipe: Optimistic protection scheme (2)
The high priority domain is at the beginning of the pipeline, so events are delivered first to it This pipeline is referred as interrupt pipeline or I-pipe There is a pipeline for each CPU Interrupt pipeline (1)
The Linux domain is always the root domain, whatever is its position in the pipeline Other domains are started by the root domain Linux starts and loads the kernel modules that implement other domains Interrupt pipeline (2)
virtualized interrupts disabling Each domain may be “stalled”, meaning that it does not accept interrupts. Hardware interrupts are not disabled however (except for the domain leading the pipeline), instead the interrupts received during that time are logged and replayed when the domain is unstalled.
I-pipe observation through procfs $ cat /proc/ipipe/Xenomai        +­­­­­ Handling ([A]ccepted, [G]rabbed,        |+­­­­ Sticky    [W]ired, [D]iscarded)        ||+­­­ Locked        |||+­­ Exclusive        ||||+­ Virtual [IRQ]  38: W..X. 418: W...V $ cat /proc/ipipe/Xenomai        +­­­­­ Handling ([A]ccepted, [G]rabbed,        |+­­­­ Sticky    [W]ired, [D]iscarded)        ||+­­­ Locked        |||+­­ Exclusive        ||||+­ Virtual [IRQ]  38: W..X. 418: W...V $ cat /proc/ipipe/Linux 0: A.... 1: A.... ... priority=100 $ cat /proc/ipipe/Linux 0: A.... 1: A.... ... priority=100
Linux options for Xenomai configuration The POSIX skin allows to recompile without changes a traditional POSIX application so that instead of using Linux real-time services, it uses Xenomai services. The POSIX skin allows to recompile without changes a traditional POSIX application so that instead of using Linux real-time services, it uses Xenomai services.
Xenomai Execution Model (1) All tasks managed by Xenomai are known by the real time nucleus: Tasks (Xenomai tasks or threads) executed in the Xenomai domain are executed in primary mode (or primary domain) [the highest priority domain; real time domain] Tasks (Linux threads or processes) executed in the Linux domain are executed in secondary mode (or secondary domain)
Xenomai Execution Model (2) To ensure that tasks in secondary domain still have real time support, it is required: (A) Common priority scheme When a Xenomai task migrates to Linux domain, the root thread's mutable priority technique is applied: Linux kernel inherits the priority of the Xenomai thread that migrates
Xenomai Execution Model (3) Common priority scheme This means that other threads in the Xenomai Domain can only preempt the migrated thread (ie. the linux kernel executing the syscall) if their priority is higher Note: This is the opposite with RTAI (Real-Time Application Interface): A thread that enters the Linux Domain inherits the priority of the Linux kernel, which is the lowest in the RTAI scheduler Note: This is the opposite with RTAI (Real-Time Application Interface): A thread that enters the Linux Domain inherits the priority of the Linux kernel, which is the lowest in the RTAI scheduler
Xenomai Execution Model (4) To ensure that tasks in secondary domain still have real time support, it is required: (B) Predictability of program execution time When a Xenomai thread enters secondary domain (Linux) it should not be preempted by: Linux domain interrupts other low priority asynchronous activity at Linux kernel level Note that it is assumed that when an interrupt is delivered to Linux domain, Linux Kernel code can be preempted whatever its priority To avoid this, Linux kernel can be starved from interrupts while the Xenomai thread is in secondary mode
Xenomai Execution Model (5) To ensure that a Xenomai tasks in secondary domain have still real time support (and predictable execution time ) While a Xenomai task migrates to Linux domain (secondary): Linux kernel inherits the priority of the Xenomai task that migrates (normal Linux kernel priority is the lowest: 0) Linux is starved from interrupts and other system events, which are blocked in an intermediate domain: The interrupt shield (to avoid that the Linux kernel interrupts the migrated Xenomai task when responding to an interrupt or other asynchronous event)
Adeos additional features The Adeos I-pipe patch implement additional features, essential for the implementation of the Xenomai real-time extension: Disables on-demand mapping of kernel-space vmalloc/ioremap areas. Disables copy-on-write when real-time processes are forking. Allow subscribing to event allowing to follow progress of the Linux kernel, such as Linux system calls, context switches, process destructions, POSIX signals, FPU faults. On ARMv5 architecture, utilizes FCSE to reduce the latency induced by cache flushes during context switches.
Real-Time Scheduler Xenomai extends the Linux kernel and is integrated as part of OS. A task with a period = 15 us, shown in light blue. While this real-time task is not being executed, Xenomai invokes the regular Linux scheduler which executes tasks as normal, shown in dark green. LinuxLinux XenomaiXenomai struct task_struct struct xnthread mapped
Xenomai features Factored RT core with skins implementing various RT APIs Seamless support for hard real-time in user-space No second-class citizen, all ports are equivalent feature- wise Each Xenomai branch has a stable user/kernel ABI Timer system based on hardware high-resolution timers Per-skin time base which may be periodic RTDM skin allowing to write real-time drivers
Two modes are defined for a thread the primary mode, where the thread is handled by Xenomai scheduler the secondary mode, when it is handled by Linux scheduler. Thanks to the services of the Adeos I-pipe service, Xenomai system calls are defined. A thread migrates from secondary mode to primary mode when such a system call is issued It migrates from primary mode to secondary mode when a Linux system call is issued, or to handle gracefully exceptional events such as exceptions or Linux signals. Xenomai real-time user-space support
Original Linux User space Kernel space Low prio high prio Linuxscheduler main Linux thread pthread_create Runs in User Mode : can only address user space Linux syscall readfile(..), printf(..) Kernel moduleKernel module Kernel module Linux Kernel Communicates with terminal to display text of printf filesystem module communicates with harddisk to read the file Kernel module
Xenomai (kernel space) User space Kernel space Low prio high prio not realtime hard realtime Linuxscheduler Real-time scheduler RT task Kernel module init_module() rt_task_start Xenomai Kernel function call Linux Kernel Shell : # insmod <module> Runs in Kernel Mode : can address User and Kernel space
Xenomai (user space) User space Kernel space Low prio high prio Linuxscheduler Real-time scheduler FIFO Sched other main Linux thread rt_thread_create Xenomai task Xenomai task rt_task_create, rt_task_start Linux syscall Runs in User Mode : can only address user space Xenomai Kernel xenomai syscall Linux Kernel Linux syscall hard realtime soft realtime
Concepts changed in Xenomai Linux kernel realtime kernel threads controlled by realtime scheduler threads controlled by Linux-Scheduler system call entry exactly the same for both kernels Ipipe (ADEOS) Common Library Linux Process POSIX API HW- Interrupts Soft- Interrupts static priorities (POSIX-counting) 99 0 99 0 realtime kernelLinux Linux + Realtime boundary implementation defined Linux
Real-time Linux task Observe the difference between real-time tasks and standard Linux programs: In real-time Linux, we can make a real-time program by programming real-time threads. None-real-time programs in real-time Linux just use the conventional Linux threads. A real-time task can be executed in kernel space using a kernel module, but it can also be executed in user space using a normal C program.
Real-time tasks in Linux userspace Running in user space instead of in kernel space gives a little extra overhead, but with the following advantages: Bugs in a user space program can only crash the program. In a kernel model, one can only use the real-time API and the limited kernel API, but in a real-time user space program the real-time API and the whole Linux API can be used. However when using the Linux API in user space, the program cannot be scheduled by the real-time scheduler (HARD real-time) but must be scheduled by the Linux scheduler (SOFT real-time). So calling a Linux system call from a real-time user space task will degrade its performance from HARD to SOFT real-time. After the call the task will return to the real-time scheduler.
RTDM Real-Time Driver Model An approach to unify the interfaces for developing device drivers and associated applications under real-time Linux. Currently available for Xenomai and RTAI RTDM-native: a port over native Linux with the real-time preemption patch. See the whitepaper on http://www.xenomai.org/documentation/xenomai-2.6/pdf/RTDM-and-Applications.pdf
The POSIX skin POSIX skin allows to recompile without changes a traditional POSIX application so that instead of using Linux real-time services, it uses Xenomai services Clocks and timers, condition variables, message queues, mutexes, semaphores, shared memory, signals, thread management Good for existing code or programmers familiar with the POSIX API Of course, if the application uses any Linux service that isn't available in Xenomai, it will switch back to secondary mode To link an application against the POSIX skin Export DESTDIR=/path/to/xenomai/ CFLAGS=`$DESTDIR/bin/xeno­config ­­posix­cflags` LDFLAGS=`$DESTDIR/bin/xeno­config ­­posix­ldflags` {CROSS_COMPILE}gcc ${CFLAGS} ­o rttest rttest.c ${LDFLAGS}
Communication with a normal task If a Xenomai real-time application using the POSIX skin wishes to communicate with a separate non-real-time application, it must use the rtipc mechanism In Xenomai application, create an IPCPROTO_XDDP socket socket(AF_RTIPC, SOCK_DGRAM, IPCPROTO_XDDP); setsockopt(s, SOL_RTIPC, XDDP_SETLOCALPOOL,&poolsz,sizeof(poolsz)); memset(&saddr, 0, sizeof(saddr)); saddr.sipc_family = AF_RTIPC; saddr.sipc_port = MYAPPIDENTIFIER; ret = bind(s, (struct sockaddr *)&saddr, sizeof(saddr)); And then the normal socket API sendto() / recvfrom() In the Linux application Open /dev/rtpX, where X is the XDDP port Use read() and write()
Conclusion Linux was not designed as a RTOS However, you can make your system hard real-time by using one of the hard real-time extensions (e.g. Xenomai), and writing your critical applications and drivers with the corresponding APIs. You can get soft real-time with the standard kernel preemption mode. Most of the latencies will be reduced, offering better quality, but probably not all of them. However, using hard real-time extensions will not guarantee that your system is hard real-time. Your system and applications will also have to be designed properly (correct priorities, use of deterministic APIs, allocation of critical resources ahead of time...).
Reference Soft, Hard and Hard Real Time Approaches with Linux, Gilad Ben- Yossef A nice coverage of Xenomai (Philippe Gerum) and the RT patch (Steven Rostedt):http://oreilly.com/catalog/9780596529680/ Real-time Linux, Insop Song Understanding the Latest Open-Source Implementations of Real- Time Linux for Embedded Processors, Michael Roeder

More Related Content

PDF
Making Linux do Hard Real-time
National Cheng Kung University
 
PDF
BPF Internals (eBPF)
Brendan Gregg
 
PPT
File system
Harleen Johal
 
PDF
The Linux Block Layer - Built for Fast Storage
Kernel TLV
 
PDF
8 memory management strategies
Dr. Loganathan R
 
PPTX
Buddy system final
Tech_MX
 
PDF
Performance Wins with eBPF: Getting Started (2021)
Brendan Gregg
 
PPTX
Page replacement algorithms
Piyush Rochwani
 
Making Linux do Hard Real-time
National Cheng Kung University
 
BPF Internals (eBPF)
Brendan Gregg
 
File system
Harleen Johal
 
The Linux Block Layer - Built for Fast Storage
Kernel TLV
 
8 memory management strategies
Dr. Loganathan R
 
Buddy system final
Tech_MX
 
Performance Wins with eBPF: Getting Started (2021)
Brendan Gregg
 
Page replacement algorithms
Piyush Rochwani
 

What's hot (20)

PDF
Process Scheduler and Balancer in Linux Kernel
Haifeng Li
 
PDF
SFO15-TR9: PSCI, ACPI (and UEFI to boot)
Linaro
 
PDF
U-Boot - An universal bootloader
Emertxe Information Technologies Pvt Ltd
 
PDF
The Linux Kernel Scheduler (For Beginners) - SFO17-421
Linaro
 
PPTX
The TCP/IP Stack in the Linux Kernel
Divye Kapoor
 
PDF
LCA14: LCA14-306: CPUidle & CPUfreq integration with scheduler
Linaro
 
PDF
HKG15-107: ACPI Power Management on ARM64 Servers (v2)
Linaro
 
PDF
The linux networking architecture
hugo lu
 
PDF
LAS16-200: SCMI - System Management and Control Interface
Linaro
 
PDF
Linux Preempt-RT Internals
哲豪 康哲豪
 
PPTX
Linux device drivers
Abhishek Sagar
 
PDF
Linux Locking Mechanisms
Kernel TLV
 
PDF
LCU14 500 ARM Trusted Firmware
Linaro
 
PDF
Linux-Internals-and-Networking
Emertxe Information Technologies Pvt Ltd
 
PPT
Linux Kernel Image
艾鍗科技
 
PPTX
Linux Interrupts
Kernel TLV
 
PPTX
Linux kernel debugging
Hao-Ran Liu
 
PPT
Kernel module programming
Vandana Salve
 
PDF
Uboot startup sequence
Houcheng Lin
 
PPTX
Linux Network Stack
Adrien Mahieux
 
Process Scheduler and Balancer in Linux Kernel
Haifeng Li
 
SFO15-TR9: PSCI, ACPI (and UEFI to boot)
Linaro
 
U-Boot - An universal bootloader
Emertxe Information Technologies Pvt Ltd
 
The Linux Kernel Scheduler (For Beginners) - SFO17-421
Linaro
 
The TCP/IP Stack in the Linux Kernel
Divye Kapoor
 
LCA14: LCA14-306: CPUidle & CPUfreq integration with scheduler
Linaro
 
HKG15-107: ACPI Power Management on ARM64 Servers (v2)
Linaro
 
The linux networking architecture
hugo lu
 
LAS16-200: SCMI - System Management and Control Interface
Linaro
 
Linux Preempt-RT Internals
哲豪 康哲豪
 
Linux device drivers
Abhishek Sagar
 
Linux Locking Mechanisms
Kernel TLV
 
LCU14 500 ARM Trusted Firmware
Linaro
 
Linux-Internals-and-Networking
Emertxe Information Technologies Pvt Ltd
 
Linux Kernel Image
艾鍗科技
 
Linux Interrupts
Kernel TLV
 
Linux kernel debugging
Hao-Ran Liu
 
Kernel module programming
Vandana Salve
 
Uboot startup sequence
Houcheng Lin
 
Linux Network Stack
Adrien Mahieux
 
Ad

Viewers also liked (13)

PDF
Xvisor: embedded and lightweight hypervisor
National Cheng Kung University
 
PDF
中輟生談教育: 完全用開放原始碼軟體進行 嵌入式系統教學
National Cheng Kung University
 
PDF
How A Compiler Works: GNU Toolchain
National Cheng Kung University
 
PDF
進階嵌入式系統開發與實做 (2014 年秋季 ) 課程說明
National Cheng Kung University
 
PDF
從線上售票看作業系統設計議題
National Cheng Kung University
 
PDF
Develop Your Own Operating Systems using Cheap ARM Boards
National Cheng Kung University
 
PDF
Explore Android Internals
National Cheng Kung University
 
PDF
給自己更好未來的 3 個練習：嵌入式作業系統設計、實做，與移植 (2015 年春季 ) 課程說明
National Cheng Kung University
 
PDF
Lecture notice about Embedded Operating System Design and Implementation
National Cheng Kung University
 
PDF
Implement Runtime Environments for HSA using LLVM
National Cheng Kung University
 
PDF
Virtual Machine Constructions for Dummies
National Cheng Kung University
 
PDF
PyPy's approach to construct domain-specific language runtime
National Cheng Kung University
 
PDF
Priority Inversion on Mars
National Cheng Kung University
 
Xvisor: embedded and lightweight hypervisor
National Cheng Kung University
 
中輟生談教育: 完全用開放原始碼軟體進行 嵌入式系統教學
National Cheng Kung University
 
How A Compiler Works: GNU Toolchain
National Cheng Kung University
 
進階嵌入式系統開發與實做 (2014 年秋季 ) 課程說明
National Cheng Kung University
 
從線上售票看作業系統設計議題
National Cheng Kung University
 
Develop Your Own Operating Systems using Cheap ARM Boards
National Cheng Kung University
 
Explore Android Internals
National Cheng Kung University
 
給自己更好未來的 3 個練習：嵌入式作業系統設計、實做，與移植 (2015 年春季 ) 課程說明
National Cheng Kung University
 
Lecture notice about Embedded Operating System Design and Implementation
National Cheng Kung University
 
Implement Runtime Environments for HSA using LLVM
National Cheng Kung University
 
Virtual Machine Constructions for Dummies
National Cheng Kung University
 
PyPy's approach to construct domain-specific language runtime
National Cheng Kung University
 
Priority Inversion on Mars
National Cheng Kung University
 
Ad

Similar to Making Linux do Hard Real-time (20)

PPT
Chapter 19 - Real Time Systems
Wayne Jones Jnr
 
ODP
Realtime
Mark Veltzer
 
PPT
UNIT V PPT.ppt
ThanmayaGoud
 
PDF
embedded system CHAPTER four about design .pdf
abdulkerimaragaw936
 
PPTX
Beneath the Linux Interrupt handling
Bhoomil Chavda
 
PPT
Rtos 2
SahibZada Ibm
 
PPT
Four Ways to Improve Linux Performance IEEE Webinar, R2.0
Michael Christofferson
 
PDF
Mastering Real-time Linux
Jean-François Deverge
 
PPTX
Real Time OS For Embedded Systems
Himanshu Ghetia
 
PDF
Lab6 rtos
indirakumar86
 
PPT
timers 2.ppt
ycelgemici1
 
PDF
RTOS implementation
Rajan Kumar
 
PDF
Cs6413 operating-systems-laboratory
Preeja Ravishankar
 
PDF
Vx works RTOS
Sai Malleswar
 
PDF
IJCER (www.ijceronline.com) International Journal of computational Engineeri...
ijceronline
 
PPT
ucos-2 introduction and characteristics
Varsha506533
 
PPT
ucos-2 introduction and characteristics
Varsha506533
 
PDF
Linux scheduler
Liran Ben Haim
 
PDF
Enea Enabling Real-Time in Linux Whitepaper
Enea Software AB
 
PPTX
Embedded os
K Senthil Kumar
 
Chapter 19 - Real Time Systems
Wayne Jones Jnr
 
Realtime
Mark Veltzer
 
UNIT V PPT.ppt
ThanmayaGoud
 
embedded system CHAPTER four about design .pdf
abdulkerimaragaw936
 
Beneath the Linux Interrupt handling
Bhoomil Chavda
 
Rtos 2
SahibZada Ibm
 
Four Ways to Improve Linux Performance IEEE Webinar, R2.0
Michael Christofferson
 
Mastering Real-time Linux
Jean-François Deverge
 
Real Time OS For Embedded Systems
Himanshu Ghetia
 
Lab6 rtos
indirakumar86
 
timers 2.ppt
ycelgemici1
 
RTOS implementation
Rajan Kumar
 
Cs6413 operating-systems-laboratory
Preeja Ravishankar
 
Vx works RTOS
Sai Malleswar
 
IJCER (www.ijceronline.com) International Journal of computational Engineeri...
ijceronline
 
ucos-2 introduction and characteristics
Varsha506533
 
ucos-2 introduction and characteristics
Varsha506533
 
Linux scheduler
Liran Ben Haim
 
Enea Enabling Real-Time in Linux Whitepaper
Enea Software AB
 
Embedded os
K Senthil Kumar
 

More from National Cheng Kung University (16)

PDF
2016 年春季嵌入式作業系統課程說明
National Cheng Kung University
 
PDF
Interpreter, Compiler, JIT from scratch
National Cheng Kung University
 
PDF
進階嵌入式作業系統設計與實做 (2015 年秋季 ) 課程說明
National Cheng Kung University
 
PDF
Construct an Efficient and Secure Microkernel for IoT
National Cheng Kung University
 
PDF
The Internals of "Hello World" Program
National Cheng Kung University
 
PDF
F9: A Secure and Efficient Microkernel Built for Deeply Embedded Systems
National Cheng Kung University
 
PDF
Open Source from Legend, Business, to Ecosystem
National Cheng Kung University
 
PDF
Summer Project: Microkernel (2013)
National Cheng Kung University
 
PDF
進階嵌入式系統開發與實作 (2013 秋季班 ) 課程說明
National Cheng Kung University
 
PDF
LLVM 總是打開你的心：從電玩模擬器看編譯器應用實例
National Cheng Kung University
 
PDF
Faults inside System Software
National Cheng Kung University
 
PDF
Hints for L4 Microkernel
National Cheng Kung University
 
PDF
Shorten Device Boot Time for Automotive IVI and Navigation Systems
National Cheng Kung University
 
PDF
Microkernel Evolution
National Cheng Kung University
 
PDF
Develop Your Own Operating System
National Cheng Kung University
 
PDF
olibc: Another C Library optimized for Embedded Linux
National Cheng Kung University
 
2016 年春季嵌入式作業系統課程說明
National Cheng Kung University
 
Interpreter, Compiler, JIT from scratch
National Cheng Kung University
 
進階嵌入式作業系統設計與實做 (2015 年秋季 ) 課程說明
National Cheng Kung University
 
Construct an Efficient and Secure Microkernel for IoT
National Cheng Kung University
 
The Internals of "Hello World" Program
National Cheng Kung University
 
F9: A Secure and Efficient Microkernel Built for Deeply Embedded Systems
National Cheng Kung University
 
Open Source from Legend, Business, to Ecosystem
National Cheng Kung University
 
Summer Project: Microkernel (2013)
National Cheng Kung University
 
進階嵌入式系統開發與實作 (2013 秋季班 ) 課程說明
National Cheng Kung University
 
LLVM 總是打開你的心：從電玩模擬器看編譯器應用實例
National Cheng Kung University
 
Faults inside System Software
National Cheng Kung University
 
Hints for L4 Microkernel
National Cheng Kung University
 
Shorten Device Boot Time for Automotive IVI and Navigation Systems
National Cheng Kung University
 
Microkernel Evolution
National Cheng Kung University
 
Develop Your Own Operating System
National Cheng Kung University
 
olibc: Another C Library optimized for Embedded Linux
National Cheng Kung University
 

Recently uploaded (20)

PDF
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
DOCX
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PDF
Encapsulation theory and applications.pdf
gurumoop
 
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
PDF
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
PDF
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PPT
Teaching material agriculture food technology
LiaRayya
 
PPT
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
PDF
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
PPTX
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
Encapsulation theory and applications.pdf
gurumoop
 
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
Teaching material agriculture food technology
LiaRayya
 
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 

Making Linux do Hard Real-time

  • 1. Making Linux do Hard Real-time Jim Huang ( 黃敬群 ) <jserv.tw@gmail.com> Jul 26, 2014
  • 2. Rights to copy Attribution – ShareAlike 3.0 You are free to copy, distribute, display, and perform the work to make derivative works to make commercial use of the work Under the following conditions Attribution. You must give the original author credit. Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one. For any reuse or distribution, you must make clear to others the license terms of this work. Any of these conditions can be waived if you get permission from the copyright holder. Your fair use and other rights are in no way affected by the above. License text: http://creativecommons.org/licenses/by-sa/3.0/legalcode Based on a work of Free Electrons © Copyright 2004-2009 © Copyright 2014 Jim Huang © Copyright 2004-2010 Free Electrons
  • 3. Agenda General Concepts Linux scheduler Interrupt Handling Latency in Linux Real-time Features in Linux RT-Preempt Linux Realtime Extension: Xenomai
  • 4. Making Linux do Hard Real-time General Concepts
  • 5. Real Time Real Time != Real Fast is determinism RTOS will deliver decent overall throughput (performance) but can sacrifice throughput for being deterministic (or predictable) Hard real-time Missing a deadline is a total system failure Soft real-time The usefulness of a result degrades after its deadline, thereby degrading the system's quality of service
  • 6. Hard Real Time A system is considered as a hard real time if it can answer to an internal or external stimulus within a given maximum amount of time. “Guaranteed worst case” Hard real time systems are used wherever failing to react in time can cause a system failure or damage, or put its users in danger. Typical examples: air traffic control vehicle subsystems control Medicine (pacemakers, etc.)
  • 7. Soft Real Time A system is considered as soft real time if it is built to react to stimuli as quickly as it can: “best effort”. However, if the system loses events or fails to process them in time, there is no catastrophic consequence on its operation. There is just a degradation in quality. Typical examples Voice over IP multimedia streaming computer games
  • 8. Standard Linux and Real Time The vanilla Linux kernel was not designed as a hard real time system: Like Unix, it is a time sharing operating system designed to maximize throughput and give a fair share of the CPU in a multi-user environment. scheduler avoids process starvation Non deterministic timing behavior of some kernel services: memory allocation, system calls... By default, processes are not preemptible when they execute system calls.
  • 9. Throughput vs. Determinism Linux as optimized for throughput rather than determinism. Linux CPUs typically utilize large, multi-layer memory caches Caches make CPUs run like a hare but, in real-time systems, the tortoise wins! CPU memory caching prevents Hard RT processes from safely utilizing more than a small fraction of the available time.
  • 10. Realtime Performance (Standard) LinuxReal-time LinuxCustomized Realtime kernel One (physical) address space with n-tasks Application program 100% source compatibility better binary compatibility API Response time: < 10 µs Response time: ~1ms Response time: < 10 µs same homogeneity One process address space with n-threads All tasks have one common API implemented by one common library All threads of a process have one common API implemented by one common library
  • 11. Jitter It means the scatter of the control period of all the tasks defined in the system The total jitter is the time difference between the minimal response time and the maximal response time
  • 12. Real-time Approaches Two major approaches real time Linux rt-preempt (PREEMPT_RT patch) Allows preemption, so minimize latencies Execute all activities (including IRQ) in “schedulable/thread” context Many of the RT patch have been merged Linux (realtime) extensions Add extra layer between hardware and the Linux kernel to manage real-time tasks separately
  • 13. Making Linux do Hard Real-time Linux Scheduler
  • 14. Linux Scheduler Decides which process to run next Allocates processor(s)’ time among run-able processes Realizes multi-tasking in a single processor machine Schedules SMP as well Affects how the system behaves, responsiveness
  • 15. Linux Scheduler Framework Kernel supports various scheduling policies by plug-in Scheduling classes contains details of the scheduling policy Each operation can be requested by the global scheduler; Allows for creating of the generic scheduler without any knowledge about different scheduler classes
  • 16. Linux Scheduler Classes SCHED_OTHER (default, non-RT) SCHED_FIFO Use FIFO real-time scheduling SCHED_RR Use round-robin real-time scheduling
  • 17. Linux Scheduler Priority Two separate priority ranges Nice value: -20 ... +19 (19 being the lowest) Real-time priority: 0 ... 99 (higher value is higher prio) ps priority: 0 ... 139 (0: lowest and 139: highest)
  • 19. Linux Scheduler Utilities How to configure priority chrt [option] ­p [prio] pid -r: SCHED_RR; -f: SCHED_FIFO sched_setscheduler() Checking priority, nice, rt-priority ps -eo pid,class,pri,nice,rtprio,comm
  • 20. Making Linux do Hard Real-time Interrupt Handling
  • 21. Interrupts (1) I/O devices can be checked if ready to transfer data by pooling (ie. continuously checking device registers) while (<Device is NOT ready for transfer data>)     do_wait(); Transfer_Data(); This operation has a busy-waiting part which consumes CPU To minimize busy-waiting, checking registers can be done by a periodic task If no data is ready to transfer the periodic task sleeps until the next period The busy waiting condition can be completely removed using interrupts
  • 22. Interrupts (2) When an I/O device is ready to transfer data it signals the CPU (issues an interrupt) The CPU interrupts the current task an calls an Interrupt Service Routine (ISR) to handle the interrupt
  • 23. Interrupts (3) The length of Linux interrupt vector is 256 32 entries are reserved to handle CPU exceptions The remaining 224 entries can point: software interrupts' handlers (like int 0x80: system calls) hardware interrupts' handlers (assigned to a IRQ line) On x86 architecture normally only 16 IRQ lines are available
  • 24. Interrupts (4) Devices are connected to two Programmable Interrupt Controllers (intel 8259x) : A maximum of 8 8259s can be connected in cascade An ISR will be executed every time an interrupt occur in the IRQ line assigned The RTC can be used to generate interrupts on IRQ 8.
  • 25. Interrupts (5) PICs can be accessed through I/O ports see how many PICs are available; IRQ lines reserved: cat /proc/ioports  cat /proc/interrupts
  • 26. Context Switching (1) The context of the interrupted task (at least the values of the CPU registers at the time of the interrupt) must be preserved, so that it can be resumed with the same state Task context can be stored in: Interrupted task local stack system stack in a set of specialized CPU registers Probably the most efficient is the last one If the CPU itself has some registers to save context, it can be done in a few cycles (or just one) Some DSPs and PowerPCs have a set of these registers
  • 27. Context Switching (2) If the ISR itself can not be interrupted any one of those 3 strategies can be used to save task context However sometimes it is needed to interrupt an ISR (to handle a fast I/O device which issues an high priority interrupt) A Programmable interrupt controller PIC can have some levels (priorities of interrupts)
  • 28. Context Switching (3) It is possible to store task context in any stack, even if several ISRs are nested is just another set of data to put on top of the stack stacks have limited capacity However CPUs have a restricted set of registers to store context, making impossible to store context of several ISRs … or even one, it depends on the CPU
  • 29. Clock Interrupts (1) In a scheduling mechanism, such as round-robin, after a time quantum or time-slice: Dispatcher removes the task currently assigned to the CPU and assigns the CPU to the first task in the ready tasks queue (switching task context)
  • 30. Clock Interrupts (2) The dispatcher can be an ISR, awaken by a periodic interrupt, generated by a timer, in each time quantum Assuming that the time-slice is fixed, the timer must generate an interrupt after a fixed number of cycles or ticks These interrupts are referred as tick-based clock interrupts
  • 31. Clock Interrupts (3) If time-slice may be different for different tasks, the timer must be programmable When a new task is assigned to the CPU, the timer is programmed with the time-slice of this task These variable time interrupts are referred as event-based clock interrupts
  • 32. Making Linux do Hard Real-time Latency in Linux
  • 33. Latency in Kernel Real time means external event should be handled within the bounded time Interrupt handler responds to the event and inform user-space process Latency Time taken from external interrupt till a user-space process to react to the interrupt
  • 34. Linux kernel latency components Typical scenario: process waiting for the completion of device I/O (signaled by an interrupt) to resume execution. Waiting task interrupt latency Interrupt handler Scheduler Running task Interrupt handler duration scheduler latency scheduler duration Process context Interrupt context Makes the task runnable kernel latency = interrupt latency + handler duration + scheduler latency + scheduler duration
  • 35. Interrupt latency Waiting task interrupt latency Interrupt handler Scheduler Running task Interrupt handler duration scheduler latency scheduler duration Makes the task runnable Time elapsed before executing the interrupt handler
  • 36. Sources of interrupt latency Time between an event happens (raising an interrupt) and the handler is called. Sources of latency: Interrupts disabled by kernel code: spinlocks, evil drivers masking out interrupts. Other interrupts processed before: Shared interrupt line: all handlers are executed (don't use shared IRQ lines for critical events) Interrupts with higher priority can preempt lower priority ones (managed by the CPU, not by the scheduler). Not a problem in a correctly designed product: you use the top priority interrupt source. Summary: the only real problem is disabled interrupts. interrupt latency Interrupt handler Interrupt handler with greater priority Linux processes handler
  • 37. Interrupt handler duration Waiting task interrupt latency Interrupt handler Scheduler Running task Interrupt handler duration scheduler latency scheduler duration Makes the task runnable Time taken to execute the interrupt handler
  • 38. Interrupt Handler Interrupt handlers are split into two parts Top-half: Process, as quickly as possible, the work during interrupt disabled, such as queue the information for the bottom- half Schedule the bottom-half bottom-half Process the required tasks from the triggered interrupt
  • 39. Kernel preemption (1) Linux kernel is a preemptive operating system When a task runs in user-space mode and gets interrupted by an interruption, if the interrupt handler wakes up another task, this task can be scheduled as soon as we return from the interrupt handler. Task A (running in user mode) Interrupt handler Wakes up Task B Task B (running in user mode) Interrupt
  • 40. Kernel preemption (2) However, when the interrupt comes while the task is executing a system call, this system call has to finish before another task can be scheduled. By default, the Linux kernel does not do kernel preemption. This means that the time before which the scheduler will be called to schedule another task is unbounded. Task A (user mode) Interrupt handler Wakes up Task B Task B (user mode) System call Task A (kernel mode) Task A (kernel mode) Interrupt ? Return from syscall
  • 41. Preemptive Kernel non-preemptive system preemptive system A concept linked to that of real time is preemption: the ability of a system to interrupt tasks at many “preemption points”.The longer the non-interruptible program units are, the longer is the waiting time (‘latency’) of a higher priority task before it can be started or resumed. GNU/Linux is “user-space preemptible”: it allows user tasks to be interrupted at any point. The job of real- time extensions is to make system calls preemptible as well.
  • 42. Interrupt handler implementation Run with interrupts disabled (at least on the current line). Hence, they need to complete and restore interrupts as soon as possible. To satisfy this requirement, interrupt handlers are often implemented in 2 parts: top half and bottom half (soft IRQ) Top half Interrupt ACK Exit Bottom half Schedule bottom half other interrupt handlers... Handle device data... Wake up waiting tasks User space... Soft IRQs are handled in interrupt context (not by the scheduler)
  • 43. Sources of interrupt handler execution time Time taken to execute your interrupt handler. Causes which can make this duration longer: Preemption by other interrupts with a greater priority. Again, not a problem if priorities are configured properly. Interrupt handler with a softirq (“bottom half”) component: run after all interrupts are serviced. => For critical processing, better not have a softirq (may require rewriting the driver if reusing an existing one). handler duration Interrupt handler Interrupt interrupt latency
  • 44. Scheduler latency Waiting task interrupt latency Interrupt handler Scheduler Running task Interrupt handler duration scheduler latency scheduler duration Makes the task runnable Time elapsed before executing the scheduler
  • 45. When is the scheduler run next? In the general case (still after a task is woken up) Upon return from interrupt context (our particular case), unless the current process is running a system call (system calls are not preemptible by default). After returning from system calls. When the current process runs an action which sleeps / waits (voluntarily or not, in kernel or user mode), or explicitly calls the scheduler.
  • 46. Sources of scheduler latency Possible sources of latency: Other interrupts coming in (even with a lower priority). Back to our case We are returning from interrupt context. The scheduler is run immediately if the current task was running in userspace. Otherwise, we have to wait for the end of the current system call. Interrupt handler Scheduler scheduler latency
  • 47. Scheduler duration Waiting task interrupt latency Interrupt handler Scheduler Running task Interrupt handler duration scheduler latency scheduler duration Makes the task runnable Time taken to execute the scheduler and switch to the new task.
  • 48. Sources of scheduler execution time Time to execute the scheduler and switch to a new task Time to execute the scheduler: depended on the number of running processes. Context switching time: time to save the state of the current process (CPU registers) and restore the new process to run. This time is constant, so if not an issue for real-time. SCHED_FIFO & SCHED_RR use bit map to find out the next task CFS (Completely Fair Scheduler) uses Red-black tree as a sorted queue Scheduler New task scheduler duration
  • 49. Other sources of latency (1) Demand paging When it starts a process, the Linux kernel first sets up the virtual address space of the program (code, stack and data). However, to improve performance and reduce memory consumption, Linux loads the corresponding pages in RAM only when they are needed. When a part of the address space is accessed (like a particular part of the code) and is not in RAM yet, a page fault is raised by the MMU. This triggers the loading of the page in RAM. This usually involves reading from disk: unexpected latencies!
  • 51. Other sources of latency (2) Linux is highly based on virtual memory provided by MMU so that memory is allocated on demand. Whenever an application accesses code or data for the first time, it is loaded on demand, which can creates huge delays. Memory allocation The Linux allocator is fast, but does not guarantee a maximum allocation time.
  • 52. Other sources of latency (3) System calls Similarly, no guaranteed response time. Device drivers System calls handlers or interrupt handlers usually not implemented with care for real-time requirements. Expect uncertainties unless you implemented all kernel support code by yourself.
  • 53. Issue: priority inversion 99 50 3 1. Low Priority task takes mutex 3. Hi Priority task block on mutex 4. Medium Priority task preempts low priority task and high priority task 2. High Priority task preempts low priority task Time TaskPriority
  • 54. Issue: priority inversion Issue: a process with more priority can preempt a process holding the lock. The top priority process could wait for a very long time. This happened with standard Linux before version 2.6.18. Acquires a lock Priority Time preempted Tries to get the same lock waits
  • 55. Solution: priority inheritance Solution: priority inheritance. The process holding the lock inherits the priority of the process waiting for the lock with the greatest priority. Acquires a lock Priority Time preempted Tries to get the lock waits Executes with the priority of the waiting process Releases the lock Acquires the lock preempted
  • 56. Issue: interrupt inversion Issue: even your top priority task can be “preempted” by any interrupt handler, even for interrupts feeding tasks with lower priority. Solution: threaded interrupts (scheduled for later execution). top priority task Any interrupt top priority task Any interrupt...
  • 57. Making Linux do Hard Real-time Linux 2.6 features for real­time Linux features for real-time
  • 58. Linux 2.6 improvements (1) Scheduler: starting from 2.6.23, Linux implements CFS. complexity of O(log N), where N is the number of tasks in the runqueue. Choosing a task can be done in constant time O(1), but reinserting a task after it has run requires O(log N) operations, because the runqueue is implemented as a red-black tree. May be built with no virtual memory support (on supported platforms). Benefits: reduced context switching time (address space shared by all processes), better performance (CPU cache still valid after switching). Full POSIX real-time API support. the standard Unix API to implement real-time applications.
  • 59. POSIX 1003.1b Real-time extensions Priority Scheduling Real-Time Signals Clocks and Timers Semaphores Message Passing Shared Memory Asynchronous and Synchronous I/O Memory Locking
  • 60. Linux 2.6 improvements (2) Threaded Interrupt Interrupt can be executed by a kernel thread, we can assign a priority to each thread RT-Mutexes RT-mutexes extend the semantics of simple mutexes by the priority inheritance protocol. BKL-free Since 2.6.37 kernel can be built completely without the use of Big Kernel Lock.
  • 61. Linux 2.6 improvements (3) Increased determinism with improved POSIX RT API support. By default, system calls still not preemptible. New CONFIG_PREEMPT option making most of the kernel code preemptible. However, even CONFIG_PREEMPT was not enough for audio users (need guaranteed latency in the milliseconds range), and of course for real-time users with stronger requirements.
  • 62. New preemption options in Linux 2.6 2 new preemption models offered by standard Linux 2.6:
  • 63. 1st option: no forced preemption CONFIG_PREEMPT_NONE Kernel code (system calls) never preempted. Default behavior in standard kernels. Best for systems making intense computations, on which overall throughput is key. Best to reduce task switching to maximize CPU and cache usage (by reducing context switching). Can also benefit from a lower timer frequency (100 Hz instead of 250 or 1000).
  • 64. 2nd option: voluntary kernel preemption CONFIG_PREEMPT_VOLUNTARY Kernel code can preempt itself Typically for desktop systems, for quicker application reaction to user input. Adds explicit rescheduling points throughout kernel code. Minor impact on throughput.
  • 65. 3rd option: preemptible kernel CONFIG_PREEMPT Most kernel code can be involuntarily preempted at any time. When a process becomes runnable, no more need to wait for kernel code (typically a system call) to return before running the scheduler. Exception: kernel critical sections (holding spinlocks). Typically for desktop or embedded systems with latency requirements in the milliseconds range. Still a relatively minor impact on throughput.
  • 66. Priority inheritance support Available since Linux 2.6.18 Implemented in kernel space locking Available in user space locking too, through fast user-space mutexes (“futex”). Priority inheritance needs to be explicited for each mutex though. See the kernel documentation for details: Documentation/pi­futex.txt Documentation/rt­mutex­design.txt
  • 67. High-resolution timers Make it possible for POSIX timers and nanosleep() to be as accurate as allowed by the hardware (typically 1 us). Together with tickless support, allow for “on-demand” timer interrupts. Available in the vanilla kernel since 2.6.21.
  • 68. Making Linux do Hard Real-time Linux rt-preempt
  • 69. Linux real-time preemption http://www.kernel.org/pub/linux/kernel/projects/rt/ led by kernel developers including Ingo Molnar, Thomas Gleixner, and Steven Rostedt Large testing efforts at RedHat, IBM, OSADL, Linutronix Goal is to improve real time performance Configurable in the Processor type and features (x86), Kernel  Features (arm) or Platform options (ppc)...
  • 70. Wrong ideas about real-time preemption Myth (1): It will improve throughput and overall performance Wrong: it will degrade overall performance. Myth (2): It will reduce latency Often wrong. The maximum latency will be reduced. The primary goal is to make the system predictable and deterministic.
  • 71. Current issues and limitations Linux RT patches not mainstream yet. Facing resistance to retain the general purpose nature of Linux. Though they are preemptible, kernel services (such as memory allocation) do not have a guaranteed latency yet! However, not really a problem: real-time applications should allocate all their resources ahead of time. Kernel drivers are not developed for real-time constraints yet (e.g. USB or PCI bus stacks) Binary-only drivers have to be recompiled for RT preempt Memory management (SLOB/SLUB allocator) still runs with preemption disabled for long periods of time Lock around per-CPU data accesses, eliminating the preemption problem but wrecking scalability
  • 72. PREEMPT_RT: complete RT preemption Replace non-preemptible constructs with preemptible ones Make OS preemptible as much as possible except preempt_disable and interrupt disable Make Threaded (schedulable) IRQs so that it can be scheduled spinlocks converted to mutexes (a.k.a. sleeping spinlocks) Not disabling interrupt and allows preemption Works well with threaded interrupts
  • 73. Toward complete RT preemption Most important aspects of Real-time Controlling latency by allowing kernel to be preemptible everywhere
  • 74. original Linux Kernel User Space User Context Interrupt Handlers Kernel Space Interrupt Context SoftIRQs Hiprio tasklets Network Stack Timers Regular tasklets ... Scheduling Points Process Thread Kernel Thread
  • 75. PREEMPT_RT User Space SO_NODELAY Interrupt Handlers Kernel Space NetworkStack Timers Tasklets Scheduling Points Process Thread Kernel Threads
  • 76. Interface changed by PREEMPT_RT spinlocks and local_irq_save() no longer disable hardware interrupts. spinlocks no longer disable preemption. Raw_ variants for spinlocks and local_irq_save() preserve original meaning for SO_NODELAY interrupts. Semaphores and spinlocks employ priority inheritance
  • 77. Threaded Interrupts Interrupt handlers run in normal kernel threads Priorities can be configured Main interrupt handler Do minimal work and wake-up the corresponding thread Thread interrupts allows to use sleeping spinlocks in PREEMPT_RT, all interrupt handlers are switched to threaded interrupt drivers in mainline get gradually converted to the new threaded interrupt API that has been merged in Linux 2.6.30.
  • 78. Threaded Interrupts The vanilla kernel Interrupts as threads Real world behavior
  • 79. Threaded Interrupt handler Almost all kernel space is now preemptible An interrupt can occur at any time The woken up processes by interrupt can start immediately Treaded IRQs: Kernel thread per ISR Priority must be set Interrupt handler threads Softirq threads Other kernel threads Real time application processes/threads
  • 80. Timer Frequency Linux timer interrupts are raised every HZ of second. Resolution of the system timer (HZ) 100 HZ to 1000 HZ depends on arch and configuration Resolution of 10 ms to 1 ms Adds overheads → Compromise between system responsiveness and global throughput.
  • 81. Effect of Timer Frequency 10 20 10 20 15 Requested sleep time, in ms HZ=100 Real sleep time, in ms
  • 82. Effect of Timer Frequency 10 20 10 20 15 Requested sleep time, in ms Real sleep time, in ms HZ=1000
  • 83. High Resolution Timers Use non-RTC interrupt timer sources to deliver timer interrupt between ticks. Allows POSIX timers and nanosleep() to be as accurate as the hardware allows This feature is transparent. When enabled it makes these timers much more accurate than the current HZ resolution. Around 1usec on typical hardware.
  • 84. Use POSIX Timer for Periodical Task const int NSEC_IN_SEC = 1000000000l, INTERVAL = 1000000l; struct timespec timeout; clock_gettime(CLOCK_MONOTONIC, &timeout); while (1) {     do_some_work(&some_data);     timeout.tv_nsec += INTERVAL;     if (timeout.tv_nsec >= NSEC_IN_SEC) {         timeout.tv_nsec ­= NSEC_IN_SEC;         timeout.tv_sec++;     }     clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &timeout, NULL); }
  • 85. Making Linux do Hard Real-time Implementing real-time constraints
  • 86. Process vs. Thread In Unix, a process is created using fork() and is composed of An address space, which contains the program code, data, stack, shared libraries, etc. One thread, that starts executing the main() function. Upon creation, a process contains one thread Additional threads can be created inside an existing process, using pthread_create() They run in the same address space as the initial thread of the process They start executing a function passed as argument to pthread_create()
  • 87. Process vs. Thread: Kernel point of view Kernel represents each thread running in the system by a structure of type task_struct From a scheduling point of view, it makes no difference between the initial thread of a process and all additional threads created dynamically using pthread_create() Address space Thread A Process after fork() Address space Thread A Thread B Same process after pthread_create()
  • 88. Creating threads Linux support the POSIX thread API To create a new thread pthread_create(pthread_t *thread, pthread_attr_t *attr, void *(*routine)(*void*), void *arg); The new thread will run in the same address space, but will be scheduled independently Exiting from a thread pthread_exit(void *value_ptr); Waiting for the termination of a thread pthread_join(pthread_t *thread, void **value_ptr);
  • 89. Scheduler policy SCHED_FIFO: the process runs until completion unless it is blocked by an I/O, voluntarily relinquishes the CPU, or is preempted by a higher priority process. SCHED_RR: the processes are scheduled in a Round Robin way. Each process is run until it exhausts a max time quantum. Then other processes with the same priority are run, and so and so... SCHED_BATCH: Does not preempt nearly as often as regular tasks would, thereby allowing tasks to run longer and make better use of caches but at the cost of interactivity. This is well suited for batch jobs. SCHED_IDLE: Lowest priority in the system SCHED_NORMAL: Used for regular processes with non-RT priority.
  • 90. Scheduling classes (1) sched_setscheduler() API can be used to change the scheduling class and priority of a process int sched_setscheduler(pid_t pid, int  policy, const struct sched_param *param); policy can be SCHED_OTHER, SCHED_FIFO,  SCHED_RR, etc. param is a structure containing the priority
  • 91. Scheduling classes (2) Priority can be set on a per-thread basis when its creation: Then the thread can be created using pthread_create(), passing the attr structure. Several other attributes can be defined this way: stack size struct sched_param parm; pthread_attr_t attr; pthread_attr_init(&attr); pthread_attr_setinheritsched(&attr,                              PTHREAD_EXPLICIT_SCHED); pthread_attr_setschedpolicy(&attr, SCHED_FIFO); parm.sched_priority = 42; pthread_attr_setschedparam(&attr, &parm);
  • 92. How fork() seems to work Parent Process Parent Process Child Process Copy A complete copy is created of the parent. Both parent and child continue execution from the same spot.
  • 93. How fork() really works Parent Process Parent Process Child Process Memory RW Memory RO RO Child process gets a copy of stack and file descriptors. Copy On Write reference is used for the memory.
  • 94. What happens during write? Parent Process Child Process Original Memory RO RO When write is attempted, a single memory page is copied and references updated. This is called: “breaking the CoW”. RW
  • 95. Locking pages in RAM A solution to latency issues with demand paging! Available through <sys/mman.h> (see man mman.h) mlock Lock a given region of process address space in RAM. Makes sure that this region is always loaded in RAM. mlockall Lock the whole process address space. munlock, munlockall Unlock a part or all of the process address space.
  • 97. Mutexes Allows mutual exclusion between two threads in the same address space Initialization/destruction pthread_mutex_init(pthread_mutex_t *mutex,  const pthread_mutexattr_t *mutexattr); pthread_mutex_destroy(pthread_mutex_t *mutex); Lock/unlock pthread_mutex_lock(pthread_mutex_t *mutex); pthread_mutex_unlock(pthread_mutex_t *mutex); Priority inheritance must be activated explicitly pthread_mutexattr_t attr; pthread_mutexattr_init (&attr); pthread_mutexattr_getprotocol(&attr,PTHREAD_PRIO_INHERIT);
  • 98. Timers timer_create(clockid_t clockid, struct sigevent *evp, timer_t *timerid) Create a timer. clockid is usually CLOCK_MONOTONIC. sigevent defines what happens upon timer expiration: send a signal or start a function in a new thread. timerid is the returned timer identifier. timer_settime(timer_t timerid, int flags, struct itimerspec *newvalue, struct itimerspec *oldvalue) Configures the timer for expiration at a given time. timer_delete(timer_t timerid), delete a timer clock_getres(), get the resolution of a clock Other functions: timer_getoverrun(), timer_gettime()
  • 99. Signals Signals are asynchronous notification mechanisms Notification occurs either By the call of a signal handler. Be careful with the limitations of signal handlers! By being unblocked from the sigwait(), sigtimedwait() or sigwaitinfo() functions. Usually better. Signal behaviour can be configured using sigaction() The mask of blocked signals can be changed with pthread_sigmask() Delivery of a signal using pthread_kill() or tgkill()
  • 100. Benchmarking (1) cyclictest measuring accuracy of sleep and wake operations of highly prioritized realtime threads https://rt.wiki.kernel.org/index.php/Cyclictest vanilla kernel PREEMPT_RT
  • 101. Benchmarking (2) QA farm running at OSADL.ORG worst case latency comparison between 2.6.32 and 2.6.33-rt https://www.osadl.org/Compare-systems.qa-farm-compare-latency.0.html Worst case latency in non-rt is over 4,000 usec, in rt around 100 usec
  • 102. Again, Response time vs. Throughput Overhead for real-time preemption Mutex instead of spin lock Priority inheritance Threaded interrupt handler Due to overhead of real-time preemption Throughput is reduced Due to the flexibility of preemption Better worst case latency
  • 103. Making Linux do Hard Real-time Linux hard real-time extensions
  • 104. Linux hard real-time extensions Three generations RTLinux RTAI Xenomai A common principle Add a extra layer between the hardware and the Linux kernel, to manage real-time tasks separately. Hardware Micro/Nano-kernel Linux kernel real­time tasks real-time tasks
  • 105. Interrupt Response Time ConfigurationConfiguration AvgAvg MaxMax MinMin XENOMAIXENOMAI 4343 5858 22 PREEMPT 1st run PREEMPT 1st run 8888 415415 2727 PREEMPT 2nd run PREEMPT 2nd run 8181 18291829 1313 Hardware: Freescale i.MX53 ARM Cortex-A8 processor operating at 1GHz. Time in micro second. PREEMPT: standard kernel with CONFIG_PREEMPT (“Preemptible Kernel (Low-Latency Desktop)) enabled cyclictest –m ­n ­p99 ­t1 ­i10000  ­1360000 XENOMAI: Kernel + Xenomai 2.6.0-rc4 + I-Pipe 1.18-03 cyclictest ­n ­p99 ­t1 ­i10000  ­1360000
  • 106. Xenomai project http://www.xenomai.org/ Started in the RTAI project (called RTAI / fusion). Skins mimicking the APIs of traditional RTOS such as VxWorks, pSOS+, and VRTXsa. Initial goals: facilitate the porting of programs from traditional RTOS to RTAI on GNU / Linux. Now an independent project and an alternative to RTAI. Many contributors left RTAI for Xenomai, frustrated by its goals and development style.
  • 107. Xenomai architecture Adeos i-pipe Xenomai RTOS (nucleus) VxWorks application glibc Xenomai libvxworks POSIX application glibc Xenomai libpthread_rt Linux application glibc VFS Network Memory ... System calls Linux kernel space Pieces added by Xenomai Xenomai skins ipipe = interrupt pipeline
  • 108. Linux vs. Xenomai A special way of passing of the interrupts between real-time kernel and the Linux kernel is needed. Each flavor of RT Linux does this is in its own way. Xenomai uses an interrupt pipeline from the Adeos project.
  • 109. From the Adeos point of view, guest OSes are prioritized domains. For each event (interrupts, exceptions, syscalls, ...), the various domains may handle the event or pass it down the pipeline. Interrupt pipeline abstraction
  • 110. Main idea support interrupt priority ensure that domains support segments of code that can not be interrupted (like critical sections) Adeos implements the Optimistic protection scheme (Stodolsky, 1993): When a segment of code is executed in a domain with interrupts disabled, that stage of the pipeline stalls, so interrupts can not be serviced nor propagated further in the pipeline i-pipe: Optimistic protection scheme (1)
  • 111. If a real time domain (like Xenomai) has higher priority it is the first in the pipeline It will receive interrupt notification first without delay (or at least with predictable latency) Then it can be decided if interrupts are propagated to low priority domains (like Linux) or not i-pipe: Optimistic protection scheme (2)
  • 112. The high priority domain is at the beginning of the pipeline, so events are delivered first to it This pipeline is referred as interrupt pipeline or I-pipe There is a pipeline for each CPU Interrupt pipeline (1)
  • 113. The Linux domain is always the root domain, whatever is its position in the pipeline Other domains are started by the root domain Linux starts and loads the kernel modules that implement other domains Interrupt pipeline (2)
  • 114. virtualized interrupts disabling Each domain may be “stalled”, meaning that it does not accept interrupts. Hardware interrupts are not disabled however (except for the domain leading the pipeline), instead the interrupts received during that time are logged and replayed when the domain is unstalled.
  • 115. I-pipe observation through procfs $ cat /proc/ipipe/Xenomai        +­­­­­ Handling ([A]ccepted, [G]rabbed,        |+­­­­ Sticky    [W]ired, [D]iscarded)        ||+­­­ Locked        |||+­­ Exclusive        ||||+­ Virtual [IRQ]  38: W..X. 418: W...V $ cat /proc/ipipe/Xenomai        +­­­­­ Handling ([A]ccepted, [G]rabbed,        |+­­­­ Sticky    [W]ired, [D]iscarded)        ||+­­­ Locked        |||+­­ Exclusive        ||||+­ Virtual [IRQ]  38: W..X. 418: W...V $ cat /proc/ipipe/Linux 0: A.... 1: A.... ... priority=100 $ cat /proc/ipipe/Linux 0: A.... 1: A.... ... priority=100
  • 116. Linux options for Xenomai configuration The POSIX skin allows to recompile without changes a traditional POSIX application so that instead of using Linux real-time services, it uses Xenomai services. The POSIX skin allows to recompile without changes a traditional POSIX application so that instead of using Linux real-time services, it uses Xenomai services.
  • 117. Xenomai Execution Model (1) All tasks managed by Xenomai are known by the real time nucleus: Tasks (Xenomai tasks or threads) executed in the Xenomai domain are executed in primary mode (or primary domain) [the highest priority domain; real time domain] Tasks (Linux threads or processes) executed in the Linux domain are executed in secondary mode (or secondary domain)
  • 118. Xenomai Execution Model (2) To ensure that tasks in secondary domain still have real time support, it is required: (A) Common priority scheme When a Xenomai task migrates to Linux domain, the root thread's mutable priority technique is applied: Linux kernel inherits the priority of the Xenomai thread that migrates
  • 119. Xenomai Execution Model (3) Common priority scheme This means that other threads in the Xenomai Domain can only preempt the migrated thread (ie. the linux kernel executing the syscall) if their priority is higher Note: This is the opposite with RTAI (Real-Time Application Interface): A thread that enters the Linux Domain inherits the priority of the Linux kernel, which is the lowest in the RTAI scheduler Note: This is the opposite with RTAI (Real-Time Application Interface): A thread that enters the Linux Domain inherits the priority of the Linux kernel, which is the lowest in the RTAI scheduler
  • 120. Xenomai Execution Model (4) To ensure that tasks in secondary domain still have real time support, it is required: (B) Predictability of program execution time When a Xenomai thread enters secondary domain (Linux) it should not be preempted by: Linux domain interrupts other low priority asynchronous activity at Linux kernel level Note that it is assumed that when an interrupt is delivered to Linux domain, Linux Kernel code can be preempted whatever its priority To avoid this, Linux kernel can be starved from interrupts while the Xenomai thread is in secondary mode
  • 121. Xenomai Execution Model (5) To ensure that a Xenomai tasks in secondary domain have still real time support (and predictable execution time ) While a Xenomai task migrates to Linux domain (secondary): Linux kernel inherits the priority of the Xenomai task that migrates (normal Linux kernel priority is the lowest: 0) Linux is starved from interrupts and other system events, which are blocked in an intermediate domain: The interrupt shield (to avoid that the Linux kernel interrupts the migrated Xenomai task when responding to an interrupt or other asynchronous event)
  • 122. Adeos additional features The Adeos I-pipe patch implement additional features, essential for the implementation of the Xenomai real-time extension: Disables on-demand mapping of kernel-space vmalloc/ioremap areas. Disables copy-on-write when real-time processes are forking. Allow subscribing to event allowing to follow progress of the Linux kernel, such as Linux system calls, context switches, process destructions, POSIX signals, FPU faults. On ARMv5 architecture, utilizes FCSE to reduce the latency induced by cache flushes during context switches.
  • 123. Real-Time Scheduler Xenomai extends the Linux kernel and is integrated as part of OS. A task with a period = 15 us, shown in light blue. While this real-time task is not being executed, Xenomai invokes the regular Linux scheduler which executes tasks as normal, shown in dark green. LinuxLinux XenomaiXenomai struct task_struct struct xnthread mapped
  • 124. Xenomai features Factored RT core with skins implementing various RT APIs Seamless support for hard real-time in user-space No second-class citizen, all ports are equivalent feature- wise Each Xenomai branch has a stable user/kernel ABI Timer system based on hardware high-resolution timers Per-skin time base which may be periodic RTDM skin allowing to write real-time drivers
  • 125. Two modes are defined for a thread the primary mode, where the thread is handled by Xenomai scheduler the secondary mode, when it is handled by Linux scheduler. Thanks to the services of the Adeos I-pipe service, Xenomai system calls are defined. A thread migrates from secondary mode to primary mode when such a system call is issued It migrates from primary mode to secondary mode when a Linux system call is issued, or to handle gracefully exceptional events such as exceptions or Linux signals. Xenomai real-time user-space support
  • 126. Original Linux User space Kernel space Low prio high prio Linuxscheduler main Linux thread pthread_create Runs in User Mode : can only address user space Linux syscall readfile(..), printf(..) Kernel moduleKernel module Kernel module Linux Kernel Communicates with terminal to display text of printf filesystem module communicates with harddisk to read the file Kernel module
  • 127. Xenomai (kernel space) User space Kernel space Low prio high prio not realtime hard realtime Linuxscheduler Real-time scheduler RT task Kernel module init_module() rt_task_start Xenomai Kernel function call Linux Kernel Shell : # insmod <module> Runs in Kernel Mode : can address User and Kernel space
  • 128. Xenomai (user space) User space Kernel space Low prio high prio Linuxscheduler Real-time scheduler FIFO Sched other main Linux thread rt_thread_create Xenomai task Xenomai task rt_task_create, rt_task_start Linux syscall Runs in User Mode : can only address user space Xenomai Kernel xenomai syscall Linux Kernel Linux syscall hard realtime soft realtime
  • 129. Concepts changed in Xenomai Linux kernel realtime kernel threads controlled by realtime scheduler threads controlled by Linux-Scheduler system call entry exactly the same for both kernels Ipipe (ADEOS) Common Library Linux Process POSIX API HW- Interrupts Soft- Interrupts static priorities (POSIX-counting) 99 0 99 0 realtime kernelLinux Linux + Realtime boundary implementation defined Linux
  • 130. Real-time Linux task Observe the difference between real-time tasks and standard Linux programs: In real-time Linux, we can make a real-time program by programming real-time threads. None-real-time programs in real-time Linux just use the conventional Linux threads. A real-time task can be executed in kernel space using a kernel module, but it can also be executed in user space using a normal C program.
  • 131. Real-time tasks in Linux userspace Running in user space instead of in kernel space gives a little extra overhead, but with the following advantages: Bugs in a user space program can only crash the program. In a kernel model, one can only use the real-time API and the limited kernel API, but in a real-time user space program the real-time API and the whole Linux API can be used. However when using the Linux API in user space, the program cannot be scheduled by the real-time scheduler (HARD real-time) but must be scheduled by the Linux scheduler (SOFT real-time). So calling a Linux system call from a real-time user space task will degrade its performance from HARD to SOFT real-time. After the call the task will return to the real-time scheduler.
  • 132. RTDM Real-Time Driver Model An approach to unify the interfaces for developing device drivers and associated applications under real-time Linux. Currently available for Xenomai and RTAI RTDM-native: a port over native Linux with the real-time preemption patch. See the whitepaper on http://www.xenomai.org/documentation/xenomai-2.6/pdf/RTDM-and-Applications.pdf
  • 133. The POSIX skin POSIX skin allows to recompile without changes a traditional POSIX application so that instead of using Linux real-time services, it uses Xenomai services Clocks and timers, condition variables, message queues, mutexes, semaphores, shared memory, signals, thread management Good for existing code or programmers familiar with the POSIX API Of course, if the application uses any Linux service that isn't available in Xenomai, it will switch back to secondary mode To link an application against the POSIX skin Export DESTDIR=/path/to/xenomai/ CFLAGS=`$DESTDIR/bin/xeno­config ­­posix­cflags` LDFLAGS=`$DESTDIR/bin/xeno­config ­­posix­ldflags` {CROSS_COMPILE}gcc ${CFLAGS} ­o rttest rttest.c ${LDFLAGS}
  • 134. Communication with a normal task If a Xenomai real-time application using the POSIX skin wishes to communicate with a separate non-real-time application, it must use the rtipc mechanism In Xenomai application, create an IPCPROTO_XDDP socket socket(AF_RTIPC, SOCK_DGRAM, IPCPROTO_XDDP); setsockopt(s, SOL_RTIPC, XDDP_SETLOCALPOOL,&poolsz,sizeof(poolsz)); memset(&saddr, 0, sizeof(saddr)); saddr.sipc_family = AF_RTIPC; saddr.sipc_port = MYAPPIDENTIFIER; ret = bind(s, (struct sockaddr *)&saddr, sizeof(saddr)); And then the normal socket API sendto() / recvfrom() In the Linux application Open /dev/rtpX, where X is the XDDP port Use read() and write()
  • 135. Conclusion Linux was not designed as a RTOS However, you can make your system hard real-time by using one of the hard real-time extensions (e.g. Xenomai), and writing your critical applications and drivers with the corresponding APIs. You can get soft real-time with the standard kernel preemption mode. Most of the latencies will be reduced, offering better quality, but probably not all of them. However, using hard real-time extensions will not guarantee that your system is hard real-time. Your system and applications will also have to be designed properly (correct priorities, use of deterministic APIs, allocation of critical resources ahead of time...).
  • 136. Reference Soft, Hard and Hard Real Time Approaches with Linux, Gilad Ben- Yossef A nice coverage of Xenomai (Philippe Gerum) and the RT patch (Steven Rostedt):http://oreilly.com/catalog/9780596529680/ Real-time Linux, Insop Song Understanding the Latest Open-Source Implementations of Real- Time Linux for Embedded Processors, Michael Roeder