Four Ways to Improve Linux Performance IEEE Webinar, R2.0
- 1. 4 Ways to Improve Linux Performance Michael Christofferson Director Product Marketing, Enea IEEE/Enea Webinar July 23, 2013
- 2. FOUNDED 1968 TEN OFFICES IN NORTH AMERICA, EUROPE AND ASIA REVENUE 67M USD NO. OF EMPLOYEES 426 Increasing data traffic in communication devices require new and innovative software solutions to handle bandwidth, performance and power requirements. Enea software is heavily used in wireless Infrastructure (Macro, small cell), gateway, terminal, military, auto, etc. More than 250M of the 325M LTE population coverage is powered by Enea Solutions Enea Solutions run in more than 50% of the world’s 8.2M radio base stations. Enea has recently released its first commercial Linux distribution, built by Yocto, and specially tailored for networking and communications Global presence, global development, and headquartered in Stockholm, Sweden Enea - Powering Communications Numbers for 2011
- 3. FOUNDED 1968 Overview of four approaches to enhancement of standard Linux performance in embedded multicore devices. Linux PREEMPT_RT CONFIG Patch Set Enea LWRT Open Event Machine Hypervisors or “thin kernel” solutions Relative performance comparisons, as well as other metrics that reflect “Pros and Cons” of each approach Agenda
- 4. Many measures of “performance” •Real-time Responsiveness – In embedded, often linked with the concept of “deterministic” response – But not always!! …. See next slide •Throughput – Discreet event processing bandwidth or rates – Does not necessarily mean short or even deterministic real-time response •High Performance Computing – Massive compute intensive applications like modeling and simulation, and mathematical related computations – Not the same as throughput What Does Performance Mean? => For embedded, it’s about Real-time response and Throughput
- 5. • Real-time systems – Have “operational deadlines from event to system response” – Must guarantee the response to external events within strict time constraints • Non-real-time systems – Cannot guarantee response time in any situation – Are often optimized for best-effort, high throughput performance • “Real-time response” means deterministic response – Can mean seconds, milliseconds, microseconds. – I.e. not necessarily short times, but usually this is the case • Real-time system classifications: – Hard: missing a deadline means total system failure – Firm: infrequent misses are tolerable, but result is useless. QoS degrades quickly – Soft: infrequent misses are tolerable, increased frequency degrades QoS more slowly => Real-time performance OFTEN is contradictory to Throughput!! What Does “Real-time” Performance Mean?
- 6. Examples of real-time systems • Hard real-time applications: – Automotive: anti-lock brakes, car engine control – Medical: heart pacemakers – Industrial: process controllers, robot control Throughput NOT an issue • Firm real-time applications: – 3G/4G baseband processing/signaling in base stations and radio network controllers – 3G/4G baseband processing/signaling in wireless modems (phones, tablets) – Many other examples in the networking space – RRU, optical transport, backhaul, too numerous to list Throughput is often an issue • Soft real-time applications: – IP network control signaling, network servers – Live audio-video systems on the edge or in data centers Throughput with “good enough” real time response IS the issue
- 7. Four Ways for Better Performance in Linux: Linux KernelLinux Kernel Vertically partition Linux in two domains: Linux KernelLinux Kernel Linux KernelLinux Kernel Add a thin real-time kernel underneath Linux: Rework the internals of Linux: Realtime KernelRealtime Kernel RT Runtime- LWRTRT Runtime- LWRT The PREEMPT_RT patch “Thin-kernel” or virtualization Vertical Partitioning + User mode Runtime RT appsRT apps Event Machine Partition Linux in two domains:, one not running Linux at all Linux KernelLinux Kernel Event MachineEvent Machine
- 9. What Problem is PREEMPT_RT Trying to Solve? Minimize Linux Interrupt Processing Delays from external event to response External Interrupt Triggered Interrupt Taken Interrupt Received in User/Thread Context Critical section with interrupts disabled HW Exception “Top Half” / ISR Exit from IRQ Reschedule Context Switch Something else is executing (probably another ISR) E.g. locks (xtime lock could be one example?) Softirqs, RCUs Priority inversion/ conflict Cache misses, etc. Signal/ Wakeup Locks, RCUs, etc. Resource Conflicts
- 10. The CONFIG_PREEMPT_RT patch set • Started 10+ years ago – Before multicore evolution; uni-core optimized technology – Many other contributors since then • Replaces most kernel spinlocks with mutexes with priority inheritance • Moves most interrupt handling to kernel threads – This means many drivers must be modified • Roughly, PREEMPT_RT patches 500+ locations in the kernel, with 11,500+ new lines of code in total. • In a multicore device, is “system wide in scope” Improves real-time performance (interrupt latency) but AT THE EXPENSE of throughput
- 11. PREEMPT_RT Throughput/RT Tradeoff A Very Simple Example Linux 3.6.4: # netperf -H localhost -t TCP_STREAM -A 16K,16K -l 120 -C -D 20 Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % U % S us/KB us/KB 87380 16384 16384 120.00 8782.10 -1.00 84.81 -1.000 1.582 Linux 3.6.4-rt10 (PREEMPT_RT): # netperf -H localhost -t TCP_STREAM -A 16K,16K -l 120 -C -D 20 Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % U % S us/KB us/KB 87380 16384 16384 120.00 4185.48 -1.00 70.21 -1.000 2.748 But this is a simple example that doesn’t always apply
- 12. Other CONFIG_PREEMPT_RT Characteristics • ALL Linux Solution – API’s / programming paradigm – Including all tools – BUT!! Requires driver modifications for all drivers • Compatible with Core Isolation/Shielding techniques – Can work reasonably well for both real-time and throughput in a “bare metal” environment, i.e. no multithreading on isolated cores • Linux SMP style load balancing, for what it’s worth • Standard Linux memory protection • Standard Linux Power Management
- 13. LWRT
- 14. LWRT and the Vertical Partitioning Concept • Partitioning of the system into separate real-time critical (shielded cores) an non-critical domains. • It is often the Linux kernel itself that introduces real-time problems. • Real-time partition does not need full POSIX/Linux API • A combination of partitioning, combined with a user-mode environment that avoids using the kernel can improve performance and real-time characteristics compared to a standard Linux. “Improve performance and realtime characteristics under Linux by partitioning the system into logical domains, and by avoiding usage of the Linux kernel and its resources more than necessary”
- 15. The LWRT Vertical Partitioning Concept (2) • Configure processes and interrupts to run with core affinity • Make minor modifications to the kernel to avoid running unnecessary kernel threads/timers on real-time cores • Avoid using/calling the kernel, and rely on a user-mode execution runtime environment Use Cases: a.When targeting interrupt latency at a 3-10 us average and 15-30 us worst case requirements b.When the application requires multi-threading performance
- 16. How does LWRT work? PthreadPthread Core 0 Core 0 Core N Core N Linux KernelLinux Kernel PthreadPthread LWRT EnvironmentLWRT Environment LWRT Kernel Module Realtime ProcessesRealtime ProcessesNon-realtime ProcessesNon-realtime Processes LWRT partitions the system into one realtime domain and one non-realtime domain. LWRT adds a user-mode runtime environment, including an optimized user-mode scheduler. LWRT adds a kernel module to catch and forward interrupts to the user-mode environment. LWRT migrates some specific kernel functionality (e.g. timers) away from the realtime domain.
- 17. What are the benefits of LWRT? PthreadPthread Core 0 Core 0 Core N Core N Linux KernelLinux Kernel PthreadPthread LWRT EnvironmentLWRT Environment LWRT Kernel Module Realtime ProcessesRealtime ProcessesNon-realtime ProcessesNon-realtime Processes LWRT provides a solution that is unencumbered by GPL, even for interrupt driven code which can be placed in user-space without any major penalty. LWRT provides very good (i.e. low-latency) interrupt response time, all the way up to user-mode. LWRT provides low latency and high throughput. LWRT does not depend on the PREEMPT_RT patch, and does not affect throughput negatively. LWRT provides optimized APIs for realtime applications, and allows the same application to use the POSIX/Linux APIs when realtime doesn’t matter. LWRT is an “all-Linux” solution, based on a single Linux Kernel. Thus, almost all tools from the existing Linux ecosystem will be available.
- 19. 0 20000 40000 60000 80000 100000 0 500 1000 1500 2000 2500 3000 3500 4000 4500 pthreads Enea’s User-Space Linux Executive Much better performance i.e. lower scheduling latency Much better real-time characteristics, i.e. less variance. Clock cycles (lower is better) Number of samples measured (ideally a single peak) Scheduling Latency – LWRT vs Pthreads
- 22. Throughput ≈ “Idle” Time Based on a Real-world LTE Example 0 500 μs 1000 μs 2000 μs1500 μs Cell N Cell 1 Cell 0 “Idle” Time In our example: “Theoretical” maximum for a system with infinitesimally little overhead is 400 μs In our example: “Theoretical” maximum for a system with infinitesimally little overhead is 400 μs
- 24. Other LWRT Characteristics • NOT ALL Linux Solution – Different API’s, programming paradigm – Does include all Linux tools, except for LWRT thread awareness – BUT, doesn’t require standard Linux driver modification • Depends on Core Isolation/Shielding • Slightly better real-time response/determinism than PREEMPT_RT – Interrupt handling model “cleaner” • Better than PREEMPT_RT for Throughput – But only if Multithreading in the application is necessary – Not for bare metal • No load balancing – the current vertical partitioning concept prohibits it • No memory protection between threading environments on a core – Best implementation requires ONE pthread per core • Not standard Linux Power Management
- 26. What does Event Machine Look Like? EMEM Core 0 Core 0 Core N Core N Linux KernelLinux Kernel EMEM EMEM EMEM EM needs a “dispatcher”EM needs a “dispatcher” EM Scheduler Realtime ApplicationsRealtime ApplicationsNon-realtime ProcessesNon-realtime Processes EM partitions the system into one realtime domain and one non-realtime domain, like LWRT. EM is a run-to-completion model for individual “contextless” work packages. NO threading or OS model . EM does not necessary need a special interrupt handling model. Needs a “scheduler” in either Linux partition OR in HW EM does not require kernel mods, nor core isolation, but it can use core shielding, i.e. non-essential Linux processes and interrupts are migrated away from the EM cores.
- 27. Event Machine • An efficient (low overhead) execution model for data plane processing. • An “event” based programming paradigm, replacing traditional threads and processes. – “Events are data associated with code – Run-to-completion model code. This means “context-less” or “state-less” code for processing • New “first class” OS primitives: queues, events, execution objects. – Can work within an RTOS environment!! See next slide • A framework for distribution and scheduling in multicore scenario. • A standardized API. • HW offloading friendly API. EOX Scheduler Core/Thread 1 EOX EOY Core/Thread N EOX EOY DispatcherDispatcher
- 28. Push versus Pull Models • Pull model – Simple design – Passive loadbalancing. – Offload a majority of scheduling decisions to HW – Core hot-plug(powersave) easier to implement. – Cache cold problems on MIMO/SIMO queues. • Push model – Cache prefetching can be improved. – Active load balancing protocols needed. – Offloading scheduling decisions to I/O co/processor ? i.e. smart HW queues. • Push/Pull – Pull whenever HW can schedule I/O. – Keep it simple. EOX Scheduler Core/Thread 1 EOX EOY Core/Thread N EOX EOY DispatcherDispatcher
- 29. Priority Processes Interrupt Processes Event Scheduling (in scheduler idle) OS + Event Machine Scheduling Model Preemption Background Jobs
- 30. Other Event Machine Characteristics • NOT ALL Linux Solution – Different API’s, programming paradigm on EM cores • This means tools as well – Requires restructuring code into simple, non-preemptive, run- to-completion models …. “Context-less” processing • Depends on Core Isolation/Shielding • Superior for max data plane THROUGHPUT • Real-time response is not part of the equation – Time to process events is not a parameter – But it “could” result in good real-time response depending on use case • Designed for best load balancing on the data plane • No memory protection EM instances on cores • Not standard Linux Power Management – But not a hard problem to solve in a “Pull” model
- 32. Virtualizes Linux Examples includes hypervisor, Xenomai, RTLinux etc Provides a highly deterministic RTOS environment for RT apps Cannot completely utilize the Linux eco-system (e.g. tools) in the realtime domain. Suitable for very high real-time requirements, inherited from classic RTOS domains CPU 0 Multicore SoC RT OS CPU 1 CPU 2 Tools CPU 3 CPU 4 CPU 5 Virtual Machine Bare Metal OR RT Apps Data Plane fast path application CPU 6 CPU 7 Linux Real Time Virtualization Solution
- 33. Hypervisor Characteristics • NOT ALL Linux Solution – Different API’s, programming paradigm for real-time cores • This means tools as well • Superior real-time response • Excellent THROUGHPUT • Memory protection across cores But best use case is for legacy migration or consolidation. Hypervisors really not discussed too much anymore in the embedded industry
- 34. Four Ways for Better Performance in Linux: Linux KernelLinux Kernel Vertically partition Linux in two domains: Linux KernelLinux Kernel Linux KernelLinux Kernel Add a thin real-time kernel underneath Linux: Rework the internals of Linux: Realtime KernelRealtime Kernel RT Runtime- LWRTRT Runtime- LWRT The PREEMPT_RT patch “Thin-kernel” or virtualization Vertical Partitioning + User mode Runtime RT appsRT apps Event Machine Partition Linux in two domains:, one not running Linux at all Linux KernelLinux Kernel Event MachineEvent Machine Enea supports PREEMPT_RT, Virtualization, LWRT Event Machine with Linux is a research topic
- 35. Thank You Visit us at enea.com