Is Linux ready for safety related applications?

2018-10-23 | ETAS Connections 2018 | Public | © Elektrobit Automotive GmbH 2018 | All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
2018-10-23, ETAS Connections 2018, Stuttgart, Germany
Alexander Much, Alexander Mattausch
Is Linux ready for
safety related applications?

2
Interesting times...
Is Linux ready for safety related applications?
Machine learning Crowd-sourced data System of systems Third party access
Evolution after SOPPersonalization Shortened
development cycles
New topics
new business models
?

3
We need to completely re-think the E/E architecture!
• Domain and then zonal architectures
• Centralized computing units
• High-speed, reliable and dependable networking
• Connected vehicle within an infrastructure eco-system
• The industry is losing with the approach „Adaptive is
just Classic on Linux“!
What comes first?
Philosophy: Mobile on wheels or wheels on mobile?
Source: https://pxhere.com/en/photo/1064249, CC0 Public Domain
Cloud and mobile first!

4
Most prominent answer of our „car“ guys:
„Of course, my car!“
People don‘t realize:
• How many security solutions are in today‘s phones
• Cloud and phones set the „state-of-the-art“
• ... not cars!
What needs to be „more“ secure?
Phone and cloud vs. vehicle
Source: https://www.kompulsa.com/wordpress/wp-content/uploads/2018/06/bigstock-Cyber-security-information-
pr-205808125.jpg, CC0 Creative Commons

What is Linux?

6
Linux is everywhere!

7
• First release 17 September 1991
• March 1994: release of version 1.0
• Linus Torvalds maintains the mainline kernel
• Development is driven by
• Kernel contains
– Process and thread handling
– Memory Management
– Networking and file systems
– Drivers
– …
• No application functionality is provided by the kernel!
Linux distributionLinux kernel
What is meant by “Linux”?
• A “distribution” consists of the Linux kernel plus applications
• Size of a distribution varies from few hundred up to 50.000
software packages
• Many distributions are for special purposes:
– Servers and desktop PCs
– Embedded systems and IoT devices
– Special use cases, e.g. system administration, networking, …
• All packages are tightly coupled within a distribution
When using the term “Linux”, we refer to the kernel from now on.

8
• Monolithic kernel
– All drivers run within and are distributed with the kernel
– ~25 Mio. lines of code
• Divided into various subsystems
– About 100 subsystem trees: e.g. networking, mm, x86, …
• DON’T BREAK USERSPACE !!!!!!
– Only new API functions are added, existing remain stable
– Further abstraction via glibc
• Internal API functions are volatile
– “mainline” functions are adapted by the kernel
maintainers
– “off-the-tree” kernel patches need to be adjusted by
patch providers
Architecture of the Linux kernel
Source: http://www.makelinux.net/kernel_map/

9
Cycle Changesets
4.15 14,866
4.16 13,630
4.17 13.541
4.18 13.283
4.19 13,657 (so far)
In 8 weeks of development for v4.19:
• 1,710 different contributors, 253 first-time contributors
• 307,000 lines added
For a start, e.g. read the report:
„2017 State of Linux Kernel Development”
A note on the community
Source: https://lwn.net/Articles/767635/
Some people think: Linux is driven by „hobbyists“.
Today, it is super-professional and improving constantly.
Some people think: Linux is driven by „hobbyists“.
Today, it is super-professional and improving constantly.

Linux development process?
(Some of you may be assessors,
well I am and I love it ;-))

11
Merge window
• Patches are collected from subsystem
branches
• Duration: 2 weeks
Stabilization phase
• Testing of new kernel version
• Defect fixes
• New rc-Kernel releases every week
Stable branches
• Only bug fixes are merged to stable
branches
• Every release gets a new patch version
• Long-term support (LTS) kernels have a
dedicated maintainer
Linux development process
4.4-rc1 4.4-rc2 4.4-rc3 4.4-rc4 4.4-rc5 4.4-rc6 4.4-rc7 4.4-rc8 4.5-rc1 4.5-rc24.3-rc7 4.4.04.3.0
4.4.1 4.4.2 4.4.3 4.4.4 4.4.54.3.1 4.3.2 4.3.3 4.3.4 4.3.5
Merge window Merge windowStabilization phase
Stable branch Stable branch (LTS)

12
• Analyze fixes over time for LTS kernel version
• Predict bug evolution using statistical methods
– Assumptions: bugs follow a negative binomial distribution
– Take confidence interval of 95%
– Verify prognosis by randomly selecting subsets of the data
(“bootstrapping”)
• Prerequisites and assumptions
– Sufficient data points are available (i.e. high patch level)
– Every patch is a bug fix
• Can also be performed on individual subsystems
OSADL approach: statistical analysis
Quality evolution of a LTS kernel
Source: Nicholas McGuire, OSADL SIL2LinuxMP project

13
Courtesy of Nicholas McGuire, OSADL/SIL2LinuxMP project
• Analysis of all commits between 2.6.12-rc1 – 4.12-rc2
• Bugs introduced by individual developers related to their
overall number of commits
• Results of the analysis
– Risk of introducing a bug with first patch: ~3%
– Maximum at ~220 patches: ~6%
• Assumptions
– 35% of all patches have a „Fixes“ tag
Bug rate analysis of individual developers
Linux deveaIs Linux ready for safety related applications?lopment process
Quality assessment of contributions

14
Involve two different parties:
• Linux kernel team
– analyze kernel bugs and their fixes
– Can be done generically
– Linux expert knowledge needed, possibly on certain submodules
• System architecture team
– Analyze impact of kernel bugs on systems
– Can the bug violate a safety requirement?
– System knowledge needed
– Needs to be done by project using the Linux kernel
Linux kernel monitoring team needs to be established!
Main content of root cause analysisRoot-cause analysis of bug fixes
Linux development process
Root cause analysis of kernel bugs
• Commit overview information
– Backport commit / up-stream commit on mainline
• Sources
– Findings – discussions on mailing lists (lkml.org)
– Related commits – backport to other LTS versions?
• Analysis of commit
– Analysis of behavioral change
– Bug introduction / bug detection
• Impact on Userspace and System
– Relevant for system analysis
• Measures for detection/avoidance
– Root cause identification, test case creation, static analysis…
• Open Actions

15
• Developed specifically for the Linux kernel
• Allows verifying and modifying code according to “semantic
patches”
– Check for common programming fault
– Support and automate internal API changes
• Samples of error patterns that are checked (and corrected)
– Size of a pointer (8 patches)
– Move dereference to after a NULL test. (20 patches)
– Add missing kfree (34 matches)
– …
• Coccinelle check scripts are provided in the kernel repository
Usage of further analysis toolsCoccinelle – a static code analysis tool
Analysis of the Linux kernel code
• SPARSE – semantic C checker (static)
– Developed by Linus Torvalds to support kernel development
– Used for type checking, lock checking etc.
• KASAN – Kernel address space sanitizer (dynamic)
– Verify access to freed memory and out-of-bounds access
– Uses compile-time instrumentation of GCC
• UBSAN – Undefined Behavior Sanitizer (dynamic)
– Compile-time instrumentation adding checks to the code
• Kmemleak/Kmemcheck (dynamic)
– Used for the detection of memory leaks by tracing memory
allocations
– Detection of uninitialized memory accesses in the kernel
(Dynamic checks are activated via kernel compilation switches)

Process & Product: Product

17
The overall system (e.g. for autonomous driving) has ASIL-D
• The ASIL is on requirements, not on components
• Decomposition of requirements (!) and diverse redundancy in
system architecture (!)
 resulting in ASIL-B(D) requirements and subsystems
• Subsystems should be (mainly) fail-safe
• Safety requirements are realized as safety mechanisms and
safety integrity mechanisms
…from the software perspective…from a system perspective
The target ASIL…
• Linux does follow a strict and professional software
development process, but not Automotive SPICE or ISO 26262
• Every kernel release is thoroughly tested by various companies
(Intel, ARM, Amazon, Google, Facebook, Microsoft, Netflix, Red
Hat, SuSE, IBM, Oracle, …)
• Allocation of safety integrity requirements only on:
– Spatial and temporal independence
– System and hardware integrity monitoring
Process approaches:
Follow OSADL SIL2LinuxMP project working on a SIL-2 certified Linux kernel for a reference project

18
• Usage of MPU or MMU for spatial independence
– ?
• Usage of secondary time source, e.g. a windowed watchdog,
that is extended to timing and execution monitoring
 See e.g. Classic AUTOSAR watchdog stack
• Activating and using hardware fault detection mechanisms
• E2E protection of communication
Security mechanismsFreedom from interference mechanisms
Freedom from interference vs. (?) security
• Usage of MMU for spatial independence
– Adding ASLR, stack protection, control flow integrity, asan and ubsan
sanitizers, automatic const, no read/exec pages, etc.
• Same, but add control flow integrity, monitoring of timing
attacks, integrity checks of external attacks on timing hardware
 internal knowledge, but cool stuff
• Monitoring attacks on hardware (resilience)
• Cryptographic signatures and encryption of messages
Security mechanisms are more expensive and complicated,
but much stronger than their Classic AUTOSAR counterparts for safety!

Functional Safety Aspects

21
• Fail-safe system
– A “Kernel Oops” is totally acceptable
– No change to Classic AUTOSAR ;-)
• ASIL-capable SoC
– SoC supports the target ASIL-B
– Next-gen
– Safety Manual for SoC is available and useful!
• External monitoring facility available
– E.g. external watchdog with separate time-base
System requirementsAssumptions to the system
System approach for derivation of safety requirements
• 3-Level Hazop
– Technologically unaware (“safety goals”)
– Technologically aware, but unspecified (“functional requirements”)
– Analysis on implementation level (“technical requirements”)
• Input to software safety architecture and application design
– Goal: remove safety requirements from Linux kernel as far as possible
• Identify safety requirements that are actually applicable to the
Linux kernel
General design rule: Safety is a system property!

22
• Take results from HAZOP
• Identify affected Linux System Calls
• Specify requirements
– From man-pages
– From POSIX standard
• Derive test cases (requirements-based and interface testing)
– Identify possibly existing test cases (e.g. LTP)
– Check if IEC criteria are met (equivalence classes, boundary values)
– Extend or create new test cases
• Coverage measurement
– Tools exist, methodology yet open
(kcov kernel interface, LCOV/GCOV)
Test suites: Linux test projectApproach
Verification of the Linux kernel and libraries
• Open-source test suite to verify the Linux kernel
• Started in 2000 by SGI
• Collaborators: IBM, Cisco, Fujitsu, SUSE, Red Hat, …
• Components under test:
– Open POSIX Test Suite
• POSIX interfaces and conformity
– Linux kernel tests
• Linux specifics (syscalls, file systems, memory, containers, …)
– Userland tools (cp, mv, cron, gzip, cpio, …)
– Networking
– Some CVEs

23
Classic AUTOSAR
components
Distributed safety management
Safety & Security Aspects of Automotive High-Performance Controllers
Classic AUTOSAR
components
Lockstep
Safety OS
WDG
Core CoreCore Core
Safety
core
Safety
core
Core…. CoreCore
Health
control
Bootloader
Hypervisor
Privileged partition
Adaptive AUTOSAR on
Linux
Health manager
Vehicle functions partition
Adaptive AUTOSAR on Linux
Container
Vehicle
function
Virtual
resources
Container
Vehicle
function
Virtual
resources
Container
Vehicle
function
Virtual
resources
Pesistency
manager
Execution
manager
Health
manager
Diagnostic
manager
Virtual
resources
Physical resources
….
Classic AUTOSAR
Safety
core
Safety
core
Lockstep
Safety OS
WDG
Health
control
Classic AUTOSAR
Monitor Control

The “system” and outlook

25
Generic Software Architecture
AUTOSAR OS
Adaptive AUTOSAR
App App
High-performance computer
Classic AUTOSAR
Hypervisor
Adaptive AUTOSAR
App
POSIX OS POSIX OS
Trusted Execution
Environment
App
Trusted OS
Classic AUTOSAR
App
Safety cores
AUTOSAR Safety OS
New CPU-intensive
(safety-relevant)
functions:
e.g. sensor fusion
Novel user functions:
e.g. App Store
Reuse of existing
vehicle functions from
Classic AUTOSAR
(SWCs)
Secure startup,
authentication
Safety-relevant vehicle
functions, monitoring
of performance
partitions
Security partition Safety partition
Virtual machineVirtual machine Virtual machine
Performance cores
Secure Boot
Performance partitions

26
EB product line
Individual building blocks
EB tresos
AutoCore OS
EB corbos
AdaptiveCore
App App
High-performance computer
EB tresos
AutoCore
EB corbos Hypervisor
EB corbos
AdaptiveCore
App
EB corbos Linux POSIX RTOS
Trusted Execution
Environment
App
Trusted OS
EB tresos
AutoCore
App
Safety cores
EB tresos
Safety OS
Security partition Safety partition
Performance cores
Secure Boot
Performance partitions
EB tresos Studio
Logging and debugging
EB corbos Studio
Code generation
Configuration
Application development
Integration and deployment
Tools
EB tresos EB corbos Services 3rd party
Software
Hardware
(SoC)

27
Solutions for interesting times
Machine learning
Crowd-sourced data System of systems
Third party access
Evolution after SOP
Personalization
Shortened development
cycles
New topics
new business models
?
High-assurance security
Automotive safety up to
ASIL-D
Real-time capable
Based on open-source
and established, well-
proven implementations
Long-term maintenance
and operations

28
Let‘s rant
Some off-slide remarks before questions
(unfinished, but maybe worthwile…)
A remark from four days ago (!): https://www.zdnet.com/article/windows-10-will-banish-spectre-slowdowns-with-googles-retpoline-patch/

www.elektrobit.com
alexander.much@elektrobit.com
Get in touch!

Is Linux ready for safety related applications?

More Related Content

What's hot (20)

Similar to Is Linux ready for safety related applications? (20)

More from Alexander Much (6)

Recently uploaded (20)

Is Linux ready for safety related applications?