Talk 160920 @ Cat System Workshop

Dynamically Hacking the Kernel
with Containers
CAT System Workshop
2016.09.20
Quey-Liang Kao
NTHUCS

About Myself
● PhD Student
– Adviser: Che-Rung Lee
● Research topics
➢ HPC (numerical), Heterogeneous computing
➢ High-end hardware virtulization (InfiniBand, GPGPU)
➢ Container technology

About Myself - the other side
● Archoholic
– Archwiki
– AUR packager
● runc-git, openscap
● kpatch
● GNOME
● BICIB
– https://github.com/NonerKao/BICIB

Hack
...
Computers.
● to modify (a computer program or electronic device) or write
(a program) in a skillful or clever way.
● to circumvent security and break into (a network, computer,
file, etc.), usually with malicious intent: Criminals hacked the
bank's servers yesterday.
http://www.dictionary.com/browse/hacking

Hack
...
Computers.
● to modify (a computer program or electronic device) or
write (a program) in a skillful or clever way.
● to circumvent security and break into (a network, computer,
file, etc.), usually with malicious intent: Criminals hacked the
bank's servers yesterday.
http://www.dictionary.com/browse/hacking

Hacking the Kernel ... [ OK ]
Dynamically ...

3 ways to modify the
kernelspace
Modules,
Live patches,
and Kernel detouring

Kernel Module: Loading
Kernel
ext.ko
Kernel Space
User Space
Process:
insmod ext.ko
syscall: init_module

Kernel Module: Using
some_ext.ko
Kernel Space
User Space
Process:
cat /dev/some_cdev
syscall: read
Kernel

Kernel
Live Patching: Building
fix.ko
Kernel Space
User Space
Process:
build fix.patch
File:
fix.patch
Current
kernel source
Bug()
Bug-fixed()

Kernel
Live Patching: Applying
fix.ko
Kernel Space
User Space
Process:
insmod fix.ko
Bug()
Bug-fixed()

Kernel
Live Patching: Applying
fix.ko
Kernel Space
User Space
Bug() Bug-fixed()
ftrace
Normal
Process

Kernel
Kernel Detouring
detour.ko
Kernel Space
User Space
Normal
Process
Special
Process
func() func()

Dynamically ... [ OK ]
with Containers ...

Virtual Machines ( KVM )
Kernel
User Space
Kernel Space
kvm.ko
VM 2VM 1
Proc 1
Libs / Conf
Guest Kernel
Proc 3
Libs /
Conf
Guest
Kernel
Proc 2
hypercall

Processes
Kernel
Process
1
User Space
Kernel Space
Libraries / Configuration
Process
2
Process
3

Container 2Container 1
Containers
Kernel
Process
1
User Space
Kernel Space
Libs / Conf
Process
2
Process
3
Libs /
Conf
Namespaces

Kernel
Kernel Detouring
detour.ko
(namespace-
aware)
Kernel Space
User Space
Normal
Process
Container
func() func()

Dynamically ... [ OK ]
with Containers ... [ OK ]

The short story
-- Just for fun!

The long story:
1. Impact
http://alaskarobotics.com/2009/11/30/gallery-walk/
+
=

The long story:
2. Trend
SaaS
PaaS
IaaS
CaaS
?

VM
Process
Libs /
Conf
Guest
Kernel
Container
The long story:
3. Towards higher isolation
Isolation++
Performance++
Process
Libs /
Conf
kvm
K-Container
Process
Libs /
Conf
Detour

VM
Process
Libs /
Conf
Guest
Kernel
Container
Side Note:
For better performance
Isolation++
Performance++
Process
Libs /
Conf
kvm
HyperV
Container
Process
Libs /
Conf
Windows
Server 10
Thin
Kernel

Possible Use Cases
● Experimental module/patch test bed
● Environment for other OSes
● Educational purpose

FreeBSD binary on Linux
01000110011100100110010101100101010000100101001101000100

Why it is not possible: Scene 1
Kernel
mov 0x1, %rax
syscall
(translate: I want to exit)

Kernel
...you want to write?
nothing specialsys_write()

Kernel
(Eventually)
SIGSEGV
End of life but no exit
...what’s wrong with this process?

Kernel
Recall: Kernel Detouring
detour.ko
(namespace-
aware)
Kernel Space
User Space
Normal
Process
Container
func() func()

Why it IS possible now: Scene 1
Kernel
mov 0x1, %rax
syscall
(translate: I want to exit)
Container

Container
Kernel detour.ko
(namespace-
aware)
system call table
system call table
for FreeBSD
#1 is sys_write, so …
...wait, this is a FreeBSD container!

Container
Kernel detour.ko
(namespace-
aware)sys_exit()
system call table
for FreeBSD

Specific Challenges ( FreeBSD )
● Corresponding system calls
– Flag numbers are not portable
– different calling/exiting conventions
● Unique system calls
– Re-implementation

General Challenges
● Insufficient isolation
● Limitation of development
– live patching should only be a temp. solution

Other Binary Compatibility Work
● Wine
– Special loader for PEs/DLLs
● FreeBSD, Windows 10
– Kernel built-in compatibility layer for Linux binary
– System call remapping/re-implementation

Step 0: Setup
● Environment ( x86_64 machine )
– Linux 4.6.2
– FreeBSD 10.2
● Tools
– kpatch: A tool for kernel livepatch
– docker

Step 1: From LivePatching to
Detouring
kernel/livepatch/core.c.orig:klp_ftrace_handler
klp_arch_set_pc(regs, (unsigned long)func->new_func);
kernel/livepatch/core.c:klp_ftrace_handler
klp_arch_set_pc(regs, (is_freebsd_container()) ?
(unsigned long)func->new_func :
(unsigned long)ip+5);

Ftrace in LivePatching
old_func()
0x1230 call __fentry__
0x1235 push %rbp
0x1236 mov %rsp, %rbp
0x1239 push %r15
...
new_func()
0x2235 push %rbp
0x2239 push %r14
...
ftrace
handler
IP
func->new_func

Ftrace in Detouring
old_func()
0x1235 push %rbp
0x1239 push %r15
...
new_func()
0x2235 push %rbp
0x2239 push %r14
...
ftrace
handler
IP
IP+5
func->new_func

Step 2: Detour-able Entry Point
arch/x86/entry/entry_64.S.orig:
ENTRY(entry_SYSCALL_64)
…
call *sys_call_table(, %rax, 8)
void detour_entry()
{
…
asm("jmp *sys_call_table(,
%rax, 8)");
}
arch/x86/entry/entry_64.S:
ENTRY(entry_SYSCALL_64)
…
call detour_entry
Assembly file is NOT
detour-able!
In some C source file

Step 3: Detoured Entry Point
void detour_entry()
...
asm("jmp *sys_call_table(, %rax, 8)");
void detour_entry()
...
asm("mov fbd_table(, %rax, 8), %rax");
asm("jmp *sys_call_table(, %rax, 8)");
asmlinkage const long long fbd_table[] = {0,60,57,...};
EXPORT_SYMBOL(fbd_table);
remapping syscall#

The work flow
1. Launch a normal container
2. Run a init script
1. which enables the specific detour modules
3. A FreeBSD environment in the container
4. On exit
1. disable detour modules

Conclusion
● The kernel detouring demo attempts to
indicate the possibility of the development of
OS containers
– as a proof-of-concept
– kpatch as a temp. solution
● Future direction
– Make more fun

Behind the scenes
● How to implement the is_freebsd_container()
function?
● How was the ContainerCon Japan?

Talk 160920 @ Cat System Workshop

More Related Content

What's hot (20)

Viewers also liked (6)

Similar to Talk 160920 @ Cat System Workshop (20)

Recently uploaded (20)

Talk 160920 @ Cat System Workshop