Noah - Robust and Flexible Operating System Compatibility Architecture - Container runtime meetup #2

Editor's Notes

• #2: Hello everyone. I’m Takahiro Shinagawa from the University of Tokyo. Today, I’d like to talk about a robust and flexible operating system compatibility architecture. This is a joint work with Mr. Saeki, Mr. Nishiwaki, and professor Honiden. This work was done mainly by Mr. Saeki in cooperation with Mr. Nishiwaki while they were master course students. Unfortunately, Mr. Saeki has already graduated and Mr. Nishiwaki is in a different field laboratory, so I’m going to make this presentation.
• #12: There are two ways to implement OS compatibility layers. One way is to implement them in kernel space. Kernel-space implementation has the advantage that it has flexibility to achieve binary compatibility. However, it is vulnerable against bugs in the OS compatibility layers. For example, the former Windows Subsystem for Linux has several bugs that could cause the blue screen of death of Windows. The other way is to implement them in user space. It has the advantage that bugs do not affect the stability of the operating system. However, user-space-only implementations are inflexible to achieve full binary compatibility because they cannot trap system call instructions or manipulate page tables, unless we use binary modification. For example, Cygwin cannot implement the copy-on-write capability in the fork() system call.
• #13: So, we propose a novel operating system compatibility architecture. In this architecture, we run each guest process in a separate VM without its OS kernel, and the process of the OS compatibility layer running on the host operating system manages the VM to emulate the execution environment for the guest application process. This architecture can achieve robustness because most of OS compatibility layers can be implemented in user space and bugs in the layers do not cause kernel crashes. It can also achieve flexibility to realize full binary compatibility because the virtualization technology allows low-level event handling such as trapping system calls and page faults, manipulating page tables, and so on. We need a host OS support to handle hardware-assisted virtualization technology, but fortunately recent operating systems provide standard virtualization interfaces and we can reuse them. So, we do not need to modify the OS kernels by ourselves.
• #14: Here is the overall design of our proposed architecture. Our system consists of three main components, that is, guest VMs, the VMM module, and monitor processes. We will explain each of them in the following slides.
• #26: This is the summary of the advantages of our architecture. It can achieve robustness because the monitor process is implemented in user space and bugs of them will not cause system crashes. It can also achieve flexibility to realize full binary compatibility and good performance with the copy-on-write capability. In addition, it inherits the advantages of using OS compatibility layers in general. For example, it can achieve low development cost because it can use the rich host OS functionalities such as system calls, useful libraries, and high-level languages such as Rust and Go. It also achieve seamlessness because there is only a single kernel and all system resources are managed by the kernel with the single management policy.
• #28: Now, we explain the implementation. Our target is Linux 4 point 6 running on x eighty-six-sixty-four processors with the support of Inter VT-x. We implemented a Linux compatibility layer for macOS called Noah, that is, we can run Linux binaries on macOS without modifications. This implementation is mature enough to run many Linux applications. For example, we can build Linux kernels and run several X11 applications on Noah. We also implemented a preliminary version for Windows that supports the copy-on-write capability so that we can confirm the advantage of our architecture. Unfortunately, it does not implement many system calls yet.
• #29: As for memory management, there are two page tables in our architecture. That is, the guest page table in the VM and nested page table in the VMM. To avoid handling two page tables, we should fix one page table and manipulate only the other. Which one to choose is a design choice. We chose to fix the guest page table and manipulate the nested page table for easy implementation and debugging. The VMM module provides the API to manipulate the nested page tables and the monitor process can map memory pages to the VM by specifying the virtual address of the monitor process. So, the monitor process can easily change the page mappings of the VMs. This approach has one limitation. Since current Intel CPUs support only up to 39-bit physical address, we cannot use the all 48-bit virtual address. Fortunately, the only problem of this in Linux is the default stack address, and we can safely move the stack to the lower address. We should note that there is no kernel in the VM, so we do not need the higher region of the guest virtual address space.
• #30: As for process management, we used different approaches on macOS and Windows. On macOS, we can use the fork system call to fork the monitor process, and implemented a subset of the clone system call. Unfortunately, Apple Hypervisor.framework does not support the fork of the process with a virtual machine. Therefore, we first save and destroy the VM before fork, and then restore the VM state after fork. On Windows, the monitor process cannot use fork because Windows does not support it. So, we implemented the fork functionality by ourself using shared memory and virtualization technology. That is, we created a memory region shared among the monitor processes, and the monitor processes trap the page faults and performs the copy-on-write. The implementation is a little bit complicated, but basically similar to the implementation in the OS kernel.
• #31: As for process management, we used different approaches on macOS and Windows. On macOS, we can use the fork system call to fork the monitor process, and implemented a subset of the clone system call. Unfortunately, Apple Hypervisor.framework does not support the fork of the process with a virtual machine. Therefore, we first save and destroy the VM before fork, and then restore the VM state after fork. On Windows, the monitor process cannot use fork because Windows does not support it. So, we implemented the fork functionality by ourself using shared memory and virtualization technology. That is, we created a memory region shared among the monitor processes, and the monitor processes trap the page faults and performs the copy-on-write. The implementation is a little bit complicated, but basically similar to the implementation in the OS kernel.
• #38: This is the result of macro benchmark using Phoronix Test Suite. We can see from the graph that some applications became slower and some applications became faster depending on their characteristics. If the application issues many simple system calls, the overhead will become higher. If the application issues many complicated system calls, the overhead will become lower. If the application causes many page faults, it could become faster. Since one of our target application is build environments, kernel build performance is a good benchmark. It was 16% percent overhead and we believe this is a reasonable performance.
• #39: This is the result of the primitive benchmark. We measured the CPU cycles of dup() system call. dup() is a simple system call but it actually call the OS kernel, so we used it to measure the breakdown of the system call. As shown in the figure, VM enter and exit took around three hundred cycles and they were not so high. The system call itself took only around two thousands cycles. Unfortunately, the VMM module took extra CPU cycles to enter and exit the VM. We believe there is still room for optimization in this part and we can further reduce the overhead on system calls.
• #40: This is the result of the micro benchmark using lmbench. This benchmark measured the performance related to the process and processor. From this figure, we found that the overhead of simple system calls was high, because the cost of context switches is high, but in complex system calls, the context switch cost became relatively lower. One interesting result is the performance of the exec system call. It became faster than macOS because the exec system call is mainly implemented by our system in a simple way, and the implementation in macOS may be very complicated due to its kernel structure.
• #41: This is another result of lmbench. The result shows that Noah incurred up to 42 percent overhead on basic file-related system calls. Another interesting result is that page fault and protection fault handling was much faster in our system than macOS. This is probably because of the same reason with the exec system call, because memory management is mainly implemented by our system.
• #42: Finally, we show the performance comparison of OS Compatibility layers. We used the Windows version of our implementation, and compared with Cygwin that is a user-level implementation of OS compatibility layers, and Windows Subsystem for Linux, shown as WSL1, as a kernel-level implementation. We can see that the performance of the dup2 system call is much slower in our system because this is a simple system call and the virtualization overhead is dominant. We can also see that write performance is comparable to the other two systems because this performance is mainly determined by the host I/O performance. Finally, we can confirm that fork performance of our system is much faster than Cygwin, especially the guest process has large data, because we support the copy-on-write capability.
• #43: So, this is the conclusion. We proposed a novel OS compatibility architecture that exploits the OS-standard virtualization technology and achieved both robustness and flexibility. In our architecture, an OS compatibility layer consists of three components, that is, virtual machines to run guest processes, the VMM module to provide API for hardware virtualization technology, and monitor processes that implement the OS compatibility functions. Our implementations can run many Linux binaries on macOS and simple Linux binaries on Windows. The overhead of Linux kernel build time on Noah was six-teen percent.