Faster recovery from operating system failure & file cache missing

Faster Recovery from Operating System
Failure and File Cache Missing
Yudai Kato, Shoichi Saito, Koichi Mouri, and Hiroshi Matsuo
Nagoya Institute of Technology, Nagoya 466-8555, Japan.

A. Jaleel
MS 1390 4692

Outline
Abstract
Introduction
Proposal
Design
Application
Implementation

Evaluation
•

Conclusion
References

Abstract
 Rebooting a computer is one method to fix OS failures. However this method
causes two problems.
a. The computer will not be available until the rebooting is finished.
b. The user data on the volatile main memory will be erased.

 To address these problems, we propose a system that can fix OS failures
within one second without losing file caches by replacing the OS encountering
a fatal error by another OS that is running on the same machine operating
simultaneously.
 One OS is dedicated to the backup of the other OS. Using the backup OS, we
can instantly conduct a failover.

Introduction
OS failure causes various problems, including loss of temporary data on the main
memory. In this research, we resolve these problems by making the restoration
process faster and protecting file caches from OS failure.
The main goals of Research
•
•
•
•
•

Making OSes robust against bugs
Constructing a high availability system in a single machine
Using a machine with minimum hardware
Quickly making failover
Minimizing overhead

For these goals,
• Two OSes are executed simultaneously on a single machine “Active OS &
Backup OS”.
• When the active OS encounters a failure, the backup OS can take over the
active OS by migrating the devices and file cashes from the active OS to itself.
• the proposed system, which does not need special hardware to perform this.

PROPOSAL
A. Outline
The amount of time for the failover of crashed OSes is the total suspension time
until the administrator detects the failure and restarts the OSes. To reduce the
computer downtime, we automatically detect OS failures and start a quick
failover instead of restarting the computer.
This proposed system provides a method to check the health of an OS kernel. If
the kernel is defunct, we simply conduct a failover
The backup OS can quickly take over functions of the active OS when it fails to
operate properly. This shortens the failover time more than rebooting. Therefore
we can reduce the downtime caused by OS failure.
B. Behavior
During failover, we migrate devices and file caches and launch applications.

DESIGN
a.
b.
c.
d.

Multiple-OS execution platform
Alive monitoring
Device migration
File cache migration

A. Multiple-OS execution platform
 For such simultaneously running of the active and backup OSes, we use Logical
PARtition (LPAR), which is a virtualization technique that emulates multiple
machines on a physical machine.
 Each OS is booted on a partition specified by kernel parameters that are given
by a bootloader.
B. Alive monitoring
How to confirm whether the active OS is dead or alive???
1. The crashed OS itself realizes the fault. So, active OS can send a dying message
to the backup OS.
2. the OS cannot realize the fault. this is caused by serious bugs, immediately
stops the OSes after they encounter the bugs .here, we use heartbeat messages at
regular intervals.

C. Device migration
The device migration mechanism is intended for migrating configuration files,
the libraries, the IP addresses and the function of the devices from the active
to the backup OS.

D. File cache migration
we can migrate the file cache that remains on the volatile main memory from
the active to the backup OS. As a result, the file cache can be written properly
to the storage device.
The steps performed during migration.
• The backup OS obtains the data structures related to the file caches from the
memory region of the active OS.
• It reconstructs file caches from the data structures.
• It writes the file caches to the storage.
APPLICATION
The proposed system enhances the availability of a operating system on a
machine using an example of an application to a Network File System (NFS)
server.
NFS is a kind of server-client application for sharing files and this exports its
local directory to the network. Since it’s a stateless application, the NFS server
can handle client requests even if the server is rebooted after a sudden
interruption.

we create an NFS server process on both the active and the backup OS. A NFS
client is running on a remote machine and communicating with the NFS server on
the active OS through the network. The NFS server on the active OS exports a
directory /export, which is a partition on the storage device. When the active OS
crashed because of bugs, alive monitoring detects it, and the backup OS starts the
failover to restore the NFS server.

First, the backup OS migrates the storage device and the NIC from the active OS to
it. The storage device contains the partition for the NFS server. Then the backup
OS mounts the partition on the same directory /export. After that, the backup OS
migrates the dirty file cache that remains on the volatile main memory to it.
Next, attaching the same IP address to the NIC used by the active OS, the backup
OS can communicate with remote NFS clients. In this stage, the backup OS
transparently finishes the failover for the client.
After that, the server begins processing the suspended operations, which are
successfully processed for the client because the previous operations are never
lost. Hence, our proposed system can protect files even if the OS crashed when the
file was being written by the NFS clients. In this way, we migrate environments
from the active to the backup OS, which transparently takes over for the crashed
OS to the NFS clients.

IMPLEMENTATION
Linux (2.6.38, processor type x86 64). Both the active and backup OSes use this
kernel.
A. Implementation of multiple-OS execution platform
For creating a LPAR partition that represents a virtual machine, every OS on the
machine must exclusively access the hardware of its partition. First, the OS is
booted from the normal bootloader, and subsequent OSes are booted by a special
bootloader based on kexec.
B. Failover
The failover of the proposed system is composed of device and file cache
migrations. First, we conduct device migration. In the normal state, both OSes
exclusively access their devices by detaching other’s devices without initializing
them.

C. Implementation of alive monitoring
Inter Processor Interrupt (IPI) Application is used which a core can communicate
with other cores through the processor’s interrupt controller. for detecting the
failure of the active OS. Both messages are sent by IPI.
After receiving this message, the backup OS notices the death of the active OS and
immediately starts failover.
The active OS sends heartbeat messages at regular intervals using interval timers. If
any message cannot be received for several seconds, the backup OS assumes that
the active OS is dead. With these two methods, alive monitoring detects the failure
of the active OS.

EVALUATION
we measure the time required to perform failover.
For the measurement, for simplicity, we deliberately crashed the active OS and
measured the length of the failover & the backup OS immediately detected the
failure of the active OS by receiving a dying message from the active OS. While the
backup OS was taking over the active OS, the backup OS performed the following:
1. Device migration
2. Mounting storage
3. File cache migration
4. Assignment of IP addresses to the NIC.

1. Device migration
2. Mounting storage
3. File cache migration
4. Assignment of IP addresses
to the NIC.

• Measured the amount of time for (3) with several different amounts of caches. The
times for (1)(2)(4) did not differ by conditions, thus the total time for (1)-(4) was
measured instead of measuring them individually. The measurement results of the
amount of time for cache migration are shown in Fig. 2.
• About 90 MBytes of the file cache can be migrated within 70 milliseconds.
After that, we measured the time for (1)-(4). The results were from 560 to 650
milliseconds. The margin of 90 milliseconds reflects the amount of time for file-cache
migration. Our proposed system can finish a failover process within one second.

•

Measured the amount of downtime for the network and the NFS on the proposed system using
a remote machine connected to a switching hub to which our proposed system was also
connected. We measured the time (a)(b)(c) shown in Fig. 3. First, we measured the network
downtime. The echo requests were sent continuously using Internet control message protocol
(ICMP) before the active OS crashed.

•

Measured the downtime using the amount of time between (a) the beginning of the first echo
request that failed to come back and (b) the time of the echo request that first came back
after the failover. The result was about 3.1 seconds, which is slightly longer than the one
second of the failover.

The start time of the suspended operation was the same as (b), and its end is
denoted by (c). The results ranged from about 3 to 13 seconds. They were much
longer than the network downtime.
We conducted the same measure using a UDP protocol, and the time was about 3
milliseconds. Therefore, if NFS uses a UDP protocol, it can communicate
immediately after the network comes back.
CONCLUSION
In this paper, we proposed a system with a failover mechanism for operating
systems. Our proposed system’s failover takes less than one second. The user data
in the file cache are never lost because they are migrated by a file cache
migration mechanism.
our proposed system can resolve problems caused by rebooting after OS failure. It
runs on commodity x86 machines and needs no modifications for applications.
During normal times, since there is no special processing except for heartbeat
messages, our proposed system is lightweight. Future work will randomly insert
bugs into kernels to measure the durability of our proposed system.

REFERENCES
[1] A. S. Tanenbaum, J. N. Herder, and H. Bos, “Can we make operating systems reliable and secure,” Computer, vol. 39, pp.
44–51, 2006.
[2] M. M. Swift, M. Annamalai, B. N. Bershad, and H. M. Levy, “Recovering device drivers,” ACM Transactions on Computer
Systems, vol. 24, pp. 333–360, 2006.
[3] A. Depoutovitch and M. Stumm, “”otherworld”: Giving applications a chance to survive os kernel crashes,” in EuroSys,
2008, pp. 181–194.
[4] A. Pfiffer, “Reducing system reboot time with kexec.” http://www.osdl.org/.
[5] M. K. Mckusick and T. J. Kowalski, “Fsck - the unix file system check program,” 1994.
[6] A. Sweeney, A. Sweeney, D. Doucette, W. Hu, C. Anderson, M. Nishimoto, and G. Peck, “Scalability in the xfs file system,”
in In Proceedings of the 1996 USENIX Annual Technical Conference, 1996, pp.1–14.
[7] M. K. Mckusick, M. K. Mckusick, G. R. Ganger, and G. R. Ganger, “Soft updates: A technique for eliminating most
synchronous writes in the fast filesystem,” in In Proceedings of the Freenix Track: 1999 USENIX Annual Technical Conference,
1999, pp. 1–17.
[8] M. Rosenblum and J. K. Ousterhout, “The design and implementation of a log-structured file system,” ACM Transactions
on Computer Systems, vol. 10, pp. 1–15, 1991.
[9] “Keepalived for linux.” [Online]. Available: http://www.keepalived.org/
[10] T. L. Borden, J. P. Hennessy, and J. W. Rymarczyk, “Multiple operating systems on one processor complex,” IBM Systems
Journal, vol. 28, no. 1, pp. 104 –123, 1989.
[11] T. Shimosawa, H. Matsuba, and Y. Ishikawa, “Logical partitioning without architectural supports,” in International
Computer Software and Applications Conference, 2008, pp. 355–364.
[12] A. Chou, J. Yang, B. Chelf, S. Hallem, and D. Engler, “An empirical study of operating systems errors,” in Symposium on
Operating Systems Principles, 2001, pp. 73–88.
[13] G. Candea, S. Kawamoto, Y. Fujiki, G. Friedman, and A. Fox,“Microreboot - a technique for cheap recovery,” in
Operating Systems Design and Implementation, 2004, pp. 31–44.
[14] P. M. Chen, W. T. Ng, S. Chandra, C. M. Aycock, G. Rajamani, and D. E. Lowell, “The rio file cache: Surviving operating
system crashes,” in Architectural Support for Programming Languages and Operating Systems, 1996, pp. 74–83.
[15] C. Giuffrida, L. Cavallaro, and A. S. Tanenbaum, “We crashed, now what?” in Proceedings of the Sixth international
conference on Hot topics in system dependability, ser. HotDep’10. Berkeley, CA, USA: USENIX Association, 2010, pp. 1–8.
[16] M. M. Swift, B. N. Bershad, and H. M. Levy, “Improving the reliability of commodity operating systems,” in Symposium on
Operating Systems Principles, 2003, pp. 207–222.
[17] N. Palix, G. Thomas, S. Saha, C. Calv`es, J. Lawall, and G. Muller,“Faults in linux: ten years later,” in Architectural
Support for Programming Languages and Operating Systems, 2011, pp. 305–318.

Thank You
Athambawa Jaleel
MS 1390 4692

Faster recovery from operating system failure & file cache missing

More Related Content

What's hot (20)

Viewers also liked (10)

Similar to Faster recovery from operating system failure & file cache missing (20)

Recently uploaded (20)

Faster recovery from operating system failure & file cache missing

Editor's Notes