Checkpoint/Restore mostly in Userspace
Download as ODP, PDF0 likes2,532 views
CRIU is a tool that provides checkpoint and restore capabilities in Linux. It allows saving the state of processes and restoring them later. This allows for failure recovery, live migration, rebootless upgrades, and other use cases. CRIU works by dumping kernel objects and process data to disk, then restoring that data to recreate the processes. It has support for Linux processes, threads, memory mappings, files, sockets, namespaces and more. CRIU is tested on real applications and works to integrate with the Linux kernel and distributions.
![How to use
● ./crtools dump -t pid [<options>]
– checkpoint a process/tree identified by pid
● ./crtools restore -t pid [<options>]
– restore - restore a process/tree identified by pid
● ./crtools show (-D dir)|(-f file) [<options>]
– show dump file(s) contents
● ./crtools check
– checks whether the kernel support is up-to-date
● ./crtools exec -t pid <syscall-string>
– exec - execute a system call by other task
15](https://image.slidesharecdn.com/criu-v2-130207132501-phpapp02/85/Checkpoint-Restore-mostly-in-Userspace-15-320.jpg)
Checkpoint/Restore mostly in Userspace
- 1. CRIU - Checkpoint/Restore in User-space Andrey Vagin <avagin@openvz.org><
- 2. What is C/R and how can it be used? C/R is the ability to save states of processes and to restore them later. Usage scenarios: – Failure recovery – Live migration – Reboot-less upgrade – Speed up of slow-boot services – HPC issues 2
- 3. History BLCR DMTCP OpenVZ Linux C/R CRIU 2003 2007 2005 2008 2011 ● Berkeley Lab Checkpoint/Restart (BLCR) (2003) – Load a kernel module and link with a library ● DMTCP: Distributed MultiThreaded CheckPointing (2004-2006) – Preload a library ● OpenVZ (2005) – OpenVZ kernel ● Linux Checkpoint/Restart by Oren Laadan (2008) – A non-mainline kernel ● CRIU (2011) 3
- 4. How does this work? Image files Kernel objects Process tree 001101 001101 101010 101010 110001 110001 011010 011010 000011 000011 010101 010101 Files 001101 101010 001101 101010 110001 110001 011010 011010 Sockets crtools 000011 010101 000011 010101 Pipes 001101 001101 101010 101010 110001 110001 011010 011010 000011 000011 010101 010101 Name-spaces 4
- 5. Kernel interfaces /proc/ ptrace Dump Restore syscalls netlink 5
- 6. Dump ● Parasite code – Receive file descriptors – Dump memory content – Prctl(), sigaction, pending signals, timers, etc. ● Ptrace – freeze processes – Inject a parasite code ● Netlink – Get information about sockets, netns ● Procfs /proc/PID/maps, /proc/PID/map_files/, /proc/PID/status, /proc/PID/mountinfo 6
- 7. Restore Namespaces ● Collect shared objects ● Restore name-spaces ● Create a process tree Processes – Restore SID, PGID – Restore objects, which should be inherited ● Files, sockets, pipes, ... ● Restore per-task properties. ● Restore memory ● Call sigreturn ● Awesome 7
- 8. Interesting moments ● How to restore shared objects? – Send file descriptors via unix sockets – Map files from /proc/self/map_files/ for restoring anon shared mappings ● How to restore memory mappings on the correct places? – Map a new code block and a stack – Unmap crtools' mappings – Remap task's mappings on the correct places ● How to resume a process? – Create a signal frame – Call sigreturn() 8
- 9. Kernel impact ~120 patches merged ~20 patches in flight ~10 new features appeared ~2 new features to come 9
- 10. New features in a kernel ● Parasite code injection (by Tejun Heo) – Read task states, that are currently retrieved by a task only about itself ● The kcmp() system call – Helps checking which kernel objects are shared between processes ● Proc map_files directory – Find out what exact file is mapped – Mappings sharing info ● A bunch of prctl extensions – Set various private stuff on task/mm objects (c/r-only feature) ● Last-pid sysctl – Restore task with desired PID value 10
- 11. New features in a kernel (net) ● TCP repair mode – Read intimate state of a TCP connection and reconstructs it from scratch on a freshly created socket ● Sockets information dumping via netlink (sock_diag) – Extendable sockets state retrieving engine ● Virtual net devices indexes – Allows to restore network devices in a namespace ● Socket peeking offset – Allows peeking sockets queues (reading without removing data from queue) 11
- 12. What are already supported? ● Linux 3.8 ● In flight – X86_64 architecture – and ARM architecture – Process tree linkage – Pending signals – Multi-threaded apps – TCP time-stamps – All kinds of memory mappings – Terminals, groups, sessions – Open files (shared and unlinked) – Established TCP connections – Unix sockets, Packet sockets – Name-spaces (net, mount, ipc) – Non-posix files (epoll, inotify) – Pipes, Fifo-s, IPC, ... 12
- 13. How is CRIU tested? ● ZDTM – a set of unit-tests ● Real-life applications – Apache, Nginx – MySQL, MongoDB, Oracle – Make && gcc – Tar & gzip – Screen – Java – LXC – VNC server + GUI applications 13
- 14. Future plans ● Support all kinds of kernel objects ● Merge all in-flight patches in the mainstream kernel ● Integrate CRIU with OpenVZ and LXC utilities ● Iterative migration – Migrate memory content before freezing applications ● Integration in distributions – CRIU was accepted to Fedora 19 14
- 15. How to use ● ./crtools dump -t pid [<options>] – checkpoint a process/tree identified by pid ● ./crtools restore -t pid [<options>] – restore - restore a process/tree identified by pid ● ./crtools show (-D dir)|(-f file) [<options>] – show dump file(s) contents ● ./crtools check – checks whether the kernel support is up-to-date ● ./crtools exec -t pid <syscall-string> – exec - execute a system call by other task 15
- 16. Checkpoint/restore of a VNC server. 16
Editor's Notes
- #4: BLCR is used a kernel module, doesn't checkpoint sockets, SysV IPC, zombies, etc. Applications should be linked with a library and executed via a helper. DMTCP uses an executer too, but doesn't require a kernel module. C/R in OpenVZ is used for checkpount/restore and migrate OpenVZ containers. It requires the OpenVZ kernel. Linux C/R is very similar on OpenVZ C/R. It is used for checkpoint/restore of LXC. CRIU combines all this project. It will work on the pure upstream kernel. It's able to dump a task without any preparation.