Live migrating a container: pros, cons and gotchas -- Pavel Emelyanov

Live migrating a container:
pros, cons and gotchas
Pavel Emelyanov
Principal engineer @ Virtuozzo

AgendaAgenda
• Why you might want to live migrate a container
• Why (and how) to avoid live migration
• Why is container live migration so complex
2

Migration in a nutshelMigration in a nutshel
• Save state
• Copy state
• Restore from state
3

Why you might want to live migrate a containerWhy you might want to live migrate a container
• Spectacular
• Load balancing
• Updating kernel
– Can avoid live migration, just C/R
• Updaring or replacing hardware
4

Why to avoid live migrationWhy to avoid live migration
5

How to avoid live migrationHow to avoid live migration
• Balance network traffic
• Microservices
• Crash-driven updates
• Planned downtime
6

Making live migration liveMaking live migration live
• State saving, transfering and restoring happens with tasks frozen
– “Scatter” container is too complex
• Save state quickly
• Big transfer should not be done at that time
• Restore from state quickly
7

Making live migration liveMaking live migration live
• Save/restore speed up is a long-running task
• Big memory transfer should not be done at frozen time
• Memory pre-copy
• Memory post-copy
8

Pre-copyPre-copy
• Copy memory while tasks are running
• Track memory changes
• goto again
9

Pre-copyPre-copy
• Pros:
– Safe: once migrated, source node can disappear
• Cons:
– Unpredictable: iterations may take long
– Non-guaranteed: “dirty” memory next round may remain big
10

Post-copyPost-copy
• Migrate all but memory
• Turn on “network swap” on destination
11

Post-copyPost-copy
Pros:
– Predictable: time to migrate can be well estimated
• Cons:
– Unsafe: src node death means death of container on destination
– Application slows down after migration
12

Live migration at lengthLive migration at length
• Memory pre-copy (iteratively, optional)
• Freeze + Save state
• Copy state
• Restore from state + Unfreeze and resume
• Memory post-copy (optional)
13

Things to deal withThings to deal with
• VM
– Environment: virtual hardware, paravirt
– CPU
– Memory
• Container
– Environment: cgroups, namespaces
– Processes and other animals
– Memory
15

Memory pre-copyMemory pre-copy
• VM
– All memory at hands
– Plain address space
• Container
– Memory
●
is scatered over the processes
●
can be (or can be not) shared
●
can be (or can be not) mapped to disk files
16

Save stateSave state
• VM
– Hardware state
●
Tree of ~100 objects
●
Fixed amount of data per each
• Container
– State of all objects
●
Graph of up to ~1000 objects
●
All have different amount of data, different reading API
17

Restore from stateRestore from state
• VM
– Copy memory in place, write state into devices
• Container
– Creation of many small objects
– Not all have sane API for creation
●
Creation sequence can be non-trivial
18

Memory post-copyMemory post-copy
• UserfaultFD from Andrea Archangeli
• VM
– Merged into 4.2
• Container
– Non-cooperative work of uffd monitor and client,
need further patching
19

And we also need this, this and this!And we also need this, this and this!
• Check for CPUs compatibility
• Check and load necessary kernel modules (iptables, filesystems)
• Non-shared filesystem should be copied
• Roll-back on source node if something fails in between
– Keep tasks frozen after dump, kill after restore
20

ImplementationImplementation
• CRIU
– Save & restore state
– Memory pre/post copy
• P.Haul
– Checks
– Orchestrate all C/R steps
– Deal with filesystem
21

P.Haul goalsP.Haul goals
• Provide engine for containers live miration using CRIU
• Perform necessary pre-checks (e.g. CPU compatibility)
• Organize memory pre-copy and/or post-copy
• Take care of file-system migration (if needed)
22

Under the hoodUnder the hood
23
CRIU CRIUp.haul p.hauldocker -d docker -d
migrate
src dst
check (CPUs, kernels)
pre-dump
memory
dump
other images
restore
memory
lazy mem
FS
FS copy
done
pre-copypost-copy
kill
freeze
time

More infoMore info
• http://criu.org
• http://criu.org/P.Haul
• criu@openvz.org
• +CriuOrg / @__criu__
• https://github.com/xemul/(criu|p.haul)
24

Thank you!
Pavel Emelyanov
@__criu__
xemul@openvz.org

Live migrating a container: pros, cons and gotchas -- Pavel Emelyanov

More Related Content

What's hot (20)

Similar to Live migrating a container: pros, cons and gotchas -- Pavel Emelyanov (20)

More from OpenVZ (20)

Recently uploaded (20)

Live migrating a container: pros, cons and gotchas -- Pavel Emelyanov

Editor's Notes