Improving Rollback in Linux via DSL approach & distributing

Energia Open Source

Improving Rollback in Linux
via DSL approach & distributing
Sunday, 6th February 2011

Presented by:
John Thomson: John.Thomson@caixamagica.pt
Researcher
Paulo Trezentos: Paulo.Trezentos@caixamagica.pt
http://twitter.com/PauloTrezentos
R&D Director

Energia Open Source

Overview
 An overview of what Caixa Mágica does.
– MANCOOSI project: European 7th Framework Programme
 Roll-back
– Definition
– Types of roll-back
– As part of a bigger system
 DSL
– Definition
– Example
 Approach
 Implementation
 Distributing data (kad/p2p)
 Conclusion

06.02.2011 Improving Rollback in Linux via DSL approach & distributing 2

Energia Open Source

A brief summary of Caixa Mágica Software
and an overview of what we do:
 Caixa Mágica is a FOSS software and solutions
provider, based in Lisbon, Portugal
 Linux- Caixa Mágica 15 is the main Linux
Distribution available in Portugal
 Caixa Mágica work with national companies,
Government and with European partners to promote Open
Source in all aspects of work
 Over 800,000 Linux CM systems installed (single & dual boot)

 Interesting development → CM16 moving to Debian base


Energia Open Source

European based research projects
 Caixa Mágica works in many multi-national European based
research projects. After EDOS (EU FP6 STREP) based project
it is now actively working on MANCOOSI-FP7 Project
 Also working on ULOOP and Timbus projects
 Work with a multitude of top-tier Universities and research
institutions


Energia Open Source

About MANCOOSI
 MANCOOSI - Managing the complexity of open-
source software. www.mancoosi.org
Solving package management issues that have
been identified through EDOS project

Other talks
 Talks yesterday by Ralf and Jaap about solvers + tools
 André Guerreiro presents Gumby in room H.1302 at 15:30
– Visualise package meta-data
 Paulo Trezentos presents APT-PBO in room H.1302 at 16:00
– A better class of solver
 Aim is to provide better tools for package management


Energia Open Source

Roll-back as simple as traversing time?


Energia Open Source

What does roll-back mean, really?
 In terms of package configuration,
“roll-back is the process of inverting the changes to the
system made by package upgrades to get back to a
particular system state.”
 Many other mechanisms out there that work on using file
system snapshots/saving the state, (next slide)
S1 S2
PkgFoo v 1.00 Roll-back, PkgFoo v 2.00
Time: 10.00pm possible? Time: 10.20pm Installation Timeline

 The mechanism is one part of Transactionally Protected
Package Management as spoken about by Jeff Johnson last
year at FOSDEM 2010.


Energia Open Source

Different types of roll-back
 Conary, a 2nd Generation Package Manager, aims to
improve current meta-installers. Used in Linux
distributions such as Foresight Linux.
 Augeas, is a configuration management tool that
makes manipulating config files much easier
 ZFS, used by Nexenta is an example of a file-system,
snapshot mechanism to snapshot several system states.
 NixOS, a revolutionary system that re-thinks how files
and resources are used to make them purely functional

 Other mechanisms e.g. etckeeper being developed by
Fedora also try and capture configuration files into a
VCS. BTRFS – another snapshot based mechanism.


Energia Open Source

Drawbacks of the various approaches
 File-system Snapshot based mechanisms
– Capture state of the whole system → down to individual files
– Agnostic of packages
– Granularity is not well matched for packages
 Package Based
– Same as installing an older version
– Maintain configurations and settings

 System Call based
– Syscall trapping - records changes to the file-system
– Capturing files alone is not always enough
 Aim is to combine the complimentary parts of these systems
to cover a wide range of situations and events


Energia Open Source

Difficulties of Roll-back
 Package maintainers and developers tend to think in the
forward direction of upgrading packages
 Working in the reverse direction is a relatively unheard of
concept. Think of a toy slinky

 Downgrade is counter-intuitive
 Already have been done if it was necessary?
 Rolling-back changes is 'only' needed when a package fails
to work on the system
 There may be cases where roll-back is impossible using
the techniques that we have investigated, or possibly at all

Energia Open Source

Roll-back is one part of a bigger picture
 Ability to undo package upgrade/install is just one benefit of
having a system that can capture the package configuration
state and maintain a deterministic, state transition model of
the system.
 By examining the current maintainer scripts and templates
provided by deb/rpm-helper, we defined a language that
assists with problems not addressed now by meta-installers.
 Transactionaly Protected Package Management (TPPM) is
what we are aiming to move towards.

TPPM

roll-back roll-back


Energia Open Source

Advantages and benefits of Roll-back/TPPM
 By building a framework in which R/B + TPPM is possible we
can capture the functional elements of maintainer scripts
Error Detection
 Simulate/test the results of
an upgrade on a model of system
 Detect errors a-priori or
potential 'slow' errors after upgrades
 Detect/(correct) sequencing orders in package installations
Roll-back
 Framework → drive roll-back. Now provide more information
to a roll-back utility than available to current meta-installers
 Identify which packages can/cannot perform roll-back
 Allow individual packages to change versions and show the
dependencies for those changes

Energia Open Source

Domain Specific Language (DSL) -package upgrades

 Our DSL is a language used to abstract from the system and
model to solve particular, identified problems
 In our case, the DSL is focused on analysing package
maintainer scripts
 Designed not to be a Turing Complete Language like BASH
but rather something which can resolve problems
 It is an evolutionary language designed to capture the details
of the vast majority of common maintainer scripts
 Increase the coverage of DSL by examining new scripts

Many thanks to University of L'Aquila for their creation of the
language and for their support in using it


Energia Open Source

System Integration
 For roll-back we will have a log of
Apt the DSL commands executed in-
sequence
model_simulator (( ))
model_simulator
 Perform roll-back → run inverse
statements associated with those
DSL_rollback_pre (( ))
DSL_rollback_pre commands in 'reverse' order
 Have the simulator to pre-check if
run_transaction ()
run_transaction () the package configuration can be
rolled-back; whether it will leave it
DSL_rollback_post (( ))
in an erroneus state
DSL_rollback_post
 Performing LIFO style roll-back we
run post commands before we run
the pre statements


Energia Open Source

Domain Specific Language (DSL) Example
 The DSL has been created to assist with some of the
problems discovered and analysed by prior research
Using cups.spec %post example from CUPS-1.4.2
%post
 dslstart postinst_init(cups)
/sbin/chkconfig add cups Matched DSL Pair
/sbin/chkconfig cups on
 dslstop postinst_init(cups)
# Restart cupsd if we are upgrading...
 dslstart post_init_restart(cups)
if test $1 gt 1; then
/sbin/service cups stop Matched DSL Pair
/sbin/service cups start
fi
 dslstop post_init_restart(cups)


Energia Open Source

Example continued, Log

rbHist dslID TID parentID DSL_CMD bhINVERSE
1 1 1 start postinst_init(cups) TRUE
2 1 1 stop postinst_init(cups) TRUE
3 1 1 start post_init_restart(cups) TRUE
4 1 1 stop post_init_restart(cups) TRUE

pkgHist id parent op pkgName pkgVer1 pkgVer2 dateTime
1 1 inst cups 0 1.4.2 2010-01-30


Energia Open Source

Transactions- what happens if a maintainer
script fails?
 If a maintainer script fails in the middle of one of the
operations will have a log like this
dslID TID parentID DSL_CMD bhINVERSE
1 1 1 start postinst_init(cups) TRUE
2 1 1 stop postinst_init(cups) TRUE
3 1 1 start post_init_restart(cups) TRUE

 Transaction has quite obviously failed. No matching end for a
DSL command reached. Odd number of elements etc.
 Perform a roll-back for all matching sub-transaction ID
elements, but in the reverse order with certain constraints.
 If a set of script elements cannot perform roll-back in the
middle of operating, then don't create a dsl tag.


Energia Open Source

Performing the roll-back
 Identify the sub-transaction/package upgrade to roll-back.
 Check that all commands in the database have an inverse for
the particular transaction.
 Run them in reverse-operation order.
 dslstart postinst_init(cups) [1]
 dslstop postinst_init(cups) [2]
 dslstart post_init_restart(cups) [3] //Restart not needed postrm

becomes
 postrm_init_restart(cups) [3] //Remove nonneeded element
 prerm_init_stop(cups) [4] //From lookup table.
 prerm_init(cups) [1+2]

 In other cases the order might be [3,2,1]. Identified by tags.

Energia Open Source

Architecture
 Modified Package as input → DSL extracted → Log storage
and simulator → Mechanism for executing roll-backs →
Maintenance of scripts etc.
User I/P
Inject DSL
dslstart postinst_init(cups)
dslend postinst_init(cups) Logs
Roll-back
SQLite DB

.spec file Apt-rpm
%post
/sbin/chkconfig --add cups Modified System State
/sbin/chkconfig cups on WP2
...
Simulator


Energia Open Source

Overall Approach
Analyse maintainer scripts in standard systems

Identify common themes in maintainer scripts

Create first version of the DSL

Modify maintainer scripts to contain DSL

Log DSL into SQLite DB

Create roll-back mechanism

Logged Data + Stored Info → Perform Roll-back


Energia Open Source

A state/time perspective of roll-back

apt-get [Maintainer Scripts]
T0 install cups In terms of DSL

apt-get DSL cups pkg
rollback apt-rpm DSL stmts
Log
cups 1.2
apt-get
T0+n rollback
TID-028 [Maintainer Scripts]-1
apt-get
rollback
roll-back
2010-02-06


Energia Open Source

Distributing Rollback-Information
 Rollback data can be distributed
– Transceive data onto Kademlia network.
– Distribute rpms/rollback data/packages state to network
– Private data submission to network? Opt-in mechanism

Kad/P2P Network

Logs Config cups pkg
SQLite DB data

Normal upgrades
Initial System State Modified System State


Energia Open Source

Todo list:
 Completed:
– Single package rollback with DSL
– Supporting framework e.g. SQLite3 and script injectors.
 Now in the process of:
– Integrating Simulator/Failure Detector into package upgrade
– Generating tool suite for automatic detection/generation of DSL
commands for a given set of packages and their scripts
– Creating web interface for changing the auto-created DSL pkgs
– Completing rollback mechanism for multiple packages txns.
– Testing
– Documentation
– Release + revisions
– Integration into build systems


Energia Open Source

Conclusion
 The need for Roll-back of packages is growing
 Not a trivial problem
 Need to consider the problems of
complex upgrade paths
 Using the DSL we aim to remove
a lot of the failure cases in
package upgrades.


Energia Open Source

Questions?
 Thank you for listening. I hope that it was interesting and
that you have some questions to ask.


Improving Rollback in Linux via DSL approach & distributing

More Related Content

What's hot (19)

Similar to Improving Rollback in Linux via DSL approach & distributing (20)

Recently uploaded (20)

Improving Rollback in Linux via DSL approach & distributing