Linux Disaster Recovery Best Practices with rear

Relax and Recover

Linux Disaster Recovery best
practices with rear
Gratien D'haese
IT3 Consultants

Who am I
● Independent Unix System Engineer since 1996
● Unix user since 1986
● Linux user since 1991
● Open Source contributor:
● The OTA CD-ROM (Linux95 before Windows95 :)
● Make CD-ROM Recovery (mkCDrec)
● Relax and Recover (rear)
● SIM Installation and Logging (similar)
● Adhocracy (adhocr)
2012-04-01 | Gratien D'haese Linux Disaster Recovery best practices with rear 2

Business Continuity Planning
●
●
A business continuity plan specifies how a
company plans to restore core business
operations when disasters occur


●
Business Process and Analysis
●
Identification of business processes and their
interrelationships
●
Prioritizations of business processes
– Downtime tolerance
●
Resource needs (must be shifted during crises)


●
Communicating, Testing, and Updating the
Plan
●
Testing (usually through walk-throughs) needed
to find weaknesses
●
Updated frequently because business conditions
change and businesses reorganize constantly
– Telephone numbers and contact numbers must be
updated even more frequently than the plan as a
whole


Disaster Recovery

●
●
A business continuity plan specifies how a
company plans to restore core business
operations when disasters occur

●
Disaster Recovery
●
Disaster recovery looks specifically at the
technical aspects of how a company can get back
into operation using backup facilities


Disaster Recovery Concerns
● Uptime
● Quick restores with minimal or no manual steps
after the recovery
● Reliability
● Avoid corrupted file systems and that system boots
after recovery
● Cost
● DR solutions need to be affordable
● Complexity
● DR plans tend to be too complex.

Avoid Big Disaster?
● Use Cloud Service provides?
●
Feb 29th, 2012 Microsoft Azure was complete down
for several hours (http://www.infoworld.com/print/187630)
– Leap year was to blame according Microsoft
● Amazon cloud EC2 went down for almost 36 hours
in some regions (April 22, 2011)
– a "networking event" caused a domino effect??
● Google gmail out of service (May, 14, 2009)
● Cloud Services are not the Holy Grail!


GPL DR Solutions
● Mondorescue http://www.mondorescue.org/
● Started in 2000
● Stable and lots of contributors
● Needs lots of pre-requisites
● Proper documentation
● Relax and recover (rear) http://rear.github.com/
or http://rear.sourceforge.net
● Started in 2006
● Evolving rapidly, less stable versions
● Simple in use and quick; friendly developers

Getting started with rear
● Download it from
● The official tar-balls
– http://sourceforge.net/projects/rear/files/rear/
● The rear-snapshot rpm's build from SVN
– http://download.opensuse.org/repositories/home:/sschapiro/
● The official source
– https://github.com/rear/rear
● The official repo's (Fedora, EPEL and SLES)
– yum install rear
– zypper install rear


Installation of rear
● E.g. on Fedora 17
# yum install rear
Installing:
rear noarch 1.12.0-1.fc17 fedora 327 k
Installing for dependencies:
at i686 3.1.13-7.fc17 fedora 61 k
bc i686 1.06.95-6.fc17 fedora 106 k
binutils i686 2.22.52.0.1-5.fc17 fedora 3.6 M
ed i686 1.5-3.fc17 fedora 72 k
ethtool i686 2:3.2-2.fc17 fedora 93 k
genisoimage i686 1.1.11-10.fc17 fedora 338 k
….
Install 1 Package (+40 Dependent packages)
Total download size: 21 M
Installed size: 65 M
Is this ok [y/N]: y

● We also need syslinux
# yum install syslinux

● Install nfs-utils, cifs-utils, rsync if required
● Do not forget openssh(-clients)


Decide on DR strategy
● Which backup mechanism to use?
● GNU tar, rsync, bacula, commercial backup
program
● Where will the backups reside?
● NFS share, CIFS share, external USB disk, tape,
local spare disk
● Remote network location
● How shall we start the rescue image
● Via CDROM (ISO image), tape (OBDR), network
(PXE), USB disk

Backup Types
● The major backup types available are
● NETFS: NFS, CIFS, USB, TAPE
● RSYNC: rsync method
● REQUESTRESTORE, EXTERNAL
● BACULA (open source backup software)
● DP, NBU, TSM, GALAXY[7] (commercial stuff)
● Some not (yet) implemented backup types
● NSR (Legato Networker)
● CDROM


BACKUP type NETFS
pxelinux OUTPUT=PXE
network BACKUP=NETFS
isolinux
OUTPUT=ISO

(NFS|CIFS|local) extlinux
disks External USB disks
Tape drive

OUTPUT=ISO OUTPUT=OBDR OUTPUT=USB
BACKUP=NETFS BACKUP=NETFS BACKUP=NETFS


Location BACKUP_URL
● BACKUP=NETFS
● BACKUP_URL can be
● File type: BACKUP_URL=file:///directory/
● NFS type: BACKUP_URL=nfs://nfs-server/directory/
● CIFS type: BACKUP_URL=cifs://samba/directory/
● USB type: BACKUP_URL=usb:///dev/sdc1/directory/
● Tape type: BACKUP_URL=tape:///dev/nst0


Backup Program
● BACKUP=NETFS
● /usr/share/rear/conf/default.conf
● By default is BACKUP_PROG=tar
● However, BACKUP_PROG=rsync is possible for
local attached storage
● BACKUP_PROG_COMPRESS_OPTIONS="--
gzip"
● BACKUP_PROG_COMPRESS_SUFFIX=".gz"
● BACKUP_PROG_EXCLUDE=( '/tmp/*'
'/dev/shm/*' )

BACKUP_PROG_COMPRESS
_OPTIONS


/etc/rear/local.conf
● Define your settings in /etc/rear/local.conf (or
/etc/rear/site.conf)
● # grep -v -E '(^#|^$)' /etc/rear/local.conf
OUTPUT=ISO
MODULES_LOAD=( vmxnet )
● Add:
BACKUP=NETFS
BACKUP_URL=nfs://server/path
● On NFS server backup => /path/$(hostname)/


Rear dump
● View system configuration:
Relax and Recover 1.12.0 / 2011-11-22 10:21:35 +0100
Dumping out configuration and system information
This is a 'Linux-i686' system, compatible with 'Linux-i386'.
System definition:
ARCH = Linux-i386
OS = GNU/Linux


Rear help
● Usage: rear [-dDsSvV] [-r KERNEL] COMMAND [--
ARGS...]
● Available options:
● -d debug mode; log debug messages
● -D debugscript mode; log every function call
● -r KERNEL kernel version to use; current: '2.6.42.3-
2.fc15.i686.PAE'
● -s simulation mode; show what scripts rear would include
● -S step-by-step mode; acknowledge each script individually
● -v verbose mode; show more output
● -V version information


Rear help
● Usage: rear [-dDsSvV] [-r KERNEL] COMMAND [--
ARGS...]
● List of commands:
– checklayout check if the disk layout has changed
– format format and label media for use with rear
– mkbackup create rescue media and backup system
– mkbackuponly backup system without creating rescue media
– mkrescue create rescue media only
– recover recover the system; only valid during rescue
– savelayout save the disk layout of the system
– shell start a bash within rear; development tool


Disaster Recovery in Practice
● Gather system information
● Store the disk layout
● Partitioning, LVM and RAID configuration
● File systems, file system labels ...
● Boot loader (GRUB, LILO, ELILO)
● Make a system backup (OS and user data)
● Create boot-able rescue media with system
configuration (and optional with backup data)
● All steps are done “online”

Rear mkrescue
● Will create an ISO image stored as
● /tmp/rear-$(hostname).iso
● On NFS server as /path/$(hostname)/rear-
$(hostname).iso
● Inspect disklayout.conf file under
/var/lib/rear/layout/
● Try to boot from the ISO image into the
RESCUE system
● Use 'dmesg' to check if devices were found


Rear mkbackup
● Create rescue image with backup archive
● Do not forget to browse through the
/tmp/rear-$(hostname).log file for errors


BUG BUG BUG!
● What now? Look in the log file
2012-03-16 09:00:57 ERROR: BUG BUG BUG! Could not
determine size of disk
sda/sda11049kB2097kB1049kBbios_grub, please file a
bug.
Please report this as a bug to the authors of Relax
and Recover
/usr/share/rear/lib/layout-functions.sh: line
390:
/sys/block/sda/sda11049kB2097kB1049kBbios_grub/size
: No such file or directory
● Which version of rear? rear -V
● Download latest snapshot and try again

Install rear-snapshot
● # rpm -e rear
warning: /etc/rear/local.conf saved as /etc/rear/local.conf.rpmsave
● # rpm -ivh /tmp/rear-snapshot-
0.0.794-5.1.noarch.rpm
● Copy the saved local.conf.rpmsave back
# cp /etc/rear/local.conf.rpmsave
/etc/rear/local.conf
● Try again: # rear -v mkrescue
● Use debugscript mode: # rear -v -D mkrescue
● See log file for the complete debugscript output

Recovery Process in detail
● Boot system from rescue media
● Restore disk layout
● Create partitions, RAID configuration and LVM
● Create file systems (mkfs, mkswap)
● Configure file systems (labels, mount points)
● Restore the backup data
● Restore the boot loader
● Inspect & Reboot


Recover with rear (1)
● Boot rescue image and select 'recover'


● Wait until you see the login prompt
●


● Start the recover process: rear -v recover

● Have a look at the 'disklayout.conf' or continue


● Rear creates a script 'diskrestore.sh' on the fly
●

●

●

●

● Oops! An error and rear stops – what now?


● View the rear log:
+++ lvm lvcreate -l 126 -n lv_root vg
Volume group "vg" has insufficient free
space (94 extents): 126 required.
● Edit the restore script and find '126' and change
it into '94' (the target disk was smaller!)
● Continue with the recover and wait until the
restore is complete
● Inspect /mnt/local directory structure



● Ready? Reboot
● That's it – wait a while for the selinux relabeling
● Verify the restored system

Get your hands dirty?
● We hope you want to dig deeper into rear!
● Getting started:
● Use: rear -s mkbackup
to see the flow of the scripts it will execute
● Depends on BACKUP method, architecture and OS
version/brand
● Be careful: rear -s recover
follows a different flow (seems logically, but you
must understand the difference)


Where is the code?
● Main script is /usr/sbin/rear
● All the other scripts live under /usr/share/rear
● Documentation is at /usr/share/doc/rear-x.y.z
● Good news! It's all written in Bash


Where to put a script?
● mkbackup method: /usr/share/rear/...
● conf/ - configuration files (/etc/rear/*.conf read last)
● prep/ - preparation work; checking the environment
● layout/save/ - save the disk layout /var/lib/rear/layout
● rescue/ - modules, network, storage,...
● build/ - populate the initial ramdisk for our rescue image
● pack/ - create the initrd and copy kernel
● output/ - create the ISO image and copy to
OUTPUT_URL
● backup/ - make the backup archive to BACKUP_URL


Where to put a script? (2)
● recover method: /usr/share/rear/...
● conf/ - read the configuration file + /etc/rear/*.conf
● setup/ - user defined scripts to run before recover
● verify/ - to check if a recover is possible at all
● layout/prepare – recreate the disk layout
● restore/ - restore the archive from BACKUP_URL
● finalize/ - do some dirty tricks for disks, grub,...
● wrapup/ - copy the recover log to /mnt/local/root/


Example script: sysreqs.sh
● A simple script to save basic system
requirements – sysreqs.sh
● OS version; rear version
● CPU, memory
● Disk space requirements
● IP addresses in use; routes
● Copy sysreqs.sh to a flow, e.g. rescue is a good
choice
● # cp /tmp/sysreqs.sh
/usr/share/rear/rescue/GNU/Linux/96_sysreqs.sh

Test the script
● # rear -s mkrescue | grep sysreqs
Source rescue/GNU/Linux/96_sysreqs.sh
● # rear -v mkrescue
● # cat /var/lib/rear/sysreqs/MinSysReqs.txt
Zowiezo - 2012-03-31 19:57

Operating system:
LSB Version: :core-4.0-amd64:core-4.0-noarch
Distributor ID: Fedora
Description: Fedora release 16 (Verne)
Release: 16
Codename: Verne

Relax and recover version:
Relax and Recover 0.0.819 / 2012-03-16 18:37:24 +0100
...

Contacts

Web-site: http://rear.sourceforge.net/
GitHib: https://github.com/rear/rear
Mailing list: rear-users@lists.sourceforge.net

Rear Maintainer - Gratien D'haese - gratien.dhaese@it3.be

Rear Maintainer - Schlomo Schapiro - schlomo@schapiro.org

Rear Developer – Jeroen Hoekx - jeroen.hoekx@hamok.be

Rear Developer – Dag Wieers - dag@wieers.com


Linux Disaster Recovery Best Practices with rear

More Related Content

What's hot (20)

Viewers also liked (19)

Similar to Linux Disaster Recovery Best Practices with rear (20)

More from Gratien D'haese (8)

Linux Disaster Recovery Best Practices with rear