Linux boot-time

Andrea Righi - andrea@betterlinux.com
Ottimizzare i tempi di boot di Linux

Agenda
● Overview
● Case study: Raspberry Pi
● Kernel optimizations
● rootfs optimizations
● Q/A

Overview of boot phases
●
Firmware (boot-loader)
● Hardware probing
● Hardware initialization
● Kernel load and decompression
●
Kernel execution
● Core init (start_kernel)
● Driver init (initcalls)
●
User-space init
● /sbin/init
● RC scripts
●
Application start (first user impression)

Generic concepts
● Identify boot time functionalities
● Measure boot time of each functionality
● Remove unnecessary functionalities
● Optimize required functions
● Defer loading of less-important features
(modularization - LKM)
● Re-order initialization / parallelization
● Asynchronous initialization

Generic concepts
● Identify boot time functionalities
● Measure boot time of each functionality
● Remove unnecessary functionalities
● Optimize required functions
● Defer loading of less-important features
(modularization - LKM)
● Re-order initialization / parallelization
● Asynchronous initialization
Premature optimization is the root of all evil.
-Donald Knuth

Optimization
● Remove features
● The fastest code is the code you don't run!
● Defer loading of less important features
● Modularization (LKM)
● Parallelization
● async initialization (kernel/async.c)
● Reduce probing delays
● Filesystem tricks

Case study
● Raspberry Pi
● Boot time functionality
● Reponsive ssh login
● Tools and techniques
used to improve boot
time
● Focusing on cold-
boot optimizations

Boot steps
● RPi boot steps:
● bootcode.bin: 1st-stage boot loader from SoC
BCM2835 ROM
● bootloader.bin: 2nd-stage boot loader from external
SD card
● kernel.img: Linux
● rootfs: external SD card
● application: ssh (dropbear)

Boot steps
● Rpi boot steps:
● bootcode.bin: 1st-stage boot loader from SoC
BCM2835 ROM
● bootloader.bin: 2nd-stage boot loader from external
SD card
● kernel.img: Linux
● rootfs: external SD card
● application: ssh (dropbear)

Measure boot-time functionality
(kernel)
● CONFIG_PRINTK_TIME
● Configure it in “Kernel hacking” section
● Adds timing informations to kernel messages
(dmesg)
# dmesg
...
[ 0.022030] initcall bcm_mbox_init+0x0/0x38 returned 0 after 0 usecs
[ 0.022054] calling bcm_power_init+0x0/0xa4 @ 1
[ 0.022065] bcm_power: Broadcom power driver
[ 0.022080] bcm_power_open() -> 0
[ 0.022090] bcm_power_request(0, 8)
[ 0.522766] bcm_mailbox_read -> 00000080, 0
[ 0.522783] bcm_power_request -> 0
[ 0.522812] initcall bcm_power_init+0x0/0xa4 returned 0 after 488281 usecs

Kernel initcall tracer
● Introduced in Linux 2.6.28
● Add “initcall_debug” to the kernel boot options
● Allows to record the timings of each initcall in
dmesg
# dmesg | grep " initcall " | sed "s/ */ /g" | sort -n -t' ' -k8 | tail -5
[ 0.701759] initcall bcm2708_fb_init+0x0/0xc returned 0 after 32518 usecs
[ 0.794306] initcall pty_init+0x0/0x3e0 returned 0 after 90313 usecs
[ 0.660366] initcall init_kprobes+0x0/0x108 returned 0 after 94523 usecs
[ 1.224203] initcall dwc_otg_driver_init+0x0/0xb8 returned 0 after 402667 usecs
[ 0.518454] initcall bcm_power_init+0x0/0xac returned 0 after 488281 usecs

Kernel initcall tracer (example)
● dmesg > dmesg.log
● cat dmesg.log | perl
scripts/bootgraph.pl >
/boot/dmesg.svg

Raspbian
● http://www.raspbian.org
● General-purpose distro based on Debian
● Optimized for the Raspberry Pi
● compilation settings adjusted to produce hard-float
code
● 2013-05-25-wheezy-raspbian.img

Raspbian: boot time
# dmesg
...
[ 3.107163] smsc95xx 1-1.1:1.0: eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1,
smsc95xx USB 2.0 Ethernet, b8:27:eb:a9:d6:24
[ 5.664780] EXT4-fs (mmcblk0p2): recovery complete
[ 5.677326] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[ 5.689427] VFS: Mounted root (ext4 filesystem) on device 179:2.
[ 5.700523] devtmpfs: mounted
[ 5.706028] Freeing init memory: 128K
[ 6.257687] init start
[ 7.337703] udevd[154]: starting version 175
[ 8.511002] Registered led device: led0
[ 16.041293] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 25.553157] smsc95xx 1-1.1:1.0: eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1
[ 28.672178] Adding 102396k swap on /var/swap. Priority:-1 extents:2 across:507900k SS
[ 30.436046] init done
- Shell after ~6.3sec
- SSH after ~30.5sec

sysv-rc-conf
● Run-level configuration for SysV like init scripts
● disable npt
● disable plymouth
● disable rsync
● disable x11-common and lightdm
● disable alsa-utils
● disable swap (dphys-swapfile)
● disable checkfs
● disable ntp

Static IP vs DHCP
● Assign a static IP to the board (save the time to
get an IP via DHCP)
● Disable CONFIG_IP_PNP in the kernel .config
● Used to mount rootfs via NFS

Raspbian: boot time after simple
optimizations
# dmesg
...
[ 3.132731] smsc95xx 1-1.1:1.0: eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1,
smsc95xx USB 2.0 Ethernet, b8:27:eb:a9:d6:24
[ 5.730131] EXT4-fs (mmcblk0p2): recovery complete
[ 5.742783] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[ 5.754881] VFS: Mounted root (ext4 filesystem) on device 179:2.
[ 5.765958] devtmpfs: mounted
[ 7.389321] udevd[154]: starting version 175
[ 8.536932] calling leds_init+0x0/0x60 [led_class] @ 229
[ 8.537065] initcall leds_init+0x0/0x60 [led_class] returned 0 after 82 usecs
[ 8.543802] calling gpio_led_driver_init+0x0/0xc [leds_gpio] @ 229
[ 8.544067] Registered led device: led0
[ 8.545242] initcall gpio_led_driver_init+0x0/0xc [leds_gpio] returned 0 after 1344 usecs
[ 24.493346] smsc95xx 1-1.1:1.0: eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1
[ 24.991030] init done
About 25 sec to login via ssh, instead of 30 sec (16% faster)

Build a minimal distro from scratch:
requirements
● build toolchain (gcc, glibc, binutils)
● git://github.com/raspberrypi/tools.git
● HINT: use hard-float toolchain (arm-bcm2708hardfp-linux-
gnueabi)
● Linux (kernel)
● git://github.com/raspberrypi/linux.git
● busybox (rootfs)
● git://busybox.net/busybox.git
● dropbear (ssh server)
● https://matt.ucc.asn.au/dropbear/

Kernel initcalls (before optimization)
# dmesg | grep " initcall " | sed "s/ */ /g" | sort -n -t' ' -k8
...
[ 0.810927] initcall iscsi_transport_init+0x0/0x154 returned 0 after 446 usecs
[ 1.225990] initcall bcm2835_thermal_driver_init+0x0/0xc returned 0 after 451 usecs
[ 1.232472] initcall pm_qos_power_init+0x0/0xac returned 0 after 713 usecs
[ 1.230686] initcall sysctl_ipv4_init+0x0/0x98 returned 0 after 755 usecs
[ 0.810406] initcall vchiq_init+0x0/0x1dc returned 0 after 1565 usecs
[ 1.228797] initcall sdhci_drv_init+0x0/0xc returned 0 after 1829 usecs
[ 0.560116] initcall inet_init+0x0/0x260 returned 0 after 2739 usecs
[ 0.808747] initcall loop_init+0x0/0x11c returned 0 after 4852 usecs
[ 0.803722] initcall brd_init+0x0/0x1c0 returned 0 after 8230 usecs
[ 0.540586] initcall genhd_device_init+0x0/0x84 returned 0 after 9765 usecs
[ 0.556774] initcall chr_dev_init+0x0/0xd8 returned 0 after 12234 usecs
[ 0.538982] initcall param_sysfs_init+0x0/0x1d4 returned 0 after 19531 usecs
[ 0.701759] initcall bcm2708_fb_init+0x0/0xc returned 0 after 32518 usecs
[ 0.794306] initcall pty_init+0x0/0x3e0 returned 0 after 90313 usecs
[ 0.660366] initcall init_kprobes+0x0/0x108 returned 0 after 94523 usecs
[ 1.224203] initcall dwc_otg_driver_init+0x0/0xb8 returned 0 after 402667 usecs
[ 0.518454] initcall bcm_power_init+0x0/0xac returned 0 after 488281 usecs

Kernel optimizations
(initcalls)
● disable kprobes (CONFIG_KPROBES): ~95ms
● reduce the number of PTYs: ~90ms
● CONFIG_LEGACY_PTY_COUNT=256 =>
CONFIG_LEGACY_PTY_COUNT=1
● disable framebuffer: ~33ms
● remove loop device: ~5ms

Kernel initcalls (after optimization)
# dmesg | grep " initcall " | sed "s/ */ /g" | sort -n -t' ' -k8
...
[ 0.570870] initcall tun_init+0x0/0x8c returned 0 after 259 usecs
[ 0.568035] initcall des_generic_mod_init+0x0/0x10 returned 0 after 341 usecs
[ 0.975476] initcall bcm2835_cpufreq_module_init+0x0/0xc returned 0 after 358 usecs
[ 0.569678] initcall pty_init+0x0/0x194 returned 0 after 375 usecs
[ 0.561542] initcall vc_mem_init+0x0/0x1b4 returned 0 after 393 usecs
[ 0.974729] initcall bcm2835_thermal_driver_init+0x0/0xc returned 0 after 393 usecs
[ 1.018751] initcall deferred_probe_initcall+0x0/0x6c returned 0 after 395 usecs
[ 0.561078] initcall bcm2708_gpio_init+0x0/0xc returned 0 after 405 usecs
[ 1.018028] initcall pm_qos_power_init+0x0/0x60 returned 0 after 506 usecs
[ 0.559402] initcall inet_init+0x0/0x260 returned 0 after 1540 usecs
[ 1.021006] initcall net_secret_init+0x0/0x1c returned 0 after 2145 usecs
[ 1.024343] initcall initialize_hashrnd+0x0/0x1c returned 0 after 3052 usecs
[ 0.557293] initcall chr_dev_init+0x0/0xd8 returned 0 after 10978 usecs
[ 0.541305] initcall param_sysfs_init+0x0/0x19c returned 0 after 19531 usecs
[ 1.016084] initcall sdhci_drv_init+0x0/0xc returned 0 after 39250 usecs
[ 0.973739] initcall dwc_otg_driver_init+0x0/0xc4 returned 0 after 392536 usecs
[ 0.522812] initcall bcm_power_init+0x0/0xa4 returned 0 after 488281 usecs

Kernel optimizations
(other optimizations)
● preset loops_per_jiffy
● At each boot the kernel calibrates a delay loop (used later by the udelay()
function)
● 1 jiffy = time between 2 timer interrupts
● CONFIG_HZ=100 => 250ms!!!
● loop
● disable console output
● remove “console=xxx” and add “quiet” to the kernel boot parameters
● use LZO kernel decompression (CONFIG_KERNEL_LZO)
● LZO is a compression algorithm that is much faster than gzip, at the cost of a
slightly degrade compression ratio (+10%)
● reduce kernel size...
● A smaller kernel is faster to load, less code also means smaller working set
(good for caches)

Kernel image (before => after)

rootfs
● Generated using busybox and dropbear
● Hints about busybox (reduce fork()s):
● CONFIG_FEATURE_SH_STANDALONE: use applets instead of fork/exec/wait
external binaries
● CONFIG_FEATURE_SH_NOFORK: call <applet>_main() directly without
spawning another task
● Build everything with -Os
● Use mklibs to strip system libraries (rootfs is mounted to /mnt)
● mklibs -v -D -d lib2 -L /mnt/lib --ldlib lib/ld-linux.so.3 --target=arm-bcm2708-
linux-gnueabi /mnt/bin/* /mnt/sbin/* /mnt/usr/sbin/* /mnt/usr/bin/*; rm -rf lib; mv
lib2 lib
● After all these stops total rootfs size is ~3.2MB

filesystem optimizations
● Split the filesystem into read-only portion and
read/write portion
● Read-only filesystem mounts faster
● Use Squashfs + tmpfs (or aufs

root filesystem: boot time

Kernel image (before => after)
1.5GB
3.2MB

Demo: custom kernel+rootfs boot time
...
[ 1.249067] waiting eth0 to come up
[ 1.345882] usb 1-1: new high-speed USB device number 2 using dwc_otg
[ 1.346059] Indeed it is in host mode hprt0 = 00001101
[ 1.556942] hub 1-1:1.0: USB hub found
[ 1.557074] hub 1-1:1.0: 3 ports detected
[ 1.836029] usb 1-1.1: new high-speed USB device number 3 using dwc_otg
[ 1.940066] smsc95xx v1.0.4
[ 1.951858] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx
USB 2.0 Ethernet, b8:27:eb:a9:d6:24
[ 2.086695] init done
[ 3.406004] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1
- Shell after ~1.2 sec
- SSH after ~3.4 sec

Conclusion
● Why a faster boot?
● Consumer electronics products require very fast
boot times (digital camera, mobile phone, etc.)
● Fast recovery upon system failures (crashes)
● You can show to your friends your new awesome
board booting in a couple of seconds ;-)

Linux boot-time

More Related Content

What's hot (20)

Viewers also liked (6)

Similar to Linux boot-time (20)

More from Andrea Righi (6)

Recently uploaded (20)

Linux boot-time