SlideShare a Scribd company logo

Vmlinux: anatomy of bzimage and how x86 64 processor is booted

Adrian Huang, profile picture
Adrian Huang

The document provides a detailed examination of the bzImage format and the boot process of the x86_64 Linux kernel, specifically focusing on kernel version 5.11. It covers the layout of bzImage, the roles of setup.bin and compressed vmlinux, physical memory layout, and the kernel's entry point during the boot process. Key concepts include CPU operation modes, memory addressing, and the intricacies of the initialization flow through GRUB and QEMU loaders.

Software
1 of 73
Downloaded 48 times
1
2
3
4
5
6
7
Most read
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
Most read
73
Most read
vmlinux: Anatomy of bzimage and how x86_64 processor is booted Adrian Huang | May, 2021 * Based on kernel 5.11 (x86_64) – QEMU * Legacy BIOS
Agenda • bzimage: high-level overview • Layout of bzImage • ELF layout • setup.bin and compressed vmlinux • Physical memory layout • Entry point of Linux – ‘start_of_setup’@0x10200 (physical memory) • From viewpoint of GRUB and QEMU loader • Initialization flow • Compressed vmlinux • ELF layout • Physical memory layout • Initialization flow
Agenda • Layout of bzImage • ELF layout • setup.bin and compressed vmlinux • Physical memory layout • Entry point of Linux – ‘start_of_setup’@0x10200 (physical memory) • From viewpoint of GRUB and QEMU loader • Initialization flow • Compressed vmlinux • ELF layout • Physical memory layout • Initialization flow • CPU architecture knowledge ✓ Near call and far call ✓ Near jump and far jump ✓ Instruction opcode • CPU Operation Mode ✓ Real mode, protected mode and long mode (64-bit mode) ➢ Memory addressing • ELF ✓ Relocation, program header,… • GNU assembly Requisite Knowledge
bzImage: High-level Overview (1/2) Boot code (Real mode -> protected mode) Compressed vmlinux Boot code (protected mode + paging -> long mode) vmlinux.bin.gz bzImage
bzImage: High-level Overview (2/2) Boot code (Real mode -> protected mode) Compressed vmlinux Boot code (protected mode + paging -> long mode) vmlinux.bin.gz bzImage setup.bin Compressed vmlinux (Protected-mode kernel) CRC bzImage
Layout of bzImage – setup.bin setup.bin Compressed vmlinux (Protected-mode kernel) .bstext CRC .bsdata Part 1 of ‘.header’ Part 2 of ‘.header’ .entrytext Kernel Boot Section: 512 bytes (MBR) Source : arch/x86/boot/header.S <- arch/x86/boot/header.S 0x0 0x200 Offset/Size Name Description 0x1F1/1 setup_sects The size of the setup in sectors 0x01FE/2 boot_flag magic number: 0xAA55 0x200/2 jump Jump instruction 0x214/4 code32_start Boot loader hook: The address to jump to in protected mode. Default: 0x100000 ".header": Real-mode kernel header 0x1F1 Part 1 of ‘header’ Part 2 of ‘header’ .inittext .initdata .text .text32 .rodata .videocards .data .signature .bss <- arch/x86/boot/header.S <- arch/x86/boot/header.S <- arch/x86/boot/tty.c <- arch/x86/boot/*.c arch/x86/boot/bioscall.S arch/x86/boot/copy.S arch/x86/boot/ pmjump.S <- arch/x86/boot/ pmjump.S <- arch/x86/boot/*.c <- arch/x86/boot/video-*.c <- arch/x86/boot/*.c <- 4-byte signature <- arch/x86/boot/*.c bzImage ELF sections
Layout of bzImage – setup.bin setup.bin Compressed vmlinux (Protected-mode kernel) .bstext CRC .bsdata Part 1 of ‘.header’ Part 2 of ‘.header’ .entrytext Kernel Boot Section: 512 bytes (MBR) Source : arch/x86/boot/header.S <- arch/x86/boot/header.S 0x0 0x200 Offset/Size Name Description 0x1F1/1 setup_sects The size of the setup in sectors 0x01FE/2 boot_flag magic number: 0xAA55 0x200/2 jump Jump instruction 0x214/4 code32_start Boot loader hook: The address to jump to in protected mode. Default: 0x100000 ".header": Real-mode kernel header 0x1F1 Part 1 of ‘header’ Part 2 of ‘header’ short jump .inittext .initdata .text .text32 .rodata .videocards .data .signature .bss <- arch/x86/boot/header.S <- arch/x86/boot/header.S <- arch/x86/boot/tty.c <- arch/x86/boot/*.c arch/x86/boot/bioscall.S arch/x86/boot/copy.S arch/x86/boot/ pmjump.S <- arch/x86/boot/ pmjump.S <- arch/x86/boot/*.c <- arch/x86/boot/video-*.c <- arch/x86/boot/*.c <- 4-byte signature <- arch/x86/boot/*.c bzImage near call long jump 1 2 3 1 CPU Real Mode (16 bits) 2 CPU Real Mode 3 CPU Real Mode -> CPU Protected Mode (32 bits) ELF sections
Layout of bzImage – compressed vmlinux setup.bin (arch/x86/boot/setup.bin) Compressed vmlinux (Protected-mode kernel) Note ELF: arch/x86/boot/compressed/vmlinux Binary: arch/x86/boot/vmlinux.bin CRC bzImage vmlinux.bin vmlinux.bin.gz How to pack vmlinux.bin.gz? arch/x86/boot/compressed .head.text .rodata..compressed (vmlinux.bin.gz) .text .rodata .data arch/x86/boot/compressed/head_64.S 0x0 .bss .pgtable arch/x86/boot/compressed/piggy.S created by arch/x86/boot/compressed/mkpiggy.c arch/x86/boot/compressed/vmlinux.bin.gz arch/x86/boot/compressed/*.c arch/x86/boot/compressed/head_64.S arch/x86/boot/compressed/efi_thunk_64.S arch/x86/boot/compressed/head_64.S ELF Sections
Layout of bzImage – compressed vmlinux Compressed vmlinux setup.bin (arch/x86/boot/setup.bin) Compressed vmlinux (Protected-mode kernel) Note ELF: arch/x86/boot/compressed/vmlinux Binary: arch/x86/boot/vmlinux.bin CRC bzImage .head.text .rodata..compressed (vmlinux.bin.gz) .text .rodata .data arch/x86/boot/compressed/head_64.S 0x0 .bss .pgtable arch/x86/boot/compressed/piggy.S created by arch/x86/boot/compressed/mkpiggy.c arch/x86/boot/compressed/vmlinux.bin.gz arch/x86/boot/compressed/*.c arch/x86/boot/compressed/head_64.S arch/x86/boot/compressed/efi_thunk_64.S arch/x86/boot/compressed/head_64.S ELF Sections
Layout of bzImage – compressed vmlinux.bin * Symbol: Equivalent to using ‘.set’ directive * https://sourceware.org/binutils/docs/as/Setting-Symbols.html Why z_input_len/input and z_output_len/output_len? * BFD: Binary File Descriptor library - https://www.gnu.org/software/binutils/
Memory layout of bzImage – Entry Point Address Where is ‘X’? BIOS use only Typically used by MBR Reserved for MBR/BIOS Boot loader 0x00000 0x00600 0x00800 0x01000 Kernel boot section stack/heap X X+0x08000 Reserved for BIOS Command line I/O memory hole Protected-mode kernel (Compressed vmlinux) X+0x10000 0x100000 0xA0000 Boot sector entry point 0000:7C00 The kernel legacy boot sector The kernel real-mode/protected mode code For use by the kernel real-mode/protected mode code Physical Memory Kernel setup code Reference: Documentation/x86/boot.rst
Entry Point of Linux - GRUB Memory addressing in real mode [GRUB] Get the memory address for real mode code 1. gs = fs = es = ds = ss = 0x1000 2. sp = GRUB_LINUX_SETUP_STACK = 0x9000 3. cs = 0x1020, ip = 0 Registers configured by GRUB Kernel boot section 0x10000 0x10200 Physical Memory GRUB loads ‘setup.bin’ at address 0x10000 0 ds = es = fs = gs = ss cs stack ss:sp = 0x1FFF0 protected mode real mode Kernel setup code
Entry Point of Linux - GRUB Memory addressing in real mode [GRUB] Get the memory address for real mode code 1. gs = fs = es = ds = ss = 0x1000 2. sp = GRUB_LINUX_SETUP_STACK = 0x9000 3. cs = 0x1020, ip = 0 Registers configured by GRUB Kernel boot section 0x10000 0x10200 Physical Memory GRUB loads ‘setup.bin’ at address 0x10000 0 ds = es = fs = gs = ss cs stack ss:sp = 0x1FFF0 protected mode real mode Kernel setup code 1. QEMU loader and GRUB load ‘setup.bin’ at address 0x10000 2. QEMU loader sets SS:SP = 1000:FFF0 while GRUB sets SS:SP 1000:9000
Entry Point of Linux: QEMU loader Kernel boot section 0x10000 0x10200 Physical Memory QEMU loader loads ‘setup.bin’ at address 0x10000 0 ds = es = fs = gs = ss cs stack ss:sp = 0x1FFF0 protected mode real mode Kernel setup code 1 2 3 4 5 6 7 ds = es = fs = gs = ss = segment_addr = 0x1000 esp = stack_addr = cmdline_addr - setup_addr – 16 = 0x20000 – 0x10000 – 16 = 0x10000 – 16 = 0xfff0 cs = 0x1020, ip = 0 Registers configured by QEMU loader 5 6 7 Prepare for far return 8 far return: change ‘cs’ by means of CPU arch itself
Entry Point of Linux: QEMU loader – Near and Far calls 3 4 5 6 7 Prepare for far return 8 far return: change ‘cs’ by means of CPU arch itself
Entry Point of Linux: QEMU loader Kernel boot section 0x10000 0x10200 Physical Memory QEMU loader loads ‘setup.bin’ at address 0x10000 0 ds = es = fs = gs = ss cs stack sp = 0x1FFF0 (ss:0xFFF0) protected mode real mode Kernel setup code Make sure setup.bin is loaded at 0x10000 Make sure vmlinux.bin is loaded at 0x100000 Address of setup.bin Address of vmlinux.bin
arch/x86/boot/setup.ld arch/x86/boot/header.S 1 2 Entry Point of Linux: GNU Linker [GNU Linker] ENTRY() command * First executable instruction in an output file → entry point * ENTRY() is one of choosing the entry point -- the `-e' entry command-line option -- the ENTRY(symbol) command in a linker control script -- the value of the symbol start, if present -- the address of the first byte of the .text section, if present; -- the address 0
arch/x86/boot/setup.ld 1 Entry Point of Linux: GNU Linker [GNU Linker] ENTRY() command * First executable instruction in an output file → entry point * ENTRY() is one of choosing the entry point -- the `-e' entry command-line option -- the ENTRY(symbol) command in a linker control script -- the value of the symbol start, if present -- the address of the first byte of the .text section, if present; -- the address 0 Kernel boot section 0x10000 0x10200 Physical Memory QEMU loader loads ‘setup.bin’ at address 0x10000 0 ds = es = fs = gs = ss cs stack sp = 0x1FFF0 (ss:0xFFF0) protected mode real mode Kernel setup code
Entry Point of Linux: start_of_setup - GDB Kernel boot section 0x10000 0x10200 Physical Memory Boot loader loads ‘setup.bin’ at address 0x10000 0 gs = fs = es = ds = ss cs stack sp = 0x1FFF0 (ss:0xFFF0) protected mode real mode Kernel setup code
Entry Point of Linux: start_of_setup - GDB Kernel boot section 0x10000 0x10200 Physical Memory Boot loader loads ‘setup.bin’ at address 0x10000 0 gs = fs = es = ds = ss cs stack sp = 0x1FFF0 (ss:0xFFF0) protected mode real mode Kernel setup code
Entry Point of Linux: start_of_setup – short jump Kernel boot section 0x10000 0x10200 Physical Memory Boot loader loads ‘setup.bin’ at address 0x10000 0 gs = fs = es = ds = ss cs stack sp = 0x1FFF0 (ss:0xFFF0) protected mode real mode Kernel setup code Offset/Size Name Description 0x1F1/1 setup_sects The size of the setup in sectors 0x01FE/2 boot_flag magic number: 0xAA55 0x200/2 jump Jump instruction 0x214/4 code32_start Boot loader hook: The address to jump to in protected mode. Default: 0x100000 ".header": Real-mode kernel header
Entry Point of Linux: start_of_setup – short jump 0x26c – 0x202 = 0x6a
Entry Point of Linux: start_of_setup Call Path Kernel boot section 0x10000 0x10200 Boot loader loads ‘setup.bin’ at address 0x10000 0 gs = fs = es = ds = ss = cs cs stack sp = 0x1FFF0 (ss:0xFFF0) protected mode real mode Kernel setup code Physical Memory lretw instruction: Far Return Operation ‘l’ prefix: far control transfer ‘w’ suffix: word (16 bits)
Entry Point of Linux: start_of_setup Call Path Kernel boot section 0x10000 0x10200 Boot loader loads ‘setup.bin’ at address 0x10000 0 gs = fs = es = ds = ss = cs cs stack sp = 0x1FFF0 (ss:0xFFF0) protected mode real mode Kernel setup code Physical Memory lretw instruction: Far Return Operation ‘l’ prefix: far control transfer ‘w’ suffix: word (16 bits) 1 1 2 2 3 3
Entry Point of Linux: start_of_setup Kernel boot section 0x10000 0x10200 Boot loader loads ‘setup.bin’ at address 0x10000 0 gs = fs = es = ds = ss = cs cs stack sp = 0x1FFF0 (ss:0xFFF0) protected mode real mode Kernel setup code Physical Memory Call Path lretw instruction: Far Return Operation ‘l’ prefix: far control transfer ‘w’ suffix: word (16 bits)
Entry Point of Linux: start_of_setup – Why to align CS? Kernel boot section 0x10000 0x10200 Boot loader loads ‘setup.bin’ at address 0x10000 0 gs = fs = es = ds = ss = cs cs stack sp = 0x1FFF0 (ss:0xFFF0) protected mode real mode Kernel setup code Physical Memory Call Path lretw instruction: Far Return Operation ‘l’ prefix: far control transfer ‘w’ suffix: word (16 bits) If cs is not align with ds, ds and es are incorrect after returning from ‘intcall’.
Entry Point of Linux: start_of_setup – data & bss section Call Path Kernel boot section 0x10000 0x10200 Boot loader loads ‘setup.bin’ at address 0x10000 0 gs = fs = es = ds = ss= cs stack sp = 0x1FFF0 protected mode real mode Kernel setup code Kernel boot section 0x10000 0x10200 0 gs = fs = es = ds = ss = cs stack sp = 0x1FFF0 protected mode real mode Kernel setup code BSS Section _end __bss_start __bss_end Data Section Physical Memory
Entry Point of Linux: start_of_setup -> main() Call Path
Entry Point of Linux: start_of_setup -> main()
Entry Point of Linux: start_of_setup -> main()
Entry Point of Linux: start_of_setup -> main() -> copy_boot_params() Call Path • copy setup header into boot parameter block (struct boot_params: arch/x86/include/uapi/asm/bootparam.h) o `struct setup_header hdr` in boot_params ▪ Contain the same fields defined in Linux boot protocol. Those fields are configured by boot loader and kernel compile/build time
Call Path • console_init() o Initialize the corresponding serial port if command line has ‘earlyprintk’ parameter Entry Point of Linux: start_of_setup -> main() -> console_init() – (1/2) Kernel boot section 0x10000 0x10200 0 gs = fs = es = ds = ss = cs stack sp = 0x1FFF0 protected mode real mode Kernel setup code BSS Section _end __bss_start __bss_end Data Section Kernel Command Line 0x20000 QEMU Loader Physical Memory
Call Path • console_init() o Initialize the corresponding serial port if command line has ‘earlyprintk’ parameter Entry Point of Linux: start_of_setup -> main() -> console_init() – (2/2) Kernel boot section 0x10000 0x10200 0 gs = fs = es = ds = ss = cs stack sp = 0x1FFF0 protected mode real mode Kernel setup code BSS Section _end __bss_start __bss_end Data Section Kernel Command Line 0x20000 Physical Memory
Call Path • init_heap() • Discussion in the next few slides • validate_cpu() o Check CPU flags o Check if long mode (x86_64) is available o [AMD – K7 Processor] Turn SSE+SSE2 on if they are missing in CPU flags • detect_memory() o Use different program interfaces (0xe820, 0xe801 and 0x88) for memory detection o 0xe820 ▪ Fill boot_params.e820_table based on e820 map Entry Point of Linux: start_of_setup -> main() -> validate_cpu() & detect_memory() Kernel boot section 0x10000 0x10200 0 gs = fs = es = ds = ss = cs stack sp = 0x1FFF0 protected mode real mode Kernel setup code BSS Section _end __bss_start __bss_end Data Section Kernel Command Line 0x20000 Physical Memory
Call Path • init_heap o Setup the heap space if the ‘CAN_USE_HEAP’ flag (0x80) is set in loadflags of the kernel setup header. Entry Point of Linux: start_of_setup -> main() -> init_heap() (1/2)
Call Path Entry Point of Linux: start_of_setup -> main() -> init_heap() (2/2) heap: allocate heap if CAN_USE_HEAP’ flag (0x80) is set No heap sp (STACK_SIZE = 0x400) Kernel boot section 0x10000 0x10200 stack protected mode real mode Kernel setup code BSS Section Unused Area __bss_start __bss_end HEAP = heap_end = _end Data Section sp (STACK_SIZE = 0x400) Kernel boot section 0x10000 0x10200 stack protected mode real mode Kernel setup code BSS Section Heap __bss_start __bss_end HEAP = _end heap_end Data Section gs = fs = es = ds = ss = cs gs = fs = es = ds = ss = cs
go_to_protected_mode GDT_ENTRY_BOOT_DS GDT_ENTRY_BOOT_CS NULL NULL 0 1 2 3 GDT_ENTRY_BOOT_TSS 4 Descriptor Table: boot_gdt System Memory 0 0xFFFFFFFF limit Base Address GDTR x86 Segmentation: Address Translation setup_gdt(): Setup 4G memory space for CS/DS Call Path
protected_mode_jump (1/6)
protected_mode_jump – ljmpl instruction: ignore ‘.Lin_pm32’ relocation (2/6) 0x30cc Jump (absolute address) to the wrong location sp (STACK_SIZE = 0x400) Kernel boot section 0x10000 0x10200 stack protected mode real mode Kernel setup code BSS Section Heap __bss_start __bss_end HEAP = _end heap_end Data Section gs = fs = es = ds = ss = cs setup.bin generation Physical Memory
protected_mode_jump – ljmpl instruction - relocation (3/6) sp (STACK_SIZE = 0x400) Kernel boot section 0x10000 0x10200 stack protected mode real mode Kernel setup code BSS Section Heap __bss_start __bss_end HEAP = _end heap_end Data Section gs = fs = es = ds = ss = cs Relocation for absolute address of ‘ljmpl’ ljmpl Physical Memory
Relocation for absolute address of ‘ljmpl’ protected_mode_jump – ljmpl instruction (4/6) sp (STACK_SIZE = 0x400) Kernel boot section 0x10000 0x10200 stack protected mode real mode Kernel setup code BSS Section Heap __bss_start __bss_end HEAP = _end heap_end Data Section gs = fs = es = ds = ss = cs ljmpl Physical Memory
protected_mode_jump – ljmpl instruction: instruction format (5/6)
protected_mode_jump – ljmpl instruction: instruction format (6/6)
Protected mode: ‘.Lin_pm32’ (1/2) [real mode] SP configuration [protected mode] SP configuration `addl %ebx, %esp` in label “.Lin_pm32” 0x1FF80 (SS:SP = 0x1000:0xFF80) Kernel boot section 0x10000 0x10200 stack protected mode real mode Kernel setup code BSS Section Heap __bss_start __bss_end HEAP = _end heap_end esp = 0x1FF80 Kernel boot section 0x10000 (ebx) 0x10200 stack protected mode real mode Kernel setup code BSS Section Heap __bss_start __bss_end HEAP = _end heap_end Data Section Data Section
Data Section 1 2 4 3 Protected mode: ‘.Lin_pm32’ (2/2) X = 0x10000 esp = 0x1FF80 Kernel boot section 0x10200 stack protected mode real mode Kernel setup code BSS Section Heap __bss_start __bss_end HEAP = _end heap_end Reserved for BIOS command line I/O memory hole Protected-mode kernel code (compressed vmlinux) X+0x10000 0xA0000 0x100000 jmpl *%eax 5 Physical Memory Call Path
Compressed vmlinux: memory layout (1/10) .head.text – startup_32 0x100000 (ebp register) 0x100200 decompressed vmlinux.bin.bz .head.text – startup_64 0x1000000 compressed vmlinux (Relocation) 0x1000000 + boot_param.init_size 0x1000000 + boot_param.init_size - _end (rbx register) vmlinux.bin.gz .text .rodata .data .bss .pgtable _end 0x100000 + _end boot_heap (size: 0x10000) boot_stack (size: 0x4000) … input_data input_data_end Memory Layout 32-bit entry point _bss
Compressed vmlinux: boot_stack & boot_heap in .bss (2/10) .head.text – startup_32 0x100000 (ebp register) 0x100200 decompressed vmlinux.bin.bz .head.text – startup_64 0x1000000 compressed vmlinux (Relocation) 0x1000000 + boot_param.init_size 0x1000000 + boot_param.init_size - _end (rbx register) vmlinux.bin.gz .text .rodata .data .bss .pgtable _end 0x100000 + _end boot_heap (size: 0x10000) boot_stack (size: 0x4000) … input_data input_data_end Memory Layout 32-bit entry point _bss
Compressed vmlinux: High-level Overview (3/10) Why relocation • Base address of 32-bit Linux kernel entry point: 0x100000 • Default base address of Linux kernel: CONFIG_PHYSICAL_START=0x1000000 • Use Case • kdump: a recuse kernel is loaded to a different address • PIE (Position independent Executable) and PIC (Position Independent Code)
Compressed vmlinux: startup_32: 32-bit entry point (4/10) 1 1
Compressed vmlinux: startup_32 (5/10) 1 1 Get the loading address
2 2 Compressed vmlinux: startup_32 (6/10)
Compressed vmlinux: startup_32: Init 4-level page table (7/10) Sign-extend Page Map Level-4 Offset Page Directory Pointer Offset Page Directory Offset Physical Page Offset 0 30 21 39 20 38 29 47 48 63 PML4E #0 PDPTE #3 Data Page Map Level-4 Table Page Directory Pointer Table Page Directory Table 40 9 9 9 Linear Address CR3 PDPTE #2 PDPTE #1 PDPTE #0 PDE #1535 PDE #1024 . . PDE #2047 PDE #1536 . . PDE #511 PDE #0 . . PDE #1023 PDE #512 . . 2MBbyte Physical Page 40 40 31 21 [Paging] Identity mapping for 0-4GB memory space
Compressed vmlinux: startup_32: Init 4-level page table (8/10) Reference: Section 4.1 “PAGING MODES AND CONTROL BITS”, Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 3 (3A, 3B, 3C & 3D): System Programming Guide
Compressed vmlinux: startup_32: Init 4-level page table (9/10)
Compressed vmlinux: far return to startup_64 (10/10) rva(startup_64) = 0x200 ebp = 0x100000 eax = 0x100000 + 0x200 = 0x100200
Compressed vmlinux: startup_64 2 3 Why to reload CS? (Commit “34bb49229f19”) When the pre-decompression code loads its first GDT in startup_64, it is still running on the CS value of the previous GDT. In the case of SEV-ES this is the EFI GDT. It can be anything depending on what has loaded the kernel (EFI, legacy boot code, container runtime, etc.)
Compressed vmlinux: [.text] .Lrelocated (1/5) 4 5 Why to call initialize_identity_maps()?
Compressed vmlinux: [.text] .Lrelocated (2/5) 4 5 Why to map boot_params and command line?
Compressed vmlinux: parse_elf (3/5) 4 ELF Header 0x1000000 decompressed vmlinux.bin.bz (vmlinux.bin – ELF format) program headers program header #0 (.text, .rodata, .pci_fixup….) 0x1200000 program header #1 (.data .vvar) program header #2 (.init.text .altinstr_aux …) 0x1a00000 0x1ac2000 program header #3 (.notes) 0x18886b0 0x1000000 program header #0 (.text, .rodata, .pci_fixup….) 0x1800000 program header #1 (.data .vvar) program header #2 (.init.text .altinstr_aux …) 0x18c2000 Physical memory Physical memory
Compressed vmlinux: handle_relocations (4/5) 4 CONFIG_RELOCATABLE • Retain relocation information (generate .rel.* or rela.* sections) when building a kernel image, so it can be loaded someplace besides the default address (CONFIG_PHYSICAL_START = 16MB). • Use case: kdump kernel (recovery kernel) handle_relocations() - Relocation if CONFIG_X86_NEED_RELOCS is set • Depend on RANDOMIZE_BASE || (X86_32 && RELOCATABLE) • Scan relocation tables (.rel.* or .rela.* sections) for symbol relocation
Compressed vmlinux: handle_relocations (5/5) 4 vmlinux.bin.bz vmlinux.bin vmlinux.relocs handle_relocations(): Perform relocation backwards from the end of the decompressed vmlinux 64-bit relocation address 0 32-bit relocation address 0 -R section_name: Remove any section matching section_name -S or strip-all: Do not copy relocation and symbol information from the source file objdump options
Recap setup.bin (arch/x86/boot/setup.bin) Compressed vmlinux (Protected-mode kernel) Note ELF: arch/x86/boot/compressed/vmlinux Binary: arch/x86/boot/vmlinux.bin CRC bzImage
[More info] bzImage = vmlinuz On a physical machine Source code: arch/x86/boot/Makefile, arch/x86/boot/install.sh
Reference • The Linux/x86 Boot Protocol, Documentation/x86/boot.rst • Intel® 64 and IA-32 Architectures Software Developer’s Manual • https://wdv4758h.github.io/notes/blog/linux-kernel-boot.html • Linux insides, https://0xax.gitbooks.io/linux-insides/content/
Appendix
gdb: Preparation for debugging real-mode of Linux kernel (1/2) Github: https://github.com/AdrianHuang/gdb-linux-real-mode
gdb: Preparation for debugging real-mode of Linux kernel (2/2) Github: https://github.com/AdrianHuang/gdb-linux-real-mode
initialize_identity_maps x86_mapping_info void *(*alloc_pgt_page)(void *) void *context unsigned long page_flag unsigned long offset alloc_pgt_data unsigned char *pgt_buf unsigned long pgt_buf_size unsigned long pgt_buf_offset bool direct_gbpages unsigned long kernpg_flag
UEFI booting flow – EFI boot stub: Entry point AddressOfEntryPoint (efi_pe_entry): 0x18d84a ImageBase = 0x1000000 Physical address of AddressofEntryPoint = 0x1000000 + 0x18d84a = 0x118d84a
UEFI booting flow – EFI Handover protocol
UEFI booting flow – EFI Handover protocol
UEFI booting flow – EFI Handover protocol Where is the address of bzimage loaded by boot loader?
UEFI booting: call path

More Related Content

PDF
Physical Memory Management.pdf
Adrian Huang
 
PDF
Decompressed vmlinux: linux kernel initialization from page table configurati...
Adrian Huang
 
PPTX
Slab Allocator in Linux Kernel
Adrian Huang
 
PDF
Process Address Space: The way to create virtual address (page table) of user...
Adrian Huang
 
PDF
Page cache in Linux kernel
Adrian Huang
 
PDF
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
Adrian Huang
 
PDF
Physical Memory Models.pdf
Adrian Huang
 
PDF
spinlock.pdf
Adrian Huang
 
Physical Memory Management.pdf
Adrian Huang
 
Decompressed vmlinux: linux kernel initialization from page table configurati...
Adrian Huang
 
Slab Allocator in Linux Kernel
Adrian Huang
 
Process Address Space: The way to create virtual address (page table) of user...
Adrian Huang
 
Page cache in Linux kernel
Adrian Huang
 
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
Adrian Huang
 
Physical Memory Models.pdf
Adrian Huang
 
spinlock.pdf
Adrian Huang
 

What's hot (20)

PPTX
Linux Kernel Booting Process (2) - For NLKB
shimosawa
 
PPTX
Linux Kernel Booting Process (1) - For NLKB
shimosawa
 
PDF
Memory Mapping Implementation (mmap) in Linux Kernel
Adrian Huang
 
PDF
Arm device tree and linux device drivers
Houcheng Lin
 
PDF
Kdump and the kernel crash dump analysis
Buland Singh
 
PPTX
Linux Initialization Process (1)
shimosawa
 
PDF
Memory Management with Page Folios
Adrian Huang
 
PPTX
Linux Initialization Process (2)
shimosawa
 
PPTX
U-Boot presentation 2013
Wave Digitech
 
PDF
U-Boot - An universal bootloader
Emertxe Information Technologies Pvt Ltd
 
PPTX
Linux kernel debugging
Hao-Ran Liu
 
PPT
U boot porting guide for SoC
Macpaul Lin
 
PDF
Part 02 Linux Kernel Module Programming
Tushar B Kute
 
PDF
malloc & vmalloc in Linux
Adrian Huang
 
PDF
Embedded Linux BSP Training (Intro)
RuggedBoardGroup
 
PPT
Introduction to Linux Kernel by Quontra Solutions
QUONTRASOLUTIONS
 
PDF
systemd
nussbauml
 
PDF
Reverse Mapping (rmap) in Linux Kernel
Adrian Huang
 
PPT
U Boot or Universal Bootloader
Satpal Parmar
 
PDF
Linux kernel
Mahmoud Shiri Varamini
 
Linux Kernel Booting Process (2) - For NLKB
shimosawa
 
Linux Kernel Booting Process (1) - For NLKB
shimosawa
 
Memory Mapping Implementation (mmap) in Linux Kernel
Adrian Huang
 
Arm device tree and linux device drivers
Houcheng Lin
 
Kdump and the kernel crash dump analysis
Buland Singh
 
Linux Initialization Process (1)
shimosawa
 
Memory Management with Page Folios
Adrian Huang
 
Linux Initialization Process (2)
shimosawa
 
U-Boot presentation 2013
Wave Digitech
 
U-Boot - An universal bootloader
Emertxe Information Technologies Pvt Ltd
 
Linux kernel debugging
Hao-Ran Liu
 
U boot porting guide for SoC
Macpaul Lin
 
Part 02 Linux Kernel Module Programming
Tushar B Kute
 
malloc & vmalloc in Linux
Adrian Huang
 
Embedded Linux BSP Training (Intro)
RuggedBoardGroup
 
Introduction to Linux Kernel by Quontra Solutions
QUONTRASOLUTIONS
 
systemd
nussbauml
 
Reverse Mapping (rmap) in Linux Kernel
Adrian Huang
 
U Boot or Universal Bootloader
Satpal Parmar
 
Linux kernel
Mahmoud Shiri Varamini
 
Ad

Similar to Vmlinux: anatomy of bzimage and how x86 64 processor is booted (20)

PPTX
Linux Kernel Tour
samrat das
 
PPTX
U-Boot Porting on New Hardware
RuggedBoardGroup
 
PPT
How to build and load linux to embedded system
Игорь Медведев
 
PDF
Grub2 Booting Process
Mike Wang
 
PDF
Linux Porting
Champ Yen
 
PPTX
Raspberry Pi tutorial
艾鍗科技
 
PPTX
“Linux Kernel CPU Hotplug in the Multicore System”
GlobalLogic Ukraine
 
PPT
Bootstrap process of u boot (NDS32 RISC CPU)
Macpaul Lin
 
PDF
005 skyeye
Sherif Mousa
 
ODP
[Defcon] Hardware backdooring is practical
Moabi.com
 
PDF
Launch the First Process in Linux System
Jian-Hong Pan
 
PDF
Beagleboard xm-setup
Premjith Achemveettil
 
ODP
Kernel compilation
mcganesh
 
PDF
LCU14 302- How to port OP-TEE to another platform
Linaro
 
PPTX
Vagrant, Ansible, and OpenStack on your laptop
Lorin Hochstein
 
PPTX
PV-Drivers for SeaBIOS using Upstream Qemu
The Linux Foundation
 
PPTX
망고100 보드로 놀아보자 7
종인 전
 
PPT
Linux Booting Steps
Anando Kumar Paul
 
PPT
Basic Linux Internals
mukul bhardwaj
 
PDF
Project ACRN configuration scenarios and config tool
Project ACRN
 
Linux Kernel Tour
samrat das
 
U-Boot Porting on New Hardware
RuggedBoardGroup
 
How to build and load linux to embedded system
Игорь Медведев
 
Grub2 Booting Process
Mike Wang
 
Linux Porting
Champ Yen
 
Raspberry Pi tutorial
艾鍗科技
 
“Linux Kernel CPU Hotplug in the Multicore System”
GlobalLogic Ukraine
 
Bootstrap process of u boot (NDS32 RISC CPU)
Macpaul Lin
 
005 skyeye
Sherif Mousa
 
[Defcon] Hardware backdooring is practical
Moabi.com
 
Launch the First Process in Linux System
Jian-Hong Pan
 
Beagleboard xm-setup
Premjith Achemveettil
 
Kernel compilation
mcganesh
 
LCU14 302- How to port OP-TEE to another platform
Linaro
 
Vagrant, Ansible, and OpenStack on your laptop
Lorin Hochstein
 
PV-Drivers for SeaBIOS using Upstream Qemu
The Linux Foundation
 
망고100 보드로 놀아보자 7
종인 전
 
Linux Booting Steps
Anando Kumar Paul
 
Basic Linux Internals
mukul bhardwaj
 
Project ACRN configuration scenarios and config tool
Project ACRN
 
Ad

More from Adrian Huang (6)

PDF
Linux Synchronization Mechanism: RCU (Read Copy Update)
Adrian Huang
 
PPTX
qemu + gdb + sample_code: Run sample code in QEMU OS and observe Linux Kernel...
Adrian Huang
 
PDF
semaphore & mutex.pdf
Adrian Huang
 
PDF
Memory Compaction in Linux Kernel.pdf
Adrian Huang
 
PDF
Linux Kernel - Virtual File System
Adrian Huang
 
PDF
Anatomy of the loadable kernel module (lkm)
Adrian Huang
 
Linux Synchronization Mechanism: RCU (Read Copy Update)
Adrian Huang
 
qemu + gdb + sample_code: Run sample code in QEMU OS and observe Linux Kernel...
Adrian Huang
 
semaphore & mutex.pdf
Adrian Huang
 
Memory Compaction in Linux Kernel.pdf
Adrian Huang
 
Linux Kernel - Virtual File System
Adrian Huang
 
Anatomy of the loadable kernel module (lkm)
Adrian Huang
 

Recently uploaded (20)

PDF
System and Network Administration Chapter 2
jaramharo
 
PDF
Odoo Companies in India – Driving Business Transformation.pdf
azhuhardeen1968
 
PPTX
history of c programming in notes for students .pptx
Vijayakumari829030
 
PPTX
CHAPTER 2 - PM Management and IT Context
KevinPutra14
 
PDF
Raksha Bandhan Grocery Pricing Trends in India 2025.pdf
Actowiz Solustions
 
PPTX
ManageIQ - Sprint 268 Review - Slide Deck
ManageIQ
 
PDF
Flood Susceptibility Mapping Using Image-Based 2D-CNN Deep Learnin. Overview ...
lawrenceomai2
 
PDF
Why TechBuilder is the Future of Pickup and Delivery App Development (1).pdf
TechBuilder
 
PDF
Design an Analysis of Algorithms II-SECS-1021-03
DilanEscobar1
 
PPTX
Transform Your Business with a Software ERP System
Canada Software Company
 
PDF
How Creative Agencies Leverage Project Management Software.pdf
Orangescrum
 
PDF
PTS Company Brochure 2025 (1).pdf.......
pts464036
 
PPTX
ISO 45001 Occupational Health and Safety Management System
EHA Soft Solutions
 
PPTX
CHAPTER 12 - CYBER SECURITY AND FUTURE SKILLS (1) (1).pptx
AnujKumar530915
 
PPTX
L1 - Introduction to python Backend.pptx
MoizAhmed480282
 
PDF
Adobe Illustrator 28.6 Crack My Vision of Vector Design
ghrom2211g
 
PDF
Addressing The Cult of Project Management Tools-Why Disconnected Work is Hold...
OnePlan Solutions
 
PPTX
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
PDF
How to Migrate SBCGlobal Email to Yahoo Easily
raymondjones7273
 
PDF
Digital Strategies for Manufacturing Companies
Virendra Rai, PMP
 
System and Network Administration Chapter 2
jaramharo
 
Odoo Companies in India – Driving Business Transformation.pdf
azhuhardeen1968
 
history of c programming in notes for students .pptx
Vijayakumari829030
 
CHAPTER 2 - PM Management and IT Context
KevinPutra14
 
Raksha Bandhan Grocery Pricing Trends in India 2025.pdf
Actowiz Solustions
 
ManageIQ - Sprint 268 Review - Slide Deck
ManageIQ
 
Flood Susceptibility Mapping Using Image-Based 2D-CNN Deep Learnin. Overview ...
lawrenceomai2
 
Why TechBuilder is the Future of Pickup and Delivery App Development (1).pdf
TechBuilder
 
Design an Analysis of Algorithms II-SECS-1021-03
DilanEscobar1
 
Transform Your Business with a Software ERP System
Canada Software Company
 
How Creative Agencies Leverage Project Management Software.pdf
Orangescrum
 
PTS Company Brochure 2025 (1).pdf.......
pts464036
 
ISO 45001 Occupational Health and Safety Management System
EHA Soft Solutions
 
CHAPTER 12 - CYBER SECURITY AND FUTURE SKILLS (1) (1).pptx
AnujKumar530915
 
L1 - Introduction to python Backend.pptx
MoizAhmed480282
 
Adobe Illustrator 28.6 Crack My Vision of Vector Design
ghrom2211g
 
Addressing The Cult of Project Management Tools-Why Disconnected Work is Hold...
OnePlan Solutions
 
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
How to Migrate SBCGlobal Email to Yahoo Easily
raymondjones7273
 
Digital Strategies for Manufacturing Companies
Virendra Rai, PMP
 

Vmlinux: anatomy of bzimage and how x86 64 processor is booted

  • 1. vmlinux: Anatomy of bzimage and how x86_64 processor is booted Adrian Huang | May, 2021 * Based on kernel 5.11 (x86_64) – QEMU * Legacy BIOS
  • 2. Agenda • bzimage: high-level overview • Layout of bzImage • ELF layout • setup.bin and compressed vmlinux • Physical memory layout • Entry point of Linux – ‘start_of_setup’@0x10200 (physical memory) • From viewpoint of GRUB and QEMU loader • Initialization flow • Compressed vmlinux • ELF layout • Physical memory layout • Initialization flow
  • 3. Agenda • Layout of bzImage • ELF layout • setup.bin and compressed vmlinux • Physical memory layout • Entry point of Linux – ‘start_of_setup’@0x10200 (physical memory) • From viewpoint of GRUB and QEMU loader • Initialization flow • Compressed vmlinux • ELF layout • Physical memory layout • Initialization flow • CPU architecture knowledge ✓ Near call and far call ✓ Near jump and far jump ✓ Instruction opcode • CPU Operation Mode ✓ Real mode, protected mode and long mode (64-bit mode) ➢ Memory addressing • ELF ✓ Relocation, program header,… • GNU assembly Requisite Knowledge
  • 4. bzImage: High-level Overview (1/2) Boot code (Real mode -> protected mode) Compressed vmlinux Boot code (protected mode + paging -> long mode) vmlinux.bin.gz bzImage
  • 5. bzImage: High-level Overview (2/2) Boot code (Real mode -> protected mode) Compressed vmlinux Boot code (protected mode + paging -> long mode) vmlinux.bin.gz bzImage setup.bin Compressed vmlinux (Protected-mode kernel) CRC bzImage
  • 6. Layout of bzImage – setup.bin setup.bin Compressed vmlinux (Protected-mode kernel) .bstext CRC .bsdata Part 1 of ‘.header’ Part 2 of ‘.header’ .entrytext Kernel Boot Section: 512 bytes (MBR) Source : arch/x86/boot/header.S <- arch/x86/boot/header.S 0x0 0x200 Offset/Size Name Description 0x1F1/1 setup_sects The size of the setup in sectors 0x01FE/2 boot_flag magic number: 0xAA55 0x200/2 jump Jump instruction 0x214/4 code32_start Boot loader hook: The address to jump to in protected mode. Default: 0x100000 ".header": Real-mode kernel header 0x1F1 Part 1 of ‘header’ Part 2 of ‘header’ .inittext .initdata .text .text32 .rodata .videocards .data .signature .bss <- arch/x86/boot/header.S <- arch/x86/boot/header.S <- arch/x86/boot/tty.c <- arch/x86/boot/*.c arch/x86/boot/bioscall.S arch/x86/boot/copy.S arch/x86/boot/ pmjump.S <- arch/x86/boot/ pmjump.S <- arch/x86/boot/*.c <- arch/x86/boot/video-*.c <- arch/x86/boot/*.c <- 4-byte signature <- arch/x86/boot/*.c bzImage ELF sections
  • 7. Layout of bzImage – setup.bin setup.bin Compressed vmlinux (Protected-mode kernel) .bstext CRC .bsdata Part 1 of ‘.header’ Part 2 of ‘.header’ .entrytext Kernel Boot Section: 512 bytes (MBR) Source : arch/x86/boot/header.S <- arch/x86/boot/header.S 0x0 0x200 Offset/Size Name Description 0x1F1/1 setup_sects The size of the setup in sectors 0x01FE/2 boot_flag magic number: 0xAA55 0x200/2 jump Jump instruction 0x214/4 code32_start Boot loader hook: The address to jump to in protected mode. Default: 0x100000 ".header": Real-mode kernel header 0x1F1 Part 1 of ‘header’ Part 2 of ‘header’ short jump .inittext .initdata .text .text32 .rodata .videocards .data .signature .bss <- arch/x86/boot/header.S <- arch/x86/boot/header.S <- arch/x86/boot/tty.c <- arch/x86/boot/*.c arch/x86/boot/bioscall.S arch/x86/boot/copy.S arch/x86/boot/ pmjump.S <- arch/x86/boot/ pmjump.S <- arch/x86/boot/*.c <- arch/x86/boot/video-*.c <- arch/x86/boot/*.c <- 4-byte signature <- arch/x86/boot/*.c bzImage near call long jump 1 2 3 1 CPU Real Mode (16 bits) 2 CPU Real Mode 3 CPU Real Mode -> CPU Protected Mode (32 bits) ELF sections
  • 8. Layout of bzImage – compressed vmlinux setup.bin (arch/x86/boot/setup.bin) Compressed vmlinux (Protected-mode kernel) Note ELF: arch/x86/boot/compressed/vmlinux Binary: arch/x86/boot/vmlinux.bin CRC bzImage vmlinux.bin vmlinux.bin.gz How to pack vmlinux.bin.gz? arch/x86/boot/compressed .head.text .rodata..compressed (vmlinux.bin.gz) .text .rodata .data arch/x86/boot/compressed/head_64.S 0x0 .bss .pgtable arch/x86/boot/compressed/piggy.S created by arch/x86/boot/compressed/mkpiggy.c arch/x86/boot/compressed/vmlinux.bin.gz arch/x86/boot/compressed/*.c arch/x86/boot/compressed/head_64.S arch/x86/boot/compressed/efi_thunk_64.S arch/x86/boot/compressed/head_64.S ELF Sections
  • 9. Layout of bzImage – compressed vmlinux Compressed vmlinux setup.bin (arch/x86/boot/setup.bin) Compressed vmlinux (Protected-mode kernel) Note ELF: arch/x86/boot/compressed/vmlinux Binary: arch/x86/boot/vmlinux.bin CRC bzImage .head.text .rodata..compressed (vmlinux.bin.gz) .text .rodata .data arch/x86/boot/compressed/head_64.S 0x0 .bss .pgtable arch/x86/boot/compressed/piggy.S created by arch/x86/boot/compressed/mkpiggy.c arch/x86/boot/compressed/vmlinux.bin.gz arch/x86/boot/compressed/*.c arch/x86/boot/compressed/head_64.S arch/x86/boot/compressed/efi_thunk_64.S arch/x86/boot/compressed/head_64.S ELF Sections
  • 10. Layout of bzImage – compressed vmlinux.bin * Symbol: Equivalent to using ‘.set’ directive * https://sourceware.org/binutils/docs/as/Setting-Symbols.html Why z_input_len/input and z_output_len/output_len? * BFD: Binary File Descriptor library - https://www.gnu.org/software/binutils/
  • 11. Memory layout of bzImage – Entry Point Address Where is ‘X’? BIOS use only Typically used by MBR Reserved for MBR/BIOS Boot loader 0x00000 0x00600 0x00800 0x01000 Kernel boot section stack/heap X X+0x08000 Reserved for BIOS Command line I/O memory hole Protected-mode kernel (Compressed vmlinux) X+0x10000 0x100000 0xA0000 Boot sector entry point 0000:7C00 The kernel legacy boot sector The kernel real-mode/protected mode code For use by the kernel real-mode/protected mode code Physical Memory Kernel setup code Reference: Documentation/x86/boot.rst
  • 12. Entry Point of Linux - GRUB Memory addressing in real mode [GRUB] Get the memory address for real mode code 1. gs = fs = es = ds = ss = 0x1000 2. sp = GRUB_LINUX_SETUP_STACK = 0x9000 3. cs = 0x1020, ip = 0 Registers configured by GRUB Kernel boot section 0x10000 0x10200 Physical Memory GRUB loads ‘setup.bin’ at address 0x10000 0 ds = es = fs = gs = ss cs stack ss:sp = 0x1FFF0 protected mode real mode Kernel setup code
  • 13. Entry Point of Linux - GRUB Memory addressing in real mode [GRUB] Get the memory address for real mode code 1. gs = fs = es = ds = ss = 0x1000 2. sp = GRUB_LINUX_SETUP_STACK = 0x9000 3. cs = 0x1020, ip = 0 Registers configured by GRUB Kernel boot section 0x10000 0x10200 Physical Memory GRUB loads ‘setup.bin’ at address 0x10000 0 ds = es = fs = gs = ss cs stack ss:sp = 0x1FFF0 protected mode real mode Kernel setup code 1. QEMU loader and GRUB load ‘setup.bin’ at address 0x10000 2. QEMU loader sets SS:SP = 1000:FFF0 while GRUB sets SS:SP 1000:9000
  • 14. Entry Point of Linux: QEMU loader Kernel boot section 0x10000 0x10200 Physical Memory QEMU loader loads ‘setup.bin’ at address 0x10000 0 ds = es = fs = gs = ss cs stack ss:sp = 0x1FFF0 protected mode real mode Kernel setup code 1 2 3 4 5 6 7 ds = es = fs = gs = ss = segment_addr = 0x1000 esp = stack_addr = cmdline_addr - setup_addr – 16 = 0x20000 – 0x10000 – 16 = 0x10000 – 16 = 0xfff0 cs = 0x1020, ip = 0 Registers configured by QEMU loader 5 6 7 Prepare for far return 8 far return: change ‘cs’ by means of CPU arch itself
  • 15. Entry Point of Linux: QEMU loader – Near and Far calls 3 4 5 6 7 Prepare for far return 8 far return: change ‘cs’ by means of CPU arch itself
  • 16. Entry Point of Linux: QEMU loader Kernel boot section 0x10000 0x10200 Physical Memory QEMU loader loads ‘setup.bin’ at address 0x10000 0 ds = es = fs = gs = ss cs stack sp = 0x1FFF0 (ss:0xFFF0) protected mode real mode Kernel setup code Make sure setup.bin is loaded at 0x10000 Make sure vmlinux.bin is loaded at 0x100000 Address of setup.bin Address of vmlinux.bin
  • 17. arch/x86/boot/setup.ld arch/x86/boot/header.S 1 2 Entry Point of Linux: GNU Linker [GNU Linker] ENTRY() command * First executable instruction in an output file → entry point * ENTRY() is one of choosing the entry point -- the `-e' entry command-line option -- the ENTRY(symbol) command in a linker control script -- the value of the symbol start, if present -- the address of the first byte of the .text section, if present; -- the address 0
  • 18. arch/x86/boot/setup.ld 1 Entry Point of Linux: GNU Linker [GNU Linker] ENTRY() command * First executable instruction in an output file → entry point * ENTRY() is one of choosing the entry point -- the `-e' entry command-line option -- the ENTRY(symbol) command in a linker control script -- the value of the symbol start, if present -- the address of the first byte of the .text section, if present; -- the address 0 Kernel boot section 0x10000 0x10200 Physical Memory QEMU loader loads ‘setup.bin’ at address 0x10000 0 ds = es = fs = gs = ss cs stack sp = 0x1FFF0 (ss:0xFFF0) protected mode real mode Kernel setup code
  • 19. Entry Point of Linux: start_of_setup - GDB Kernel boot section 0x10000 0x10200 Physical Memory Boot loader loads ‘setup.bin’ at address 0x10000 0 gs = fs = es = ds = ss cs stack sp = 0x1FFF0 (ss:0xFFF0) protected mode real mode Kernel setup code
  • 20. Entry Point of Linux: start_of_setup - GDB Kernel boot section 0x10000 0x10200 Physical Memory Boot loader loads ‘setup.bin’ at address 0x10000 0 gs = fs = es = ds = ss cs stack sp = 0x1FFF0 (ss:0xFFF0) protected mode real mode Kernel setup code
  • 21. Entry Point of Linux: start_of_setup – short jump Kernel boot section 0x10000 0x10200 Physical Memory Boot loader loads ‘setup.bin’ at address 0x10000 0 gs = fs = es = ds = ss cs stack sp = 0x1FFF0 (ss:0xFFF0) protected mode real mode Kernel setup code Offset/Size Name Description 0x1F1/1 setup_sects The size of the setup in sectors 0x01FE/2 boot_flag magic number: 0xAA55 0x200/2 jump Jump instruction 0x214/4 code32_start Boot loader hook: The address to jump to in protected mode. Default: 0x100000 ".header": Real-mode kernel header
  • 22. Entry Point of Linux: start_of_setup – short jump 0x26c – 0x202 = 0x6a
  • 23. Entry Point of Linux: start_of_setup Call Path Kernel boot section 0x10000 0x10200 Boot loader loads ‘setup.bin’ at address 0x10000 0 gs = fs = es = ds = ss = cs cs stack sp = 0x1FFF0 (ss:0xFFF0) protected mode real mode Kernel setup code Physical Memory lretw instruction: Far Return Operation ‘l’ prefix: far control transfer ‘w’ suffix: word (16 bits)
  • 24. Entry Point of Linux: start_of_setup Call Path Kernel boot section 0x10000 0x10200 Boot loader loads ‘setup.bin’ at address 0x10000 0 gs = fs = es = ds = ss = cs cs stack sp = 0x1FFF0 (ss:0xFFF0) protected mode real mode Kernel setup code Physical Memory lretw instruction: Far Return Operation ‘l’ prefix: far control transfer ‘w’ suffix: word (16 bits) 1 1 2 2 3 3
  • 25. Entry Point of Linux: start_of_setup Kernel boot section 0x10000 0x10200 Boot loader loads ‘setup.bin’ at address 0x10000 0 gs = fs = es = ds = ss = cs cs stack sp = 0x1FFF0 (ss:0xFFF0) protected mode real mode Kernel setup code Physical Memory Call Path lretw instruction: Far Return Operation ‘l’ prefix: far control transfer ‘w’ suffix: word (16 bits)
  • 26. Entry Point of Linux: start_of_setup – Why to align CS? Kernel boot section 0x10000 0x10200 Boot loader loads ‘setup.bin’ at address 0x10000 0 gs = fs = es = ds = ss = cs cs stack sp = 0x1FFF0 (ss:0xFFF0) protected mode real mode Kernel setup code Physical Memory Call Path lretw instruction: Far Return Operation ‘l’ prefix: far control transfer ‘w’ suffix: word (16 bits) If cs is not align with ds, ds and es are incorrect after returning from ‘intcall’.
  • 27. Entry Point of Linux: start_of_setup – data & bss section Call Path Kernel boot section 0x10000 0x10200 Boot loader loads ‘setup.bin’ at address 0x10000 0 gs = fs = es = ds = ss= cs stack sp = 0x1FFF0 protected mode real mode Kernel setup code Kernel boot section 0x10000 0x10200 0 gs = fs = es = ds = ss = cs stack sp = 0x1FFF0 protected mode real mode Kernel setup code BSS Section _end __bss_start __bss_end Data Section Physical Memory
  • 28. Entry Point of Linux: start_of_setup -> main() Call Path
  • 29. Entry Point of Linux: start_of_setup -> main()
  • 30. Entry Point of Linux: start_of_setup -> main()
  • 31. Entry Point of Linux: start_of_setup -> main() -> copy_boot_params() Call Path • copy setup header into boot parameter block (struct boot_params: arch/x86/include/uapi/asm/bootparam.h) o `struct setup_header hdr` in boot_params ▪ Contain the same fields defined in Linux boot protocol. Those fields are configured by boot loader and kernel compile/build time
  • 32. Call Path • console_init() o Initialize the corresponding serial port if command line has ‘earlyprintk’ parameter Entry Point of Linux: start_of_setup -> main() -> console_init() – (1/2) Kernel boot section 0x10000 0x10200 0 gs = fs = es = ds = ss = cs stack sp = 0x1FFF0 protected mode real mode Kernel setup code BSS Section _end __bss_start __bss_end Data Section Kernel Command Line 0x20000 QEMU Loader Physical Memory
  • 33. Call Path • console_init() o Initialize the corresponding serial port if command line has ‘earlyprintk’ parameter Entry Point of Linux: start_of_setup -> main() -> console_init() – (2/2) Kernel boot section 0x10000 0x10200 0 gs = fs = es = ds = ss = cs stack sp = 0x1FFF0 protected mode real mode Kernel setup code BSS Section _end __bss_start __bss_end Data Section Kernel Command Line 0x20000 Physical Memory
  • 34. Call Path • init_heap() • Discussion in the next few slides • validate_cpu() o Check CPU flags o Check if long mode (x86_64) is available o [AMD – K7 Processor] Turn SSE+SSE2 on if they are missing in CPU flags • detect_memory() o Use different program interfaces (0xe820, 0xe801 and 0x88) for memory detection o 0xe820 ▪ Fill boot_params.e820_table based on e820 map Entry Point of Linux: start_of_setup -> main() -> validate_cpu() & detect_memory() Kernel boot section 0x10000 0x10200 0 gs = fs = es = ds = ss = cs stack sp = 0x1FFF0 protected mode real mode Kernel setup code BSS Section _end __bss_start __bss_end Data Section Kernel Command Line 0x20000 Physical Memory
  • 35. Call Path • init_heap o Setup the heap space if the ‘CAN_USE_HEAP’ flag (0x80) is set in loadflags of the kernel setup header. Entry Point of Linux: start_of_setup -> main() -> init_heap() (1/2)
  • 36. Call Path Entry Point of Linux: start_of_setup -> main() -> init_heap() (2/2) heap: allocate heap if CAN_USE_HEAP’ flag (0x80) is set No heap sp (STACK_SIZE = 0x400) Kernel boot section 0x10000 0x10200 stack protected mode real mode Kernel setup code BSS Section Unused Area __bss_start __bss_end HEAP = heap_end = _end Data Section sp (STACK_SIZE = 0x400) Kernel boot section 0x10000 0x10200 stack protected mode real mode Kernel setup code BSS Section Heap __bss_start __bss_end HEAP = _end heap_end Data Section gs = fs = es = ds = ss = cs gs = fs = es = ds = ss = cs
  • 37. go_to_protected_mode GDT_ENTRY_BOOT_DS GDT_ENTRY_BOOT_CS NULL NULL 0 1 2 3 GDT_ENTRY_BOOT_TSS 4 Descriptor Table: boot_gdt System Memory 0 0xFFFFFFFF limit Base Address GDTR x86 Segmentation: Address Translation setup_gdt(): Setup 4G memory space for CS/DS Call Path
  • 39. protected_mode_jump – ljmpl instruction: ignore ‘.Lin_pm32’ relocation (2/6) 0x30cc Jump (absolute address) to the wrong location sp (STACK_SIZE = 0x400) Kernel boot section 0x10000 0x10200 stack protected mode real mode Kernel setup code BSS Section Heap __bss_start __bss_end HEAP = _end heap_end Data Section gs = fs = es = ds = ss = cs setup.bin generation Physical Memory
  • 40. protected_mode_jump – ljmpl instruction - relocation (3/6) sp (STACK_SIZE = 0x400) Kernel boot section 0x10000 0x10200 stack protected mode real mode Kernel setup code BSS Section Heap __bss_start __bss_end HEAP = _end heap_end Data Section gs = fs = es = ds = ss = cs Relocation for absolute address of ‘ljmpl’ ljmpl Physical Memory
  • 41. Relocation for absolute address of ‘ljmpl’ protected_mode_jump – ljmpl instruction (4/6) sp (STACK_SIZE = 0x400) Kernel boot section 0x10000 0x10200 stack protected mode real mode Kernel setup code BSS Section Heap __bss_start __bss_end HEAP = _end heap_end Data Section gs = fs = es = ds = ss = cs ljmpl Physical Memory
  • 42. protected_mode_jump – ljmpl instruction: instruction format (5/6)
  • 43. protected_mode_jump – ljmpl instruction: instruction format (6/6)
  • 44. Protected mode: ‘.Lin_pm32’ (1/2) [real mode] SP configuration [protected mode] SP configuration `addl %ebx, %esp` in label “.Lin_pm32” 0x1FF80 (SS:SP = 0x1000:0xFF80) Kernel boot section 0x10000 0x10200 stack protected mode real mode Kernel setup code BSS Section Heap __bss_start __bss_end HEAP = _end heap_end esp = 0x1FF80 Kernel boot section 0x10000 (ebx) 0x10200 stack protected mode real mode Kernel setup code BSS Section Heap __bss_start __bss_end HEAP = _end heap_end Data Section Data Section
  • 45. Data Section 1 2 4 3 Protected mode: ‘.Lin_pm32’ (2/2) X = 0x10000 esp = 0x1FF80 Kernel boot section 0x10200 stack protected mode real mode Kernel setup code BSS Section Heap __bss_start __bss_end HEAP = _end heap_end Reserved for BIOS command line I/O memory hole Protected-mode kernel code (compressed vmlinux) X+0x10000 0xA0000 0x100000 jmpl *%eax 5 Physical Memory Call Path
  • 46. Compressed vmlinux: memory layout (1/10) .head.text – startup_32 0x100000 (ebp register) 0x100200 decompressed vmlinux.bin.bz .head.text – startup_64 0x1000000 compressed vmlinux (Relocation) 0x1000000 + boot_param.init_size 0x1000000 + boot_param.init_size - _end (rbx register) vmlinux.bin.gz .text .rodata .data .bss .pgtable _end 0x100000 + _end boot_heap (size: 0x10000) boot_stack (size: 0x4000) … input_data input_data_end Memory Layout 32-bit entry point _bss
  • 47. Compressed vmlinux: boot_stack & boot_heap in .bss (2/10) .head.text – startup_32 0x100000 (ebp register) 0x100200 decompressed vmlinux.bin.bz .head.text – startup_64 0x1000000 compressed vmlinux (Relocation) 0x1000000 + boot_param.init_size 0x1000000 + boot_param.init_size - _end (rbx register) vmlinux.bin.gz .text .rodata .data .bss .pgtable _end 0x100000 + _end boot_heap (size: 0x10000) boot_stack (size: 0x4000) … input_data input_data_end Memory Layout 32-bit entry point _bss
  • 48. Compressed vmlinux: High-level Overview (3/10) Why relocation • Base address of 32-bit Linux kernel entry point: 0x100000 • Default base address of Linux kernel: CONFIG_PHYSICAL_START=0x1000000 • Use Case • kdump: a recuse kernel is loaded to a different address • PIE (Position independent Executable) and PIC (Position Independent Code)
  • 49. Compressed vmlinux: startup_32: 32-bit entry point (4/10) 1 1
  • 50. Compressed vmlinux: startup_32 (5/10) 1 1 Get the loading address
  • 52. Compressed vmlinux: startup_32: Init 4-level page table (7/10) Sign-extend Page Map Level-4 Offset Page Directory Pointer Offset Page Directory Offset Physical Page Offset 0 30 21 39 20 38 29 47 48 63 PML4E #0 PDPTE #3 Data Page Map Level-4 Table Page Directory Pointer Table Page Directory Table 40 9 9 9 Linear Address CR3 PDPTE #2 PDPTE #1 PDPTE #0 PDE #1535 PDE #1024 . . PDE #2047 PDE #1536 . . PDE #511 PDE #0 . . PDE #1023 PDE #512 . . 2MBbyte Physical Page 40 40 31 21 [Paging] Identity mapping for 0-4GB memory space
  • 53. Compressed vmlinux: startup_32: Init 4-level page table (8/10) Reference: Section 4.1 “PAGING MODES AND CONTROL BITS”, Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 3 (3A, 3B, 3C & 3D): System Programming Guide
  • 54. Compressed vmlinux: startup_32: Init 4-level page table (9/10)
  • 55. Compressed vmlinux: far return to startup_64 (10/10) rva(startup_64) = 0x200 ebp = 0x100000 eax = 0x100000 + 0x200 = 0x100200
  • 56. Compressed vmlinux: startup_64 2 3 Why to reload CS? (Commit “34bb49229f19”) When the pre-decompression code loads its first GDT in startup_64, it is still running on the CS value of the previous GDT. In the case of SEV-ES this is the EFI GDT. It can be anything depending on what has loaded the kernel (EFI, legacy boot code, container runtime, etc.)
  • 57. Compressed vmlinux: [.text] .Lrelocated (1/5) 4 5 Why to call initialize_identity_maps()?
  • 58. Compressed vmlinux: [.text] .Lrelocated (2/5) 4 5 Why to map boot_params and command line?
  • 59. Compressed vmlinux: parse_elf (3/5) 4 ELF Header 0x1000000 decompressed vmlinux.bin.bz (vmlinux.bin – ELF format) program headers program header #0 (.text, .rodata, .pci_fixup….) 0x1200000 program header #1 (.data .vvar) program header #2 (.init.text .altinstr_aux …) 0x1a00000 0x1ac2000 program header #3 (.notes) 0x18886b0 0x1000000 program header #0 (.text, .rodata, .pci_fixup….) 0x1800000 program header #1 (.data .vvar) program header #2 (.init.text .altinstr_aux …) 0x18c2000 Physical memory Physical memory
  • 60. Compressed vmlinux: handle_relocations (4/5) 4 CONFIG_RELOCATABLE • Retain relocation information (generate .rel.* or rela.* sections) when building a kernel image, so it can be loaded someplace besides the default address (CONFIG_PHYSICAL_START = 16MB). • Use case: kdump kernel (recovery kernel) handle_relocations() - Relocation if CONFIG_X86_NEED_RELOCS is set • Depend on RANDOMIZE_BASE || (X86_32 && RELOCATABLE) • Scan relocation tables (.rel.* or .rela.* sections) for symbol relocation
  • 61. Compressed vmlinux: handle_relocations (5/5) 4 vmlinux.bin.bz vmlinux.bin vmlinux.relocs handle_relocations(): Perform relocation backwards from the end of the decompressed vmlinux 64-bit relocation address 0 32-bit relocation address 0 -R section_name: Remove any section matching section_name -S or strip-all: Do not copy relocation and symbol information from the source file objdump options
  • 62. Recap setup.bin (arch/x86/boot/setup.bin) Compressed vmlinux (Protected-mode kernel) Note ELF: arch/x86/boot/compressed/vmlinux Binary: arch/x86/boot/vmlinux.bin CRC bzImage
  • 63. [More info] bzImage = vmlinuz On a physical machine Source code: arch/x86/boot/Makefile, arch/x86/boot/install.sh
  • 64. Reference • The Linux/x86 Boot Protocol, Documentation/x86/boot.rst • Intel® 64 and IA-32 Architectures Software Developer’s Manual • https://wdv4758h.github.io/notes/blog/linux-kernel-boot.html • Linux insides, https://0xax.gitbooks.io/linux-insides/content/
  • 66. gdb: Preparation for debugging real-mode of Linux kernel (1/2) Github: https://github.com/AdrianHuang/gdb-linux-real-mode
  • 67. gdb: Preparation for debugging real-mode of Linux kernel (2/2) Github: https://github.com/AdrianHuang/gdb-linux-real-mode
  • 68. initialize_identity_maps x86_mapping_info void *(*alloc_pgt_page)(void *) void *context unsigned long page_flag unsigned long offset alloc_pgt_data unsigned char *pgt_buf unsigned long pgt_buf_size unsigned long pgt_buf_offset bool direct_gbpages unsigned long kernpg_flag
  • 69. UEFI booting flow – EFI boot stub: Entry point AddressOfEntryPoint (efi_pe_entry): 0x18d84a ImageBase = 0x1000000 Physical address of AddressofEntryPoint = 0x1000000 + 0x18d84a = 0x118d84a
  • 70. UEFI booting flow – EFI Handover protocol
  • 71. UEFI booting flow – EFI Handover protocol
  • 72. UEFI booting flow – EFI Handover protocol Where is the address of bzimage loaded by boot loader?