SlideShare a Scribd company logo

Memory Management with Page Folios

Adrian Huang, profile picture
Adrian Huang

The document describes a memory management system using memory folios to address problems with legacy page caching and compound pages. Memory folios provide a unified interface for accessing pages and simplify operations on high-order and compound pages. Folios also improve page cache performance by maintaining a shorter LRU list with one entry per folio rather than per page.

Software
Related topics:
Memory Management
1 of 34
Downloaded 59 times
1
2
3
4
5
6
Most read
7
Most read
8
Most read
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
* Based on kernel 6.3 (x86_64) – QEMU * 2-socket CPUs (4 cores/socket) * 16GB memory * Kernel parameter: nokaslr norandmaps * KASAN: disabled * Userspace: ASLR is disabled * Legacy BIOS Memory Management with Page Folios Adrian Huang | May, 2023
Agenda • Problem description ✓[Background] Normal high-order page & compound page ✓Legacy page cache • Memory folio’s goal • Solution ✓Page cache with memory folio ✓page struct vs folio struct ✓[Example] total_mapcount() implementation difference between legacy approach and folio
Normal high-order page & compound page Page (head page) flags |= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page Page Page Page Normal high-order page pages = alloc_pages(GFP_KERNEL, 2); Four physically contiguous pages: Init compound page metadata during page allocation pages = alloc_pages(GFP_KERNEL | __GFP_COMP, 2); Four physically contiguous pages: not a compound page
Compound page Page (head page) flags |= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only . . . Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page (Tail page) _compound_pad_1 (compound_head) Compound page – Use Cases • Mainly used in huge page ✓ hugetlbfs (also called HugeTLB Pages or persistent huge pages) ➢ Reserved inside the kernel and cannot be used for other purposes. ➢ Cannot be swapped out. ➢ Two allocation methods: • Pre-allocated to the kernel huge page pool with appending kernel parameter. • [Dynamically allocated huge pages of the default size] Example: `echo 10 > /proc/sys/vm/nr_hugepages` ➢ User application calls the mmap system call or shared memory system calls (shmget and shmat) to request the huge page allocation. ➢ Used by database for many years ➢ Manual configuration for hugetlb pages is required. ➢ Application change is required. (via open/mmap)
Compound page Page (head page) flags |= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only . . . Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page (Tail page) _compound_pad_1 (compound_head)
Compound page Page (head page) flags |= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only . . . Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page (Tail page) _compound_pad_1 (compound_head) Compound page – Use Cases • Mainly used in huge page ✓ Transparent Huge Page (THP) ➢ Support the automatic promotion and demotion of page sizes ➢ Transparent to the application: No need to modify application. ➢ Control via /sys/kernel/mm/transparent_hugepage/enabled:
Compound page Page (head page) flags |= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only . . . Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page (Tail page) _compound_pad_1 (compound_head) Compound page – Use Cases • Mainly used in huge page • kmalloc: allocation size > 8192 bytes o Check kmalloc_order() • Memory folio When to configure compound page? • Condition: page order >= 1 && __GFP_COMP allocation flag is set • alloc_pages -> … -> prep_new_page -> prep_compound_page
Compound page: Problem Description #1 Page (head page) flags |= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only . . . Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page (Tail page) _compound_pad_1 (compound_head) [Problem Description] No unified interface: Ambiguity • Some functions may deal with PAGES_SIZE unit (4KB): They’re unaware of compound pages and huge pages • Some functions accept the page head *only* • Some functions accept the page head or page tail ✓ Call compound_head() to get the page head: waste instructions to get the page head → Performance impact ✓ compound_head() users: ➢ get_page(): This function is called quite frequently ➢ put_page(): This function is called quite frequently ➢ …
Legacy page cache 4KB 4KB 512B 512B 512B 512B [file] file->f_pos (continuous file position) Page cache 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B sector .. Disk 4KB Page Cache Buffer Cache (Buffer head) Kernel Space read()/write()/sendfile() User Space mmap()
Legacy page cache: Problem Description #2 1. Page cache occupies most of memory pages!!! 2. Each page cache (no “compound page” concept) is added to active/inactive lru list • Long lru list: lock contention & cache misses
Agenda • Problem description ✓Normal high-order page & compound page ✓Legacy page cache • Memory folio’s goal • Solution ✓Page cache with memory folio ✓page struct vs folio struct ✓[Example] total_mapcount() implementation difference between legacy approach and folio
Memory folio’s goal • Unified interface ✓All accesses via folio struct (head page/tail page in a compound page) • [Page Cache] Shorter LRU list ✓Original: one struct page per 4KB to be added to LRU list ✓Folio: one struct page (page head) per 8KB, 16KB, 32KB, 64KB, 128KB….and so on (include THP) to be added to LRU list • [Anonymous page] THP support ✓create_huge_pmd -> do_huge_pmd_anonymous_page -> __do_huge_pmd_anonymous_page * THP: Transparent Huge Page
Agenda • Problem description ✓Normal high-order page & compound page ✓Legacy page cache • Memory folio’s goal • Solution ✓Page cache with memory folio ✓page struct vs folio struct ✓[Example] total_mapcount() implementation difference between legacy approach and folio
Page cache with folio 4KB 512B … 512B [file] file->f_pos (continuous file position) folio sector … … .. Disk page page page page 4KB 4KB 4KB 4KB 4KB 4KB 4KB 512B … 512B 512B … 512B 512B … 512B 512B … 512B page page page page 512B … 512B 512B … 512B 512B … 512B Kernel Space read()/write()/sendfile() User Space mmap() • Folio is the container of struct page(s) ✓ All accesses via folio struct ✓ No tail page → fewer run-time checks
Page cache with folio 4KB 512B … 512B [file] file->f_pos (continuous file position) folio sector … … .. Disk page page page page 4KB 4KB 4KB 4KB 4KB 4KB 4KB 512B … 512B 512B … 512B 512B … 512B 512B … 512B page page page page 512B … 512B 512B … 512B 512B … 512B Kernel Space read()/write()/sendfile() User Space mmap() Folio’s page order: readahead mechanism • CONFIG_TRANSPARENT_HUGEPAGE is enabled ✓ Minimum: order 2 (4 pages) ✓ Maximum: order 9 (512 pages) • CONFIG_TRANSPARENT_HUGEPAGE is disabled ✓ Minimum: order 2 (4 pages) ✓ Maximum: order 8 (256 pages) • Commit 793917d997df (“mm/readahead: Add large folio readahead”): merged in 5.18 kernel • Default readahead size: 128KB (32 pages)
Page cache with folio 4KB 512B … 512B [file] file->f_pos (continuous file position) folio sector … … .. Disk page page page page 4KB 4KB 4KB 4KB 4KB 4KB 4KB 512B … 512B 512B … 512B 512B … 512B 512B … 512B page page page page 512B … 512B 512B … 512B 512B … 512B Kernel Space read()/write()/sendfile() User Space mmap() 1. Short LRU list: Only the head page of folio is added LRU list → Performance improvement 2. 45% improvement for lru-file-mmap-read (vm-scalability): Matthew Wilcox’s PDF file Folio’s page order: readahead mechanism • CONFIG_TRANSPARENT_HUGEPAGE is enabled ✓ Minimum: order 2 (4 pages) ✓ Maximum: order 9 (512 pages) • CONFIG_TRANSPARENT_HUGEPAGE is disabled ✓ Minimum: order 2 (4 pages) ✓ Maximum: order 8 (256 pages) • Commit 793917d997df (“mm/readahead: Add large folio readahead”): merged in 5.18 kernel • Default readahead size: 128KB (32 pages)
Page cache with folio: backtrace * kernel: 6.3
Agenda • Problem description ✓Normal high-order page & compound page ✓Legacy page cache • Memory folio’s goal • Solution ✓Page cache with memory folio ✓page struct vs folio struct ✓[Example] total_mapcount() implementation difference between legacy approach and folio
folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 _flags_2 _head_2 void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Page #0 (head) flags … Page #1 (tail) flags compound_head compound_dtor compound_order compound_mapcount compound_nr … Page #N (tail) flags _compound_pad_1 hpage_pinned_refcount deferred_list . . . page struct vs folio struct Compound pages
folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 _flags_2 _head_2 void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Page #0 (head) flags … Page #1 (tail) flags compound_head compound_dtor compound_order compound_mapcount compound_nr … Page #2 (tail) flags _compound_pad_1 hpage_pinned_refcount deferred_list . . . page struct vs folio struct Compound pages folio’s benefit • [Example] 512KB compound page ✓ page struct: Need to maintain 128 page structs (1 head page and 127 tail pages) ✓ folio struct: Maintain 3 page structs regardless of the size of compound pages.
folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 → won’t be used _flags_2 _head_2 void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 → won’t be used struct struct list_head _deferred_list page struct vs folio struct folio struct’s members • _entire_mapcount ✓ The compound page is mapped via a single PMD (huge page). • _nr_pages_mapped ✓ Number of individual subpages (PTE: 4KB pages) are mapped. ✓ Scenario: Two processes map the same memory range ✓ One process maps the entire 2MB compound page (Transparent Huge Page - THP): mapped via a single PMD ✓ The other process maps some 4KB pages within this 2MB memory area: mapped via PTEs ✓ Benefit about THP: No need to split the huge page if other processes map 4KB pages within the same memory area. • _folio_nr_pages ✓ Number of pages in this folio. ✓ _folio_nr_pages = 1 << order, where order > 0.
folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 _flags_2 _head_2 void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Page #0 (head) flags … Page #1 (tail) flags compound_head compound_dtor compound_order compound_mapcount compound_nr … Page #2 (tail) flags _compound_pad_1 hpage_pinned_refcount deferred_list . . . folio struct vs legacy compound page Page #1 of folio and legacy page struct has the same mapping
Agenda • Problem description ✓Normal high-order page & compound page ✓Legacy page cache • Memory folio’s goal • Solution ✓Page cache with memory folio ✓page struct vs folio struct ✓[Example] total_mapcount() implementation difference between legacy approach and folio
[kernel v5.11] total_mapcount() page Page cache and anonymous pages struct union page_pool used by netstack struct slab, slob and slub struct Tail pages of compound page struct Second tail page of compound page struct Page table pages: 1. PMD huge PTE 2. x86 pgd page <-> mm_struct struct ZONE_DEVICE pages struct rcu_head: free a page by RCU struct union atomic_t _mapcount: the number of this page is referenced by page table unsigned int page_type unsigned int active: used by slab int units: used by slob atomic_t _refcount … Case 1: Singleton page(s) Get _mapcount directly total_mapcount() users: • Huge page • rmap (reverse mapping)
page (head) Page cache and anonymous pages struct union page_pool used by netstack struct slab, slob and slub struct Tail pages of compound page struct Second tail page of compound page struct Page table pages: 1. PMD huge PTE 2. x86 pgd page <-> mm_struct struct ZONE_DEVICE pages struct rcu_head: free a page by RCU struct union atomic_t _mapcount: the number of this page is referenced by page table unsigned int page_type unsigned int active: used by slab int units: used by slob atomic_t _refcount … Case 2: Compound page && hugetlb (hugetlbfs) page page (first tail) compound_head struct union compound_dtor compound_order compound_mapcount compound_nr = 1 << compound_nr struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . page (second tail) _compound_pad_1 struct union hpage_pinned_refcount deferred_list struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . . . . _compound_pad_1 (compound_head) page (second tail) Get compound_mapcount directly compound_mapcount: • Map count of the whole compound page (does not include mapped sub-pages) Steps: 1. Get the head page based on any page (page head or page tail) 2. Read ‘compound_mapcount’ of the first tail page A. page[1].compound_mapcount [kernel v5.11] total_mapcount()
page (head) Page cache and anonymous pages struct union page_pool used by netstack struct slab, slob and slub struct Tail pages of compound page struct Second tail page of compound page struct Page table pages: 1. PMD huge PTE 2. x86 pgd page <-> mm_struct struct ZONE_DEVICE pages struct rcu_head: free a page by RCU struct union atomic_t _mapcount: the number of this page is referenced by page table unsigned int page_type unsigned int active: used by slab int units: used by slob atomic_t _refcount … Case 3: [Anonymous page] Compound page && transparent huge page page (first tail) compound_head struct union compound_dtor compound_order compound_mapcount compound_nr = 1 << compound_nr struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . page (second tail) _compound_pad_1 struct union hpage_pinned_refcount deferred_list struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . . . . _compound_pad_1 (compound_head) page (second tail) Steps: 1. Get the head page based on any page (page head or page tail) 2. Read ‘compound_mapcount’ of the first tail page A. page[1].compound_mapcount: Map count of the whole compound page 3. `Accumulate each subpage._mapcount`: A. One process maps 2MB range as a single huge page (a single PMD) B. Another process maps 512 individual PTEs 4. `Accumulate each subpage._mapcount` + page[1].compound_mapcount [kernel v5.11] total_mapcount()
page (head) Page cache and anonymous pages struct union page_pool used by netstack struct slab, slob and slub struct Tail pages of compound page struct Second tail page of compound page struct Page table pages: 1. PMD huge PTE 2. x86 pgd page <-> mm_struct struct ZONE_DEVICE pages struct rcu_head: free a page by RCU struct union atomic_t _mapcount: the number of this page is referenced by page table unsigned int page_type unsigned int active: used by slab int units: used by slob atomic_t _refcount … Case 3: [Page cache] Compound page && transparent huge page page (first tail) compound_head struct union compound_dtor compound_order compound_mapcount compound_nr = 1 << compound_nr struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . page (second tail) _compound_pad_1 struct union hpage_pinned_refcount deferred_list struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . . . . _compound_pad_1 (compound_head) page (second tail) Steps: 1. Get the head page based on any page (page head or page tail) 2. Read ‘compound_mapcount’ of the first tail page A. page[1].compound_mapcount: Map count of the whole compound page 3. `Accumulate each subpage._mapcount`: A. One process maps 2MB range as a single huge page (a single PMD) B. Another process maps 512 individual PTEs 4. `Accumulate each subpage._mapcount` + page[1].compound_mapcount - page[1].compound_mapcount * page[1].compound_nr A. File pages has compound_mapcount included in _mapcount [kernel v5.11] total_mapcount()
[kernel v6.3] total_mapcount() and folio_mapcount() folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 … void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Case 1: Singleton page – Not a compound page 1 2
[kernel v6.3] total_mapcount() and folio_mapcount() folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 … void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Case 2: Compound page is mapped via PMD (huge page) 1 3 2 Get _entire_mapcount 4 5 Get _nr_pages_mapped
[kernel v6.3] total_mapcount() and folio_mapcount() folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 … void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Case 3: Compound page is mapped via PMD (huge page) and some subpages are mapped by PTE 1 2 mapcount = folio’s _entire_mapcount + sum(each subpage’s _mapcount)
Reference • Memory Folios • LWN - A memory-folio update • LWN - An introduction to compound pages • LWN - Huge pages part 1 (Introduction) • Documentation/mm/transhuge.rst
backup
Learn new C standard (C11) from folio: Generic Selection * Reference from: ISO/IEC 9899:201x • C99 defines type-generic macros in the standardized library: the type of argument is detected automatically, and the corresponding function is invoked based on that type. ✓ Example: sqrt(X), ➢ X is double → invoke sqrt() ➢ X is float → invoke sqrtf() ➢ X is long double → invoke sqrtl() • However, programmers cannot define their own type- generic macros in C99. • In C11, programmers can define their own type-generic macros:
Some Functions

More Related Content

PDF
Memory Mapping Implementation (mmap) in Linux Kernel
Adrian Huang
 
PDF
Memory Compaction in Linux Kernel.pdf
Adrian Huang
 
PDF
Reverse Mapping (rmap) in Linux Kernel
Adrian Huang
 
PDF
Physical Memory Models.pdf
Adrian Huang
 
PDF
Anatomy of the loadable kernel module (lkm)
Adrian Huang
 
PDF
Decompressed vmlinux: linux kernel initialization from page table configurati...
Adrian Huang
 
PDF
Page cache in Linux kernel
Adrian Huang
 
PDF
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
Adrian Huang
 
Memory Mapping Implementation (mmap) in Linux Kernel
Adrian Huang
 
Memory Compaction in Linux Kernel.pdf
Adrian Huang
 
Reverse Mapping (rmap) in Linux Kernel
Adrian Huang
 
Physical Memory Models.pdf
Adrian Huang
 
Anatomy of the loadable kernel module (lkm)
Adrian Huang
 
Decompressed vmlinux: linux kernel initialization from page table configurati...
Adrian Huang
 
Page cache in Linux kernel
Adrian Huang
 
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
Adrian Huang
 

What's hot (20)

PDF
Process Address Space: The way to create virtual address (page table) of user...
Adrian Huang
 
PDF
malloc & vmalloc in Linux
Adrian Huang
 
PDF
Physical Memory Management.pdf
Adrian Huang
 
PPTX
Slab Allocator in Linux Kernel
Adrian Huang
 
PDF
Linux Kernel - Virtual File System
Adrian Huang
 
PPTX
qemu + gdb + sample_code: Run sample code in QEMU OS and observe Linux Kernel...
Adrian Huang
 
PDF
Vmlinux: anatomy of bzimage and how x86 64 processor is booted
Adrian Huang
 
PDF
Kdump and the kernel crash dump analysis
Buland Singh
 
PDF
semaphore & mutex.pdf
Adrian Huang
 
PDF
spinlock.pdf
Adrian Huang
 
PPTX
Linux Kernel Booting Process (2) - For NLKB
shimosawa
 
PDF
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
Anne Nicolas
 
PDF
Linux Synchronization Mechanism: RCU (Read Copy Update)
Adrian Huang
 
PDF
Linux kernel debugging
libfetion
 
PPTX
Linux Kernel Booting Process (1) - For NLKB
shimosawa
 
PPTX
Linux Memory Management
Ni Zo-Ma
 
PDF
Memory management in Linux kernel
Vadim Nikitin
 
PDF
The Linux Block Layer - Built for Fast Storage
Kernel TLV
 
PDF
XPDDS17: Shared Virtual Memory Virtualization Implementation on Xen - Yi Liu,...
The Linux Foundation
 
PDF
Arm device tree and linux device drivers
Houcheng Lin
 
Process Address Space: The way to create virtual address (page table) of user...
Adrian Huang
 
malloc & vmalloc in Linux
Adrian Huang
 
Physical Memory Management.pdf
Adrian Huang
 
Slab Allocator in Linux Kernel
Adrian Huang
 
Linux Kernel - Virtual File System
Adrian Huang
 
qemu + gdb + sample_code: Run sample code in QEMU OS and observe Linux Kernel...
Adrian Huang
 
Vmlinux: anatomy of bzimage and how x86 64 processor is booted
Adrian Huang
 
Kdump and the kernel crash dump analysis
Buland Singh
 
semaphore & mutex.pdf
Adrian Huang
 
spinlock.pdf
Adrian Huang
 
Linux Kernel Booting Process (2) - For NLKB
shimosawa
 
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
Anne Nicolas
 
Linux Synchronization Mechanism: RCU (Read Copy Update)
Adrian Huang
 
Linux kernel debugging
libfetion
 
Linux Kernel Booting Process (1) - For NLKB
shimosawa
 
Linux Memory Management
Ni Zo-Ma
 
Memory management in Linux kernel
Vadim Nikitin
 
The Linux Block Layer - Built for Fast Storage
Kernel TLV
 
XPDDS17: Shared Virtual Memory Virtualization Implementation on Xen - Yi Liu,...
The Linux Foundation
 
Arm device tree and linux device drivers
Houcheng Lin
 
Ad

Similar to Memory Management with Page Folios (20)

PDF
Page Cache in Linux 2.6.pdf
ycelgemici1
 
PPTX
Linux Memory Management with CMA (Contiguous Memory Allocator)
Pankaj Suryawanshi
 
PDF
Understand
Kalimuthu Velappan
 
PDF
Practical ,Transparent Operating System Support For Superpages
Nadeeshani Hewage
 
PDF
AOS Lab 7: Page tables
Zubair Nabi
 
PDF
Vmreport
meru2ks
 
PPT
Windows memory manager internals
Sisimon Soman
 
PPT
Chapter 04
Google
 
PDF
Buddy system
ChenYiHuang5
 
PDF
Кирилл Шутемов aka “kas” - О Transparent Hugepages и Huge zero page
Minsk Linux User Group
 
PPTX
Virtual Memory Managementddddddddffffffffffffff.pptx
ananyachouhan70
 
PPTX
Abhaycavirtual memory and the pagehit.pptx
wemoji5816
 
PDF
Unit 5
pm_ghate
 
PPT
Segmentation with paging methods and techniques
nikhilrana24112003
 
PPT
Paging and Segmentation
sathish sak
 
PPT
Linux memory
ericrain911
 
PPT
Linux Memory Management
Rajan Kandel
 
PPT
Memory Management
Ramasubbu .P
 
PPT
Computer memory management
Kumar
 
PPT
memory
Kishore Kumar
 
Page Cache in Linux 2.6.pdf
ycelgemici1
 
Linux Memory Management with CMA (Contiguous Memory Allocator)
Pankaj Suryawanshi
 
Understand
Kalimuthu Velappan
 
Practical ,Transparent Operating System Support For Superpages
Nadeeshani Hewage
 
AOS Lab 7: Page tables
Zubair Nabi
 
Vmreport
meru2ks
 
Windows memory manager internals
Sisimon Soman
 
Chapter 04
Google
 
Buddy system
ChenYiHuang5
 
Кирилл Шутемов aka “kas” - О Transparent Hugepages и Huge zero page
Minsk Linux User Group
 
Virtual Memory Managementddddddddffffffffffffff.pptx
ananyachouhan70
 
Abhaycavirtual memory and the pagehit.pptx
wemoji5816
 
Unit 5
pm_ghate
 
Segmentation with paging methods and techniques
nikhilrana24112003
 
Paging and Segmentation
sathish sak
 
Linux memory
ericrain911
 
Linux Memory Management
Rajan Kandel
 
Memory Management
Ramasubbu .P
 
Computer memory management
Kumar
 
memory
Kishore Kumar
 
Ad

Recently uploaded (20)

PPTX
VVF-Customer-Presentation2025-Ver1.9.pptx
blackmambaettijean
 
PDF
Understanding Forklifts - TECH EHS Solution
TECH EHS Solution
 
PDF
System and Network Administration Chapter 2
jaramharo
 
PDF
AI in Product Development-omnex systems
omnex systems
 
PDF
Design an Analysis of Algorithms II-SECS-1021-03
DilanEscobar1
 
PDF
Nekopoi APK 2025 free lastest update
umarali35kp
 
PDF
Adobe Illustrator 28.6 Crack My Vision of Vector Design
ghrom2211g
 
PPTX
Operating system designcfffgfgggggggvggggggggg
rmskumara09
 
PDF
System and Network Administraation Chapter 3
jaramharo
 
PPT
Introduction Database Management System for Course Database
ReinertYosua
 
PDF
Which alternative to Crystal Reports is best for small or large businesses.pdf
Varsha Nayak
 
PPTX
Agentic AI : A Practical Guide. Undersating, Implementing and Scaling Autono...
IT IDOL Technologies
 
PDF
Claude Code: Everyone is a 10x Developer - A Comprehensive AI-Powered CLI Tool
William Chong
 
PDF
Flood Susceptibility Mapping Using Image-Based 2D-CNN Deep Learnin. Overview ...
lawrenceomai2
 
PDF
2025 Textile ERP Trends: SAP, Odoo & Oracle
Aetos Technologies
 
PDF
How to Choose the Right IT Partner for Your Business in Malaysia
Asian Business Services & Technologies
 
PPTX
Online Work Permit System for Fast Permit Processing
SHEQ Network Limited
 
PPTX
Agentic AI Use Case- Contract Lifecycle Management (CLM).pptx
mhxyzzp
 
PDF
How Creative Agencies Leverage Project Management Software.pdf
Orangescrum
 
PDF
Softaken Excel to vCard Converter Software.pdf
markwillsonmw004
 
VVF-Customer-Presentation2025-Ver1.9.pptx
blackmambaettijean
 
Understanding Forklifts - TECH EHS Solution
TECH EHS Solution
 
System and Network Administration Chapter 2
jaramharo
 
AI in Product Development-omnex systems
omnex systems
 
Design an Analysis of Algorithms II-SECS-1021-03
DilanEscobar1
 
Nekopoi APK 2025 free lastest update
umarali35kp
 
Adobe Illustrator 28.6 Crack My Vision of Vector Design
ghrom2211g
 
Operating system designcfffgfgggggggvggggggggg
rmskumara09
 
System and Network Administraation Chapter 3
jaramharo
 
Introduction Database Management System for Course Database
ReinertYosua
 
Which alternative to Crystal Reports is best for small or large businesses.pdf
Varsha Nayak
 
Agentic AI : A Practical Guide. Undersating, Implementing and Scaling Autono...
IT IDOL Technologies
 
Claude Code: Everyone is a 10x Developer - A Comprehensive AI-Powered CLI Tool
William Chong
 
Flood Susceptibility Mapping Using Image-Based 2D-CNN Deep Learnin. Overview ...
lawrenceomai2
 
2025 Textile ERP Trends: SAP, Odoo & Oracle
Aetos Technologies
 
How to Choose the Right IT Partner for Your Business in Malaysia
Asian Business Services & Technologies
 
Online Work Permit System for Fast Permit Processing
SHEQ Network Limited
 
Agentic AI Use Case- Contract Lifecycle Management (CLM).pptx
mhxyzzp
 
How Creative Agencies Leverage Project Management Software.pdf
Orangescrum
 
Softaken Excel to vCard Converter Software.pdf
markwillsonmw004
 

Memory Management with Page Folios

  • 1. * Based on kernel 6.3 (x86_64) – QEMU * 2-socket CPUs (4 cores/socket) * 16GB memory * Kernel parameter: nokaslr norandmaps * KASAN: disabled * Userspace: ASLR is disabled * Legacy BIOS Memory Management with Page Folios Adrian Huang | May, 2023
  • 2. Agenda • Problem description ✓[Background] Normal high-order page & compound page ✓Legacy page cache • Memory folio’s goal • Solution ✓Page cache with memory folio ✓page struct vs folio struct ✓[Example] total_mapcount() implementation difference between legacy approach and folio
  • 3. Normal high-order page & compound page Page (head page) flags |= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page Page Page Page Normal high-order page pages = alloc_pages(GFP_KERNEL, 2); Four physically contiguous pages: Init compound page metadata during page allocation pages = alloc_pages(GFP_KERNEL | __GFP_COMP, 2); Four physically contiguous pages: not a compound page
  • 4. Compound page Page (head page) flags |= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only . . . Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page (Tail page) _compound_pad_1 (compound_head) Compound page – Use Cases • Mainly used in huge page ✓ hugetlbfs (also called HugeTLB Pages or persistent huge pages) ➢ Reserved inside the kernel and cannot be used for other purposes. ➢ Cannot be swapped out. ➢ Two allocation methods: • Pre-allocated to the kernel huge page pool with appending kernel parameter. • [Dynamically allocated huge pages of the default size] Example: `echo 10 > /proc/sys/vm/nr_hugepages` ➢ User application calls the mmap system call or shared memory system calls (shmget and shmat) to request the huge page allocation. ➢ Used by database for many years ➢ Manual configuration for hugetlb pages is required. ➢ Application change is required. (via open/mmap)
  • 5. Compound page Page (head page) flags |= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only . . . Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page (Tail page) _compound_pad_1 (compound_head)
  • 6. Compound page Page (head page) flags |= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only . . . Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page (Tail page) _compound_pad_1 (compound_head) Compound page – Use Cases • Mainly used in huge page ✓ Transparent Huge Page (THP) ➢ Support the automatic promotion and demotion of page sizes ➢ Transparent to the application: No need to modify application. ➢ Control via /sys/kernel/mm/transparent_hugepage/enabled:
  • 7. Compound page Page (head page) flags |= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only . . . Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page (Tail page) _compound_pad_1 (compound_head) Compound page – Use Cases • Mainly used in huge page • kmalloc: allocation size > 8192 bytes o Check kmalloc_order() • Memory folio When to configure compound page? • Condition: page order >= 1 && __GFP_COMP allocation flag is set • alloc_pages -> … -> prep_new_page -> prep_compound_page
  • 8. Compound page: Problem Description #1 Page (head page) flags |= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only . . . Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page (Tail page) _compound_pad_1 (compound_head) [Problem Description] No unified interface: Ambiguity • Some functions may deal with PAGES_SIZE unit (4KB): They’re unaware of compound pages and huge pages • Some functions accept the page head *only* • Some functions accept the page head or page tail ✓ Call compound_head() to get the page head: waste instructions to get the page head → Performance impact ✓ compound_head() users: ➢ get_page(): This function is called quite frequently ➢ put_page(): This function is called quite frequently ➢ …
  • 9. Legacy page cache 4KB 4KB 512B 512B 512B 512B [file] file->f_pos (continuous file position) Page cache 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B sector .. Disk 4KB Page Cache Buffer Cache (Buffer head) Kernel Space read()/write()/sendfile() User Space mmap()
  • 10. Legacy page cache: Problem Description #2 1. Page cache occupies most of memory pages!!! 2. Each page cache (no “compound page” concept) is added to active/inactive lru list • Long lru list: lock contention & cache misses
  • 11. Agenda • Problem description ✓Normal high-order page & compound page ✓Legacy page cache • Memory folio’s goal • Solution ✓Page cache with memory folio ✓page struct vs folio struct ✓[Example] total_mapcount() implementation difference between legacy approach and folio
  • 12. Memory folio’s goal • Unified interface ✓All accesses via folio struct (head page/tail page in a compound page) • [Page Cache] Shorter LRU list ✓Original: one struct page per 4KB to be added to LRU list ✓Folio: one struct page (page head) per 8KB, 16KB, 32KB, 64KB, 128KB….and so on (include THP) to be added to LRU list • [Anonymous page] THP support ✓create_huge_pmd -> do_huge_pmd_anonymous_page -> __do_huge_pmd_anonymous_page * THP: Transparent Huge Page
  • 13. Agenda • Problem description ✓Normal high-order page & compound page ✓Legacy page cache • Memory folio’s goal • Solution ✓Page cache with memory folio ✓page struct vs folio struct ✓[Example] total_mapcount() implementation difference between legacy approach and folio
  • 14. Page cache with folio 4KB 512B … 512B [file] file->f_pos (continuous file position) folio sector … … .. Disk page page page page 4KB 4KB 4KB 4KB 4KB 4KB 4KB 512B … 512B 512B … 512B 512B … 512B 512B … 512B page page page page 512B … 512B 512B … 512B 512B … 512B Kernel Space read()/write()/sendfile() User Space mmap() • Folio is the container of struct page(s) ✓ All accesses via folio struct ✓ No tail page → fewer run-time checks
  • 15. Page cache with folio 4KB 512B … 512B [file] file->f_pos (continuous file position) folio sector … … .. Disk page page page page 4KB 4KB 4KB 4KB 4KB 4KB 4KB 512B … 512B 512B … 512B 512B … 512B 512B … 512B page page page page 512B … 512B 512B … 512B 512B … 512B Kernel Space read()/write()/sendfile() User Space mmap() Folio’s page order: readahead mechanism • CONFIG_TRANSPARENT_HUGEPAGE is enabled ✓ Minimum: order 2 (4 pages) ✓ Maximum: order 9 (512 pages) • CONFIG_TRANSPARENT_HUGEPAGE is disabled ✓ Minimum: order 2 (4 pages) ✓ Maximum: order 8 (256 pages) • Commit 793917d997df (“mm/readahead: Add large folio readahead”): merged in 5.18 kernel • Default readahead size: 128KB (32 pages)
  • 16. Page cache with folio 4KB 512B … 512B [file] file->f_pos (continuous file position) folio sector … … .. Disk page page page page 4KB 4KB 4KB 4KB 4KB 4KB 4KB 512B … 512B 512B … 512B 512B … 512B 512B … 512B page page page page 512B … 512B 512B … 512B 512B … 512B Kernel Space read()/write()/sendfile() User Space mmap() 1. Short LRU list: Only the head page of folio is added LRU list → Performance improvement 2. 45% improvement for lru-file-mmap-read (vm-scalability): Matthew Wilcox’s PDF file Folio’s page order: readahead mechanism • CONFIG_TRANSPARENT_HUGEPAGE is enabled ✓ Minimum: order 2 (4 pages) ✓ Maximum: order 9 (512 pages) • CONFIG_TRANSPARENT_HUGEPAGE is disabled ✓ Minimum: order 2 (4 pages) ✓ Maximum: order 8 (256 pages) • Commit 793917d997df (“mm/readahead: Add large folio readahead”): merged in 5.18 kernel • Default readahead size: 128KB (32 pages)
  • 17. Page cache with folio: backtrace * kernel: 6.3
  • 18. Agenda • Problem description ✓Normal high-order page & compound page ✓Legacy page cache • Memory folio’s goal • Solution ✓Page cache with memory folio ✓page struct vs folio struct ✓[Example] total_mapcount() implementation difference between legacy approach and folio
  • 19. folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 _flags_2 _head_2 void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Page #0 (head) flags … Page #1 (tail) flags compound_head compound_dtor compound_order compound_mapcount compound_nr … Page #N (tail) flags _compound_pad_1 hpage_pinned_refcount deferred_list . . . page struct vs folio struct Compound pages
  • 20. folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 _flags_2 _head_2 void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Page #0 (head) flags … Page #1 (tail) flags compound_head compound_dtor compound_order compound_mapcount compound_nr … Page #2 (tail) flags _compound_pad_1 hpage_pinned_refcount deferred_list . . . page struct vs folio struct Compound pages folio’s benefit • [Example] 512KB compound page ✓ page struct: Need to maintain 128 page structs (1 head page and 127 tail pages) ✓ folio struct: Maintain 3 page structs regardless of the size of compound pages.
  • 21. folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 → won’t be used _flags_2 _head_2 void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 → won’t be used struct struct list_head _deferred_list page struct vs folio struct folio struct’s members • _entire_mapcount ✓ The compound page is mapped via a single PMD (huge page). • _nr_pages_mapped ✓ Number of individual subpages (PTE: 4KB pages) are mapped. ✓ Scenario: Two processes map the same memory range ✓ One process maps the entire 2MB compound page (Transparent Huge Page - THP): mapped via a single PMD ✓ The other process maps some 4KB pages within this 2MB memory area: mapped via PTEs ✓ Benefit about THP: No need to split the huge page if other processes map 4KB pages within the same memory area. • _folio_nr_pages ✓ Number of pages in this folio. ✓ _folio_nr_pages = 1 << order, where order > 0.
  • 22. folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 _flags_2 _head_2 void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Page #0 (head) flags … Page #1 (tail) flags compound_head compound_dtor compound_order compound_mapcount compound_nr … Page #2 (tail) flags _compound_pad_1 hpage_pinned_refcount deferred_list . . . folio struct vs legacy compound page Page #1 of folio and legacy page struct has the same mapping
  • 23. Agenda • Problem description ✓Normal high-order page & compound page ✓Legacy page cache • Memory folio’s goal • Solution ✓Page cache with memory folio ✓page struct vs folio struct ✓[Example] total_mapcount() implementation difference between legacy approach and folio
  • 24. [kernel v5.11] total_mapcount() page Page cache and anonymous pages struct union page_pool used by netstack struct slab, slob and slub struct Tail pages of compound page struct Second tail page of compound page struct Page table pages: 1. PMD huge PTE 2. x86 pgd page <-> mm_struct struct ZONE_DEVICE pages struct rcu_head: free a page by RCU struct union atomic_t _mapcount: the number of this page is referenced by page table unsigned int page_type unsigned int active: used by slab int units: used by slob atomic_t _refcount … Case 1: Singleton page(s) Get _mapcount directly total_mapcount() users: • Huge page • rmap (reverse mapping)
  • 25. page (head) Page cache and anonymous pages struct union page_pool used by netstack struct slab, slob and slub struct Tail pages of compound page struct Second tail page of compound page struct Page table pages: 1. PMD huge PTE 2. x86 pgd page <-> mm_struct struct ZONE_DEVICE pages struct rcu_head: free a page by RCU struct union atomic_t _mapcount: the number of this page is referenced by page table unsigned int page_type unsigned int active: used by slab int units: used by slob atomic_t _refcount … Case 2: Compound page && hugetlb (hugetlbfs) page page (first tail) compound_head struct union compound_dtor compound_order compound_mapcount compound_nr = 1 << compound_nr struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . page (second tail) _compound_pad_1 struct union hpage_pinned_refcount deferred_list struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . . . . _compound_pad_1 (compound_head) page (second tail) Get compound_mapcount directly compound_mapcount: • Map count of the whole compound page (does not include mapped sub-pages) Steps: 1. Get the head page based on any page (page head or page tail) 2. Read ‘compound_mapcount’ of the first tail page A. page[1].compound_mapcount [kernel v5.11] total_mapcount()
  • 26. page (head) Page cache and anonymous pages struct union page_pool used by netstack struct slab, slob and slub struct Tail pages of compound page struct Second tail page of compound page struct Page table pages: 1. PMD huge PTE 2. x86 pgd page <-> mm_struct struct ZONE_DEVICE pages struct rcu_head: free a page by RCU struct union atomic_t _mapcount: the number of this page is referenced by page table unsigned int page_type unsigned int active: used by slab int units: used by slob atomic_t _refcount … Case 3: [Anonymous page] Compound page && transparent huge page page (first tail) compound_head struct union compound_dtor compound_order compound_mapcount compound_nr = 1 << compound_nr struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . page (second tail) _compound_pad_1 struct union hpage_pinned_refcount deferred_list struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . . . . _compound_pad_1 (compound_head) page (second tail) Steps: 1. Get the head page based on any page (page head or page tail) 2. Read ‘compound_mapcount’ of the first tail page A. page[1].compound_mapcount: Map count of the whole compound page 3. `Accumulate each subpage._mapcount`: A. One process maps 2MB range as a single huge page (a single PMD) B. Another process maps 512 individual PTEs 4. `Accumulate each subpage._mapcount` + page[1].compound_mapcount [kernel v5.11] total_mapcount()
  • 27. page (head) Page cache and anonymous pages struct union page_pool used by netstack struct slab, slob and slub struct Tail pages of compound page struct Second tail page of compound page struct Page table pages: 1. PMD huge PTE 2. x86 pgd page <-> mm_struct struct ZONE_DEVICE pages struct rcu_head: free a page by RCU struct union atomic_t _mapcount: the number of this page is referenced by page table unsigned int page_type unsigned int active: used by slab int units: used by slob atomic_t _refcount … Case 3: [Page cache] Compound page && transparent huge page page (first tail) compound_head struct union compound_dtor compound_order compound_mapcount compound_nr = 1 << compound_nr struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . page (second tail) _compound_pad_1 struct union hpage_pinned_refcount deferred_list struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . . . . _compound_pad_1 (compound_head) page (second tail) Steps: 1. Get the head page based on any page (page head or page tail) 2. Read ‘compound_mapcount’ of the first tail page A. page[1].compound_mapcount: Map count of the whole compound page 3. `Accumulate each subpage._mapcount`: A. One process maps 2MB range as a single huge page (a single PMD) B. Another process maps 512 individual PTEs 4. `Accumulate each subpage._mapcount` + page[1].compound_mapcount - page[1].compound_mapcount * page[1].compound_nr A. File pages has compound_mapcount included in _mapcount [kernel v5.11] total_mapcount()
  • 28. [kernel v6.3] total_mapcount() and folio_mapcount() folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 … void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Case 1: Singleton page – Not a compound page 1 2
  • 29. [kernel v6.3] total_mapcount() and folio_mapcount() folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 … void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Case 2: Compound page is mapped via PMD (huge page) 1 3 2 Get _entire_mapcount 4 5 Get _nr_pages_mapped
  • 30. [kernel v6.3] total_mapcount() and folio_mapcount() folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 … void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Case 3: Compound page is mapped via PMD (huge page) and some subpages are mapped by PTE 1 2 mapcount = folio’s _entire_mapcount + sum(each subpage’s _mapcount)
  • 31. Reference • Memory Folios • LWN - A memory-folio update • LWN - An introduction to compound pages • LWN - Huge pages part 1 (Introduction) • Documentation/mm/transhuge.rst
  • 33. Learn new C standard (C11) from folio: Generic Selection * Reference from: ISO/IEC 9899:201x • C99 defines type-generic macros in the standardized library: the type of argument is detected automatically, and the corresponding function is invoked based on that type. ✓ Example: sqrt(X), ➢ X is double → invoke sqrt() ➢ X is float → invoke sqrtf() ➢ X is long double → invoke sqrtl() • However, programmers cannot define their own type- generic macros in C99. • In C11, programmers can define their own type-generic macros: