Linux Memory Basics for SysAdmins - ChinaNetCloud Training

ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2013 ChinaNetcloud Training Program
ChinaNetCloud Training
Linux Memory Basics
By ChinaNetCloud
Pioneers in OaaS – Operations-as-a-Service
October, 2013
www.ChinaNetCloud.com

Copyright 2013 ChinaNetCloud 2
Introduction
●
Linux memory is complex and interesting
●
Class summarizes:
●
Types of memory
●
How it's used
●
How to troubleshoot memory issues

Overview, Purpose & Use
●
Linux memory is one of the most important
areas for engineers to understand
●
Used by everything
●
Often not well-understood
●
Often the cause of problems

Introduction
●
Memory is simple, in theory
●
Complex in real use
●
Plenty of strange things, too
●
Important to understand
●
Important to monitor

Memory Types I
●
Physical RAM – Fixed amount, real RAM
●
Virtual Memory – Virtual, can swap
●
Shared – Between processes
●
Oracle & Postgres use this
●
Slab – Kernel memory
●
/proc/slabinfo & slabtop utility
●
Includes big RAM users
– tcpmem & inode cache

Memory Types II
●
Page Cache – File system cache
●
Dirty
●
Changed file system data waiting to write to disks
●
Important for high write systems

/proc/meminfo
Very useful but complex
●
These are items not found by 'free' or 'top'
●
MemTotal: Total usable ram (RAM minus kernel binary code)
●
MemFree: Total free memory, same as free's 'free' output
●
SwapCached: Memory swapped out, then back in, but still
also in the swapfile

/proc/meminfo
●
Active – Recently used, not reclaimed unless necessary
●
Inactive – Less recently used, likely to be reclaimed
●
Dirty – File / Page Cache waiting to be written to disk
●
Writeback – Memory which is now being written to the disk
●
Mapped – Memory Mapped files, such as libraries
●
Includes Mongo, Varnish, and many others
●
Slab – Kernel memory, usually 256-512MB

Memory Page Size
●
4KB page size
●
Important as many stats are in pages
– Be careful of units ! KB vs. Pages
●
sysctl items like tcpmem are mixed
●
Do not confuse with disk sector size of 0.5KB

Free -m
●
Total – Physical in machine
●
Used – by Apps plus Cache, Buffers and maybe Shared
●
Free – Totally unused RAM, not important, usually very small
●
Shared – by different processes like Oracle / Postgres
●
Buffers – Raw block cache to/from disks, not impotrant
●
Cached (Page Cache) – Used by disk files cached in RAM
●
Swap – Total, Used, Free – Used should be small
●
-/+ buffers/cache – Important numbers, inside box is key
●
PAY ATTENTION to the BOX number – it's all that matters
total used free shared buffers cached
Mem: 3961 3901 60 0 121 1232
-/+ buffers/cache: 2546 1414
Swap: 4683 756 3927

Top, ps Memory Output
●
VIRT – Virtual size
●
Includes mapped libraries, often very large
●
Not too useful – IGNORE THIS
●
RSS – Resident Set Size
●
Most important !
●
Real RAM used by applications, not including swap
●
SHR – Shared, such as Oracle / Postgres
●
Be careful to see processes not threads
●
Threads will share RSS, see htop in thread mode

Swap
●
Simple, but not simple
●
Managed by kswapd
●
Actual swap used calc
●
SwapTotal – SwapCached – SwapFree
●
Goal is zero swap on servers
●
Never let a system actively swap
●
But some systems will have small swap
●
Common to see 100-200MB, more on big system
●
NUMA defaults can cause swapping

Problem with Swap
●
Swapping FREEZES the swapping process
●
So, a 32MB byte swap of MySQL RAM
●
Freezes ALL of MySQL for many seconds
●
Very bad for applications
●
Bad for any single process system
●
MySQL, Nginx, HAProxy, etc. (not Apache)
●
Also uses valuable IO, slowing DBs, etc.
●
Goal is to NEVER swap real applications

Swappiness
●
Tells kernel which is more important
●
File system Cache vs. App Memory
●
Default is 60 – Means cache more important
●
Stupid for servers – will swap even with free RAM
●
Set to 1, always (used to be 0, now use 1)
●
Won't swap until all RAM used by applications
●
Note some swap anyway (see next slide)
●
Set by sysctl
●
Check in /proc/sys/vm/swappiness

Swap still happens
●
Linux likes to swap a little
●
Even with free RAM and swappiness = 0
●
Some kernel data wants to swap
●
Can pre-swap SwapCached data
●
Still a mystery, but it's okay
●
Very large systems (64GB+ might swap 1-2GB)
●
Turning off can cause kswapd to go crazy
●
Watch vmstat swapin/out, make sure very small

NUMA - Can cause swap
●
NUMA – Non-Uniform Memory Access
●
Each CPU has its own RAM
●
Slower to use 'other' CPU's RAM
●
Standard now on all Intel servers
●
Causes RAM inbalance on big RAM processes
●
Like MySQL, MongoDB, Java
●
Install & use numa-utils package, numactl app
●
Set all big processes to full interleave:
●
numactrl –interleave all mysqld . . .

Dirty Memory & Ratios
●
Causes strange problems & performance issues
●
See /proc/meminfo
●
Dirty is the changed FileSystem cache data
●
Must be written to disk
●
Can cause big problems if too high
●
Two ratios control
●
/proc/sys/vm/dirty_background_ratio / _bytes
●
/proc/sys/vm/dirty_ratio / _bytes
●
Freezes all writing processes when dirty > dirty_ratio
●
Calculation not clear, but keep dirty data < 20% of RAM

OOM Killer
●
Kills process when no more RAM or Swap
●
Chooses process to die via a score
●
Biggest RAM user (MySQL) often dies first
●
See /proc/<pid>/oom_score
●
Always a message in dmesg - /var/log/kernel
●
Can adjust score to control (advanced)
●
Best practice to monitor swap and log

Summary
●
Memory important to understand
●
Especially basics & definitions
●
Understand swap & swappiness
●
Watch NUMA on big systems
●
Memory is cheap – buy more !

About ChinaNetCloud

ChinaNetCloud
Sales@ChinaNetCloud.com
www.ChinaNetCloud.com
Beijing Office:
Lee World Business Building #305
57 Happiness Village Road,
Chaoyang District
Beijing, 100027 China
Silicon Valley Office:
California Avenue
Palo Alto, 94123 USA
Shanghai Headquarters:
X2 Space 1-601, 1238 Xietu Lu
Shanghai, 200032 China
T: +86-21-6422-1946 F: +86-21-
6422-4911

Linux Memory Basics for SysAdmins - ChinaNetCloud Training

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Linux Memory Basics for SysAdmins - ChinaNetCloud Training (20)

More from ChinaNetCloud (20)

Recently uploaded (20)

Linux Memory Basics for SysAdmins - ChinaNetCloud Training