SlideShare a Scribd company logo
How to Diagnose Problems Quickly on Linux Servers Richard Cunningham (@rythie)
Not working
Running Slow Two Types of Problems
Basics ps  –  List of processes (use ps -ef) top  – Similar to ps but shows memory usage and is interactive dmesg  – Messages directly from the kernel lsof  – list open file and network ports /var/log/  - Where most/all of the logs live du  – you may be out of disk space!
Understanding 'top' top - 17:59:20 up 272 days, 22:17,  1 user,  load average: 0.27, 0.33, 0.29 Tasks: 142 total,  1 running, 141 sleeping,  0 stopped,  0 zombie Cpu(s):  1.4%us,  0.5%sy,  0.3%ni, 95.3%id,  1.9%wa,  0.0%hi,  0.1%si,  0.5%st Mem:  2051088k total,  1986640k used,  64448k free,  48156k buffers Swap:  4194300k total,  129028k used,  4065272k free,  810744k cached PID USER  PR  NI  VIRT  RES   SHR S %CPU %MEM  TIME+  COMMAND  1156 mysql  20  0  793m  467m  5060 S  2.0 23.4  16960:15 /usr/local/mysql-5.1 4130 richie  20  0 10916  940   668 R  2.0  0.0  0:00.01 top -cbn1  17311 backend  20  0  115m  40m  6240 S  2.0  2.0  12:54.17 /usr/local/bin/php  1 root  20  0 10408  508   480 S  0.0  0.0  6:32.75 init [3]  2 root  20  0  0  0   0 S  0.0  0.0  0:00.00 [kthreadd]  3 root  20  0  0  0   0 S  0.0  0.0  5:56.83 [ksoftirqd/0]
/var/log ls -ltr /var/log  – see most recently modified tail -f /var/log/message  – watch for new lines at the end of a log file grep  – most of those logs have too much irrelevant stuff in them, grep helps
Slowness iostat  – I/O device statistics free  – how much free memory you have sar  – historical logs of performance
Understanding 'free' $ free -m total  used  free  shared  buffers  cached Mem:  3019  2908  111   0  225  1187 -/+ buffers/cache:  1495  1524 Swap:  0  0  0
What are System Calls? open(), read(), write()... 011100101101010... Compiled Program (including libraries) Kernel Hardware
strace – typical output open("/etc/configfile.conf", O_RDONLY)  = 3 read(3, "#this is the config file for...”, 4096) = 32 write(1, "hello", 5)  = 5
strace - invocation strace -p <process id> strace <program name>
Wireshark
Wireshark
Wireshark
Wireshark Use tcpdump -s0 -w filename To capture network traffic to a file for later analysis in wireshark (i.e. copy it to you local machine) Wireshark available for OSX, Windows and Linux and is Open Source

More Related Content

PDF
Clear cache memory
PPT
Linux monitoring and Troubleshooting for DBA's
TXT
PPTX
Terminal commands ubuntu 2
PPTX
Linux file commands
ODP
Linux monitoring
PDF
Using the command line on macOS
ODP
import rdma: zero-copy networking with RDMA and Python
Clear cache memory
Linux monitoring and Troubleshooting for DBA's
Terminal commands ubuntu 2
Linux file commands
Linux monitoring
Using the command line on macOS
import rdma: zero-copy networking with RDMA and Python

What's hot (20)

PDF
2345014 unix-linux-bsd-cheat-sheets-i
DOCX
Basic linux commands
DOCX
System ctl file that is created at time of installation
PPTX
Parallel computing in bioinformatics t.seemann - balti bioinformatics - wed...
PDF
PPT
Cache presentation on Mapping and its types
PDF
Virtual memory 20070222-en
RTF
Useful linux-commands
PDF
Introducing NetBSD 5.0
PPT
Linux50commands
PDF
Cli1 Bibalex
PPTX
От sysV к systemd
PPT
Bootloader and MMU (english)
PPTX
Cache memory
DOCX
Basic command for linux
PDF
Linux Commands - 3
PDF
Kernel Recipes 2019 - Faster IO through io_uring
TXT
PDF
Linux : Booting and runlevels
2345014 unix-linux-bsd-cheat-sheets-i
Basic linux commands
System ctl file that is created at time of installation
Parallel computing in bioinformatics t.seemann - balti bioinformatics - wed...
Cache presentation on Mapping and its types
Virtual memory 20070222-en
Useful linux-commands
Introducing NetBSD 5.0
Linux50commands
Cli1 Bibalex
От sysV к systemd
Bootloader and MMU (english)
Cache memory
Basic command for linux
Linux Commands - 3
Kernel Recipes 2019 - Faster IO through io_uring
Linux : Booting and runlevels
Ad

Similar to How to Diagnose Problems Quickly on Linux Servers (20)

PDF
MeetBSD2014 Performance Analysis
PDF
Linux Systems Performance 2016
PDF
PDF
LISA2019 Linux Systems Performance
PDF
Linux Performance Tools 2014
PDF
Essential Linux Commands for DBAs
PDF
YOW2020 Linux Systems Performance
PPT
Servers and Processes: Behavior and Analysis
PDF
Linux Server Deep Dives (DrupalCon Amsterdam)
PDF
Linux Performance Profiling and Monitoring
PDF
OSDC 2015: Georg Schönberger | Linux Performance Profiling and Monitoring
PDF
Linux Performance Tools
PDF
Velocity 2015 linux perf tools
PDF
Hotsos Advanced Linux Tools
PPTX
Broken Linux Performance Tools 2016
PPTX
Refining Linux
PPTX
Designing Tracing Tools
PPTX
System administration
PDF
Red Hat Enterprise Linux Troubleshooting Guide Cane B
PDF
linux monitoring and performance tunning
MeetBSD2014 Performance Analysis
Linux Systems Performance 2016
LISA2019 Linux Systems Performance
Linux Performance Tools 2014
Essential Linux Commands for DBAs
YOW2020 Linux Systems Performance
Servers and Processes: Behavior and Analysis
Linux Server Deep Dives (DrupalCon Amsterdam)
Linux Performance Profiling and Monitoring
OSDC 2015: Georg Schönberger | Linux Performance Profiling and Monitoring
Linux Performance Tools
Velocity 2015 linux perf tools
Hotsos Advanced Linux Tools
Broken Linux Performance Tools 2016
Refining Linux
Designing Tracing Tools
System administration
Red Hat Enterprise Linux Troubleshooting Guide Cane B
linux monitoring and performance tunning
Ad

Recently uploaded (20)

PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
PDF
Building Integrated photovoltaic BIPV_UPV.pdf
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
PPT
Teaching material agriculture food technology
PDF
Machine learning based COVID-19 study performance prediction
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
PPTX
Programs and apps: productivity, graphics, security and other tools
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
PDF
Approach and Philosophy of On baking technology
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
PDF
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
PPTX
Big Data Technologies - Introduction.pptx
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
Building Integrated photovoltaic BIPV_UPV.pdf
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Per capita expenditure prediction using model stacking based on satellite ima...
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
Teaching material agriculture food technology
Machine learning based COVID-19 study performance prediction
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
Programs and apps: productivity, graphics, security and other tools
Advanced methodologies resolving dimensionality complications for autism neur...
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
Approach and Philosophy of On baking technology
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Dropbox Q2 2025 Financial Results & Investor Presentation
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Big Data Technologies - Introduction.pptx
Agricultural_Statistics_at_a_Glance_2022_0.pdf

How to Diagnose Problems Quickly on Linux Servers

  • 1. How to Diagnose Problems Quickly on Linux Servers Richard Cunningham (@rythie)
  • 3. Running Slow Two Types of Problems
  • 4. Basics ps – List of processes (use ps -ef) top – Similar to ps but shows memory usage and is interactive dmesg – Messages directly from the kernel lsof – list open file and network ports /var/log/ - Where most/all of the logs live du – you may be out of disk space!
  • 5. Understanding 'top' top - 17:59:20 up 272 days, 22:17, 1 user, load average: 0.27, 0.33, 0.29 Tasks: 142 total, 1 running, 141 sleeping, 0 stopped, 0 zombie Cpu(s): 1.4%us, 0.5%sy, 0.3%ni, 95.3%id, 1.9%wa, 0.0%hi, 0.1%si, 0.5%st Mem: 2051088k total, 1986640k used, 64448k free, 48156k buffers Swap: 4194300k total, 129028k used, 4065272k free, 810744k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1156 mysql 20 0 793m 467m 5060 S 2.0 23.4 16960:15 /usr/local/mysql-5.1 4130 richie 20 0 10916 940 668 R 2.0 0.0 0:00.01 top -cbn1 17311 backend 20 0 115m 40m 6240 S 2.0 2.0 12:54.17 /usr/local/bin/php 1 root 20 0 10408 508 480 S 0.0 0.0 6:32.75 init [3] 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 [kthreadd] 3 root 20 0 0 0 0 S 0.0 0.0 5:56.83 [ksoftirqd/0]
  • 6. /var/log ls -ltr /var/log – see most recently modified tail -f /var/log/message – watch for new lines at the end of a log file grep – most of those logs have too much irrelevant stuff in them, grep helps
  • 7. Slowness iostat – I/O device statistics free – how much free memory you have sar – historical logs of performance
  • 8. Understanding 'free' $ free -m total used free shared buffers cached Mem: 3019 2908 111 0 225 1187 -/+ buffers/cache: 1495 1524 Swap: 0 0 0
  • 9. What are System Calls? open(), read(), write()... 011100101101010... Compiled Program (including libraries) Kernel Hardware
  • 10. strace – typical output open(&quot;/etc/configfile.conf&quot;, O_RDONLY) = 3 read(3, &quot;#this is the config file for...”, 4096) = 32 write(1, &quot;hello&quot;, 5) = 5
  • 11. strace - invocation strace -p <process id> strace <program name>
  • 15. Wireshark Use tcpdump -s0 -w filename To capture network traffic to a file for later analysis in wireshark (i.e. copy it to you local machine) Wireshark available for OSX, Windows and Linux and is Open Source
  • 16. The End Richard Cunningham (@rythie)