Refining Linux
Download as PPTX, PDF0 likes590 views
The document discusses optimizing Linux performance at various levels. It covers recompiling the kernel to optimize settings, monitoring CPU, memory, I/O, and network usage. Specific tools like vmstat, iostat, top, and htop are recommended. Optimizing includes adjusting settings like the I/O scheduler, disk cache policy, and TCP parameters. The kernel loads and decompresses to initialize essential subsystems before probing hardware and loading additional modules.
![Changing Schedulers
Which schedules do you have compiled
/sys/block/sda/queue # cat scheduler
anticipatory deadline [cfq]
Compile noop
/sys/block/sda/queue # modprobe noop-iosched
anticipatory deadline [cfq] noop
Change
/sys/block/sda/queue # echo anticipatory > scheduler
http://www.linuxhowtos.org/System/iosched.htm](https://image.slidesharecdn.com/refininglinux-150504220119-conversion-gate01/85/Refining-Linux-19-320.jpg)
Refining Linux
- 2. Jason Murray, D.CS Cornwall-Lebanon SD jasonmurray72@gmail.com @jasonmurray72 slideshare.net/jasonmurray72
- 3. Agenda Linux Intro What is Optimization? Performance Server Optimization Performance Monitoring System Monitoring Tools Benchmark Tools
- 6. Kernels Linux – Monolithic Kernel Architecture: provides all services Windows – Micro Kernel Architecture: small core group of services that executes other services and applications
- 7. Recompile the Kernel make mrproper - This target cleans up the build environment of any stale files and dependencies that might have been left over from a previous kernel build. All previous kernel configurations will be cleaned (deleted) from the build environment. make clean - This target does not do as thorough a job as the “mrproper” target. It only deletes most generated files. It does not delete the kernel configuration file (.config). make menuconfig - This target invokes a text-based editor interface with menus, option lists, and text-based dialog boxes for configuring the kernel. make xconfig - This is an X Window System–based kernel configuration tool that relies on the Qt graphical development libraries. These libraries are used by KDE-based applications. make gconfig - This target also invokes an X Window System–based kernel configuration tool, but it relies on the GTK2 (GIMP) toolkit. This GTK2 toolkit is heavily used in the GNOME desktop world. make help - This target will show you all the other possible make targets and also serves as a quick online help system.
- 8. What is Optimization? Server Design Stability Availability Performance General Server Response Hardware Response Service Response
- 9. Performance Hardware Safe Clean Environment Software Enhancement in parameter improvements can compensate and improve older hardware performance Avoid a GUI At least unload the GUI after using it
- 10. Server Optimization Fine Tuning Services Hardware Kernel Network Security Database Storage Memory
- 11. Recommendations Custom installs Only load the essentials Partitioning Planning Know the purpose of your server
- 12. Performance Monitoring CPU Network I/O Memory
- 13. CPU Performance Metrics Context Switch Run Queue CPU Utilization Load Average
- 14. Context Switch Context Switch - CPU moves from one thread to another During context switch, the kernel stores current thread state in memory Simultaneously, the kernel retrieves previously stored thread state from memory & loads it to the CPU Context switching affords multitasking of CPU High levels of context switching can cause performance degradation Example: pidstat 2 5 Display five reports of CPU statistics for every active task in the system at two second intervals.
- 15. Run Queue Run Queue – total number of active processes in current queue for CPU Processes in sleep state of I/O wait state are not in the run queue High number of processes in the run queue can cause performance degradation Example: vmstat The r column in the table represents run queue processes
- 16. CPU Utilization CPU Utilization – percentage that the CPU is being used Higher percentage of CPU utilization can cause performance degradation Example: top Real time view of cpu utilization
- 17. Load Average Load Average – average CPU load over a specified time period Load average is displayed for the last 1 minute, 5 minutes, and 15 minutes Easy monitoring for overall load Load Average is calculated by combining the total number of processes in the run queue and the total number of processes in the uninterruptable task status Example: uptime “0.25 1.20 1.90” – System load coming down .25 - Load average for last 1 minute 1.20 – Load average for last 5 minutes 1.90 – Load average for last 15 minutes
- 18. Disk I/O Optimization 4 different I/O schedulers CFQ (default) – distributes all I/O bandwidth evenly among all processes requesting I/O NOOP – acts as a basic FIFO queue expecting the hardware controller to handle the performance operations of the requests; attempts to use as little CPU as possible Anticipatory – designed to increase performance on systems that have slow disks; reorders all disk I/O operations to optimize disk seeks Deadline – places I/O requests in a priority queue so each is guaranteed to run within a certain time; often used in real-time operating
- 19. Changing Schedulers Which schedules do you have compiled /sys/block/sda/queue # cat scheduler anticipatory deadline [cfq] Compile noop /sys/block/sda/queue # modprobe noop-iosched anticipatory deadline [cfq] noop Change /sys/block/sda/queue # echo anticipatory > scheduler http://www.linuxhowtos.org/System/iosched.htm
- 20. Scheduler Agility Changing schedulers on the fly allows for benchmark testing of your specific applications and hardware Notes The change will not be instant. All current processes in run queue must complete first. Once you find the optimal scheduler for your specific situation, apply the change for subsequent reboots
- 21. Scheduler Suggestions Use NOOP or Deadline on SSD drives Benchmark each scheduler to determine your optimal solution CFQ is not always the best scheduler for your system Example: Your system is running a raid
- 22. I/O Optimization Consistent high I/O wait indicates an issue with the disk subsystem Monitor BI & BO (Block in & Block out) Reads/second & writes/second can also identify trouble areas Check TPS (transactions/second) – total of BI & BO Example: vmstat
- 23. vmstat Procs r: The number of processes waiting for run time. b: The number of processes in uninterruptible sleep. Memory swpd: the amount of virtual memory used. free: the amount of idle memory. buff: the amount of memory used as buffers. cache: the amount of memory used as cache. inact: the amount of inactive memory. (-a option) active: the amount of active memory. (-a option) Swap si: Amount of memory swapped in from disk (/s). so: Amount of memory swapped to disk (/s). IO bi: Blocks received from a block device (blocks/s). bo: Blocks sent to a block device (blocks/s). System in: The number of interrupts per second, including the clock. cs: The number of context switches per second. CPU These are percentages of total CPU time. us: Time spent running non- kernel code. (user time, including nice time) sy: Time spent running kernel code. (system time) id: Time spent idle. Prior to Linux 2.5.41, this includes IO- wait time. wa: Time spent waiting for IO. Prior to Linux 2.5.41, included in idle. st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown.
- 24. Disk I/O Optimization File System Max File Size Max Partition Size Journaling Notes Fat16 2 GB 2 GB No Legacy Fat32 4 GB 8 TB No Legacy NTFS 2 TB 256 TB Yes (For Windows Compatibility) NTFS-3g is installed by default in Ubuntu, allowing Read/Write support ext2 2 TB 32 TB No Legacy ext3 2 TB 32 TB Yes Standard linux filesystem for many years. Best choice for super-standard installation. ext4 16 TB 1 EB Yes Modern iteration of ext3. Best choice for new installations where super-standard isn't necessary. reiserFS 8 TB 16 TB Yes No longer well-maintained. JFS 4PB 32PB Yes (metadata) Created by IBM - Not well maintained. XFS 8 EB 8 EB Yes (metadata) Created by SGI. Best choice for a mix of stability and advanced journaling.
- 25. Disk I/O Optimization Disk Cache Policy hdparm -a /dev/sda Readahead value at least 1024 Write Policy Writeback Read Policy Adaptive readahead Penalties Calculate the IOPS
- 26. Disk Optimization Solutions Bonnie++ Hdparm Upgrades BIOS Firmware
- 27. Network Monitoring Total number of packets Received Sent Dropped
- 28. TCP Optimization – Large Production Servers Port Availability Echo 1024 65000 > /proc/sys/net/ipv4/ip_local_port_range Memory Allocation with Socket Buffers Echo 262143 > /proc/sys/net/core/rmem_max Echo 262143 > /proc/sys/net/core/rmem_default TCP Stack Workload Reduction Echo 0 >/proc/sys/net/ipv4/tcp_sack Echo 0 > /proc/sys/net/ipv4/tcp_timestamps
- 29. Memory Optimization Virtual Memory – swap space available on the disk & physical memory (user space & kernel space) Kernel’s file system cache takes advantage of the unused RAM During a swap, Linux writes the least used memory pages from the physical memory to the swap space on the disk High levels of swapping will degrade performance, because the disk is much slower Vmstat command Recommendation Benchmark tests prove that ECC RAM is faster
- 30. Additional Helpful Commands Netstat – lists network sockets Ps – snapshot of current processes Atop – interactive monitor for the load averages Mtop – displays mysql threads Iostat – reports cpu & I/O statistics Xosview – displays CPU usage, load average, memory usage, swap space usage, network usage, interrupts, and serial port status
- 31. Kernel Optimization The first thing to get loaded is the kernel. Keep in mind that no operating system exists in memory at this point, and PCs (by their unfortunate design) have no easy way to access all of their memory. Thus, the kernel must load completely into the first megabyte of available random access memory (RAM). In order to accomplish this, the kernel is compressed. The head of the file contains the code necessary to bring the CPU into protected mode (thereby removing the memory restriction) and decompress the remainder of the kernel. Kernel Execution. With the kernel in memory, it can begin executing. It knows only whatever functionality is built into it, which means any parts of the kernel compiled as modules are useless at this point. At the very minimum, the kernel must have enough code to set up its virtual memory subsystem and root file system (usually, the ext3 file system). Once the kernel has started, a hardware probe determines what device drivers should be initialized. From here, the kernel can mount the root file system. (You could draw a parallel of this process to that of Windows being able to recognize and access its C drive.) The kernel mounts the root file system and starts a program called init.
- 32. Boot Loaders MBR – starting on page 142 of Linux Administration GRUB Modify boot record & kernel initiation Splash screen Hidden menus Title Root Initrd – preload modules in kernel LILO
- 33. Database Optimization MySQL – use MySQL tuning scripts Optimize table
- 34. OpenLDAP Optimization Important Decision – Which attributes will you build indexes on Optimal Parameters – change before data entry /etc/openldap.conf Cachesize 10000 Dbcachesize 100000 Sizelimit 10000 Loglevel 0 dbcacheNoWsync Index cn,uid Index uidnumber Index gid Index gidnumber Index mail
- 35. Apache Optimization Good Benchmark Scores – many initial daemons Example MinSpareServers 20 MaxSpareServers 80 StartServers 32 If Apache is recompiled MaxClients 256 MaxRequestsPerChild 10000 Notes – A high number of max servers and an honest number of spare servers will work fine. For the most part the StartServers help with the instant load that benchmarks generate.
- 36. Apache Management Default Directory - /var/www/html Apache Config File - /etc/httpd/httpd.conf /etc/httpd/conf.d/phpMyAdmin.conf PHP Config File - /etc/php.ini /etc/init.d/httpd start|stop|restart Service httpd start|stop|restart /var/log/httpd/error_log /var/log/httpd/access_log
- 37. Poor Performance Website Consider Optimizers Memcache Adjusting Apache Minimizing Apache modules Alternative to apache - nginx
- 38. Benchmarking Benchmarking is helping; HOWEVER, Its impossible to duplicate real world situations Understand the GOAL for benchmarking Measuring the performance of ONE particular element very accurately Benchmarks are excellent tools if you understand what they are doing and trying to accomplish
- 39. Benchmark Tools Bonnie++ - file system benchmarking tool Dbench – I/O workload generator to a file system or networked CIFS or NFS server Http_load – throughput tester for a web server Dkftpbench – FTP download benchmark analyzer Tiobench – multi-threaded I/O benchmark tool Ttcp – network throughput measuring tool Netperf – network performance tester
- 40. More Helpful Tools Lsof – list open files Tcpdump – network packet analyzer Htop – process monitoring Iotop – monitors disk I/O IPTraf – real time IP LAN monitoring Psacct – monitor user activity Monit – process & services monitoring Nethogs – monitor per process network bandwidth Iftop – network bandwidth monitoring Monitorix – system & network monitoring
- 41. Troubleshooting Methodology Clearly understand the problem Monitor & collect data Eliminate and narrow issues Make one change at a time & OBSERVE Document everything!
- 42. References Davor Guttierrez - @dguttierrez Soyinka, W. (2008). Linux Administration: A Beginner's Guide. McGraw-Hill, Inc.. http://www.tecmint.com/command-line-tools-to-monitor-linux- performance/ http://www.thegeekstuff.com/2011/07/iostat-vmstat-mpstat- examples/