Checking Linux system performance with sar
Sar is a system utility that gives us many ways to examine performance on a Linux system. It provides details on all aspects of system performance including system load, CPU usage, memory use, paging, swapping, disk usage, device load, network activity, etc.
The name “sar” stands for “system activity report,” and it can display current performance, provide reports that are based on log files stored in your system’s /var/log/sa (or /var/log/sysstat) folder, or be set up to automatically produce daily reports. It’s part of sysstat – a collection of system performance monitoring tools.
To check if sar is available on your system, run a command like this:
$ which sar /usr/bin/sar
If not, you will need to install it with a command like “yum install sysstat” or “apt install sysstat”. You will then be able to run commands like these which collect performance details from your system:
$ sar -d 5 2 Linux 5.11.15-300.fc34.x86_64 (dragonfly) 06/16/2021 _x86_64_ (2 CPU) 12:38:42 PM DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util 12:38:47 PM sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:38:47 PM sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:38:47 PM sdb 0.40 0.00 6.40 0.00 16.00 0.01 8.50 0.34 12:38:47 PM sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:38:47 PM zram0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:38:47 PM DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util 12:38:52 PM sda 2.40 0.00 388.80 0.00 162.00 0.01 2.00 0.34 12:38:52 PM sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:38:52 PM sdb 1.40 0.00 21.60 0.00 15.43 0.01 2.00 0.28 12:38:52 PM sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:38:52 PM zram0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util Average: sda 1.20 0.00 194.40 0.00 162.00 0.00 2.00 0.17 Average: sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: sdb 0.90 0.00 14.00 0.00 15.56 0.01 3.44 0.31 Average: sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: zram0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Note that this report shows device usage with activity displayed for /dev/sda and /dev/sdb. It includes two 5-second reports plus the averaging. If you delve into the sar man page, you’ll find explanations of each data column. In the above example, the fields included in the above example include:
DEV the device tps Total number of transfers per second rkB/s Number of kilobytes read from the device per second wkB/s Number of kilobytes written to the device per second dkB/s Number of kilobytes discarded for the device per second areq-sz Average size (in kilobytes) of the I/O requests aqu-sz Average queue length of the requests await Average time (in milliseconds) for I/O requests %util Percentage of elapsed time during which I/O requests were issued to the device
The next command pulls data on memory usage with three 5-second reports plus the averages.
$ sar -r 5 3 Linux 5.11.15-300.fc34.x86_64 (dragonfly) 06/16/2021 _x86_64_ (2 CPU) 12:09:54 PM kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 12:09:59 PM 3089584 5231848 471144 7.75 35944 2257060 2126524 17.50 645168 2021508 48 12:10:04 PM 3089584 5231848 471140 7.75 35944 2257060 2126524 17.50 645168 2021552 48 12:10:09 PM 3089584 5231848 471140 7.75 35944 2257060 2126524 17.50 645168 2021560 144 Average: 3089584 5231848 471141 7.75 35944 2257060 2126524 17.50 645168 2021540 80
Using a wide terminal window will clearly make the details a lot easier to view.
Looking at current performance stats
The sar commands shown above are all using real time data. In other words, sar is collecting the data as directed and displaying it. No log files are involved. Older stats will not be available.
The default display is CPU. So, if you don’t specify an option, you’ll see something like this:
$ sar 10 2 Linux 5.11.15-300.fc34.x86_64 (dragonfly) 06/16/2021 _x86_64_ (2 CPU) 10:17:50 AM CPU %user %nice %system %iowait %steal %idle 10:18:00 AM all 0.05 0.00 0.15 0.05 0.00 99.75 10:18:10 AM all 0.05 0.00 0.20 0.00 0.00 99.75 Average: all 0.05 0.00 0.18 0.03 0.00 99.75
To save the output from a command like this in a file, you could run the command like this:
$ sar -o sarfile 6 2
Keep in mind that the data file created (sarfile in this example) will not be a text file that you can display with a cat or more command. However, you can display it using a sar command that includes the file name as an argument:
$ sar -f sarfile Linux 5.11.15-300.fc34.x86_64 (dragonfly) 06/15/2021 _x86_64_ (2 CPU) 10:15:52 AM CPU %user %nice %system %iowait %steal %idle 10:15:58 AM all 0.17 0.00 0.17 0.25 0.00 99.42 10:16:04 AM all 0.08 0.00 0.33 0.00 0.00 99.58 Average: all 0.12 0.00 0.25 0.12 0.00 99.50
Looking at previous performance stats
You can easily set sar up to run automatically on your system, store its collected stats in daily files, produce daily reports and remove the oldest of the files after a given number of days (default is 28). This allows you to review performance on recent dates and compare what you’re seeing with current stats. This can often help you answers questions like “What’s different today?” — often the first question you’re likely to ask yourself when problems arise.
Once you’ve collected data using sar, you can look at it with sar commands like this command that extracts data from the sa14 data file:
$ sar -f /var/log/sa/sa14 -b Linux 5.11.15-300.fc34.x86_64 (dragonfly) 06/14/2021 _x86_64_ (2 CPU) 02:54:22 PM LINUX RESTART (2 CPU) 03:00:19 PM tps rtps wtps dtps bread/s bwrtn/s bdscd/s 03:10:19 PM 1.02 0.02 1.00 0.00 0.01 69.05 0.00 03:20:19 PM 1.05 0.02 1.03 0.00 0.01 75.03 0.00 03:30:19 PM 0.99 0.02 0.97 0.00 0.28 74.35 0.00
To do this, you need to first enable sar to run automatically and save performance stats in data files. The process for doing this is slightly different depending on which Linux distribution you are using.
Fedora
On Fedora, you would use commands like these to enable sar to collect daily stats:
# systemctl start sysstat.service # systemctl enable sysstat.service # systemctl status sysstat.service
After this, sar will store its data files in the /var/log/sa directory. A couple days after setting it up, you can expect to see something like this:
$ ls -l total 1096 -rw-r--r--. 1 root root 103292 Jun 14 23:50 sa14 <== data files -rw-r--r--. 1 root root 273544 Jun 15 23:50 sa15 -rw-r--r--. 1 root root 156240 Jun 16 13:30 sa16 -rw-r--r--. 1 root root 158740 Jun 15 00:07 sar14 <== daily reports -rw-r--r--. 1 root root 420712 Jun 16 00:07 sar15
The numbers in the file names (e.g., 14) represent the day of the month. The three “sa” files (sa14, sa15 and sa16) are the data files for each of the three days and the sar14 and sar15 files are the performance reports for the first two days. On Fedora, you can determine how many days’ worth of data will be saved by using this command:
$ grep HISTORY /etc/sysconfig/sysstat HISTORY=28
Ubuntu
To enable sar on Ubuntu and related systems, edit the /etc/default/sysstat file and change ENABLED=”false” to ENABLED=”true”.
ENABLED="true"
The log files will be stored in /var/log/sysstat.
Viewing daily reports
To view any of the daily sar reports (e.g., sar15), you can simply use commands like more or cat. These files are simple text files.
Once a day has ended (shortly after midnight), a full-day report will be available with a name like “sar15”. While files like “sa14” are used to collect the raw data, files like “sar14” are text files that contain the daily reports. They contain quite a lot of performance details.
$ wc -l sar15 4701 sar15 $ more sar15 Linux 5.11.15-300.fc34.x86_64 (dragonfly) 2021-06-15 _x86_64_ (2 CPU) 00:00:19 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle 00:10:19 all 0.07 0.00 0.12 0.05 0.00 0.01 0.01 0.00 0.00 99.74 00:10:19 0 0.03 0.01 0.05 0.02 0.00 0.00 0.01 0.00 0.00 99.88 00:10:19 1 0.10 0.00 0.20 0.08 0.00 0.01 0.01 0.00 0.00 99.60 00:20:19 all 0.07 0.00 0.10 0.04 0.00 0.00 0.01 0.00 0.00 99.77 00:20:19 0 0.01 0.00 0.02 0.02 0.00 0.00 0.00 0.00 0.00 99.94 00:20:19 1 0.12 0.00 0.18 0.07 0.00 0.01 0.01 0.00 0.00 99.60 00:30:19 all 0.06 0.00 0.10 0.06 0.00 0.01 0.01 0.00 0.00 99.76 …
To extract data from one of the daily data files (not the report files), you would use a command like this using the -f option to specify the data file to be used:
$ sar -f sa15 | head -22 Linux 5.11.15-300.fc34.x86_64 (dragonfly) 06/15/2021 _x86_64_ (2 CPU) 12:00:19 AM CPU %user %nice %system %iowait %steal %idle 12:10:19 AM all 0.07 0.00 0.14 0.05 0.00 99.74 12:20:19 AM all 0.07 0.00 0.11 0.04 0.00 99.77 12:30:19 AM all 0.06 0.00 0.12 0.06 0.00 99.76 12:40:19 AM all 0.06 0.00 0.12 0.05 0.00 99.77 12:50:19 AM all 0.07 0.00 0.11 0.05 0.00 99.77 01:00:19 AM all 0.07 0.02 0.12 0.05 0.00 99.74 01:10:19 AM all 0.07 0.00 0.11 0.05 0.00 99.77 01:20:19 AM all 0.30 0.00 0.12 0.04 0.00 99.53 01:30:19 AM all 0.07 0.00 0.11 0.04 0.00 99.77 01:40:19 AM all 0.07 0.00 0.11 0.03 0.00 99.78 01:50:19 AM all 0.07 0.00 0.12 0.03 0.00 99.78 02:00:19 AM all 0.06 0.00 0.12 0.03 0.00 99.78 02:10:19 AM all 0.06 0.00 0.13 0.04 0.00 99.77 02:20:19 AM all 0.06 0.00 0.12 0.04 0.00 99.78 02:30:19 AM all 0.06 0.00 0.12 0.02 0.00 99.79 02:40:19 AM all 0.07 0.00 0.12 0.03 0.00 99.79 02:50:19 AM all 0.07 0.94 0.20 0.05 0.00 98.76 03:00:19 AM all 0.18 0.00 0.12 0.03 0.00 99.67 03:10:19 AM all 0.06 0.00 0.12 0.04 0.00 99.78
You can specify a start (-s) and end (-e) times in addition to the file name as in this next example.
$ sar -s 11:00 -e 12:00 -f sa15 Linux 5.11.15-300.fc34.x86_64 (dragonfly) 06/15/2021 _x86_64_ (2 CPU) 11:00:19 AM CPU %user %nice %system %iowait %steal %idle 11:10:19 AM all 0.06 0.00 0.17 0.05 0.00 99.73 11:20:19 AM all 0.05 0.00 0.17 0.05 0.00 99.73 11:30:19 AM all 0.05 0.00 0.17 0.05 0.00 99.73 11:40:19 AM all 0.18 0.00 0.17 0.05 0.00 99.60 11:50:19 AM all 0.06 0.02 0.18 0.05 0.00 99.69 Average: all 0.08 0.00 0.17 0.05 0.00 99.70
Options, options and options
You can use sar to display stats related to:
- Paging (sar -B)
- I/O and transfer rates (sar -b)
- Block devices (sar -d)
- Currently mounted file systems(sar -F)
- Huge page utilization(sar -H)
- Interrupts (sar -I)
- Power management (sar -m)
- Network statistics (sar -n)
- Run queue and load average (sar -q)
- Memory usage (sar -r)
- Swap utilization (sar -S)
- CPU utilization (sar -u)
- Status of inode, file and other kernel tables (sar -v)
- Swapping (sar -W)
- Context switches (sar -w)
- TTY device activity (sar -y)
In fact, the easiest way to get a listing of all of these options and what they represent is to ask sar for help. Notice that for some options, like networking, you have a choice of which variety of activity (e.g., TCP or sockets) you want to see.
$ sar –help
Usage: sar [ options ] [ <interval> [ <count> ] ]
Main options and reports (report name between square brackets): -B Paging statistics [A_PAGE] -b I/O and transfer rate statistics [A_IO] -d Block devices statistics [A_DISK] -F [ MOUNT ] Filesystems statistics [A_FS] -H Hugepages utilization statistics [A_HUGE] -I { <int_list> | SUM | ALL } Interrupts statistics [A_IRQ] -m { <keyword> [,...] | ALL } Power management statistics [A_PWR_...] Keywords are: CPU CPU instantaneous clock frequency FAN Fans speed FREQ CPU average clock frequency IN Voltage inputs TEMP Devices temperature USB USB devices plugged into the system -n { <keyword> [,...] | ALL } Network statistics [A_NET_...] Keywords are: DEV Network interfaces EDEV Network interfaces (errors) NFS NFS client NFSD NFS server SOCK Sockets (v4) IP IP traffic (v4) EIP IP traffic (v4) (errors) ICMP ICMP traffic (v4) EICMP ICMP traffic (v4) (errors) TCP TCP traffic (v4) ETCP TCP traffic (v4) (errors) UDP UDP traffic (v4) SOCK6 Sockets (v6) IP6 IP traffic (v6) EIP6 IP traffic (v6) (errors) ICMP6 ICMP traffic (v6) EICMP6 ICMP traffic (v6) (errors) UDP6 UDP traffic (v6) FC Fibre channel HBAs SOFT Software-based network processing -q [ <keyword> [,...] | PSI | ALL ] System load and pressure-stall statistics Keywords are: LOAD Queue length and load average statistics [A_QUEUE] CPU Pressure-stall CPU statistics [A_PSI_CPU] IO Pressure-stall I/O statistics [A_PSI_IO] MEM Pressure-stall memory statistics [A_PSI_MEM] -r [ ALL ] Memory utilization statistics [A_MEMORY] -S Swap space utilization statistics [A_MEMORY] -u [ ALL ] CPU utilization statistics [A_CPU] -v Kernel tables statistics [A_KTABLES] -W Swapping statistics [A_SWAP] -w Task creation and system switching statistics [A_PCSW] -y TTY devices statistics [A_SERIAL]
That’s a LOT of choices! You can, however, use the -A option to collect the data provided by all of the options included included in the string “bBdFHSvwWy”.
Wrap-Up
While sar is easy to install and use, it provides a lot of valuable details on how your system is performing and might help you drill down to the cause of a problem when one arises.
The only “problem” with sar is that it generates a LOT of data and you’ll have to become accustomed to using it and familiar with what routine performance on your system looks like before unusual performance or reasons for concern will jump out at you.