An introduction to dsar: distributed sar utility
This blog post is about an open-source tool that I created that can read node_exporter (Prometheus format) endpoints and print (performance) data just like the sar and other utilities would on unix-based systems. The main difference is the tool can read the data from a remote system, without the need to install the sar utility locally or on the remote system. It can also read multiple machines at the same time, and print the data, which is useful if you run a cluster.
The tool can be found at github.com/FritsHoogland/dsar, and if you just want an executable, you can find these here. I created RPMs for EL 7 and 8, a DEB for Ubuntu 22.04 and just the executable for Mac OSX on Intel and ARM. If there is a Linux version that I missed, please file an issue and I'll look into it.
Besides the similarity with sar of printing system data in an ad-hoc manner, the dsar utility does not provide the functionality to be called by cron and generate historical reports. If you need that, then sar is what you need.
The specific function of dsar is to have the ability to instantly be able to look at performance figures of one or more servers, without the need to logon, and thus prevent having shell sessions open on multiple machines running sar, iostat, vmstat, etc.
A unique benefit of dsar is that it can create graphs of certain frequently used performance figures, such as CPU usage, memory usage layout, block device performance statistics and network statistics. A graph in certain cases makes evaluating performance and resource usage over a period of time much easier than a long list of numbers.
So how does it work? If you install the dsar utility, it should be available at the CLI. The OSX package contains only the executable, which you should place in /usr/local/bin to make it available. By default, it will use localhost and port 9100, which is the default node_exporter port.
This is what that looks like (on my Mac mini M1--which requires node_exporter to be installed running):
➜ dsar
hostname time CPU %usr %nice %sys %iowait %steal %idle
localhost:9100:metrics 16:15:51 all 14.81 0.00 9.11 0.00 0.00 76.08
localhost:9100:metrics 16:15:52 all 12.31 0.00 9.77 0.00 0.00 77.92
localhost:9100:metrics 16:15:53 all 12.34 0.00 9.32 0.00 0.00 78.34
localhost:9100:metrics 16:15:54 all 11.90 0.00 8.99 0.00 0.00 79.11
(Ctrl-C / interrupt to stop dsar)
This is what the previous article about CPU percentages also showed, so if you read that, you did not see anything new. However, there are many more options to use to show not only CPU usage statistics, but also disk IO, network IO, memory, etc:
| Output option | Tool output format (description) |
| sar-u (default) | sar -u (cpu utilization) |
| sar-u-ALL | sar -u ALL (cpu: adds irq, softirq, guest, guest nice) |
| sar-d | sar -d (block device) |
| sar-S | sar -S (swap usage) |
| sar-W | sar -W (swapping statistics) |
| sar-q | sar -q (queue length and load) |
| sar-B | sar -B (paging statistics) |
| sar-n-DEV | sar -n DEV (network statistics) |
| sar-n-EDEV | sar -n EDEV (network error statistics) |
| sar-n-SOCK | sar -n SOCK (network socket statistics) |
| sar-n-SOCK6 | sar -n SOCK6 (network IPv6 socket statistics) |
| sar-n-SOFT | sar -n SOFT (network software based statistics) |
| sar-r | sar -r (memory statistics) |
| sar-r-ALL | sar -r ALL (memory: adds anonymous, slab, kernelstack, pagetable and vmused) |
| iostat | iostat (I/O statistics for block devices and partitions) |
| iostat-x | iostat -x (I/O statistics extended) |
| xfs-iops | - (shows logical IO statistics, specific to XFS filesystems) |
| mem-relevant | - (shows the memory statistics that commonly are relevant) |
| vmstat | vmstat (virtual memory statistics) |
| psi | - (pressure stall information statistics) |
On top of that, for some of the really common options, dsar can create a graph of the figures, to make visual inspection of certain key performance figures even more easy. This is done of the -g or --graph switch is added:
CPU:
Memory:
Disk IO:
Network:
