Thursday, May 20, 2010

column.sh - columizator

It has been a long time since I've last written on this blog. I am indeed very busy by my work so I don't have much time to write. Nonetheless a positive aspect of this is that I happen to write some tools for me to alleviate some tasks.

I am pretty sure that many if not all of you have already been annoyed by the output format such as vmstat(8), iostat(8), ... They are great commands because they produce very valuable information but they are often very difficult to read, especially on busy servers when you need them most, because of the misalignment. We cannot blame them because it is the Unix way of doing thing: do one thing and do it well. It's not their job to pretty print the output. The "column" script exposed here will realign the output for you.

This command acts like a standard Unix command: it may be used alone or as a filter. It takes one mandatory argument, namely a keyword that will be used to recognize the «caption» line.


jlehen@warg:~$ ./column -h
Pretty-print columns, for iostat or vmstat.
OS: All.

Usage: column [-g]

Keyword is used to identify the caption line. This may be an
awk/nawk regex (thus don't forget to escape "/").
Options:
-g Change the output to what I called "giant mode".
This is useful for iostat with an awful lot of disks.


Let's take an example. Here is a classical vmstat(8) output on a busy Solaris server:

05:26:32 kthr memory page disk faults cpu
05:26:32 r b w swap free re mf pi po fr de sr m0 m1 m2 m1 in sy cs us sy id
05:26:32 0 29 0 139019936 26258128 4776 17739 38816 8 0 0 0 4 4 4 0 29969 129465 33184 21 25 54
05:26:33 0 40 0 138993232 26235120 5240 19492 47250 39 31 0 0 7 7 7 0 30665 133160 34489 23 25 51
05:26:34 0 28 0 139002648 26230848 4878 18717 36828 16 16 0 0 3 3 3 0 31368 144148 35850 22 24 54
05:26:35 1 26 0 138978136 26211456 4046 14760 30778 8 8 0 0 4 4 4 0 26295 158433 27845 20 20 61
05:26:36 0 29 0 138980472 26212968 4469 15525 33361 23 16 0 0 4 4 4 0 25826 118810 27653 21 19 60
05:26:37 0 22 0 139004424 26227824 4208 16734 29367 8 8 0 0 4 4 4 0 24879 122903 28154 23 19 58
05:26:38 0 18 0 139010608 26232864 3958 15324 27984 0 0 0 0 2 2 2 0 22422 105347 23563 17 17 66
05:26:39 0 25 0 139022192 26237584 4076 15124 27770 31 31 0 0 2 2 2 0 25046 115101 26467 17 17 65
05:26:40 0 21 0 139035416 26248720 5205 15215 38873 0 0 0 0 1 1 1 0 26961 131668 28818 22 18 60
05:26:41 0 17 0 139014056 26239552 5132 17257 31959 0 0 0 0 0 0 0 0 23780 116586 25043 18 19 63
05:26:42 0 13 0 139028280 26250352 3661 17215 17719 23 16 0 0 3 3 3 0 22048 110885 23717 15 17 67
05:26:43 0 11 0 138993472 26220968 3796 19241 16468 0 0 0 0 0 0 0 0 21640 115744 23173 18 15 66
05:26:44 0 12 0 138972904 26199192 2607 14008 15327 8 8 0 0 2 2 2 0 20428 120489 21064 18 15 67
05:26:45 0 11 0 138973008 26194376 2122 9228 21017 23 23 0 0 5 5 5 0 24543 123981 25423 20 16 64
05:26:46 2 12 0 138977112 26194960 1929 12010 23333 0 0 0 0 1 1 1 0 32145 144927 34596 26 16 59
05:26:47 1 21 0 139032632 26240256 2361 11192 32759 16 16 0 0 2 2 2 0 36018 204602 39325 27 21 51
05:26:49 1 25 0 139043840 26245952 2628 9895 35478 24 24 0 0 3 3 3 0 32645 136756 34903 26 20 54
05:26:50 0 23 0 139043552 26246304 1942 9952 21395 0 0 0 0 0 0 0 0 24948 121252 25918 21 14 66
05:26:51 0 19 0 139053752 26249176 1949 7929 26141 0 0 0 0 6 7 7 0 24490 113321 26220 19 14 67


This is difficult to read because of the large amount of memory and swap. Let's look at the same output now filtered through column (keyword used to match caption line is "swap"):

05:26:32 kthr memory page disk faults cpu
05:26:32 r b w swap free re mf pi po fr de sr m0 m1 m2 m1 in sy cs us sy id
05:26:32 0 29 0 139019936 26258128 4776 17739 38816 8 0 0 0 4 4 4 0 29969 129465 33184 21 25 54
05:26:33 0 40 0 138993232 26235120 5240 19492 47250 39 31 0 0 7 7 7 0 30665 133160 34489 23 25 51
05:26:34 0 28 0 139002648 26230848 4878 18717 36828 16 16 0 0 3 3 3 0 31368 144148 35850 22 24 54
05:26:35 1 26 0 138978136 26211456 4046 14760 30778 8 8 0 0 4 4 4 0 26295 158433 27845 20 20 61
05:26:36 0 29 0 138980472 26212968 4469 15525 33361 23 16 0 0 4 4 4 0 25826 118810 27653 21 19 60
05:26:37 0 22 0 139004424 26227824 4208 16734 29367 8 8 0 0 4 4 4 0 24879 122903 28154 23 19 58
05:26:38 0 18 0 139010608 26232864 3958 15324 27984 0 0 0 0 2 2 2 0 22422 105347 23563 17 17 66
05:26:39 0 25 0 139022192 26237584 4076 15124 27770 31 31 0 0 2 2 2 0 25046 115101 26467 17 17 65
05:26:40 0 21 0 139035416 26248720 5205 15215 38873 0 0 0 0 1 1 1 0 26961 131668 28818 22 18 60
05:26:41 0 17 0 139014056 26239552 5132 17257 31959 0 0 0 0 0 0 0 0 23780 116586 25043 18 19 63
05:26:42 0 13 0 139028280 26250352 3661 17215 17719 23 16 0 0 3 3 3 0 22048 110885 23717 15 17 67
05:26:43 0 11 0 138993472 26220968 3796 19241 16468 0 0 0 0 0 0 0 0 21640 115744 23173 18 15 66
05:26:44 0 12 0 138972904 26199192 2607 14008 15327 8 8 0 0 2 2 2 0 20428 120489 21064 18 15 67
05:26:45 0 11 0 138973008 26194376 2122 9228 21017 23 23 0 0 5 5 5 0 24543 123981 25423 20 16 64
05:26:46 2 12 0 138977112 26194960 1929 12010 23333 0 0 0 0 1 1 1 0 32145 144927 34596 26 16 59
05:26:47 1 21 0 139032632 26240256 2361 11192 32759 16 16 0 0 2 2 2 0 36018 204602 39325 27 21 51
05:26:49 1 25 0 139043840 26245952 2628 9895 35478 24 24 0 0 3 3 3 0 32645 136756 34903 26 20 54
05:26:50 0 23 0 139043552 26246304 1942 9952 21395 0 0 0 0 0 0 0 0 24948 121252 25918 21 14 66
05:26:51 0 19 0 139053752 26249176 1949 7929 26141 0 0 0 0 6 7 7 0 24490 113321 26220 19 14 67



The -g option is meant to be used when the output is giant. For instance, when a system has a lot of disks, the iostat(8) command prints so much lines that the caption line is swept out of your terminal immediately..

Let's take a sample output of iostat(8). I won't show a lot of disks because it is worthless, but keep in mind than when there are tenths or hundreds of disks, the caption line no longer appear on the screen:

extended device statistics
device r/s w/s kr/s kw/s wait actv svc_t %w %b
md0 5.0 8.1 36.3 155.5 0.0 0.1 6.7 0 6
md1 3.0 8.1 20.2 155.5 0.0 0.1 7.3 0 6
md2 2.0 8.1 16.2 155.5 0.0 0.1 5.1 0 4
md10 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
md11 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
md12 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
md50 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
md51 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
md52 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
md60 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
md61 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
md62 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ramdisk1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
sd0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
[...]


Filtering with column -g actv (using "device" would be useless as the very first line also contains it):

extended device statistics
device:md0 r/s:5.0 w/s:8.1 kr/s:36.3 kw/s:155.5 wait:0.0 actv:0.1 svc_t:6.7 %w:0 %b:6
device:md1 r/s:3.0 w/s:8.1 kr/s:20.2 kw/s:155.5 wait:0.0 actv:0.1 svc_t:7.3 %w:0 %b:6
device:md2 r/s:2.0 w/s:8.1 kr/s:16.2 kw/s:155.5 wait:0.0 actv:0.1 svc_t:5.1 %w:0 %b:4
device:md10 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
device:md11 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
device:md12 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
device:md50 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
device:md51 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
device:md52 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
device:md60 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
device:md61 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
device:md62 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
device:ramdisk1 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
device:sd0 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
[...]


This nice script is available here.
Beware that is used /bin/sh. If you want to use it on Solaris, use /bin/ksh instead. I have yet to find a shebang line that will work on *BSD, Linux and Solaris. If you know one, contact me on < jeremie le-hen org > (up to you to but the "@" and "." where you think it best fits...).

No comments: