Performance plotting with atop and Gnuplot
August 7, 2009
linux
A customer wanted some performance figures for a ESX cluster we built. We used our own atop and the trusted workhorse of plotting Gnuplot to make some performance plots.
Creating the data
As said, we used atop
for this, it has a nice data gathering
mode which prints out lines like these:
13:40:07 cpu %usr %nice %sys %irq %softirq %steal %wait %idle
13:40:08 all 0 0 0 0 1 0 4 95
13:40:09 all 0 0 1 0 0 0 0 99
We were interested in the values of %usr
, %sys
, %irq
+ %softirq
,
%wait
and %idle
.
Normalizing the data
To matter slightly easier we decided to normalize the data a bit;
- convert the time stamp to Unix epoch
- kill the first 5 lines of output as this is a header
In short, the following tiny Perl program was used for the conversion.
#!/usr/bin/perl -w
use Time::Local;
use strict;
<>;<>;<>;<>;<>;<>; # kill first 5 lines
my @l;
while(<>) {
chomp;
if (/^#/) { print; next }
@l = split;
my ($h, $min, $sec) = split /:/, $l[0];
shift @l;
$h =~ s/^0//; # don't make it look octal
$min =~ s/^0//;
$sec =~ s/^0//;
my $time = timelocal($sec, $min, $h, (localtime)[3,4,5]);
print "$time @l", "\n"; # show the normalized line
}
Generating the plot
This was actually the most difficult step as Gnuplot is such a complicated program - you can do a lot of plotting with it. So it was a matter of getting it to plot exactly as we needed it to plot.
We wanted to plot the data as a histogram were each bar in the plot
would show the percentage of time spend in the different statuses
(usr
, irq
, sys
and idle
). Gnuplot calls this a rowstacked
histogram. With this knowledge we came to the following Gnuplot
script (script.gnuplot
):
set yrange [0:100]
set xrange [0:]
set ylabel "% of total"
set xlabel "seconds"
set style data histograms
set style histogram rowstacked
set style fill solid
set key outside
set terminal jpeg
# output to /tmp/a.jpg
set output "/tmp/a.jpg"
# read the data from /tmp/a
plot '/tmp/a' u 3 t '%usr' lt rgb 'red', \
'' u 5 t '%sys' lt rgb 'black', \
'' u 6 t '%irq' lt rgb 'gray', '' u 7 t '%softirq' lt rgb 'gray', \
'' u 9 t '%wait' lt rgb 'yellow' , '' u 10 t '%idle' lt rgb 'blue'
Gnuplot would read its data from /tmp/a
and generate a jpg
named
/tmp/a.jpg
.
The full Monty
After gathering the performance data, making the plots was as easy as:
./normalize < performance.data > /tmp/a && \
gnuplot gnuplot.script && \
rm /tmp/a
Then /tmp/a.jpg
could be renamed to something more suitable.