[hobbit] iostatcpu & iostatdisk graphs

[Jeff Newman]

I know sar was somewhat challenging to implement here.

First, the lowest interval I could achieve was 2 seconds (but thats
just my craziness) so this would apply even to 5 minute increments.
One thing I would love by the way is still to be able to customize the

One note: I had to change the perms on sar to allow hobbit to run it
(/usr/lib/sa/sa1 and sa2 on AIX)

The first piece was a client script like this:

while read aline; do
CPUNUM=`echo $aline | awk '{print $1}'`
PUSR=`echo $aline | awk '{print $2}'`
PSYS=`echo $aline | awk '{print $3}'`
PWIO=`echo $aline | awk '{print $4}'`
PIDL=`echo $aline | awk '{print $5}'`

echo "cpu"$CPUNUM"pcntusr: $PUSR" >> /tmp/hobbit_sar"$CPUNUM".msg
echo "cpu"$CPUNUM"pcntsys: $PSYS" >> /tmp/hobbit_sar"$CPUNUM".msg
echo "cpu"$CPUNUM"pcntwio: $PWIO" >> /tmp/hobbit_sar"$CPUNUM".msg
echo "cpu"$CPUNUM"pcntidl : $PIDL" >> /tmp/hobbit_sar"$CPUNUM".msg


$BB $BBDISP "status $MACHINE.sar green `date`
`cat /tmp/hobbit_sar"$CPUNUM".msg`
"
rm /tmp/hobbit_sar"$CPUNUM".msg
done < /tmp/hobbit_sar.tmp
rm /tmp/hobbit_sar.tmp


The second piece was the server side script:

if [ "$TESTNAME" = "sar" ] ; then

         NUM=`cat $FNAME | grep cpu | head -1 | cut -c 4`
         if [ $NUM -eq 1 ] ; then
            NEXT=`cat $FNAME | grep cpu | head -1 | cut -c 5`
            if [ $NEXT != "p" ] ; then
               NUM=`cat $FNAME | grep cpu | head -1 | cut -c 4-5`
            fi
         fi

         USR=`grep pcntusr $FNAME | awk -F: '{print $2}'`
         SYS=`grep pcntsys $FNAME | awk -F: '{print $2}'`
         WIO=`grep pcntwio $FNAME | awk -F: '{print $2}'`
         IDL=`grep pcntidl $FNAME | awk -F: '{print $2}'`
         echo "DS:cpu"$NUM"pcntusr:GAUGE:3:0:U"
         echo "DS:cpu"$NUM"pcntsys:GAUGE:3:0:U"
         echo "DS:cpu"$NUM"pcntwio:GAUGE:3:0:U"
         echo "DS:cpu"$NUM"pcntidl:GAUGE:3:0:U"
         if [ $NUM -eq 0 ] ; then
            NUM=00
         fi
         if [ $NUM -eq 1 ] ; then
            NUM=01
         fi
         if [ $NUM -eq 2 ] ; then
            NUM=02
         fi
         if [ $NUM -eq 3 ] ; then
            NUM=03
         fi
         if [ $NUM -eq 4 ] ; then
            NUM=04
         fi
         if [ $NUM -eq 5 ] ; then
            NUM=05
         fi
         if [ $NUM -eq 6 ] ; then
            NUM=06
         fi
         if [ $NUM -eq 7 ] ; then
            NUM=07
         fi
         if [ $NUM -eq 8 ] ; then
            NUM=08
         fi
         if [ $NUM -eq 9 ] ; then
            NUM=09
         fi
         echo "sar,cpu"$NUM".rrd"
         echo "$USR:$SYS:$WIO:$IDL"

fi

The reason for the number conversion had to do with the way
the charts appeared. sar,cpu1.rrd, sar,cpu10.rrd acts differently than
sar,cpu01.rrd and sar,cpu10.

Then i needed a hobbit graph definition:

        FNPATTERN sar,(.*).rrd
        TITLE CPU sar
        YAXIS %
        DEF:usr at RRDIDX@=@RRDFN@:cpu at RRDIDX@pcntusr:AVERAGE
        DEF:sys at RRDIDX@=@RRDFN@:cpu at RRDIDX@pcntsys:AVERAGE
        DEF:wio at RRDIDX@=@RRDFN@:cpu at RRDIDX@pcntwio:AVERAGE
        DEF:idl at RRDIDX@=@RRDFN@:cpu at RRDIDX@pcntidl:AVERAGE
        -u 100
        -l 0
        -r
        -E
        -Y
        AREA:usr at RRDIDX@#0000FF:cpu at RRDIDX@pcntusr
        GPRINT:usr at RRDIDX@:LAST: \: %5.1lf (cur)
        GPRINT:usr at RRDIDX@:MAX: \: %5.1lf (max)
        GPRINT:usr at RRDIDX@:MIN: \: %5.1lf (min)
        GPRINT:usr at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
        STACK:sys at RRDIDX@#FF0000:cpu at RRDIDX@pcntsys
        GPRINT:sys at RRDIDX@:LAST: \: %5.1lf (cur)
        GPRINT:sys at RRDIDX@:MAX: \: %5.1lf (max)
        GPRINT:sys at RRDIDX@:MIN: \: %5.1lf (min)
        GPRINT:sys at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
        STACK:wio at RRDIDX@#800080:cpu at RRDIDX@pcntwio
        GPRINT:wio at RRDIDX@:LAST: \: %5.1lf (cur)
        GPRINT:wio at RRDIDX@:MAX: \: %5.1lf (max)
        GPRINT:wio at RRDIDX@:MIN: \: %5.1lf (min)
        GPRINT:wio at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
        STACK:idl at RRDIDX@#008000:cpu at RRDIDX@pcntidl
        GPRINT:idl at RRDIDX@:LAST: \: %5.1lf (cur)
        GPRINT:idl at RRDIDX@:MAX: \: %5.1lf (max)
        GPRINT:idl at RRDIDX@:MIN: \: %5.1lf (min)
        GPRINT:idl at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n

Then I added "sar" to TEST2RRD in hobbitserver.cfg, as well as: "sar::1"
in the graphs.

Because I wanted things at a certain interval, I hand-created the rrd's for
each server (depending on the number of CPU's):


/usr/local/rrdtool/bin/rrdtool create sar,cpu00.rrd --step 2 \
 DS:cpu0pcntwio:GAUGE:3:0:U \
 DS:cpu0pcntsys:GAUGE:3:0:U \
 DS:cpu0pcntusr:GAUGE:3:0:U \
 DS:cpu0pcntidl:GAUGE:3:0:U \
 RRA:AVERAGE:0.5:1:86400 \
 RRA:AVERAGE:0.5:6:86400 \
 RRA:AVERAGE:0.5:24:86400 \
 RRA:AVERAGE:0.5:288:86400
<etc etc etc... for each cpu>


That's what I had to do to get sar stuff working here. Of course, I probably
did it some wacky way because of the --step I needed.

-Jeff




On 7/18/06, Henrik Stoerner <henrik at hswn.dk> wrote:
> On Mon, Jul 17, 2006 at 09:48:17PM -0500, Craig Cook wrote:
> > I have noticed in 4.2 RC there are iostat commands
> > (hobbitclient-sunos.sh -> [iostatcpu] & [iostatdisk])
> >
> > But there is no definition listed in hobbitgraph.cfg...
> >
> > Is this a "work-in-progress"?
>
> Yes. I added it to the client scripts because that way it would
> be easy to use once I get the server-end adapted to use it.
>
> Doing so will require some new logic in Hobbit to decide which
> of the many sources of load-measurements to use - e.g. vmstat
> or sar data could provide some of those same data. We need to
> rethink how to collect and present these data.
>
> Regards,
> Henrik
>
>
> To unsubscribe from the hobbit list, send an e-mail to
> hobbit-unsubscribe at hswn.dk
>
>
>