urgent rrd help needed - im desperate!

[Jeff Newman]

Ok, i've been working on this for 6+ hours, and am totally stuck.
Here is the script on the client:


====================================

#!/bin/sh

BB=/usr/local/hobbit/client/bin/bb
BBDISP=xxx.xxx.xxx.xxx
MACHINE=xxxxxx

sar -P ALL 1 1 | grep -E "^[0-9]|^(  *)" | grep -v \- | grep -v cpu | cut -c 9->
 /tmp/hobbit_sar.$$ 2>&1; mv /tmp/hobbit_sar.$$ /tmp/hobbit_sar.tmp </dev/null >
/dev/null


while read aline; do
CPUNUM=`echo $aline | awk '{print $1}'`
PUSR=`echo $aline | awk '{print $2}'`
PSYS=`echo $aline | awk '{print $3}'`
PWIO=`echo $aline | awk '{print $4}'`
PIDL=`echo $aline | awk '{print $5}'`

echo "cpu"$CPUNUM"pcntusr : $PUSR" >> /tmp/hobbit_sar"$CPUNUM".msg
echo "cpu"$CPUNUM"pcntsys : $PSYS" >> /tmp/hobbit_sar"$CPUNUM".msg
echo "cpu"$CPUNUM"pcntwio : $PWIO" >> /tmp/hobbit_sar"$CPUNUM".msg
echo "cpu"$CPUNUM"pcntidl : $PIDL" >> /tmp/hobbit_sar"$CPUNUM".msg

$BB $BBDISP "status $MACHINE.sar,"$CPUNUM" green `date`
`cat /tmp/hobbit_sar"$CPUNUM".msg`
"
rm /tmp/hobbit_sar"$CPUNUM".msg
done < /tmp/hobbit_sar.tmp
rm /tmp/hobbit_sar.tmp

========================================

It sends the data just fine. (output from sh -x)
+ BB=/usr/local/hobbit/client/bin/bb
+ BBDISP=167.76.113.220
+ MACHINE=stlfan3
+ sar -P ALL 1 1
+ grep -E ^[0-9]|^(  *)
+ grep -v -
+ grep -v cpu
+ cut -c 9-
+ 1> /tmp/hobbit_sar.20750 2>& 1
+ mv /tmp/hobbit_sar.20750 /tmp/hobbit_sar.tmp
+ 0< /dev/null 1> /dev/null
+ 0< /tmp/hobbit_sar.tmp
+ read aline
+ + awk {print $1}
+ echo 0 2 8 1 89
CPUNUM=0
+ + awk {print $2}
+ echo 0 2 8 1 89
PUSR=2
+ + awk {print $3}
+ echo 0 2 8 1 89
PSYS=8
+ + awk {print $4}
+ echo 0 2 8 1 89
PWIO=1
+ + awk {print $5}
+ echo 0 2 8 1 89
PIDL=89
+ echo cpu0pcntusr : 2
+ 1>> /tmp/hobbit_sar0.msg
+ echo cpu0pcntsys : 8
+ 1>> /tmp/hobbit_sar0.msg
+ echo cpu0pcntwio : 1
+ 1>> /tmp/hobbit_sar0.msg
+ echo cpu0pcntidl : 89
+ 1>> /tmp/hobbit_sar0.msg
+ date
+ cat /tmp/hobbit_sar0.msg

+ /usr/local/hobbit/client/bin/bb --debug 167.76.113.220 status stlfan3.sar,0 gr
een Fri Mar 17 11:58:59 EST 2006

cpu0pcntusr : 2
cpu0pcntsys : 8
cpu0pcntwio : 1
cpu0pcntidl : 89

2006-03-17 11:58:59 Transport setup is:
2006-03-17 11:58:59 bbdportnumber = 1984
2006-03-17 11:58:59 bbdispproxyhost = NONE
2006-03-17 11:58:59 bbdispproxyport = 0
2006-03-17 11:58:59 Recipient listed as 'xxx.xx.xxx.xxx'
2006-03-17 11:58:59 Standard BB protocol on port 1984
2006-03-17 11:58:59 Will connect to address xxx.xx.xxx.xxx port 1984
2006-03-17 11:58:59 Connect status is 0
2006-03-17 11:58:59 Sent 121 bytes
2006-03-17 11:58:59 Closing connection
+ rm /tmp/hobbit_sar0.msg
+ read aline

<and so on, incrementing cpu numbers as expected.>

On the hobbit server, I want this to work like "disk" where there
are multiple file systems under one disk column. I manually created
the RRD's (again for a custom time step)

-rw-r--r--   1 hobbit hobbit 22121176 Mar 17 10:31 sar,0.rrd
-rw-r--r--   1 hobbit hobbit 22121176 Mar 17 10:31 sar,1.rrd
-rw-r--r--   1 hobbit hobbit 22121176 Mar 17 10:31 sar,2.rrd
-rw-r--r--   1 hobbit hobbit 22121176 Mar 17 10:31 sar,3.rrd

The DS names in the rrd dump look fine:
       <ds>
                <name> cpu0pcntusr </name>
                <type> GAUGE </type>
for example.

Note this all doesn't work if the files are just "sar0.rrd, sar1rrd
etc..." without the ,'s

Unfortunately, on the web page, it gives me 3 columns, a sar,0 sar,1 sar,2
Which is nitpicky, but if I can just have a "sar" column with the
others under it
would be great (like the disk problem). Here is the REAL problem.

Looking at the sar,0 button for example, I see the data update there,
HOWEVER, NONE of the /usr/local/hobbit/data/rrd/xxxx/sar,#.rrd files
ever get updated!!!
In addition, there isn't even a link for a graph in the page.

No errors in /var/log/hobbit/rrd*, or any others that I have looked at
on the client or server. Here are the .cfg files. I have tried many
variations, these are just how they are now:

hobbitserver.cfg - here are the lines that I have "sar" in:


TEST2RRD="cpu=la,disk,inode,qtree,memory,$PINGCOLUMN=tcp,http=tcp,dns=tcp,dig=tcp,time=ntpstat,vmstat,iostat,netstat,temperature,apache,bind,sendmail,mailq,nmailq=mailq,sar,socks,bea,iishealth,citrix,bbgen,bbtest,bbproxy,hobbitd,HiFlowNet="ncv",sock="ncv",qmeter="ncv",rtt="ncv"

Note, I have tried sar="ncv", sar,sar0,sar1, etc.. maybe I havn't tried
the right variation :-(

GRAPHS="la,disk,inode,qtree,memory,users,vmstat,iostat,tcp.http,tcp,netstat,mrtg::1,temperature,ntpstat,apache,bind,sendmail,mailq,socks,bea,iishealth,citrix,bbgen,bbtest,bbproxy,hobbitd,ncv,HiFlowNet,sock,rtt,sar,sar0,sar1,sar2,sar3"

again, tried many variations.. and also again, maybe I havn't tried
the right one.

bb-hosts:
only tried putting "xxx.xxx.xxx.xxx  xxx   # conn sar
that didn't help


hobbitgraph.cfg:

[sar]
        FNPATTERN sar(.*).rrd
        TITLE CPU sar
        YAXIS %
        DEF:p at RRDIDX@=@RRDFN@:cpu at RRDIDX@pcntusr:AVERAGE
        DEF:p at RRDIDX@=@RRDFN@:cpu at RRDIDX@pcntsys:AVERAGE
        DEF:p at RRDIDX@=@RRDFN@:cpu at RRDIDX@pcntwio:AVERAGE
        DEF:p at RRDIDX@=@RRDFN@:cpu at RRDIDX@pcntidl:AVERAGE
        LINE2:p at RRDIDX@#@COLOR@:@RRDPARAM@
        -u 100
        -l 0
        GPRINT:p at RRDIDX@:LAST: \: %5.1lf (cur)
        GPRINT:p at RRDIDX@:MAX: \: %5.1lf (max)
        GPRINT:p at RRDIDX@:MIN: \: %5.1lf (min)
        GPRINT:p at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n

If anyone has anything else they need to see, let me know.
I need to get this working quickly, and am at the end of my rope!
I've done other custom graph's with custom RRD's, and never had this
problem before.

By the way, the host is sending other custom data just fine with no problems.

Thanks for any help!

-Jeff