<font size=2 face="sans-serif">Ok so after that convoluted message, I
ended up having much more success after hacking at it some more.</font>
<br>
<br><font size=2 face="sans-serif">If anyone is interested in the fix,
or the believed answers to my questions below, let me know and I'll document
my trials.</font>
<br>
<br><font size=2 face="sans-serif">Regards,</font>
<br><font size=2 face="sans-serif">Matt.</font>
<br>
<br><font size=2 face="sans-serif">Unix System Administrator<br>
Computer Science Corporation<br>
General Dynamics Land Systems<br>
38500 Mound Rd.<br>
Sterling Heights, MI. 48310<br>
Desk: (586) 825-8294<br>
Oracle IM: moldvanm<br>
<br>
This is a PRIVATE message. If you are not the intended recipient, please
delete without copying and kindly advise us by e-mail of the mistake in
delivery. <br>
NOTE: Regardless of content, this e-mail shall not operate to bind CSC
to any order or other contract unless pursuant to explicit written agreement
or government initiative expressly permitting the use of e-mail for such
purpose.</font>
<br>
<br>
<br>
<table width=100%>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">From:</font>
<td><font size=1 face="sans-serif">Matthew Moldvan/GIS/CSC@CSC</font>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">To:</font>
<td><font size=1 face="sans-serif">xymon@xymon.com</font>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">Date:</font>
<td><font size=1 face="sans-serif">08/19/2010 04:00 PM</font>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">Subject:</font>
<td><font size=1 face="sans-serif">[xymon] Adding a custom RRD to graphs
and monitoring</font></table>
<br>
<hr noshade>
<br>
<br>
<br><font size=2 face="sans-serif">All,</font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
This is stressing me out, hopefully someone takes the time to go through
my ramblings below and help me out. Lots of information so please
bear with me.</font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
For the past few days I've been trying to add a custom script (iostat information)
and have the data graphed, but I'm not having any luck (mostly due to not
understanding the RRD definitions in hobbitgraph.cfg).</font><font size=3>
<br>
</font><font size=2 face="sans-serif"><br>
I've read through a ton of the how-tos on the subject, but all of them
seem to vary a bit on the details. My resulting graphs look like
this: </font><a href=http://imgur.com/4Nwrp.jpg><font size=2 color=blue face="sans-serif">http://imgur.com/4Nwrp.jpg</font></a><font size=3>
<br>
</font><font size=2 face="sans-serif"><br>
So far I've got a script running on two systems reporting data back to
the main page. This brings up my first question: When sending
information to be graphed, is the data passed in as a bb status message
or a bb data message? </font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
I thought I had it working at one point by sending similar data below through
a status message, but I'd like to pass only a status message and HTML through
the "bb status" command and keep the actual data passed for the
RRD in the "bb data" command if that works. I also tried
wrapping the data below in HTML comments as below, but no luck.</font><font size=3>
<br>
</font><font size=2 face="sans-serif"><br>
"<!---</font><font size=3> </font><font size=2 face="sans-serif"><br>
data like below (note newlines between HTML comment tags)</font><font size=3>
</font><font size=2 face="sans-serif"><br>
--->"</font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
Sample data:</font><font size=3> </font><font size=2 face="sans-serif"><br>
c1t50060E80104AAE50d1 : 0.82
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c1t50060E80104AAE50d2 : 0.07
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c1t50060E80104AAE50d3 : 1.71
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c1t50060E80104AAE50d4 : 0.46
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c3t50060E80104AAE52d0 : 1.31
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c3t50060E80104AAE52d1 : 1.53
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c3t50060E80104AAE52d2 : 0.09
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c3t50060E80104AAE52d3 : 3.14
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c3t50060E80104AAE52d4 : 0.61
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c3t50060E80104AAE52d12 : 0.06
<br>
c0t0d0 : 11.70
</font><font size=3>
</font><font size=2 face="sans-serif"><br>
c1t50060E80104AAE50d0 : 0.87 </font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
I've seen it both ways in the examples. I tried sending both, but
that doesn't seem to be working. From what I understand if I specify
a test as NCV in the TEST2RRD section, one of the running processes (hobbitd
or hobbitrrd) will read in the "name : value" pair and pass that
to an RRD update/create command? Does that require integer values
or are floating point up to a certain precision acceptable? Currently
I'm passing .2f from the nawk script and getting a bunch of "nans"
in the RRD output (could be various reasons, though).</font><font size=3>
<br>
</font><font size=2 face="sans-serif"><br>
Here go the details (NOTE: All host names and IP addresses have been scrubbed
to protect the innocent):</font><font size=3> <br>
</font><font size=2 face="sans-serif"><b><br>
Script output:</b></font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
+ /opt/xymon/client/bin/bb <xymon.server.ip> 'data <client.fqdn>.trends</font><font size=3>
</font><font size=2 face="sans-serif"><br>
c1t50060E80104AAE50d1 : 0.82
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c1t50060E80104AAE50d2 : 0.07
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c1t50060E80104AAE50d3 : 1.71
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c1t50060E80104AAE50d4 : 0.46
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c3t50060E80104AAE52d0 : 1.31
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c3t50060E80104AAE52d1 : 1.53
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c3t50060E80104AAE52d2 : 0.09
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c3t50060E80104AAE52d3 : 3.14
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c3t50060E80104AAE52d4 : 0.61
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c3t50060E80104AAE52d12 : 0.06
<br>
c0t0d0 : 11.70
</font><font size=3>
</font><font size=2 face="sans-serif"><br>
c1t50060E80104AAE50d0 : 0.87
</font><font size=3> </font><font size=2 face="sans-serif"><br>
'
</font><font size=3>
</font><font size=2 face="sans-serif"><br>
+ /opt/xymon/client/bin/bb <xymon.server.ip> 'status <client.fqdn>.iostat
green Thu Aug 19 10:47:28 EDT 2010
<br>
c1t50060E80104AAE50d1 : 0.82
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c1t50060E80104AAE50d2 : 0.07
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c1t50060E80104AAE50d3 : 1.71
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c1t50060E80104AAE50d4 : 0.46
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c3t50060E80104AAE52d0 : 1.31
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c3t50060E80104AAE52d1 : 1.53
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c3t50060E80104AAE52d2 : 0.09
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c3t50060E80104AAE52d3 : 3.14
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c3t50060E80104AAE52d4 : 0.61
</font><font size=3> </font><font size=2 face="sans-serif"><br>
c3t50060E80104AAE52d12 : 0.06
<br>
c0t0d0 : 11.70
</font><font size=3>
</font><font size=2 face="sans-serif"><br>
c1t50060E80104AAE50d0 : 0.87
</font><font size=3> </font><font size=2 face="sans-serif"><br>
'
</font><font size=3>
<br>
</font><font size=2 face="sans-serif"><b><br>
Another question: </b>I've seen some examples sending as "bb data
<client.fqdn>.trends", is that correct, or if I'm using the
"bb data" command do I have to specify the test name as above?</font><font size=3>
<br>
</font><font size=2 face="sans-serif"><b><br>
The RRD files are thus being created for every disk as such:</b></font><font size=3>
<br>
</font><font size=2 face="sans-serif"><br>
-rw-r--r-- 1 xymon 495 19648 Aug 19 11:06 iostat,c0t0d0.rrd</font><font size=3>
</font><font size=2 face="sans-serif"><br>
-rw-r--r-- 1 xymon 495 19648 Aug 18 23:22 iostat,c0t1d0.rrd</font><font size=3>
</font><font size=2 face="sans-serif"><br>
-rw-r--r-- 1 xymon 495 19648 Aug 19 11:06 iostat,c1t50060E80104AAE50d0.rrd</font><font size=3>
</font><font size=2 face="sans-serif"><br>
...snip...</font><font size=3> </font><font size=2 face="sans-serif"><br>
-rw-r--r-- 1 xymon 495 19648 Aug 18 23:22 iostat,c3t50060E80104AAE52d8.rrd</font><font size=3>
</font><font size=2 face="sans-serif"><br>
-rw-r--r-- 1 xymon 495 19648 Aug 18 23:22 iostat,c3t50060E80104AAE52d9.rrd</font><font size=3>
<br>
</font><font size=2 face="sans-serif"><b><br>
An rrdtool dump <whatever>.rrd does confirm that some values are
making it into the RRDs (assuming so by "last_ds" in dump output
below):</b></font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
[root@<hostname> <fqdn.rrd.dir>]# rrdtool dump iostat,c3t50060E80104AAE52d9.rrd
| more</font><font size=3> </font><font size=2 face="sans-serif"><br>
<?xml version="1.0" encoding="utf-8"?></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<!DOCTYPE rrd SYSTEM "</font><a href=http://oss.oetiker.ch/rrdtool/rrdtool.dtd><font size=2 color=blue face="sans-serif"><u>http://oss.oetiker.ch/rrdtool/rrdtool.dtd</u></font></a><font size=2 face="sans-serif">"></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<!-- Round Robin Database Dump --></font><font size=3> </font><font size=2 face="sans-serif"><br>
<rrd></font><font size=3> </font><font size=2 face="sans-serif"><br>
<version>0003</version></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<step>300</step> <!-- Seconds
--></font><font size=3> </font><font size=2 face="sans-serif"><br>
<lastupdate>1282188121</lastupdate>
<!-- 2010-08-18 23:22:01 EDT --></font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
<ds></font><font size=3> </font><font size=2 face="sans-serif"><br>
<name> lambda
</name></font><font size=3> </font><font size=2 face="sans-serif"><br>
<type> GAUGE
</type></font><font size=3> </font><font size=2 face="sans-serif"><br>
<minimal_heartbeat>0</minimal_heartbeat></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<min>6.0000000000e+02</min></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<max>NaN</max></font><font size=3>
<br>
</font><font size=2 face="sans-serif"><br>
<!-- PDP Status
--></font><font size=3> </font><font size=2 face="sans-serif"><b><br>
<last_ds>2.05</last_ds></b></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<value>NaN</value></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<unknown_sec>
121 </unknown_sec></font><font size=3> </font><font size=2 face="sans-serif"><br>
</ds></font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
<!-- Round Robin Archives --></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<rra></font><font size=3> </font><font size=2 face="sans-serif"><br>
<cf>AVERAGE</cf></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<pdp_per_row>1</pdp_per_row>
<!-- 300 seconds --></font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
<params></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<xff>5.0000000000e-01</xff></font><font size=3>
</font><font size=2 face="sans-serif"><br>
</params></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<cdp_prep></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<ds></font><font size=3> </font><font size=2 face="sans-serif"><br>
<primary_value>NaN</primary_value></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<secondary_value>0.0000000000e+00</secondary_value></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<value>NaN</value></font><font size=3> </font><font size=2 face="sans-serif"><br>
<unknown_datapoints>0</unknown_datapoints></font><font size=3>
</font><font size=2 face="sans-serif"><br>
</ds></font><font size=3> </font><font size=2 face="sans-serif"><br>
</cdp_prep></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<database></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<!-- 2010-08-16 23:25:00 EDT / 1282015500 --> <row><v>NaN</v></row></font><font size=3>
</font><font size=2 face="sans-serif"><br>
...snip, all others are NaN also...</font><font size=3> </font><font size=2 face="sans-serif"><br>
<!-- 2010-08-18 23:20:00 EDT / 1282188000 --> <row><v>NaN</v></row></font><font size=3>
</font><font size=2 face="sans-serif"><br>
</database></font><font size=3>
</font><font size=2 face="sans-serif"><br>
</rra></font><font size=3> </font><font size=2 face="sans-serif"><br>
<rra></font><font size=3> </font><font size=2 face="sans-serif"><br>
<cf>AVERAGE</cf></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<pdp_per_row>6</pdp_per_row>
<!-- 1800 seconds --></font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
<params></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<xff>5.0000000000e-01</xff></font><font size=3>
</font><font size=2 face="sans-serif"><br>
</params></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<cdp_prep></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<ds></font><font size=3> </font><font size=2 face="sans-serif"><br>
<primary_value>0.0000000000e+00</primary_value></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<secondary_value>0.0000000000e+00</secondary_value></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<value>NaN</value></font><font size=3> </font><font size=2 face="sans-serif"><br>
<unknown_datapoints>4</unknown_datapoints></font><font size=3>
</font><font size=2 face="sans-serif"><br>
</ds></font><font size=3> </font><font size=2 face="sans-serif"><br>
</cdp_prep></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<database></font><font size=3>
</font><font size=2 face="sans-serif"><br>
<!-- 2010-08-06 23:30:00 EDT / 1281151800 --> <row><v>NaN</v></row></font><font size=3>
</font><font size=2 face="sans-serif"><br>
...snip, all NaNs til the end...</font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
Relevant lines from /etc/xymon/hobbitserver.cfg:</font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
[root@<hostname> ~]# egrep 'TEST2RRD|GRAPHS' /etc/xymon/hobbitserver.cfg
<br>
# TEST2RRD defines the status- and data-messages you want to collect RRD
data</font><font size=3> </font><font size=2 face="sans-serif"><br>
TEST2RRD="cpu=la,disk,inode,qtree,memory,$PINGCOLUMN=tcp,http=tcp,dns=tcp,dig=tcp,time=ntpstat,vmstat,vmio=ncv,<b>iostat=ncv</b>,netstat,temperature,apache,bind,sendmail,mailq,nmailq=mailq,socks,bea,iishealth,citrix,bbgen,bbtest,bbproxy,hobbitd,files,procs=processes,ports,clock,lines,ops,stats,cifs,JVM,JMS,HitCache,Session,JDBCConn,ExecQueue,JTA,TblSpace,RollBack,MemReq,InvObj,snapmirr,snaplist,snapshot"</font><font size=3>
</font><font size=2 face="sans-serif"><br>
GRAPHS="la,disk,inode,qtree,files,processes,memory,users,vmstat:vmstat0|vmstat1|vmstat2|vmstat3|vmstat4|vmstat5|vmstat6|vmstat7|vmstat8|vmstat9,<b>iostat</b>,vmio,tcp.http,tcp,netstat,ifstat,mrtg::1,ports,temperature,ntpstat,apache,bind,sendmail,mailq,socks,bea,iishealth,citrix,bbgen,bbtest,bbproxy,hobbitd,clock,lines,ops,stats,cifs,JVM,JMS,HitCache,Session,JDBCConn,ExecQueue,JTA,TblSpace,RollBack,MemReq,InvObj,snapmirr,snaplist,snapshot,devmon::1,if_load::1,temp,<b>ncv</b>"
<br>
- (a tip from the web said "ncv" had to be in the GRAPHS portion
and said "not sure why just trust me" ...)</font><font size=3>
<br>
</font><font size=2 face="sans-serif"><br>
Relevant lines from /etc/xymon/hobbitgraph.cfg:</font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
[iostat]</font><font size=3> </font><font size=2 face="sans-serif"><br>
TITLE I/O Utilization - Overall</font><font size=3>
</font><font size=2 face="sans-serif"><br>
FNPATTERN iostat(.*).rrd</font><font size=3>
</font><font size=2 face="sans-serif"><br>
YAXIS Stats</font><font size=3> </font><font size=2 face="sans-serif"><br>
DEF:p@RRDIDX@=@RRDFN@:lambda:AVERAGE</font><font size=3>
</font><font size=2 face="sans-serif"><br>
LINE1.5:p@RRDIDX@#@COLOR@:@RRDPARAM@</font><font size=3>
</font><font size=2 face="sans-serif"><br>
GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n</font><font size=3>
<br>
</font><font size=2 face="sans-serif"><br>
Anyone know of a link that explains some of the terminology above? I
checked the rrdcreate man page, but didn't see the parts about "@RRDIDX@"
and @RRDFN@" and the other stuff. p@RRDIDX@ seems to be in a
lot of examples I've seen, and all my data is making it in with those variables
(is that what they are?) without having multiple DEF statements.</font><font size=3>
<br>
</font><font size=2 face="sans-serif"><br>
The above is generating the image I included a link to above (</font><a href=http://imgur.com/4Nwrp.jpg><font size=2 color=blue face="sans-serif">http://imgur.com/4Nwrp.jpg</font></a><font size=2 face="sans-serif">).</font><font size=3>
<br>
</font><font size=2 face="sans-serif"><br>
Thanks again to anyone that can help out ... I've been pulling my hair
out about this for a few days.</font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
Regards,</font><font size=3> </font><font size=2 face="sans-serif"><br>
Matt.</font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
Unix System Administrator<br>
Computer Science Corporation<br>
<br>
This is a PRIVATE message. If you are not the intended recipient, please
delete without copying and kindly advise us by e-mail of the mistake in
delivery. <br>
NOTE: Regardless of content, this e-mail shall not operate to bind CSC
to any order or other contract unless pursuant to explicit written agreement
or government initiative expressly permitting the use of e-mail for such
purpose.</font><font size=3> This is an e-mail from General Dynamics Land
Systems. It is for the intended recipient only and may contain confidential
and privileged information. No one else may read, print, store, copy, forward
or act in reliance on it or its attachments. If you are not the intended
recipient, please return this message to the sender and delete the message
and any attachments from your computer. Your cooperation is appreciated.
</font>
<br>
<br>