[hobbit] Sending data from script to an rrd

[Henrik Stoerner]

On Wed, Oct 18, 2006 at 04:54:44PM -0400, Rob Munsch wrote:

> I still do not quite get how the actual data gets to rrd.  It seems to
> me that there is an isolated client script that sends data which
> determines the status color.. then a DIFFERENT server-side script is
> used to send the SAME DATA to RRD for graphing.  ?!?! yesno?
> 
> Why isn't there a way to use the existing data that the client's sending
> for graphing... for CPU, MEM, etc. isn't that already happening?  or
> have i failed another concept here.

Writing clear documentation is difficult when you've designed all of
this and know how everything fits together. So blame any missing or
misleading docs on me :-)

The first thing to get your head around is that Hobbit has a very clear
separation between *collecting* data, *analyzing* data, and *presenting*
data.

Collecting data is done on the client; that's your script running
ipmitool (or whatever you use to collect data) and sends it to the
Hobbit server.

When the Hobbit server - specifically, the "hobbitd" process that
receives messages from the network - gets your data via a "status" 
message, it creates a column with the color your script decided upon. 
But it doesn't do any analysis of the data. Instead, it just shoves the
raw message out through one or more of the Hobbit "channels", and lets
the hobbitd_* worker modules do whatever they want to do with it.
One of these worker modules is "hobbitd_rrd", which (as the name
implies) handles updating of RRD files. It has to pick out the
interesting numbers from the raw status message, and put them into an
RRD file. hobbitd_rrd knows how to handle several different kinds of 
status messages (cpu, disk, memory, network response times ...), but for
custom tests that you write yourself, you'll have to provide the code
to pick out those numbers you want to put into the graph. So this is
where the second script comes into play: This script is the one with the
"echo" statements that you cannot quite figure out how should work.
Basically, whenever hobbitd_rrd sees a status message which is one of
those listed in the "--extra-tests" option, then it hands over the
parsing of the status message data to a script you've written. This
script must then return the data that should go into the RRD file.

So this is the *analyzing* part, and it runs on the Hobbit server. It is
done by your second script, which gets invoked by the hobbitd_rrd
process.

The final part is the *presentation* of the data. Hobbit knows how to
present the status message itself, so the only thing that is missing is
the graph that should be created based on the data in the RRD file.
For this, you must first create a definition in hobbitgraph.cfg,
describing what RRD file your graph uses, which datasets should go on
the graph, whether they are lines or stacked areas etc. Next, you must
associate the graph definition with the name of your status column; to do
that, you add it to the TEST2RRD definition in hobbitserver.cfg. So if
your column name is "ipmi" and the graph definition is also "ipmi", then
you just add "ipmi" to the TEST2RRD setting. If the columnname and the
graph definition differs, you add a COLUMNNAME=GRAPHNAME to TEST2RRD.
Last, you must add the graph to the GRAPHS definition (yes, this is
messy - I know) so it will show up also on the "trends" status. And here
you use the filename of the RRD file (without ".rrd").


This is quite a bit of work you need to do. It's flexible, but it takes
some time to get used to. There *is* a simpler mechanism called the
"ncv" graphs, but this only works if the data in your status message
is formatted as name-colon-value pairs which then go directly into an
RRD file.

I hope this helps.

Regards,
Henrik