[hobbit] Adding custom LARRD graphs

[Henrik Stoerner]

On Mon, Jan 24, 2005 at 03:14:54PM -0700, Charles Jones wrote:
> How difficult is it to add custom graphs?  

Well, it does require some programming - it's an advantage if you 
are familiar with C, since that is the language Hobbit is written in.
I don't think it's terribly hard, but I am biased.


> For example we have several Oracle standby databases that "resync"
> (import binary database changelogs) for several hours every night. I
> have a perl script which parses one of the logfiles created by this
> process, and gets values like the total time it took to import each
> binary log. Having a graph of the average time it takes to process
> would be VERY handy for metric and scaling evaluations.

The hard part usually is getting the data, and you have that already
with your perl script.

Next step is getting the data to the Hobbit server. This is easy;
decide on a unique name for the type of data you're handling - e.g.
"orasync" - and use the "bb" utility to send it off as a "data"
message to Hobbit. E.g. the following script runs your perl script,
stores the output in a temporary file, and uses the "bb" utility from
a Big Brother client installation to send this datafile to Hobbit
in a "data" message:

   #!/bin/sh

   /foo/perlscript >/tmp/datafile
   
   BBHOME=/usr/local/bbc
   export BBHOME
   . $BBHOME/etc/bbdef.sh

   $BB $BBDISP "data $MACHINE.orasync

   `cat /tmp/datafile`
   "

Now the fun bit starts. Hobbit will automatically pass data-messages
to all tasks monitoring the "data" channel. hobbitd_larrd is one of
them, so you can either add some code to this "worker module", or you
can create your own module from scratch using your favorite
programming language. An example of a hobbitd worker module is in the
hobbitd_sample.c application included with Hobbit.

Assuming you just add stuff to the existing hobbitd_larrd module,
you must do two things:

1) Write a routine do_orasync_larrd() similar to the other ones
   in the hobbitd/larrd/*.c files, that receives the message, 
   picks out the numbers that you want to store, and saves it in
   an RRD file;

2) Add a line to do_larrd.c at the end of the file, so when it sees
   the "orasync" message, it calls your do_orasync_larrd() routine.

You need to learn about RRDtool to really do this; the "rrdcreate"
and "rrdgraph" manpages include some tips on how to define RRD's
and how you can setup graphs.

Take a look at one of the simple ones, e.g. hobbitd/larrd/do_bbgen.c
which picks out a single value from the status message that bbgen
sends when updating the web-pages - the do_bbgen_larrd routine simply
finds the string "TIME TOTAL <some number>", picks out the number
(which is the time bbgen takes to generate the webpages and stores it
in an RRD file.

The data stored in the RRD file is described in the bbgen_params
variable (or "orasync_params" in your case):

static char *bbgen_params[] = { "rrdcreate", rrdfn, 
      "DS:runtime:GAUGE:600:0:U", 
      rra1, rra2, rra3, rra4, NULL };

The first and last line of this is static; you only change the
"DS:..." line to define the data you store in the rrd.

When you have the data you want, put all of it in the "rrdvalues"
string, with the timestamp in front, and call the
create_and_update_rrd routine to do the work of saving the data.


So now you have an RRD file. Time to put it into hobbitgraph.cfg.
This really depends on the kind of data you are handling.


The last step is to add "orasync" to the GRAPHS definition in
hobbitserver.cfg. This causes the bb-larrdcolumn tool to include the
orasync graph on the "trends" column page.


The first time you do it, it seems complex, and I admit: it isn't
exactly trivial because there are many pieces that need to fit
together. But once you get the first simple graph working you'll
see that it isn't all that hard. And I'll be happy to help you if you
run into problems along the way.


Henrik