[hobbit] Tips to get custom availability metrics?

[Henrik Stoerner]

On Tue, Feb 21, 2006 at 10:37:51AM -0500, Maschino, Shawn (GE Indust, Plastics) wrote:
> 
>             To better describe my situation - I have all my servers in
> Hobbit, and then a database that has server details - like site,
> criticality, ...  and I'm looking to be able to run reports like - "what
> was my conn test uptime for anything at site XYZ", or "what was my http
> uptime for our business critical apps".  
> 
> I can easily query the database and load the server list into an array
> as the source list to process, but when it comes to knowing how to
> process the Hobbit data files to get the % uptime for that list of
> servers I am lost.  I suppose in a perfect solution I'd be looking to be
> able to populate and array with the list of hostnames, pass that and the
> test name that I'm looking for to a routine, and get back a single
> number as the % uptime for that list of hosts for that test.

Hobbit has some built in availability reporting, which also lets you
output the result in a CSV format for importing into a spreadsheet.
But that only gives you availability for each of the status columns in
Hobbit - which is not very useful, when you want to combine multiple
statuses into a single availability measure.

The easiest way would be if you know what combinations you want to
measure in advance. E.g. if you have 4 webservers handling you business
app, and you run an http check on all 4, then you can make a combotest
out of those 4. E.g. if you add this to the bbcombotest.cfg file:

   app01.http=(web01.http+web02.http+web03.http+web04.http)>=2

then you'll get a status for the host "app01" where the "http" column
will be green if 2 or more of the web01-04 http columns are green.

Then you can run the normal Hobbit availability report, and it will 
give you the percentage available for the combined test also, which
is what you want.


If that is not sufficient - e.g. you want to do ad-hoc queries that
combine hosts/tests you had not planned in advance - then it gets
complicated. The historical logs for each status are in the
~hobbit/data/hist/HOSTNAME.COLUMN files, but they only log when a status
changes color. In the Hobbit source, lib/availability.c is the file that
deals with calculating availability - you're free to have a look at how
it works. It does have a debugging utility built-in, but this won't
build in the current version (I rarely use it nowadays). If you pickup
the current snapshot you can see how it's used: E.g. running

   ./lib/availability /var/lib/hobbit/hist/osiris.hobbitd \
      `date +%s --date="01 Jan 2006"` `date +%s`  

will call the routine that determines how much time the hobbitd
column has had the various color from Jan 1st until now.


Regards,
Henrik