[hobbit] ext scripts & the Hobbit methodology for implementation.

[Henrik Stoerner]

On Thu, Jan 24, 2008 at 10:24:24AM -0500, s_aiello at comcast.net wrote:
> What I mean by BB legacy ext scripts, are client based ext scripts that send 
> reports in via the bb status command. The Hobbit method is more server 
> centric. In that clients report raw data to the server and the server 
> processes that data, applying thresholds, formating reports, and possibly 
> trending. Now to send raw data from the client there are two choices, data or 
> client. Now in the past, I believe I saw a thread where Henrik mentioned that 
> data was the preferred method for ext scripts & that client was more for the 
> internal use between hobbit clients & server. But the lure of the client 
> command is to great to ignore. Especially with the ability to access that raw 
> data via the clientlog command.

I think this is the real "killer" difference between the client- and the
data-message types. If the client-data wasn't directly available, you
could just as well use a data-message.

> So that brings me to my first problem when sending data via the client command 
> the data needs to be structure in a very particular format. Where OSTYPE has 
> to match a predefined Hobbit OSTYPE and a minimum set of SECTIONNAME need to 
> be filled for the Hobbit server to accept the data. If they are missing the 
> data just ends up in /dev/null. 

Eh - not quite, you can still get at it with the clientlog command. It
just doesn't generate any cpu/disk/memory... statuses because hobbitd_client 
- the only program that normally listens in on the Hobbit "client" channel - 
doesn't know how to interpret the data.

The only requirement for a client-message really is that the hostname
must be listed in bb-hosts. The operating-system type (and the optional
client "class") is used to pick out an entry from the client-local.cfg
file which is returned to the client system, but there is no requirement
that these must be pre-defined anywhere in Hobbit.


> So I was curious if in the upcoming 4.3 (or newer version), that the client 
> command could be opened up for more general usage ?

I think what you really want to be able to do is to combine client-side
data from multiple sources - e.g. you have the Hobbit client running and
feeding the normal client data into Hobbit, and then you have a number
of custom add-ons that generate data, and you want this data to be
available in Hobbit as if it had been part of the "normal" client data.
Like
    Hobbit client sends data (uptime, vmstat, ps-listing etc.)
    Add-on script A sends additional data (eg. JVM performance metrics)
    Add-on script B sends additional data (eg. custom application data)
    User views the "clientlog" info on Hobbit and sees all three

It isn't difficult to implement, most of the pieces are already there.
Some things that will need to be done in the Hobbit daemon:
- store client-ext-FOO data separately from client-ext-BAR (to know what
  to overwrite when an update arrives).
- should there be a new protocol command for extensions? I think that
  might be necessary.
- Should there be a "client-local.cfg"-like file for extensions? It
  could be useful.
- for performance reasons, it would probably make sense to have some way
  of splitting extension data off from the normal client data, and then
  run a "client" channel worker handling each of the extension scripts.
  That way we dont need to send all of the client data through all of
  the add-on handlers who will just throw it away.


> Now secondly after the data is actually accepted by the Server, the ability to 
> do something with it is needed. Now that shouldn't be a problem with being 
> able to attatch a script to a Hobbit channel or by scheduling a command to 
> run clientlog every X minutes and process the data. But how does the script 
> apply thresholds ? Now I am working under the assumption that for my custom 
> data I would be able to add custom definitions to the hobbit-clients.cfg 
> file. But how does my script access those thresholds ? Since Hobbit already 
> has all the internals to acess this file, I would think that a Hobbit utility 
> would fit the need. Maybe even a new BB command ? i.e. "bin/bb 
> 1.2.3.4 "threshold server", where there could be an optional 
> parameter "stats=" which could be CPU, PROCS, DISK, etc. By storing the 
> custom ext thresholds in the hobbit-clients.cfg file, this allows the cgi 
> hobbit-confreport.sh to report on the custom thresholds along with the 
> built-in client thresholds.

It's hard for me to see how difficult this would be to implement. I know
the client configuration handling is a bit tricky in how it's
implemented, I don't know what it takes to provide a generic interface
to this for custom client-modules.

A "BB" command to fetch the configuration? No, don't like that idea. The
hobbitd daemon should not have to bother with configuration files it
doesn't need for itself.

Another complication is that the program using the configuration info is
"long-lived", so it will have to reload the configuration when it
changes. And that has some performance implications - the first attempt
at the current client-configuration performed very poorly because the
way it scanned all of the client configuration rules was extremely slow,
simply because of all the pattern-matching of hostnames, testnames,
colors, time-specifications etc. hobbitd_client now caches the list of 
relevant configuration rules for each host once it has determined what 
they are, and this gave quite a performance boost. Having 5-10 add-on 
handlers making the same mistake would kill your server.

I'll have to think about this.


> Sorry for the long winded email, I am just trying to move my ext scripts to 
> the Hobbit methodology. It just makes more sense.. and offers many more 
> options & power. And this was just my take on it. Thoughts ?

It's a good idea, and if it can be done right then it would fit in
really well with the Hobbit philosophy of providing good basic
monitoring, but also enabling you to add whatever extras you need.
So I like it, but it does require some thought to get right.


Regards,
Henrik