[hobbit] Configuration database backend.

[Henrik Stoerner]

On Fri, Dec 08, 2006 at 09:05:49AM +0200, Buchan Milne wrote:

> For me the big reason isn't to ease configuration, but rather to leverage 
> existing configuration information we have on our hosts. Our linux machines 
> are deployed automatically based on configuration information in our config 
> database, and includes IP addresses, function, hardware, site information 
> etc, sufficient to deploy the machine ready for production in 10 minutes. 
> Then, monitoring has to be added ....

Doing an automated extract of data from the DB and dumping that in
bb-hosts format into a file or directory which is then included in the
bb-hosts file is probably a fairly simple thing to setup.

> I think there are potential advantages, but IMHO:
> -a means for providing a standard way of configuring external checks should be 
> done first
> -Henrik needs to agree with the architecture

I'm all for GUI front-ends to build the configuration files, whether
they use a DB backend or something else.

I'm very much against requiring a database to be running for Hobbit to
work. The basic configuration-files (bb-hosts, hobbit-alerts.cfg and
hobbit-clients.cfg) should be accessible without relying on any
extra infrastructure components. Too often, those break just when you
need your monitoring the most.

(For a real-life scenario, consider this: Someone accidentally hits the 
 BIG RED "emergency power-off" button that cuts all power to your machine 
 room, bypassing the UPS and diesel generators. 1200 servers turned off 
 immediately, including Hobbit, databases, network switches, routers, 
 fileservers etc.  When power returned, the Hobbit servers did a quick 
 filesystem check and were back online in less than 10 minutes (thanks to 
 using journaling filesystems everywhere). All of the DB systems required 
 manual intervention to get up and running, some of which took several 
 hours to complete. Hobbit was the only system available for checking which
 servers were running and which ones weren't. True story).

So - using a DB for configuration is OK, as long as you dump the DB
config into simple text-based files that can be used by Hobbit. If you
want to use a DB backend for storing monitoring data that Hobbit
collects, put it in an external Hobbit worker module, so Hobbit doesn't
stop if the DB fails.


Regards,
Henrik