[hobbit] Latest Snapshot

[Henrik Stoerner]

On Thu, Jan 26, 2006 at 12:04:16PM -0500, Michael Dunne wrote:

>  I downloaded the latest snapshot and am curious if there is any
> documentation for the new "Critical Systems" enhancements. The new features
> look perfect for our organization and I'd love to start testing.

Docs are usually the last thing that gets done :-)

Fortunately, this is one of those things that doesn't need much 
documentation just to use.

There are two new CGI's - they both install automatically when
you run "make install". 

The "hobbit-nkview.sh" provides a view of the current critical 
systems, as defined in a new configuration file hobbit-nkview.cfg; 
the old "NK" tags in the bb-hosts file have been deprecated since 
they cannot hold all the info that is needed. So this is a 
replacement for the bbnk.html page that Hobbit inherited from
the bbgen tool. The new view is dynamic - so it's always up-to-date;
it allows you to prioritize systems so the most important ones
are listed first; it will automatically hide alerts that are
older than some threshold so a "noisy" system doesn't clutter the
view of the operators; and it allows the operator to acknowledge
alerts - both to get them off his display, and to inform others
that the alert has been recognized and some action has been taken.
The acknowledgement feature is going to be used a lot more in
the future; it is designed to allow acknowledgments to be sent
by different groups to allow for alerts that can escalate to
e.g. higher-level management regardless of any acks from a
technician.

In my production setup, the configuration of what systems and
alerts are "critical" is handled by a separate group who wants
some say about what can and cannot go on their monitor (e.g. 
they won't accept responsibility for monitoring stuff until the
operational procedures have been documented etc.); this is a
group of people who are very much "point-and-click" types. So
hand-editing text configuration files is out of the question.
Hence, a web-interface to configuring the critical systems
view was built; this is the "hobbit-nkedit.sh" CGI. So by using 
this tool, you can configure the "critical systems" view with
what status messages show up there; what their priority should be;
what times of the day/week they are monitored; any special
instructions the operators should see when there is an alert;
and for systems that are going into or out of service you can 
even define the date where this happens and the monitoring will
automatically start or stop at the right time. For groups of 
hosts with identical monitoring setups, you can define a template
for how they are configured on the "critical systems" view, and
then "clone" this template to all of the hosts that should use
the same template. So you only need to define the monitoring
once.

To use this, you basically build the current snapshot, then do
a "make install" on top of your current setup. You will need to
add a couple of menu items to your ~hobbit/server/www/menu/menu_items.js
file, to get the links to these two new CGI's; see the default file
that gets built in hobbitd/wwwfiles/menu/menu_items.js when you 
build Hobbit. In the "Views" menu, it is the "Critical systems"
item, and in the "Administration" menu it is the "Edit critical systems"
item. Note that the hobbit-nkedit.sh wrapper for the CGI with the
current snapshot gets installed in the public CGI directory; it 
should go in the secured (password-protected) CGI directory. This
has been fixed for the next snapshot.

Another thing you must do it to arrange for some way of allowing
the hobbit-nkedit CGI to update the hobbit-nkview.cfg file. The
way it is done, this CGI (which runs with your web-server userid)
must have access to update the ~hobbit/server/etc/ directory.  The 
easiest way of doing that is to make the program "suid hobbit". So: 

   chown hobbit ~hobbit/server/bin/hobbit-nkedit.cgi 
   chmod u+s ~hobbit/server/bin/hobbit-nkedit.cgi 

I haven't yet decided if that is the best way of doing it; I have
a bad feeling about making CGI programs suid, but this may be the
one occasion where it is needed. Feedback on this is welcome.


I'm actually doing a presentation tomorrow for our 24x7 monitoring
group, showing them how these new features work. It has been very
much designed based on input from them, so hopefully it should work
in a way that is useful to the operators. Once I get some confirmation
that there are no serious bugs in the new tools, I'll put it into
a proper release version - probably sometime in February.


Regards,
Henrik