[hobbit] conn alerts based on ping time

[Henrik Stoerner]

On Fri, Jan 13, 2006 at 11:01:11AM -0700, Charles Jones wrote:

> Is there any chance that Hobbit will soon support comparing the ping 
> replies to specifiied values for green, yellow, and red?

"soon" is asking for a lot :-)

As I wrote in another thread - yes, that is going to show up in Hobbit
sometime. It won't be specifically for the "conn" test, but rather I
am planning a facility so that all of the data that goes into the RRD's
can trigger a change of status color. And this would then in turn
trigger all of the normal effects like alerts, history logging and such.

> Somethign like:
> 
> 1.2.3.4 myhost.com # conn:200:500
> 
> This would make myhost.com's conn test go yellow if the ping was between 
> 200 and 500ms, and red if it was over 500ms.

The bb-hosts file is getting overloaded, and at some point we'll have to
think of a better way of configuring Hobbit. But yes - some way of
setting thresholds for these data and changing the status color if
they are exceeded.

> Since hobbit already graphs the numeric values of the ping replies, this 
> seems like it would be fairly easy to add?

As always, the devil's in the detail. The particular reason why this is
not as easy as it sounds is that the color of a network test status is
currently determined exclusively by the network test tool - which may
run on a separate host from the main hobbit server, which is the one
that looks at the network test output and picks out the ping time that
goes into the graph.

So by the time Hobbit learns what the ping time is, we're long past
the point where the status color has been decided.

The idea then is to have some sort of status color modification
mechanism - so even though the network tester reports "green", the
RRD module can tell Hobbit "no, make it be red because the ping
time is higher than 3 seconds".

I just need to come up with a good way of configuring it, and design
the status-override mechanism to work sanely with history logs and
such (you don't want the RRD override to show up in the history logs
in a way so the status flip-flops between green from the network
tester and red from the RRD override).


Henrik