[hobbit] grouping methods

[Sloan]

Josh Luthman wrote:
> Not sure what the real reasoning is behind this but if you have 1000
> servers monitored behind 3 hobbit servers each, figure one Hobbit
> server goes down you lost 1000/3000 being monitored.  If you have 3000
> servers being monitored behind 1 hobbit server, that one point of
> failure leaves you blind of all 3000 servers.
>   

We do it with redundancy. Each server in our various data centers is 
monitored by two bb servers, with one of the two set up to send 
notifications, but in all other aspects the monitoring is active/active, 
and we get only one notification for alerts, rather than a pair of 
redundant notifications.

We've not had a bb server go down in all the years we've been using it, 
but sometimes wan connectivity goes away due to circumstances beyond our 
control, and a bb server in Arizona can't talk to the corresponding bb 
server in California, so the normally passive monitoring server goes 
into failover mode, and begins sending notification for alerts, since it 
can't verify that the other bb server is alive.

Thus, we always receive notifications for all alerts, and in the worst 
case we may get redundant notifications in the case of a split brain 
situation, which is the lesser of the evils.

Once this notification failover capability makes it into hobbit, we can 
finally switch from bb to hobbit.

Joe