Re: [hobbit] Remote hobbit server's ports & procs flapping Green/Clear on secondary BBDISPLAY

[Bill Arlofski <waa-hobbitml (at) revpol.com>]

Henrik StÃrner wrote:
> On Tue, Feb 17, 2009 at 08:20:32AM -0500, Bill Arlofski wrote:
>> I have noticed that two tests on one remote server continually go green and
>> then clear at the same time.   That is, just now at 08:03:08 procs shows that
>> it when from CLEAR to GREEN, then at the same time when BACK from GREEN to
>> CLEAR. This continues with every 5  minute cycle.
> 
> Check the BBDISP and BBDISPLAY settings in your hobbitserver.cfg. It
> sounds as if you have BBDISP=0.0.0.0 and BBDISPLAYS="ip1 ip2" - that
> causes hobbitd_client (the module analyzing client data) to send the
> results to both of your Xymon servers, causing double updates for each
> status. The reason the status is flapping between clear and green is
> that your hobbit-clients.cfg is different on the two servers.

Hi Henrik... Thanks for the fast and detailed reply.


Ahhh... OK.. I have now copied the hobbit-clients.cfg from my client's hobbit
server to my central xymon server's hobbit-clients.cfg file and the flapping
stopped.

However.. I still have a question based on your description:

> What happens is:
> 
>  - Client sends the client data to server1 and server2.
>  - server1 passes the client data to the local hobbitd_client task.
>  - server1's hobbitd_client analyses the data using the
>    hobbit-clients.cfg found on server1. Then it generates status
>    updates; these are sent to the servers listed in BBDISPLAYS,
>    i.e. both server1 and server2.
>  - server2 does exactly the same, and sends another set of
>    status updates to both server1 and server2.


At my client's site, their hobbit server monitors several other local servers
with the hobbit client installed (cpu, disk, procs etc) and other network
equipment (ping telnet, ssh etc)

Their server has my server and their server in the
~/server/etc/hobbitserver.cfg's BBDISPLAYS (with just 0.0.0.0 in BBDISP)

On my server I only have itself in the BBDISP, and BBDISPLAYS=""

I have the BBLOCATION set to their site name in their server and in front of
all the tests in bb-hosts I have # NET="TheirSiteName"   On my server, I have
# NET:"MySiteName" in front of all of my local tests.

Their server also runs some local client scripts to gather NUT (Network UPS
tools) data from their UPSes. So I found that I needed to make the same
changes to BBDISP and BBDISPLAYS their ~/client/etc/hobbit-clients.cfg file
otherwise MY Xymon server didn't get their UPS report data.

> The solution is to make sure you have identical hobbit-clients.cfg
> files on both servers,

I think that this is what fixed it.

> and to avoid updating everything twice you
> should only list the local server in hobbitserver.cfg's BBDISP setting.
> (But keep both servers in the hobbitclient.cfg's BBDISP / BBDISPLAYS 
> setting).

If I do as you explained then I get the UPS data from the client scripts on my
client's hobbit server, but all the other tests for my client's site go purple
on my xymon system...

So in the end, I found that I needed to make sure that BBDISP=0.0.0.0  and
BBDISPLAY="my.xymon.ip their.hobbit.ip"  on their hobbit server's
~/server/etc/hobbitserver.cfg and ~/client/etc/hobbitclient.cfg files

So currently, with things configured as described it does not appear that any
of their data is being sent twice to my xymon server, nothing is flapping, and
I am seeing the UPS data from their client scripts too.

So I think ALL is well now.  Thanks again for the quick reply!

--
Bill Arlofski
Reverse Polarity, LLC