[Xymon] Purple storm

[Henrik Størner]

On 12-04-2012 05:28, Poppy, Ben wrote:
> To be honest, I'm not sure what the cause is 100%.

Me neither.

> The setup we have, should not have any dependencies on our DR site.
> Our xymon servers at our primary site use DNS servers in our primary
> site. They monitor a bunch of servers at our DR site, but the
> dependency ends there (and all that should mean is the servers show
> RED when DR site is down).

Aha - but you DO have tests in each setup that checks systems on the 
other site ? Would that happen to include any DNS or NTP checks ?

I suspect that you have each of your Xymon's setup to test availability 
of the DNS servers on both the primary and the DR site. That could be a 
problem.

> Another bit of information, during this 5 hour outage, both of our
> xymon servers went from showing properly (where DR servers were
> showing RED conn as they weren't reachable, but the servers we
> monitor in our primary site were up), to everything going purple in
> conn (and other tests).. It would alternate back and forth over the
> course of the outage (I didn't detect a regular timeframe of when it
> switched from RED to PURPLE)..

The interesting thing is that they switch to purple, indicating that 
something is stalled.

I have seen something like this happen when we had a number of DNS 
checks in the Xymon servers, and network access to these failed (broken 
switch to a customer network). This caused xymon to stall on these DNS 
checks, and all of the network tests went purple.

I know that this is difficult to test, because obviously you cannot just 
cut the connection between the two sites to try it out. But you could 
try applying this patch which changes the DNS lookup code to use the 
same kind of timeout settings as the development version - the 4.3.x 
versions suffer from a common misunderstanding about how the C-ARES 
library handles timeout that make DNS timeouts take much too long.

One possible way of testing it would be if you can firewall access from 
e.g. your DR site Xymon server to the primary site's DNS server. If you 
are running Xymon on a Linux server, then "iptables" can do that for 
you. If your primary site DNS server is 10.1.2.3, then

   iptables -I OUTPUT 1 -d 10.1.2.3 -j DROP
   iptables -I INPUT 1 -s 10.1.2.3 -j DROP

will cause all traffic to/from this server to be dropped.


Regards,
Henrik
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dnstimeout.patch
Type: text/x-diff
Size: 1878 bytes
Desc: not available
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20120412/2bb48ecb/attachment.patch>